V1 - significant resource usage increase

Users
1

I’ve been using v1 for ~6 months without issue, with a postgres backend and a rocksdb index.

In the past day, the memory usage has doubled from hovering around 700M to 1.5G, and even basic queries (retrieval of a single xt/id) can spike the cpu to nearly 200%. The cpu spike is likely caused by low memory…but I’m trying to identify why the memory usage has increased.

There’s been no major increase in data storage, although updates are fairly regular.

I would consider my usage low volume:

:xtdb.index/index-version 22
:xtdb.kv/estimate-num-keys 48770
:xtdb.kv/kv-store "xtdb.rocksdb.RocksKv"
:xtdb.kv/size 16672512
:xtdb.version/revision "8d2ae275b897ba1287b33be9b716d504eb50cbf1"
:xtdb.version/version "1.24.3"

|Attribute|Count (across all versions)|
|---|---|
|:xt/id|3083|
|:val|2966|
|:type|1837|
|:xt/fn|90|
|:xtdb.api/evicted?|22|
|:id|5|

:val is a nested json data structure of no more than 50k in size.

The index store is up to date:

[Thread-0] INFO xtdb.tx - Secondary indices out of date (tx '192676'), catching up...
Thread-0] INFO xtdb.tx - Secondary indices up to date, continuing...

Even with no load, the resources are relatively high - about double - compared to previous observations.

I would appreciate any suggestions as to why this is ocurring - I’m concerned that if I give it any load, it will all come crashing down on me.

2

Hi @tempire thanks for sharing some of the details.

the memory usage has doubled from hovering around 700M to 1.5G

Is that the resident usage as observed from the OS? Or the JVM’s heap / heap+off-heap usage?

Low memory can definitely cause things to grind to a halt, but assuming you’re not seeing OOM/allocation errors then it doesn’t sound like an issue with the heap usage.

Which platform are you running on? Is this Linux/x86? RocksDB being a native component makes understanding memory usage and behaviours more complex than a typical JVM app.

Feel free to put some time in my calendar this week and we can look at a few diagnosis steps: https://calendar.app.google/nGfowoQdBmQx9e29A

In the meantime, giving the process more memory should be sufficient to relieve the bottleneck for now without extra investigation.