Copying some of the follow-up Clojurians Slack discussion here for posterity (which followed the 1.23.0 release that fairly dramatically improved indexing times for various users):
Hukka 11 days ago
Hmm. Testing index building time. On local machine between 1.20.0 and 1.23.0 it goes from 230 seconds to 70s. But on cloud run from 210s to 180s. Local machine has loads more ram and CPU, so I tested more cores. 2 cpus lowers it to 90s and 4 to 70s. Very interesting that it now can use at least two cores with almost perfect efficiency, four helping a bit more.
3 replies
wotbrew 11 days ago
We’ve permitted more Rocks compaction jobs on multi-core machines (2 or ncpus-1, whichever is higher)
wotbrew 11 days ago
Indexing is pipelined across multiple threads too, which you want cores available for. (edited)
Hukka 11 days ago
ncpu-1 means it wasn’t as bad at scaling as I thought
