I have a document that includes an :event/click-type attribute, yet when I include [event :event/click-type] in a query, the document is not returned. All other attributes work fine.
A weird bug in XT perhaps? I don’t see any bugs in my code. I’ve ran this code on two different machines (both indexing from the same tx log in production) and gotten the same result. I’m also going to ingest a dump of the transaction log on my local machine and see if I get the same results; I’ll update here once that’s done. It looks like I have at least 7 documents exhibiting this behavior.
Here’s the entity history for the document above. The first transaction doesn’t include :event/click-type. The second transaction (2 seconds after the first) adds that key. So I’m guessing this is related somehow, since :event/click-type is the only key that doesn’t work in the query.
Update: when testing this locally, it works. I didn’t actually ingest the tx log exactly as returned by open-tx-log though (I’m unable to iterate through the tx log without getting an OOM error); instead I call entity-history on all docs in the DB and dump those as a list of operations, which I then ingested locally. If that’s worth exploring more, I could try to get the actual tx log and see if replaying that causes the error to manifest locally.
This isn’t super urgent; I can work around this particular query easily, though obviously it’s a little disconcerting and it would be nice to get to the bottom of this
Well that looks pretty weird. I haven’t been able to reproduce obvious anything either
This seems like the most sensible course of action to me - and I’d be quite glad to pair on the debugging once you have something you can run locally (just DM me to coordinate).
Ah, this sounds like a bug in its own right - I will do some testing of my own (and potentially open another thread/issue…).
In the meantime, if you could confirm the XT version and storage config used, that might be helpful to us or others
This seems like the most sensible course of action to me - and I’d be quite glad to pair on the debugging once you have something you can run locally (just DM me to coordinate).
That’d be great. I’ll keep you posted. Next thing I’ll try, hopefully soon (or not, I guess with the holidays!), is to download the index to my local machine and see if I can get around the OOM error there. Relevant discussion we had on Slack last month: https://clojurians.slack.com/archives/CG3AM2F7V/p1636609241148400
I mentioned in that thread that the first part of the tx log consists of a bunch of 1000-document transactions, which you said should probably be fine. Another detail I’ve remembered since then is that a lot of those documents are relatively large, since they contain email contents from newsletters (now I store the contents in object storage with a foreign key in XT). Maybe the combination of large TXes + large docs is pushing XT over the edge?
Anyway, from the Slack convo, I was able to finish indexing after enough restarts, so maybe if I set that up locally I’ll be able to get through it eventually even if I keep getting OOMs.
Regarding the tx log iteration OOM, does running the following cause an OOM also? (or is this roughly what you were doing anyway?)
On the OOM front, I think it may be because we didn’t have fetch-size configured (meaning results would be cached by the driver) - please you try using open-tx-log again with the latest dev-SNAPSHOT release? The commit with the fix hasn’t been pushed yet, but essentially we just add a couple of parameters to the prepare call here: xtdb/jdbc.clj at e2f51ed99fc2716faa8ad254c0b18166c937b134 · xtdb/xtdb · GitHub
Ah, sorry, and thanks for pointing this out! So it seems to be the case that depending on SNAPSHOTs worked fine before we moved everything off of Clojars, but Maven Central seems to prefer to keep releases & snapshots in separate repositories
To resolve, try adding this to your deps.edn: :mvn/repos {"sonatype snapshots" {:url "https://s01.oss.sonatype.org/content/repositories/snapshots"}}
Or to your project.clj: :repositories [["sonatype snapshots" {:url "https://s01.oss.sonatype.org/content/repositories/snapshots"]]
An extreme workaround is download the snapshot jar directly: https://s01.oss.sonatype.org/content/repositories/snapshots/com/xtdb/xtdb-jdbc/dev-SNAPSHOT/xtdb-jdbc-dev-20211221.134401-3.jar (note: you can browse for full coordinates on this site)
and use com.xtdb/xtdb-jdbc {:local/root "/path/to/jar/xtdb-jdbc-dev-20211221.134401-3.jar"} - you probably want to leave the other xtdb-* deps at 1.20.0 or whatever, mainly because xtdb-core brings in all the other modules, which then all need to be manually downloaded and added too…though I’m probably missing some crucial POM scoping concept to avoid this)
The root cause of this issue has now - we strongly suspect - been resolved, and the fix will be included in the next release. It boils down to a failed match operation very subtly preventing related documents which are first-seen in that transaction from ever being indexed properly thereafter. The issue for this has been closed on that basis (https://github.com/xtdb/xtdb/issues/1683) but feel free to re-open it as required
I don’t have any updates on the open-tx-log issue yet.
