"full bitemporality"

From https://xtdb.com/v2:

Where XTDB has previously been optimized for point-in-time historical queries, v2 now supports ‘cross-time’ queries, e.g. temporal joins and temporal range scans. This unlocks the complete history of data for rich analysis.

I would love to hear more details on this optimization, and some examples would be wonderful. What can v2 do (or do better) in this regard than v1, or than, say, a simply-immutable database like SurrealDB, Fluree, or TerminusDB.

Hey @djradon - fun topic!

So the big difference is that the v2 primary index (an Apache Arrow-based storage format) supports ad hoc scanning and temporal filtering at much higher throughput that what can be achieved with v1. This is because, abstractly, the v2 engine is able to handle complex query logic (Allen interval / period operators) and push much of the work down into efficient low-level scans, whereas v1 typically requires frequent object allocations and fully-realized handling of intermediate results (for complex queries, to re-iterate).

More concretely, v2 offers a declarative language for expressing complex temporal questions and the engine is able to accelerate many of these queries due to the index design (and we will publish more on this soon!). In contrast, v1 forces you to write more code and wire independent queries together achieve the same outcomes, but can’t approach the same level of performance as v2 in its current form, hence v2 :slightly_smiling_face:

We’re working on a set of these currently, but you can see some in the docs here: Control the timeline | XTDB + Avoiding a lossy database | XTDB

None of the systems you mentioned cover valid time, and the tl:dr; there is that if your application ever needs to expose time-travel functionality to users then you probably want to carefully consider the tradeoffs of having it around vs. not. For some industries it’s a non-starter to not account for these requirements properly upfront (whether your database of choice can accelerate it or not).

Hope that helps - and I’ll definitely chime in again with more examples once they’re live,

Jeremy