[ANN] 1.21.0's out!

Users
1

Here’s 1.21.0!

1.21.0 introduces ‘projection/aggregate expressions’, and also contains a number of performance improvements and bugfixes.

;; project.clj, e.g.
[com.xtdb/xtdb-core "1.21.0"]

;; deps.edn, e.g.
{com.xtdb/xtdb-core {:mvn/version "1.21.0"}}

A couple of the fixes below require an ‘index version bump’. This means that you’ll need to clear your XTDB query indices, and re-index from the transaction log as per the docs. The new index version is 20.

In green/blue production settings, we recommend doing this by starting a new cluster of XTDB nodes, waiting for them to catch up with the tx-log, switching over, and decommissioning the old nodes. If you’ve got any questions/concerns about this, please do get in touch via hello@xtdb.com - we’re happy to help out.

New: Projection/Aggregate expressions

We have added support for simple Clojure expressions in :find projections and aggregations. You should be able to remove :where clauses that were only included to generate intermediate values, and move these to the :find clause instead.

For example, the change to TPC-H Q1 looks like this:

;; before
'{:find [l_returnflag
         l_linestatus
         (sum ret_2)
         (sum ret_4)
         ...]
  :where [[l :l_shipdate l_shipdate]
          [(<= l_shipdate #inst "1998-09-02")]
          [l :l_extendedprice l_extendedprice]
          [l :l_discount l_discount]
          [l :l_tax l_tax]
          [l :l_returnflag l_returnflag]
          [l :l_linestatus l_linestatus]
          [(- 1 l_discount) ret_1]
          [(* l_extendedprice ret_1) ret_2]
          [(+ 1 l_tax) ret_3]
          [(* ret_2 ret_3) ret_4]
          ...]
  ...}

;; after
'{:find [l_returnflag
         l_linestatus
         (sum (* l_extendedprice (- 1 l_discount)))
         (sum (* (* l_extendedprice (- 1 l_discount))
                 (+ 1 l_tax)))
         ...]
  :where [[l :l_shipdate l_shipdate]
          [l :l_extendedprice l_extendedprice]
          [l :l_discount l_discount]
          [l :l_tax l_tax]
          [l :l_returnflag l_returnflag]
          [l :l_linestatus l_linestatus]
          [(<= l_shipdate #inst "1998-09-02")]
          ...]
  ...}

This has an accompanied performance improvement, due to simplifying the main phase of the query engine execution.

Performance fixes

We’ve been spending some more time with our heads in a profiler - you should see some query performance benefits, particularly if you make heavy use of or clauses and sub-queries.

  • pre-encoding attributes in get-attr, seems to save ~5% on TPC-H Q1, which has a handful of :where triple clauses internally converted to get-attr calls.
  • reducing the decoding/re-encoding of sub-query bound variables, and caching encoded literals in sub-queries - 10-15% on TPC-H Q19, which makes heavy use of literals within or clauses.

Import tx-time from upstream sources

Long requested, now available!

submit-tx and friends now take an options map, for now only accepting a ::xt/tx-time key to override the tx-time of the transaction.

This means you can import transaction times from external sources (for example, when bulk loading existing bitemporal data into XT). The transaction time mustn’t be earlier than any other transaction currently in XTDB (i.e. make sure you import the transactions in order) and it mustn’t be later than the current time on the transaction log (e.g. Kafka).

Mac M1 RocksDB ARM build support

RocksJava has taken a while to add support M1 chips, but the latest builds have finally landed in Maven :tada:

Elsewhere in this release

  • #1603: fix race condition using Postgres SERIAL / MySQL auto_increment.
  • #1618: no longer a requirement to include the same logic vars in all legs of or / or-join clauses.
  • #1654: Lucene text searches now respect query :timeout
  • #1657: NIO Checkpointer no longer throws path exceptions
  • #1660: fix to ensure Jetty server in mvn-uberjar initialises correctly.
  • #1683: (index version bump required) fix for inaccessible documents first inserted in a failed match, then subsequently put.
  • #1669: fix pull-many remote API client implementation.
  • #1700: bump Kafka to 2.6.1 to bring in jackson-databind regression fix.
  • #1702: ‘second-chance’ cache implementation stability - better hash distribution for integer values.
  • #1710: pull doesn’t through an error when attempting forward joins with a non-id type
  • #1714: pull-many returns results in the same order and cardinality as the input entity ids
  • #1724: LMDB checkpointer is closed properly during closure of the node
  • #1725: edn HTTP API is extended with literal reader support for encoded types
  • #1730: listeners are closed properly during closure of the node
  • #1732: entity-history start-tx-time has correct behaviour when between two tx-times
  • #1737: Kafka feature usage changed to avoid multi-threaded consumer errors
  • #1750: bump encore to 3.21.0 (shadowed) to fix Clojure 1.11 warning

As always, a big thanks to everyone contributing to this release by raising/fixing issues, and helping us with repros!

Cheers,

XT Team

8 Likes
2

In case anyone is confused, I’m shamelessly reusing this post as the 1.21.0 official announcement (at least to begin with … @jarohen might replace or rename it later).