We’re really proud to announce the release of XTDB 2.0.0-beta1
For those of you that are new to the 2.x train, the release notes contain a high-level summary of what we’ve been aiming to achieve with 2.x, the differences to XT1, and how to get started.
To try it out, you can:
spin XTDB up using a local docker image, and connect to it using your usual Postgres tooling:
docker run -ti --name xtdb -p 5432:5432 ghcr.io/xtdb/xtdb:2.0.0b1
psql -h localhost
INSERT INTO users RECORDS {_id: 'jms', name: 'James'}, {_id: 'jdt', name: 'Jeremy'};
Given the the Clojure community has preferred Datalog for its expressivity and logic flow, do you see XTQL as being on par (if not better) to Datalog in that regard?
I remember also reading the across time queries motivated your decision to change to an SQL-like dialect; could you not have improved upon Datalog? I imagine the biggest motivator was tapping into SQL community, so I’m wondering what we may have lost or gained from a more idealistic, expressivity standpoint (as someone who has not yet used Datomic or XTDB v1 in production, but was fond of their ideas).
As a small caveat to your first point here - whilst (Datomic-inspired, Clojure-flavoured) Datalog gets a lot of attention and praise from across the Clojure community, anecdotally I would estimate that most Clojure projects still use/prefer SQL databases regardless. There will be many reasons for that of course, but not necessarily relating to Datalog itself, and likely more around the operational implications of self-hosting a database and the lack of a fully-managed DBaaS Datalog offering (RDS/Aurora/etc.). Not to mention the sheer convenience of SQL database ecosystems with rich tooling and abundant expertise.
To answer your actual question though, I expect the main perceived ‘gap’ in XTQL by Clojure users, when compared to Datalog, would be the inability to reach out to clojure.core as the standard library. Another gap would be the absence of rules (/ recursive CTEs) …which are rather important for many people’s definition of ‘Datalog’
However, if you’re not too worried about either of those aspects specifically though then I’d say XTQL is already on fairly similar footing to the Datalog you’d find in 1.x (or elsewhere) in terms of composability/expressiveness/logic.
We tried, but found that the introduction of first-class tables made the syntax much noisier than we’d have liked. Including support for a clojure.core standard library and interop would be interesting in theory, but in practice would be very difficult to incorporate technically into the design of the v2 query planner and execution engine.
I think it will be a collective effort to determine that ultimately, but we will certainly hope and aim for that answer to be ‘nothing’.
Somewhat off-topic, but Logica is also a really interesting approach for combining the power of (non-Clojure) Datalog with SQL engines - worth a look!
I’ll answer from the p.o.v. of a company that has a Clojure development team, working with other teams that don’t know Clojure and don’t know Datalog – they only know SQL, in terms of database access.
As interested as our Clojure team has been in Datomic and XT1, adoption would have been a non-starter due to the lack of SQL support, since it would have locked those other teams out, or at least made it much, much harder for them to build tooling around “our” databases.
XT2 providing full SQL support means that we could potentially adopt it. The Clojure team could then use either SQL (what we use now with existing databases) or XTQL – and the other teams could happily interact with all “our” databases using SQL.
Datalog is interesting but it is incredibly niche: far more niche than even Clojure itself.
As a small caveat to your first point here - whilst (Datomic-inspired, Clojure-flavoured) Datalog gets a lot of attention and praise from across the Clojure community, anecdotally I would estimate that most Clojure projects still use/prefer SQL databases regardless.
Very true! Good qualification.
Thanks for the well thought out response.
Congrats to the team on the beta release and wishing XTDB v2 lots of success.
To add to the others’ answers above: we’ve additionally added the expressivity of sub-queries, as well as a few of SQL’s constructs - we’re also looking to add support for IN, quantified comparisons (> ANY(...), > ALL(...) etc) which we couldn’t have realistically added to Datomic’s Datalog dialect (sometimes referred to as ‘EDN Datalog’) without diverging too far and losing whatever benefit we had in using the ‘same’ language.
One of the primary aims of XTQL was to also appeal to people outside of the Clojure industry, who likely have SQL experience but wanted to experiment with a data-oriented query language rather than trying to compose SQL strings - to these people, without the Clojure background, Datomic’s Datalog is likely a bigger jump.
Good questions, thanks
James
P.S. if you look at ‘vanilla’ Datalog/Prolog, they’re more relation-oriented than Datomic’s triple model - Datomic themselves have taken a new path here too (inspired by RDF triples, I’d assume)
