Wire protocol for XTQL

Hey again :slightly_smiling_face:

The specifics of XTQL aside for a moment, there are a number of interesting options in the mix in the wire protocol space, all with tradeoffs:

  1. Postgres wire protocol (‘pgwire’) - well specified, but showing its age and very complex, and each driver does all kinds of things that are beyond the protocol itself which makes it awkward to validate correctness (meaning running the test suites for every single driver out there!). There are at least many other databases who have gone down this path with varying degrees of success (CockroachDB, CrateDB etc.)
  2. FlightSQL - built for analytics primarily, and uses gRPC so it is very efficient, but the spec and implementations are still in flux. The transactional aspects aren’t as developed, but it’s a promising candidate nonetheless
  3. HTTP - My understanding is that HTTP/3 eliminates most of the overhead concerns with using HTTP vs. other TCP-based transports, although we’re not actually shipping with HTTP/3 support today. More generally, HTTP architectures also neatly correspond with the non-interactive transactional semantics of XTDB and the ease of caching immutable data (e.g. using ETags)

As things stand we have explored all 3 (we have a FlightSQL module too) and the pgwire option is looking increasingly promising to ship something both optimally stable and usable (even if it’s not optimally efficient).

Sending XTQL (as edn S-expressions) over pgwire is something we have considered and haven’t ruled out. It could even be the best bet for getting XTQL out there. But for the moment though our priority is to make sure the SQL stack works smoothly and intuitively for new users, and we’ll revisit XTQL later down the road (since it’s undoubtedly more of an expert tool for the average user, requiring learning etc.).

1 Like