Wire protocol for XTQL

Users
1

Hello :smiley:

I was curious if there was any plans for a wire protocol implementation to cover the same bases that the HTTP API does?

Honestly I’m not sure myself if such a thing would be worth the effort, but many other databases I see have an explicit wire protocol in place. The HTTP API is nice but I’m wondering if there are performance implications of relying on it where TCP would suffice. Could there possibly be some way to leverage the existing pgwire implementation so that you could send XTQL in some kind of “special” command?

Any thoughts appreciated!

2

As a follow up to this, my performance concerns with the HTTP server were centered around if TCP connections could be kept alive or not. I didn’t realize HTTP 1.1 does keep connections alive by default, and testing with curl on the XTDB Jetty server confirms that they are kept alive

3

Hey again :slightly_smiling_face:

The specifics of XTQL aside for a moment, there are a number of interesting options in the mix in the wire protocol space, all with tradeoffs:

  1. Postgres wire protocol (‘pgwire’) - well specified, but showing its age and very complex, and each driver does all kinds of things that are beyond the protocol itself which makes it awkward to validate correctness (meaning running the test suites for every single driver out there!). There are at least many other databases who have gone down this path with varying degrees of success (CockroachDB, CrateDB etc.)
  2. FlightSQL - built for analytics primarily, and uses gRPC so it is very efficient, but the spec and implementations are still in flux. The transactional aspects aren’t as developed, but it’s a promising candidate nonetheless
  3. HTTP - My understanding is that HTTP/3 eliminates most of the overhead concerns with using HTTP vs. other TCP-based transports, although we’re not actually shipping with HTTP/3 support today. More generally, HTTP architectures also neatly correspond with the non-interactive transactional semantics of XTDB and the ease of caching immutable data (e.g. using ETags)

As things stand we have explored all 3 (we have a FlightSQL module too) and the pgwire option is looking increasingly promising to ship something both optimally stable and usable (even if it’s not optimally efficient).

Sending XTQL (as edn S-expressions) over pgwire is something we have considered and haven’t ruled out. It could even be the best bet for getting XTQL out there. But for the moment though our priority is to make sure the SQL stack works smoothly and intuitively for new users, and we’ll revisit XTQL later down the road (since it’s undoubtedly more of an expert tool for the average user, requiring learning etc.).

1 Like
4

Cool stuff, and yeah good points about favoring SQL at least initially to get more adoption. In an ideal setting you’d have full support for all of these options, but of course you need to spend time wisely. Looking forward to seeing this evolve : )

1 Like