[copying the below from the ‘Clojurians’ Slack for posterity, apologies for the formatting - hope it’s of interest! we’ve also open a new issue to track Elixir support]
torkantorkan Monday at 9:50 AM
Hi! Congrats on getting the 2.0 beta up and running! Not sure if this is the correct place to ask, but here goes: Is there a ballpark timeline/roadmap for implementing the non-clojure client libs?
jarohenjarohen Monday at 10:03 AM
hey, thanks @torkan tbh, we’re not as likely to now that we have better PostgreSQL tooling compatibility - we’re already using XT2 from both NodeJS and Python using their normal Postgres drivers.We’re currently going through a bug bash on the queries the different drivers/tools throw at us, so if you run into anything with yours, please do let us know
torkantorkan Monday at 10:08 AM
A very understandable approach Some of the de-facto drivers/client libs in other languages have quite a bit of very postgres-specific behavior that in practice makes xtdb incompatible atm
torkantorkan Monday at 10:10 AM
For example, Postgrex in Elixir queries the db for expected types, which makes it just crash
jarohenjarohen Monday at 10:10 AM
Heh, yep, we’ve been running into all of those we should be significantly better on this now though, so if you’re hitting any specific issues with your driver of choice, could you file a bug?
jarohenjarohen Monday at 10:11 AM
aha, Elixir, niice
jarohenjarohen Monday at 10:11 AM
one we haven’t tried, openly
jarohenjarohen Monday at 10:11 AM
we also have a ‘nightly’ Docker label which contains a good few fixes over the b1 tag - could you give that a try?
torkantorkan Monday at 10:12 AM
Ah nice, will definitely try it out
jarohenjarohen Monday at 10:13 AM
are you using Postgrex?
torkantorkan Monday at 10:13 AM
Yup!
torkantorkan Monday at 10:16 AM
I assume 99% of all elixir users would want to be able to use Ecto (a db-agnostic abstraction for defining schemas/queries/etc), which in turn has an adapter for using Postgrex (edited)
torkantorkan Monday at 10:22 AM
On nightly:
09:21:18 | DEBUG xtdb.pgwire | Interpreting SQL: SELECT t.oid, t.typname, t.typsend, t.typreceive, t.typoutput, t.typinput, coalesce(d.typelem, t.typelem), coalesce(r.rngsubtype, 0), ARRAY ( SELECT a.atttypid FROM pg_attribute AS a WHERE a.attrelid = t.typrelid AND a.attnum > 0 AND NOT a.attisdropped ORDER BY a.attnum ) FROM pg_type AS t LEFT JOIN pg_type AS d ON t.typbasetype = d.oid LEFT JOIN pg_range AS r ON r.rngtypid = t.oid OR r.rngmultitypid = t.oid OR (t.typbasetype <> 0 AND r.rngtypid = t.typbasetype) WHERE (t.typrelid = 0) AND (t.typelem = 0 OR NOT EXISTS (SELECT 1 FROM pg_catalog.pg_type s WHERE s.typrelid != 0 AND s.oid = t.typelem))
torkantorkan Monday at 10:23 AM
I’m assuming this is due to Postgrex using binary-mode, and wants to know how to handle the response and map the types
jarohenjarohen Monday at 10:44 AM
right, yep - my guess would be that we don’t have pg_range yet, but will file an issue for it and see what we can repro thanks for getting in touch!
torkantorkan Monday at 10:45 AM
Ah right, so your long-term ambition is to basically be more or less completely “pg-compatible”?
torkantorkan Monday at 10:49 AM
I’d imagine it’ll be hard to replicate the pg_type table for example, since it’s all built on the assumption that tables conform to a given schema
jarohenjarohen Monday at 11:00 AM
yep, that’s the plan (at least for outside of Clojure) - the idea being we get so much benefit from tooling/libraries if we can achieve that (edited)
jarohenjarohen Monday at 11:00 AM
pg_type’s one of the better ones - the difference with us is that rather than returning the schema specified through create/alter table, we return the schema inserted through the documents themselves
tatuttatut Monday at 11:01 AM
fwiw, I’ve been using the HTTP API (not in production) and it is very simple to use from basically any language that has http client and json support
torkantorkan Monday at 11:03 AM
Being pg-compatible is definitely the way I would go as well, so makes total sense!
jarohenjarohen Monday at 11:03 AM
@tatut mm, it’s good, and I can’t pretend I don’t enjoy being able to ‘curl’ my database - but the API itself is bespoke to us, so we’d also need to have written database exploration UIs, BI tools, etc etc, which (in addition to the database itself!) is realistically too much for our relatively small team
torkantorkan Monday at 11:04 AM
And yes, I was wondering about the http api as well, it would probably be a nice workaround until regular wire protocol “just works”
jarohenjarohen Monday at 11:04 AM
well, or rely on open-source contributions like xtdb-inspector, of course
jarohenjarohen Monday at 11:05 AM
@torkan exactly, yep - it was our quickest route to being able to get data in and out of XT2
torkantorkan Monday at 11:08 AM
There’s no docs for it, right? I’m fine with just reading the code for the remote client, just wondering if I’ve missed it somewhere
tatuttatut Monday at 11:08 AM
yes, the API is obviously xtdb2 specific… I see that, but a maximal xtdb2 client library probably would go ways beyond “generic SQL” imo
tatuttatut Monday at 11:09 AM
I enjoyed making my Prolog library… GitHub - tatut/swixt: SWI-Prolog library for XTDB v2 and even that still doesn’t make use of everything xtdb2
tatut/swixt
SWI-Prolog library for XTDB v2
Stars
5
Language
Prolog
Added by GitHub
torkantorkan Monday at 11:16 AM
Nice, thanks for sharing!
jarohenjarohen Monday at 11:25 AM
@tatut we don’t have a lot of extensions in the HTTP API outside of the query languages themselves - it’s mostly status information (which we’re looking to expose through more standard monitoring interfaces too)
tatuttatut Monday at 11:37 AM
why not expose those as virtual tables?
jarohenjarohen Monday at 11:46 AM
that too, yep
tatuttatut Monday at 11:48 AM
I really liked that the API is drop dead simple, query and tx… that’s it
2
jarohenjarohen Today at 10:01 AM
@torkan #3737 if you’d like to follow its progress
1
torkantorkan Today at 10:02 AM
Excellent! Thanks!
1
torkantorkan Today at 10:02 AM
Exciting that my days of the insanity that is update-in-place might be coming to an end in the not too distant future
1
torkantorkan Today at 10:05 AM
Have you guys landed on “how much of postgres” you will aim to emulate btw?
jarohenjarohen Today at 10:06 AM
Enough™ haha
1
jarohenjarohen Today at 10:07 AM
joking aside, it’ll be enough to get the tools working and understanding what’s in the database, but we’re saying that the end-user should be expecting to talk XTDB rather than Postgres
torkantorkan Today at 10:08 AM
Yeah, completely understandable really
Also sent to the channel
jarohenjarohen Today at 10:09 AM
i.e. we’re explicitly not going down the ‘we’re drop-in compatible, you won’t be able to tell the difference’ route (as a number of other Postgres wire-compatible dbs have) - it’s a completely reasonable avenue, just not the one we want to go down (edited)
torkantorkan Today at 10:12 AM
Yes, I’ll definitely miss pg_trgrm for easily-setup searching, but other than that it’s still an excellent trade-off imo
jarohenjarohen Today at 10:14 AM
well, I won’t rule it out, there’s nothing technical stopping us (or someone else, given it’s F/OSS!) implementing that on top of XT - it’s the usual “time/resources” issue (and indeed, we’ve added a number of Postgres SQL functions/extensions already)
torkantorkan Today at 10:15 AM
Ah, so you’re doing a little of that as well, very cool!
torkantorkan Today at 10:21 AM
Not sure how familiar you are with the Elixir ecosystem, but if setting up a new Phoenix project with postgres works, then you’re golden
1
jarohenjarohen Today at 10:43 AM
personally, not at all, I’m afraid but we have a few Elixir folk at JUXT I’m hoping to entice into writing the test harness
torkantorkan Today at 10:44 AM
Haha, well then they’ll set you straight once they start trying it out
1
torkantorkan Today at 10:50 AM
Ecto (the main db lib/wrapper) is fairly opinionated, and assumes it will be handling automatic migrations etc when using Postgres, but all of that can be easily bypassed
tatuttatut Today at 10:52 AM
yes, I did an Erlang library for xtdb v1 that mapped erlang records and/or elixir structs to documents… and I didn’t use ecto at all. but perhaps v2 can use it better due to being SQL
torkantorkan Today at 10:58 AM
Yeah, writing an ecto-adapter for v1 would be quite an undertaking I’m guessing
tatuttatut Today at 10:59 AM
xtdb v2 does more than postgresql, as you can so neatly use NEST_MANY to pull linked data… no need to do what most SQL abstractions spend lots of code doing: turning rectangular resutls back into nested data
1
jarohenjarohen Today at 11:01 AM
it was definitely a missing piece for us, yeah
spend lots of code doing: turning rectangular results back into nested data
spent far too much of my pre-XTDB life doing exactly that (edited)
torkantorkan Today at 11:02 AM
Haha yes
torkantorkan Today at 11:04 AM
That one will be probably off limits when just using the standard ecto postgres-adapter though, due to all the automatic type casting etc that relies on the table schema introspections
torkantorkan Today at 11:07 AM
But I don’t see it as unlikely that there will be enough community excitement to make that happen
torkantorkan Today at 11:23 AM
Fwiw I just reached out to the ecto-team in the elixir slack, if I hear anything worthwhile back from them I’ll post it here