Three questions

Users
1

XTDB seems neat. I haven’t used it yet but plan to try it out soon. I guess I have three questions already though:

  • client side s3 encryption?
    the official s3 libraries support client side encryption. I kind of wish that the xtdb documented a way to turn it on. Is this something xtdb supports or would it even be a good idea to use?
  • convert between storage types?
    Is it easy to go from local storage to s3 storage? or in-memory storage to local? I guess I’m mostly just wondering how different the data is inside each storage.
  • tail queries?
    One of the best features I’ve seen is the ability to take a query and turn it into a feed of all of the things that have changed to the data that the query returned. I’ve used it in lovefield. Materialize has a TAIL query that does it. rethinkdb has a changes call that turns a query into it. Think this is something that is possible?

Thanks

2

Hey @sixcorners sorry to keep you waiting! Most of the team is offline at conferences this week (OSFF NYC / GOTO Copenhagen).

I don’t see why this wouldn’t be possible or a good idea, assuming you can distribute the keys to all the nodes easily enough. Is it a hard requirement for your security policies to have this?

This is not currently supported but we would like to make it easy enough to use existing tools like pg_dump. For now you would need to run some queries to pull all the data out and re-insert it into a fresh database. In theory something even smarter should be possible, given that the object store and filesystem directories are essentially storing the same structure, but we’ve not created any tooling to handle migration at this level yet. Local storage is really only intended for testing and development purposes.

It’s essentially all the same (e.g. the filesystem structure looks like the s3 structure), but moving from the memory/filesystem to Kafka in practice would require some work at our end.

Doing this efficiently is quite a tricky engineering challenge, but definitely something we would like to make happen. A CDC-like feed at the very least would be a useful addition (right now you need to poll the xt.txs table and join against _system_from to figure out what has changed across the database) - we’ll definitely be working in this area before GA.

Thank you for the questions!