How does Xtdb compare to SurrealDB?

I’ve asked about Dolt and and Rama. It’s useful to understand the differences, use-cases, and overlap between new databases; and I appreciate the XTDB team’s insights.

With the recent release of v2 beta, another SQL-like database in mind is SurrealDB. Its features include built-in search, live queries, and change-feeds that make it handy for building modern applications; along with work on versioned temporal tables; and runs in both server and browser. It has a huge feature set that feels surreal. Who knows if they’ll meet expectations, as they are still relatively new, they’re still working on fixing performance and have yet to release benchmarks.

One critical difference from XTDB that other versioned databases such as Dolt and Surreal don’t have is bitemporality; most only track system time.

But other than that, do ya’ll have any thoughts on SurrealDB given how similar it is in its goals of being a versioned improvement over SQL? What features do you see XTDB working on in near future? How would you suggest people compare such options?

Okay so first up is that SurrealDB doesn’t offer actual SQL. Somewhat confusingly they instead provide ‘SurrealQL’, which is definitely a distinct language. XTDB could have continued down this route of avoiding full SQL support in v2, but instead we decided to rise to the challenge. For better or worse the heyday of the no-SQL era is over, and the market has moved rather unanimously towards SQL. To not attempt to support SQL is quite possibly the biggest gamble for any database company to make right now.

Our bet for v2 instead is that enough flexibility can be found within the bounds of what the SQL standard grammar allows to deliver almost all of the value that any non-SQL language might offer. You could argue that this process is essentially how all new SQL features have been developed over the past 50 years, but now the industry has many, much smaller & nimbler teams working away freely and unconstrained by the slow machinations of standards bodies.

DuckDB are working in this “Friendlier SQL” direction likewise. And as an example on our side, we just added support for inserting documents using a RECORDS construct that we think is neat: INSERT INTO docs RECORDS {_id: 'hello'} (and again, this neatly slots into the existing SQL grammar mould)

All that said, I do think there is (/continues to be) real opportunity for a database like SurrealDB, or TypeDB, or InstantDB, or Convex (etc.) to capitalise on the problems that aren’t traditionally associated with the RDBMS-lives-on-the-server view of the world. In particular, offline-first, sync, and full-stack support. Firebase proved that this market exists. However the opportunity likely also requires a strong enough DBaaS / Backend-aaS offering that targets this wave of developers who don’t ever want to touch SQL, and who probably don’t ever want to do database ops either.

It would be a mistake to assume that SQL can’t or shouldn’t be part of such a stack though, e.g. InstantDB (another Clojure team!) uses Aurora Postgres behind the scenes for their DBaaS.

To understand any of these initiatives you need to appreciate that all databases need to perform joins at some point, and they all need storage engines that run on the cloud somewhere. Maybe the whole end-to-end stack can be ‘solved’ without SQL in one go, by a single team, and still turn a profit by delivering happy customers with world class performance, but in my experience most of the best core database engineers in the world aren’t getting carried away by lofty visions like that, instead they’re busy working on some gritty details within SQL systems, or some other highly specialised ~database systems which aren’t even attempting to be “general purpose” for mass market developers (I would count the likes of Rama or kdb or TigerBeetle or RelationalAI here).

Note that both Rama and SurrealDB (and XTDB v1) lean on RocksDB for storage and, ultimately, performance. But then so does MyRocks! With v2 we’ve taken a step back and pivoted to a column-oriented design which definitely has different performance characteristics. Theoretically faster for large queries and better for long-term retention. In general though we’re not so focused on supporting highly-transactional workloads, the kind for which RocksDB was originally designed.

These are all also things we would like to offer in due course, just don’t expect a client-side TypeScript SDK coming from our team any time soon. Maybe Electric Clojure is the real answer for full-stack, real-time stuff anyway!

offline-first, sync, and full-stack support

On this I should also mention that these were the very same topics which drew me into thinking deeply about databases in the first place. Most of the problems that I had once assumed were intractable because of how ~all SQL systems seemed to work at the time (and this was long before Postgres added JSONB), I have since realised really boil down to: handling of dynamic typing, gradual schema, and the performance of multi-way joins.

Thanks for prompting me to brain-dump on these comparisons, I do think about it all an awful lot, so it’s not much trouble really, and otherwise I will probably never get around to writing up these thoughts as blog posts…that I really should try to make more time for! :sweat_smile:

2 Likes

At the very least it’ll make the inevitable XTDB vs so-and-so database comparison writeup easier :slight_smile:

Most of the problems that I had once assumed were intractable because of how ~all SQL systems seemed to work at the time (and this was long before Postgres added JSONB), I have since realised really boil down to: handling of dynamic typing, gradual schema, and the performance of multi-way joins.

This would indeed make a great blog post!

Thanks again for the sharing your analysis; as I’m spending an awful lot of time thinking about databases as well.

1 Like