entity-history I can get the history of an entity which has been deleted at the valid-time of the db value. However
open-tx-log is the only way I found to discover entities which have been deleted. I’m evaluating ways to do a migration and doing it in a transaction function seems like a good idea but I can’t use
open-tx-log in there. Is there a way to get the IDs of deleted entities from a db value that I missed?
@ema This morning at standup @refset_xt mentioned he was working on a way to get deleted entities from a db value … I assume that was referring to this, but I just wanted to check in that he spoke to you through a different channel? Just wanted to close the thread if there was a resolution.
No, I haven’t talked to @refset_xt but it might still be what I’m looking for.
Hey @ema unfortunately (contrary to the above!), I don’t have a useful suggestion to offer here other than either figuring out a different way to access
open-tx-log (e.g. using an impure transaction function…which has the potential to be unsafe/inconsistent if you’re not very careful) or adding some sort of userspace deletion tombstone entities that would be visible to queries. Ultimately the problem here is that XT’s indexes simply aren’t designed to handle this kind of requirement efficiently currently.
It is however something that definitely will be possible in our future architecture & query engine, although I can’t give an ETA for that currently.
All that said, please can you say more about the need for migration that you are hoping to resolve? Like, are you trying to truncate the history? Or is it about re-indexing entities into different shapes? Or something else? Maybe there are other options.
Have you considered simply not using
::xt/delete and just transacting an otherwise empty document, or using a
:deleted? true attribute?
This is what we’re currently doing. To ensure consistency we assert that the maximum tx-id we get from
open-tx-log is one less than the tx-id we get from the db-basis in the transaction function.
This migration is about going from :(product|contract|…)/documents attributes to just :documents so that when we query for documents we don’t need to do an or on all the different attributes.
Replacing with an empty entity instead of deleting it would be a possibility but I guess we would need a migration to get there as well