Update if entity has changed - feedback request

Users
1

I’m a clojure and xtdb n00b, but I’ve come up with this functionality, and I’d like some feedback.

Is this a reasonable way to update the record only if it’s changed? Some documents may be over 1Mb in size.

It’s working thus far, but I want to make sure I’m not setting myself for failure down the road.

POST /_xtdb/submit-tx

{
  "tx-ops": [
    [
      "put",
      {
        "xt/id": "hash",
        "xt/fn": "(fn [m] (hash (into (sorted-map) m)))"
      }
    ]
    [
      "put",
      {
        "xt/id": "update-if-different",
        "xt/fn":"
          (fn [ctx new-entity]
            (if (not= (hash new-entity)
                      (hash (xtdb.api/entity
                              (xtdb.api/db ctx)
                              (get new-entity :xt/id))))
              [[:xtdb.api/put new-entity]]
            )
        )"
      }
    ]
  ]
}

POST /_xtdb/submit-tx

{
  "tx-ops": [
    [ "fn" => "update-if-different", { "xt/id" => "555555", name => "bruh"}]
  ]
}
2

Hey @tempire apologies for the delayed response. In general XTDB v1 will deduplicate when documents are exactly the same - you shouldn’t need to attempt to detect or avoid this. Write amplification may only become a problem if your 1MB documents change by a couple of bytes each time, as each version is then stored under a distinct internal hash.

I think your approach shown shouldn’t interfere with the native mechanism though, so hopefully you are already observing the behaviour you are looking for :slight_smile: (though there is a slight processing cost to using a transaction function each time for this)

3

@refset Thanks for the response.

That would be ideal, although I haven’t seen that behavior - it still creates another record in the history for the xt/id.

1 Like
4

it still creates another record in the history for the xt/id.

Hmm, I think that would imply that the not= is always returning true and therefore the :xtdb.api/put is happening each time. Have you already tried printing the hash values to check how they compare manually?

5

I haven’t, but I’m submitting the same hard coded json in a script. There’s no question that it’s the same.

I suspect the functionality you’re speaking of works if you’re using clojure, but not the http api. The function works well enough, though. Not trying to become a clojure expert, just interested in bitemporality.

CleanShot 2024-05-15 at 19.19.13@2x
CleanShot 2024-05-15 at 19.19.13@2x2610×940 151 KB

1 Like
6

Interesting, then I still can’t see/think why that might be happening :thinking: (the HTTP API shouldn’t behave any differently here). In any case, the additional entries in the entity-history should not result in any significant space consumption if they are all actually resolving to the exact same version of a document.

7

Fair enough; I’m more interested in keeping the history entries tidy, as I’m using them to know when a record was changed.

1 Like