Entity/document ids with nested maps and equality semantics

Note that if you’re using a map as an :xt/id that the equality semantics are based on the nippy serialization of the map, where {:xt/id {:my/ident (int 123)}} is not the same as {:xt/id {:my/ident 123}} - the latter is a long and because xt/id serializes the map using nippy, which preserves the native types, and then uses the hash as an identifier, they will not result in the same xt/id.

If, however, you’re using {:xt/id (int 123)} the int is converted to a long by xtdb so a document {:xt/id (int 123)} will be stored as {:xt/id 123} or more concretely {:xt/id (long 123)}

Note the following example :-

(let [d 1                                                                                                
      node (xt/start-node {})]                                                                           
    (xt/await-tx                                                                                           
      node                                                                                                 
      (xt/submit-tx                                                                                        
        node                                                                                               
        [                                                                                                  
         [::xt/put {:xt/id (int d)}]                                                                 
         [::xt/put {:xt/id {:n (int d)}}]]))                                                                    
    (let [db (xt/db node)]                                                                                 
      [                                                                                                    
        (xt/entity db d)                                                                                   
        (xt/entity db {:n d})]))                                                                          
[{:xt/id 1} nil]

Or,

[                                                                                                      
 (=                                                                                                    
   (c/new-id {:x (int 1)})                                                                             
   (c/new-id {:x (long 1)}))                                                                           
 (=                                                                                                    
   (c/new-id (int 1))                                                                                  
   (c/new-id (long 1)))])                                                                                
[false true]

It is worth noting, if you’re reading documents from JSON, some JSON deserialization will chose ints for numbers that fit into 32 bits (ie ints). So if you’re seeing documents transacted from deserialized json which you can’t query with (eg) {:xt/id {:doc-id 123}} make sure you try {:xt/id {:doc-id (int 123)}}, or change the deserialization options for your JSON parser to use longs instead of ints.

3 Likes

Yet another strike against nippy’s surface convenience. The research currently going into XTDB 2.x is looking to provide much stricter data types, which should eliminate these kinds of problems.

1 Like