Trying out [2.x]

Hi guys,
I’m giving a first try to 2.x, and so far, no joy… I’m probably doing something silly, but just in case (running on JDK21):

Deps:

{:mvn/repos {"ossrh-snapshots" {:url "https://s01.oss.sonatype.org/content/repositories/snapshots"}}
 :paths ["src" "resources"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}
        com.xtdb.labs/xtdb-api {:mvn/version "2.0.0-SNAPSHOT"}
        com.xtdb.labs/xtdb-core {:mvn/version "2.0.0-SNAPSHOT"}}
 :aliases
 {:dev {:extra-paths ["test" "dev" "perf"]
        :jvm-opts ["--add-opens=java.base/java.nio=ALL-UNNAMED"
                             "-Dio.netty.tryReflectionSetAccessible=true"]
        :extra-deps {com.github.jpmonettas/flow-storm-dbg {:mvn/version "3.7.4"}
                     com.github.mhjort/trombi {:mvn/version "1.0.0"}
                     com.github.mhjort/trombi-gatling-highcharts-reporter {:mvn/version "1.0.0"}
                     http-kit/http-kit {:mvn/version "2.8.0-alpha3"}
                     djblue/portal {:mvn/version "0.48.0"}}}
  :run-m {:main-opts ["-m" "fi.varela.xt2exp"]}
  :run-x {:ns-default fi.varela.xt2exp
          :exec-fn greet
          :exec-args {:name "Clojure"}}
  :build {:deps {io.github.clojure/tools.build
                 {:git/tag "v0.9.2" :git/sha "fe6b140"}}
          :ns-default build}
  :test {:extra-paths ["test"]
         :extra-deps {org.clojure/test.check {:mvn/version "1.1.1"}
                      io.github.cognitect-labs/test-runner
                      {:git/tag "v0.5.1" :git/sha "dfb30dd"}}} :neil {:project {:name fi.varela/xt2exp}}}}

I’m running this with -A:dev

Code:

(ns fi.varela.xt2exp
  (:require [xtdb.node :as xt.node]
            [xtdb.api :as xt]
            [clojure.java.io :as io])
  (:gen-class))

(def node (xt.node/start-node (let [node-dir (io/file "/tmp/xt2")]
  {:xtdb.log/local-directory-log {:root-path (io/file node-dir "log")}
   :xtdb.object-store/file-system-object-store {:root-path (io/file node-dir "objects")}})))

(xt/status node)
;; => {:latest-completed-tx nil, :latest-submitted-tx nil}

(def some-doc {:xt/id :toto
               :entity/type :person
               :person/name "Toto"
               :person/age 44
               :person/others #{:tato :tito}})


(def tx (xt/submit-tx node [[:put :person some-doc]]))
;; => Execution error (UnsupportedOperationException) at org.apache.arrow.memory.util.MemoryUtil/directBuffer (MemoryUtil.java:174).
;;    sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available

Full stack trace:

2. Unhandled clojure.lang.Compiler$CompilerException
   Error compiling NO_SOURCE_FILE at (21:9)
   #:clojure.error{:phase :execution,
                   :line 21,
                   :column 9,
                   :source "NO_SOURCE_FILE"}
             Compiler.java: 3719  clojure.lang.Compiler$InvokeExpr/eval
             Compiler.java:  457  clojure.lang.Compiler$DefExpr/eval
             Compiler.java: 7199  clojure.lang.Compiler/eval
             Compiler.java: 7149  clojure.lang.Compiler/eval
                  core.clj: 3215  clojure.core/eval
                  core.clj: 3211  clojure.core/eval
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 1990  clojure.core/with-bindings*
                  core.clj: 1990  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  218  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  217  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java: 1583  java.lang.Thread/run

1. Caused by java.lang.UnsupportedOperationException
   sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available

           MemoryUtil.java:  174  org.apache.arrow.memory.util.MemoryUtil/directBuffer
             ArrowBuf.java:  229  org.apache.arrow.memory.ArrowBuf/getDirectBuffer
             ArrowBuf.java:  224  org.apache.arrow.memory.ArrowBuf/nioBuffer
         WriteChannel.java:  133  org.apache.arrow.vector.ipc.WriteChannel/write
    MessageSerializer.java:  303  org.apache.arrow.vector.ipc.message.MessageSerializer/writeBatchBuffers
    MessageSerializer.java:  276  org.apache.arrow.vector.ipc.message.MessageSerializer/serialize
          ArrowWriter.java:  147  org.apache.arrow.vector.ipc.ArrowWriter/writeRecordBatch
          ArrowWriter.java:  133  org.apache.arrow.vector.ipc.ArrowWriter/writeBatch
                  util.clj:  417  xtdb.util/build-arrow-ipc-byte-buffer/fn/fn/fn/write-batch!
                  util.clj:  428  xtdb.util/root->arrow-ipc-byte-buffer/fn
                  util.clj:  416  xtdb.util/build-arrow-ipc-byte-buffer/fn/fn/fn
                  util.clj:  408  xtdb.util/build-arrow-ipc-byte-buffer/fn/fn
                  util.clj:  408  xtdb.util/build-arrow-ipc-byte-buffer/fn
                  util.clj:  408  xtdb.util/build-arrow-ipc-byte-buffer
                  util.clj:  404  xtdb.util/build-arrow-ipc-byte-buffer
                  util.clj:  426  xtdb.util/root->arrow-ipc-byte-buffer
                  util.clj:  425  xtdb.util/root->arrow-ipc-byte-buffer
           tx_producer.clj:  387  xtdb.tx-producer/serialize-tx-ops
           tx_producer.clj:  367  xtdb.tx-producer/serialize-tx-ops
           tx_producer.clj:  403  xtdb.tx-producer.TxProducer/submitTx
                  node.clj:   89  xtdb.node.Node/submit_tx_AMPERSAND_
                   api.clj:  157  xtdb.api/submit-tx&
                   api.clj:  139  xtdb.api/submit-tx&
                   api.clj:  184  xtdb.api/submit-tx
                   api.clj:  160  xtdb.api/submit-tx
                   api.clj:  182  xtdb.api/submit-tx
                   api.clj:  160  xtdb.api/submit-tx
                  AFn.java:  156  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
             Compiler.java: 3714  clojure.lang.Compiler$InvokeExpr/eval
             Compiler.java:  457  clojure.lang.Compiler$DefExpr/eval
             Compiler.java: 7199  clojure.lang.Compiler/eval
             Compiler.java: 7149  clojure.lang.Compiler/eval
                  core.clj: 3215  clojure.core/eval
                  core.clj: 3211  clojure.core/eval
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 1990  clojure.core/with-bindings*
                  core.clj: 1990  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  218  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  217  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java: 1583  java.lang.Thread/run

Am I doing something wrong, or is this some other issue?

Cheers,
Martín

Hey Martin, thanks for giving it a go!

Yep, this is a known issue with Java 21 - the Arrow folks are working on supporting it. In the meantime, we’re all using Java 17 internally - maybe give that a try?

Looking forward to running the our new query language by you :slight_smile:

Cheers,

James

1 Like

Alright! Thanks for the quick reply, I’ll give it a go!
Cheers,
Martín

The switch to 17 did the trick. I think there might be a bug with the document binding special operator :xt/*. I was trying to figure out how to pull a whole document, and checked also with the only example I found in the tests:

(xt/q node '{:find [tx]
                         :where [(match :xt/txs {:xt/* tx})]})
;; => Execution error (IllegalArgumentException) at xtdb.expression/continue-read$fn (expression.clj:254).
;;    No method in multimethod 'read-value-code' for dispatch value: :clj-form

Full stack trace:

1. Unhandled java.lang.IllegalArgumentException
   No method in multimethod 'read-value-code' for dispatch value: :clj-form

              MultiFn.java:  156  clojure.lang.MultiFn/getFn
              MultiFn.java:  233  clojure.lang.MultiFn/invoke
            expression.clj:  254  xtdb.expression/continue-read/fn
                  core.clj: 7366  clojure.core/map-indexed/fn/fn
  TransformerIterator.java:   79  clojure.lang.TransformerIterator/step
  TransformerIterator.java:   97  clojure.lang.TransformerIterator/hasNext
                   RT.java:  517  clojure.lang.RT/chunkIteratorSeq
                  core.clj: 2679  clojure.core/sequence
                  core.clj: 2664  clojure.core/sequence
            expression.clj:  254  xtdb.expression/continue-read
            expression.clj:  253  xtdb.expression/continue-read
               RestFn.java:  467  clojure.lang.RestFn/invoke
            expression.clj:  427  xtdb.expression/eval15557/fn/fn
            expression.clj: 1246  xtdb.expression/eval16542/fn/fn/fn
                  core.clj: 2770  clojure.core/map/fn
              LazySeq.java:   42  clojure.lang.LazySeq/sval
              LazySeq.java:   51  clojure.lang.LazySeq/seq
                   RT.java:  535  clojure.lang.RT/seq
                  core.clj:  139  clojure.core/seq
                  core.clj:  662  clojure.core/apply
                  core.clj: 2800  clojure.core/mapcat
                  core.clj: 2800  clojure.core/mapcat
               RestFn.java:  423  clojure.lang.RestFn/invoke
            expression.clj: 1241  xtdb.expression/eval16542/fn/fn
            expression.clj: 1545  xtdb.expression/fn/fn
                Delay.java:   42  clojure.lang.Delay/deref
                  core.clj: 2337  clojure.core/deref
                  core.clj: 2323  clojure.core/deref
            expression.clj: 1589  xtdb.expression//reify
               project.clj:   92  xtdb.operator.project.ProjectCursor/reify
                  scan.clj:  295  xtdb.operator.scan.TrieCursor/fn
                  scan.clj:  272  xtdb.operator.scan.TrieCursor/tryAdvance
               project.clj:   84  xtdb.operator.project.ProjectCursor/tryAdvance
               project.clj:   84  xtdb.operator.project.ProjectCursor/tryAdvance
                rename.clj:   31  xtdb.operator.rename.RenameCursor/tryAdvance
               project.clj:   84  xtdb.operator.project.ProjectCursor/tryAdvance
              operator.clj:   72  xtdb.operator/wrap-cursor/reify
              operator.clj:  196  xtdb.operator.CursorResultSet/hasNext
                   RT.java:  517  clojure.lang.RT/chunkIteratorSeq
                  core.clj: 5754  clojure.core/iterator-seq
                  core.clj: 5746  clojure.core/iterator-seq
                   api.clj:   79  xtdb.api/q&/reify
    CompletableFuture.java:  684  java.util.concurrent.CompletableFuture/uniApplyNow
    CompletableFuture.java:  662  java.util.concurrent.CompletableFuture/uniApplyStage
    CompletableFuture.java: 2168  java.util.concurrent.CompletableFuture/thenApply
                   api.clj:   75  xtdb.api/q&
                   api.clj:   14  xtdb.api/q&
                   api.clj:   62  xtdb.api/q&
                   api.clj:   14  xtdb.api/q&
                   api.clj:  131  xtdb.api/q
                   api.clj:   82  xtdb.api/q
                      REPL:   39  fi.varela.xt2exp/eval33574
                      REPL:   39  fi.varela.xt2exp/eval33574
             Compiler.java: 7194  clojure.lang.Compiler/eval
             Compiler.java: 7149  clojure.lang.Compiler/eval
                  core.clj: 3215  clojure.core/eval
                  core.clj: 3211  clojure.core/eval
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 1990  clojure.core/with-bindings*
                  core.clj: 1990  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  218  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  217  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  833  java.lang.Thread/run

``

On a more “global” note, at a first impression, the datalog dialect in 2.x does not feel very datalog-ish, but rather closer to SQL-in-EDN (again, to take with a grain of salt, because I’ve used this for all of 5 minutes, and been on 1.x for about three years).

Some comments (from what I could gather from the docs and what I tried):

  • it might be nice to have an entity API, which could take table and id, to just fetch a document. Going through q feels a bit clunky
  • I really miss the old pull syntax. You can do the same with left-joins, but it is significantly clunkier.
  • Related to clunky-ness (and without having actually tried, but just from looking at the syntax), it seems that building queries programmatically would be quite a bit more involved than in 1.x

I gather that the old pull stuff can’t really be implemented as it was because of the time range queries (where you’re probably matching against more than one version of the same document).
It may also be that I’m a bit of a corner case, in that the bitemporal aspects have been mostly secondary to my use cases, but having the nice join syntax via pull and the simple graph queries have been the main “selling points” for 1.x (the ease of use of a document store with the power of Datomic-like queries, minus the Datomic-like hassle of the schemas).

I haven’t been following close enough to know this yet, but I suspect that the (E,A,V) indexing is not as before, since there’s no global IDs anymore (though a concatenation of the table name and the entity id would still be unique globally, right?), and the non-uniqueness of the id to document mapping, across time ranges probably makes it impossible to keep the “normal” Datalog semantics…

Sorry for the rambling post, I’m just trying to figure out how this works now :slight_smile:
Hopefully I’ll understand more after @refset’s call on the new Datalog stuff.

1 Like