Arrow-vector conflicting dependency

Users
1

Hello,

We’re seeing a dependency resolution problem on the org.apache.arrow/arrow-vector transitive dependency included in the xtdb2 maven artifact.

The possible problem we’ve identified in the maven artifact is that differing versions of the class org.apache.arrow.vector.complex.DenseUnionVector is pulled in multiple times, once via the transitive dependency declared in:

xtdb-core-2.0.0-SNAPSHOT.pom

    <dependency>
      <groupId>org.apache.arrow</groupId>
      <artifactId>arrow-vector</artifactId>
      <version>14.0.0</version>
      <scope>compile</scope>
    </dependency>

and again by a newer / different version of this class that is bundled in the jar itself:

xtdb-core-2.0.0-SNAPSHOT.jar

/org/apache/arrow/vector/complex/DenseUnionVector.class

resulting in some machines resolving the wrong version of DenseUnionVector on their classpath resulting in:

Caused by: java.lang.IllegalArgumentException: No matching method setOffset found taking 2 args for class org.apache.arrow.vector.complex.DenseUnionVector
    at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:127)
    at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:102)
    at xtdb.vector.writer$eval37956$fn__37957$__GT_child_writer__37963$write_value_BANG___37966.invoke(writer.clj:866)
    at xtdb.vector.writer$duv_child_writer$reify__37939.writeObject(writer.clj:782)
    at xtdb.vector.writer$eval37800$fn__37805.invoke(writer.clj:470)
    at xtdb.vector.writer$eval37361$fn__37373$G__37352__37380.invoke(writer.clj:36)
    at xtdb.vector.writer$eval37916$fn__37923.invoke(writer.clj:752)
    at xtdb.vector.writer$eval37361$fn__37373$G__37352__37380.invoke(writer.clj:36)
    at xtdb.vector.writer$eval37894$fn__37899.invoke(writer.clj:709)
    at xtdb.vector.writer$eval37361$fn__37373$G__37352__37380.invoke(writer.clj:36)
    at xtdb.log$__GT_put_writer$write_put_BANG___48518.invoke(log.clj:269)
    at xtdb.log$write_tx_ops_BANG_.invokeStatic(log.clj:388)
    at xtdb.log$write_tx_ops_BANG_.invoke(log.clj:371)
    at xtdb.log$serialize_tx_ops.invokeStatic(log.clj:409)
    at xtdb.log$serialize_tx_ops.invoke(log.clj:395)
    at xtdb.log$submit_tx_AMPERSAND_.invokeStatic(log.clj:448)
    at xtdb.log$submit_tx_AMPERSAND_.invoke(log.clj:443)
    at xtdb.node.impl.Node.submitTxAsync(impl.clj:50)
    at xtdb.api$submit_tx_AMPERSAND_.invokeStatic(api.clj:219)
    at xtdb.api$submit_tx_AMPERSAND_.invoke(api.clj:191)
    at xtdb.api$submit_tx.invokeStatic(api.clj:261)
    at xtdb.api$submit_tx.invoke(api.clj:233)
    at xtdb.api$submit_tx.invokeStatic(api.clj:259)
    at xtdb.api$submit_tx.invoke(api.clj:233)
2

More info, we’re using polylith and the problem happens only on some machines, when running tests.

It might be that we’ve run into this issue Classpath stability for tests · Issue #261 · polyfy/polylith · GitHub and therefore the problem might not be specific to the xtdb2 artifact.

Will continue investigating further next week.

2 Likes
3

Yep, it’ll be dependent on classpath ordering. Unfortunately I don’t think we have a lot of choice but to try to override the DenseUnionVector class from upstream as it stands - there are a number of bugs that we haven’t managed to get patches accepted for yet.

(edit: tbh, we should probably just deploy our own fork at this point, they’ve been in flight long enough…)

Hope you get to the bottom of it :slight_smile:

James

1 Like