Elasticsearch Driver

I’ve started work on an Elasticsearch driver:

I am running into an issue, which I think is maybe related to Clojure uberjar packaging of the es jdbc driver…

I get the following error trying to connect:

Server sent bad type [action_request_validation_exception]. Original type was [Validation Failed: 1: The [0.0.0] version of the [jdbc] client is not compatible with Elasticsearch version [7.10.1];]. [org.elasticsearch.action.ActionRequestValidationException: Validation Failed: 1: The [0.0.0] version of the [jdbc] client is not compatible with Elasticsearch version [7.10.1]; at org.elasticsearch.action.ValidateActions.addValidationError(ValidateActions.java:26) at org.elasticsearch.xpack.sql.action.AbstractSqlQueryRequest.validate(AbstractSqlQueryRequest.java:239) at org.elasticsearch.xpack.sql.action.SqlQueryRequest.validate(SqlQueryRequest.java:79) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:144) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:83) at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:86) at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:75) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:412) at org.elasticsearch.xpack.sql.plugin.RestSqlQueryAction.lambda$prepareRequest$0(RestSqlQueryAction.java:116) at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115) at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:88) at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:258) at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:340) at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:191) at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:319) at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:384) at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:309) at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:28) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.base/java.lang.Thread.run(Thread.java:832) ]

I’ve looked in the code of the JDBC driver and it seems to pull its client version out of some metadata, which maybe isn’t being packaged up properly: https://discuss.elastic.co/t/this-version-of-the-jdbc-driver-is-only-compatible-with-elasticsearch-version-vunknown-unknown/211894/8

I am pretty new to maven & lein and am not sure the right way/place to fix this.

Any ideas would be appreciated (PR fine too!)

Alan

1 Like

Hi Alan!

Thanks for giving this a shot - I wrote the Athena driver, so I’d love to help on this if possible.

You’re right, it looks like there’s some code that’s trying to pull the client version out of the metadata, but when it can’t it defaults to 0.0.0 - https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/sql/sql-client/src/main/java/org/elasticsearch/xpack/sql/client/ClientVersion.java#L112

Looking at the built jar, it doesn’t look like the current way of packing the plugin includes the raw jar, but instead pulls in the class files themselves:

➜ jar tvf metabase/plugins/elasticsearch.metabase-driver.jar | grep -i version
  5939 Sat Dec 05 00:59:10 PST 2020 shadow/org/elasticsearch/xpack/sql/client/ClientVersion.class
  5026 Sat Dec 05 00:59:08 PST 2020 shadow/org/elasticsearch/xpack/sql/proto/SqlVersion.class

So yea, likely something to do with that code that detects the version, but I’m also unsure how to patch that in. Maybe there’s a way to include a custom manifest entry?

➜ cat META-INF/MANIFEST.MF
Manifest-Version: 1.0
Created-By: Leiningen 2.9.4
Built-By: dacort
Build-Jdk: 11.0.8
Leiningen-Project-ArtifactId: elasticsearch-driver
Leiningen-Project-GroupId: metabase
Leiningen-Project-Version: 1.0.0-elasticsearch-jdbc-7.10.1
Main-Class: clojure.main

That worked!

I added the following to the uberjar portion of project.clj:

:manifest      {"X-Compile-Elasticsearch-Version" "7.10.1"}

which includes it in the manifest file:

➜ unzip -qc target/uberjar/elasticsearch.metabase-driver.jar META-INF/MANIFEST.MF | grep X-Compile
X-Compile-Elasticsearch-Version: 7.10.1

But then… :frowning:

current license is non-compliant for [jdbc]

Per https://www.elastic.co/downloads/jdbc-client it looks like the JDBC client requires either platinum subscription or trial.

Hope that helps anyway, good luck!

Yes! That got me unblocked.

I was able today to get sql queries working select 1 and even sent a real query to the backend. But the results are blowing up; I think I can debug it a bit further on my own.

Pretty exciting to almost have something working, though I anticipate it being a bit more work to discover tables and such as I have no idea if “information_schema” is implemented by elastic search or how aggregations or other things might appear.

I will try to lean things towards how Postgres treats json/jsonb.

But it’s a good start!! Kind of amazed I got this far in 8 hours or so… it’s been a while since I touched clojure!

I worked on this a bit more today and mapped some more types. I now have results actually working!

There is an issue with timestamp that I’m working through but it works if cast to text.

Hi,
Recent day ,I find a problem when I try to add elasticsearch.metabase-driver.jar to metabase,then it comes a error below(My Elasticsearch version is 6.7.0 while the JDBC driver is 7.10 )
This version of the JDBC driver is only compatible with Elasticsearch version 7.10 or newer; attempting to connect to a server version 6.7.0
How can I fix this problem?

Hi! I never got this working all the way so it’s not worth it to fix the version issue.

Unless you plan to finish the es code in clojure :slight_smile: