Connection Refused to Dremio

#1

I am writing a Metabase driver plugin to Dremio.
I have already installed the Dremio JDBC driver with the Clojure project.
Now I am trying to implement the multifunction for sql-jdbc.conn/connection-details->spec
My code looks like this:

(defmethod sql-jdbc.conn/connection-details->spec :dremio
[_ {:keys [user password db host port instance ssl]
:or {user “dbuser”, password “dbpassword”, db “”, host “localhost”}
:as details}]
(-> {:applicationName config/mb-app-id-string
:url host
:subprotocol “dremio”
:dbname db
:database db
:password password
:classname “com.dremio.jdbc.Driver”
:connection-uri “jdbc:dremio:direct=127.0.0.1:31010”
:user user
:instanceName instance
:encrypt (boolean ssl)
(sql-jdbc.common/handle-additional-options details, :seperator-style :semicolon)))

The connection-url property should override some of the database connection attributes.
I am running Dremio locally on a Docker container, with the Dremio services at port 31010.
However, when I do the Metabase setup, entering connection attributes in the “Add your data” section, and hit the “Next” button, I would get the following “Connection Refused” error message from the Metabase logs:

ERROR rpc.BasicClient :: Failed to establish connection
java.util.concurrent.ExecutionException: cdjd.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:31010
at cdjd.io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54)
at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.operationComplete(BasicClient.java:284)
at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.operationComplete(BasicClient.java:272)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:500)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:493)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:472)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:413)
at cdjd.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:538)
at cdjd.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:531)
at cdjd.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:111)
at cdjd.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:323)
at cdjd.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
at cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:685)
at cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
at cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
at cdjd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
at cdjd.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at cdjd.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: cdjd.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:31010
Caused by: java.net.ConnectException: Connection refused

I have tried countless permutations of the connection attributes.
Only this permutation, with this connection-url, produces the “Connection refused” error.
Most of the others would produce a TimeoutException, suggesting that the service does not exist.
So I suspect that my connection string is correct, but its request is just not reaching the Dremio service.
What could be going on? Please help!

No method in multimethod 'connection-details->spec' for dispatch value
#2

Hi @reydavid
I’m not a driver wizard, but I have a couple of notes.

It says Connection refused: localhost/127.0.0.1:31010 - but why does it contain both localhost and 127.0.0.1? I can see your URI is jdbc:dremio:direct=127.0.0.1:31010, so I’m not sure where localhost comes from.

Have you tried using tcpdump or Wireshark on the Dremio container, to make sure that you’re even seeing a connection to the container from Metabase?

Have you looked at some of the other third-party drivers for inspiration? Example:
https://github.com/dacort/metabase-athena-driver
https://github.com/enqueue/metabase-clickhouse-driver

Just for reference: https://github.com/metabase/metabase/issues/5562

Pinging @dacort to see if he might have some hints or knows who to ask

#3

Thanks for the usual prompt response, @flamber
I will try verifying network access through tcdump or Wireshark.

1 Like
#4

Hi @flamber
My Metabase docker container is running on my host machine, and my Dremio instance is running remotely at, say, 52.170.198.158
My browser could access the Dremio instance and I can run queries on it.
But when I use tcpdump to capture traffic on port 31010 (the Dremio coordinator), I could see nothing.
Perhaps my tcpdump command is wrong? Or I am doing something else wrong?

image
image.png698×282 23.4 KB

#5

Here’s more info, @flamber
I created a bash session inside my Metabase docker container and installed telnet in it.
I can ping localhost and the remote ip address just fine.
When I telnet to localhost or 127.0.0.1 I get a “Connection refused” error.
But when I telnet to the remote ip address, telnet just hangs or becomes unresponsive until I exit with Ctrl-C.
What do these mean?
image

#6

Hey Reynaldo,

I’m a little confused - in your first post you mention your connection-uri is 127.0.0.1, but then later on you mention your Dremio instance is running on a remote IP address. Why are you trying to connect to dremio on 127.0.0.1 when it’s on a remote IP address? In your second telnet example where you connect to 52.170.198.158, it looks like that works. So should your connection-uri be 52.170.198.158?

#7

Hi @dacort
Sorry for the confusion, but you’re right. Early in the thread, I was running Dremio on a separate Docker container in my local host. Thus, I was pointing at 127.0.0.1. But when I kept getting the connection errors, I switched target destinations and began pointing to the remote IP address.
My current connection string is:
“jdbc:dremio:direct=52.170.198.158:31010”

So are you saying that my telnet call to the remote ip address, where telnet seems to just hang, indicates that the docker container has access to it? And that the problem may just be with my connection configuration somehow?

#8

A minor correction - I think if the telnet command does actually connect then it will say “Connected to x.x.x.x”, but that doesn’t show up in your screenshot so it looks like it may be hanging trying to connect to that remote host. If that’s the case, you likely need to open up the port somehow. But it definitely seems like a network configuration issue at this point.

#9

Hi @dacort
I am using netstat inside my Metabase docker container to check for any traffic to the remote Dremio instance. I use the command:
netstat -a
I try to connect dozens of times, sometimes going through minor configuration permutations. I almost never get any indication from netstat that any request is going out from the container. But in the container logs, I usually get the error message:
Failure in connecting to Dremio: cdjd.com.dremio.exec.rpc.RpcException: HANDSHAKE_COMMUNICATION : [USER]: Channel closed null <--> null (user client)

Once every few dozen times, I would catch this message from netstat:

image
image.png1210×374 59.1 KB

Herein the 2nd row I see that a connection to 52.170.198.158:31010 is ESTABLISHED !
But a split-second later, it disappears from netstat.

image
image.png1160×242 37.3 KB

Was that a true connection where the container reached the remote server and the remote serve replied back? Or was that a fluke? Why does that line almost instantly disappear? Why doesn’t it stay? What does it mean?

#10

Hey @reydavid,

This error below seems significant. Based on the ESTABLISHED connection, the error message below, and the fact that you immediately receive the error it seems like Dremio is actively refusing the connection.

Doing a quick google search of that error turns up this post. I’d make sure you’re using Java 8 as mentioned in that post, and if that doesn’t work maybe try a different version of the JDBC driver. Somebody on that post mentions that V3.0.6 works ok.

Hope that helps!

Damon

#11

Thanks, @dacort for your insight. I will look into it.

#12

@dacort OMG!! That was it!
I already have Java 8, so I tried downgrading to Dremio jdbc driver v3.0.6.
That was it! Now I can connect!
Thank you thank you thank you!
I had been banging my head against the wall for 2 weeks on this! You saved my sanity, @dacort

#13

Hi @reydavid,
My team tried and failed to connect Dremio to Metabase and finally gave up. I was hoping someone would solve this. Will you be willing to make your approach public or outline the steps for others to follow?

#14

Hi @joe_jk
What I tried to do was create a connection from Metabase to Dremio, with Metabase as the client. I registered the Dremio JDBC driver into my local Maven repository. Then I imported the jdbc driver into my Metabase driver project. I first used the Dremio jdbc driver v3.3.2. I failed to establish a connection for the longest time.
Then @dacort gave me a tip that worked. I uninstalled v3.3.2 and replaced it with v3.0.6. That’s how I got my successful connection.

#15

@reydavid Are you planning on publishing your driver?

#16

Yes, @dacort . I plan to publish it.

2 Likes
#17

@reydavid, great work! please let us know about your progress!