Connecting Databricks Spark to Metabase

@relferreira Permissions or ownership of the files in /plugins/ ? There should be some errors during startup that should hint at the problem, but I’m not totally sure.

Hello @flamber

I couldn’t find any error during startup. Both methods logs:

04-06 23:29:44 INFO driver.impl :: Registered driver :sparksql-databricks (parents: [:hive-like]) 🚚
...
Metabase Initialization COMPLETE

Here is a diff of the permissions on both methods

Docker run (works)

bash-5.0# ls -lh
total 68K    
drwxr-xr-x    1 root     root        4.0K Feb 25 20:38 app
drwxr-xr-x    1 root     root        4.0K Jan 13 22:57 bin
drwxr-xr-x    5 root     root         360 Apr  6 23:29 dev
drwxr-xr-x    1 root     root        4.0K Apr  6 23:29 etc
drwxr-xr-x    1 root     root        4.0K Apr  6 23:29 home
drwxr-xr-x    1 root     root        4.0K Jan 13 22:57 lib
drwxr-xr-x    2 root     root        4.0K Jan  2 19:19 lib64
drwxr-xr-x    5 root     root        4.0K Oct 21 13:39 media
drwxr-xr-x    2 metabase metabase    4.0K Apr  6 23:29 metabase.db
drwxr-xr-x    2 root     root        4.0K Oct 21 13:39 mnt
drwxr-xr-x    1 root     root        4.0K Jan  2 19:21 opt
drwxr-xr-x   17 root     root         544 Apr  6 18:09 plugins
dr-xr-xr-x  206 root     root           0 Apr  6 23:29 proc
drwx------    2 root     root        4.0K Oct 21 13:39 root
drwxr-xr-x    2 root     root        4.0K Oct 21 13:39 run
drwxr-xr-x    2 root     root        4.0K Oct 21 13:39 sbin
drwxr-xr-x    2 root     root        4.0K Oct 21 13:39 srv
dr-xr-xr-x   13 root     root           0 Apr  6 23:29 sys
drwxrwxrwt    1 root     root        4.0K Apr  6 23:29 tmp
drwxr-xr-x    1 root     root        4.0K Jan  2 19:20 usr
drwxr-xr-x    1 root     root        4.0K Oct 21 13:39 var
bash-5.0# ls -lh plugins
total 140M   
-rw-r--r--    1 root     root      529.7K Feb 25 12:36 bigquery.metabase-driver.jar
-rw-r--r--    1 root     root      454.6K Feb 25 12:36 druid.metabase-driver.jar
-rw-r--r--    1 root     root        1.0M Feb 25 12:36 google.metabase-driver.jar
-rw-r--r--    1 root     root      703.8K Feb 25 12:36 googleanalytics.metabase-driver.jar
-rw-r--r--    1 root     root        4.4M Feb 25 12:36 mongo.metabase-driver.jar
-rw-r--r--    1 root     root       67.6K Feb 25 12:36 oracle.metabase-driver.jar
-rw-r--r--    1 root     root      137.6K Feb 25 12:36 presto.metabase-driver.jar
-rw-r--r--    1 root     root        2.3M Feb 25 12:36 redshift.metabase-driver.jar
-rw-r--r--    1 root     root       16.9M Apr  6 12:12 simba-spark-jdbc41-2.6.3.1003.jar
-rw-r--r--    1 root     root       27.6M Feb 25 12:36 snowflake.metabase-driver.jar
-rw-r--r--    1 root     root       17.0M Apr  6 12:11 sparksql-databricks.metabase-driver.jar
-rw-r--r--    1 root     root       59.9M Feb 25 12:36 sparksql.metabase-driver.jar
-rw-r--r--    1 root     root        6.8M Feb 25 12:36 sqlite.metabase-driver.jar
-rw-r--r--    1 root     root        1.2M Feb 25 12:36 sqlserver.metabase-driver.jar
-rw-r--r--    1 root     root       49.6K Feb 25 12:36 vertica.metabase-driver.jar

custom Dockerfile (error described in topic)

bash-5.0# ls -lh
total 72K    
drwxr-xr-x    1 root     root        4.0K Apr  6 18:41 app
drwxr-xr-x    1 root     root        4.0K Jan 13 22:57 bin
drwxr-xr-x    5 root     root         360 Apr  6 23:27 dev
drwxr-xr-x    1 root     root        4.0K Apr  6 23:27 etc
drwxr-xr-x    1 root     root        4.0K Apr  6 23:27 home
drwxr-xr-x    1 root     root        4.0K Jan 13 22:57 lib
drwxr-xr-x    2 root     root        4.0K Jan  2 19:19 lib64
drwxr-xr-x    5 root     root        4.0K Oct 21 13:39 media
drwxr-xr-x    2 metabase metabase    4.0K Apr  6 23:27 metabase.db
drwxr-xr-x    2 root     root        4.0K Oct 21 13:39 mnt
drwxr-xr-x    1 root     root        4.0K Jan  2 19:21 opt
drwxrwxrwx    1 root     root        4.0K Apr  6 19:46 plugins
dr-xr-xr-x  206 root     root           0 Apr  6 23:27 proc
drwx------    2 root     root        4.0K Oct 21 13:39 root
drwxr-xr-x    2 root     root        4.0K Oct 21 13:39 run
drwxr-xr-x    2 root     root        4.0K Oct 21 13:39 sbin
drwxr-xr-x    2 root     root        4.0K Oct 21 13:39 srv
dr-xr-xr-x   13 root     root           0 Apr  6 23:27 sys
drwxrwxrwt    1 root     root        4.0K Apr  6 23:27 tmp
drwxr-xr-x    1 root     root        4.0K Jan  2 19:20 usr
drwxr-xr-x    1 root     root        4.0K Oct 21 13:39 var
bash-5.0# ls -lh plugins
total 139M   
-rw-r--r--    1 root     root      529.7K Feb 25 12:36 bigquery.metabase-driver.jar
-rw-r--r--    1 root     root      454.6K Feb 25 12:36 druid.metabase-driver.jar
-rw-r--r--    1 root     root        1.0M Feb 25 12:36 google.metabase-driver.jar
-rw-r--r--    1 root     root      703.8K Feb 25 12:36 googleanalytics.metabase-driver.jar
-rw-r--r--    1 root     root        4.4M Feb 25 12:36 mongo.metabase-driver.jar
-rw-r--r--    1 root     root       67.6K Feb 25 12:36 oracle.metabase-driver.jar
-rw-r--r--    1 root     root      137.6K Feb 25 12:36 presto.metabase-driver.jar
-rw-r--r--    1 root     root        2.3M Feb 25 12:36 redshift.metabase-driver.jar
-rw-r--r--    1 root     root       16.9M Apr  6 12:12 simba-spark-jdbc41-2.6.3.1003.jar
-rw-r--r--    1 root     root       27.6M Feb 25 12:36 snowflake.metabase-driver.jar
-rw-r--r--    1 root     root       17.0M Apr  6 12:11 sparksql-databricks.metabase-driver.jar
-rw-r--r--    1 root     root       59.9M Feb 25 12:36 sparksql.metabase-driver.jar
-rw-r--r--    1 root     root        6.8M Feb 25 12:36 sqlite.metabase-driver.jar
-rw-r--r--    1 root     root        1.2M Feb 25 12:36 sqlserver.metabase-driver.jar
-rw-r--r--    1 root     root       49.6K Feb 25 12:36 vertica.metabase-driver.jar

@relferreira Not sure why /app is from April 6th - it should be April 2nd, when 0.35.1 was released.
Try just using docker cp to copy the drivers into place - if that works, then I guess you need to investigate what’s going on with Docker, but then it’s likely not a problem with Metabase.

@flamber I updated the driver to work with Metabase 0.34:

Now everything works if I use the following Dockerfile:

FROM metabase/metabase-head:latest

ENV MB_DB_CONNECTION_TIMEOUT_MS=60000

COPY sparksql-databricks.metabase-driver.jar /app/plugins/

But using this one, it doesn’t connect to the database

FROM metabase/metabase

ENV MB_DB_CONNECTION_TIMEOUT_MS=60000

COPY sparksql-databricks.metabase-driver.jar /plugins/

Do you know why?

By the way, thanks a lot for the quick replies.

@chandan try the version in my repository, I think it is working fine now.

@relferreira
It would be really nice if you used version tags instead of nothing or latest, since it’s much easier to see that you’re actually using the same version.

Why? Besides the obvious differences between the two Dockerfiles?
metabase/metabase-head is not the same as metabase/metabase - and is build differently.

Hello @flamber, I can obviously see the difference. I was politely asking how I can make de plugin system work using the metabase/metabase one.

@relferreira Given the amount of people that posts logs, where the answer is in their own post, then I don’t know if you had noticed the differences.

I don’t know why it doesn’t read the plugins directory - it should - you can see that in the log during startup, and that’s where it will extract the built-in drivers too.
So given that it’s extracting correctly, I’m not sure what’s going on. Post the entire log from startup.

Have you tried to docker cp into the official image?

@flamber with docker cp it works
I think you are right, there is something wrong with my Docker.

Thank’s a lot for the help :slight_smile:

1 Like

Status update, a couple years later: I've tried all three versions of the community Databricks Spark SQL driver, and none managed to connect to my Databricks instance. When I try to add a database via their drivers, Metabase fails to connect:

(My Databricks instance is on Azure, and has compute and SQL clusters running.)