Metabase x.41.x+ - How to build drivers?

Driver Development
1

Hello,

Trying to build a driver for 1.40 and it seems we can't use the command lein install-for-building-drivers anymore. Any guidance on how to proceed?

2

Hi @rcronin
Why can't you use that command on v40?
If you are talking about master (upcoming v41), then a lot has changed:
https://github.com/metabase/metabase/wiki/Migrating-from-Leiningen-to-tools.deps

3

If I run that command with V1.40, I get the following error: 'install-for-building-drivers' is not a task. See 'lein help'.

Thanks for including that link. What I'm not sure is the equivalent command for lein install-for-building-drivers as it adds the build to clojars so the driver can pickup the latest version of metabase-core.

4

@rcronin Which branch and commit did you base off from? It sounds like you are on master
install-for-building-drivers just does the same thing as lein uberjar
https://github.com/metabase/metabase/blob/release-x.40.x/project.clj#L279

5

Last branch working for me was 0.38-integration.

If that's the case that it's the equivalent of lein uberjar then I suppose clojure -T:build uberjar would work on the master branch?

6

@rcronin I was asking which branch+commit you were seeing this error on.
Okay, so you are saying the it hasn't worked for half a year. No one else has reported this.
Yes, that's the equivalent Clojure command.

7

My apologies. Latest branch I'm seeing this issue. When I use the any of the 0.38 branches, the lein install-for-building-drivers works fine.

8

@rcronin I think we're not understanding each other, but that's fine. Stop using lein it has been removed going forward. We are still in process of updating a lot of the documentation. If you are having problems, then wait until documentation has been finished.

9

@flamber - perhaps and my apologies. I did figure it out. Thanks for your help Here were roughly my steps...might not be the solution long term but it did work.

  1. From metabase repo root folder, run the following commands:
    clojure -X:deps prep
    cd modules/drivers
    clojure -X:deps prep
    cd ../..
  2. Run clojure -T:build uberjar
  3. Navigate to folder where jar file is created and run mvn install:install-file -Durl=file:repo -DgroupId=local -DartifactId=metabase-core -Dversion=1.40 -Dpackaging=jar -Dfile=metabase.jar
  4. Navigate to driver repository/folder and modify contents of project.clj and change metabase dependency to match version created
  5. Build driver and success
2 Likes
10

@rcronin You might want to follow this issue too, since when closed, then we should have updated docs everywhere: https://github.com/metabase/metabase/issues/17267

11

@rcronin Also worth following: https://github.com/metabase/metabase/pull/17606

12

Very helpful. Thanks!

13

I managed to install metabase to a local maven repo as @rcolaco described. But it would be even better if Metabase could provide an updated JAR (e.g. on Clojars) containing only the API required by drivers.

This would make it much easier for driver developers.

Many drivers on GitHub still use metabase-core 1.0.0-SNAPSHOT from 2019.

14

Hi @flamber

Could the Metabase team please update the driver documentation https://github.com/metabase/metabase/wiki/Writing-A-Driver to suit the v0.41+ style of doing things as in https://github.com/metabase/metabase/wiki/Migrating-from-Leiningen-to-tools.deps? I have tried the solutions in this thread and reviewed the updated Sudoku driver https://github.com/metabase/sudoku-driver but have not had success with updating from v0.40 style lein commands.

The issues include 1) getting the custom driver to build (particularly without having to specify a full / absolute path to the metabase repo in the driver's deps.edn) - and is the metabase-core jar required anymore?, 2) getting the Metabase driver tests to run, 3) getting custom driver tests to run, 4) environment values being mysteriously overwritten e.g. put Java-11 on the path but Java-8 gets used instead, or set mb_jetty_port to 8211 and 3001 gets used instead (msg "Warning: environ value 8211 for key :mb-jetty-port has been overwritten with 3001").

15

For the env var issue you're having make sure you delete .lein-env from your Metabase directory if it exists as described in https://github.com/metabase/metabase/wiki/Migrating-from-Leiningen-to-tools.deps#custom-env-var-values

16

RE building the driver:

  1. I think you'll have to specify an absolute path for now because the driver build script is being stricter than it needs to be. Don't forget you can specify that stuff inline tho, it doesn't need to be hardcoded in your deps.edn file. Worst-case scenario you could put together a shell script to determine that and pass it in. We're working on fixing this in the future
  2. You do not need any metabase-core JARs anymore
  3. To get tests to run, you just need to make sure the driver sources and tests are on the same classpath. I'd make a profile in ~/.clojure/deps.edn that adds those drivers as :extra-paths (or specify inline with -Sdeps) then just run tests from the Metabase directory e.g. clojure -X:dev:user/my-driver:test as described in https://github.com/metabase/sudoku-driver#hacking-on-the-driver-locally and https://github.com/metabase/metabase/wiki/Migrating-from-Leiningen-to-tools.deps#example-commands
  4. Besides the fix I mentioned above make sure your $JAVA_HOME is set correctly

I'd also suggest taking a look at https://github.com/dacort/metabase-trino-driver which is another example of a Metabase driver written with the new tooling, also https://github.com/metabase/metabase/pull/17606

17

@camsaul thanks for these tips. It would be really appreciated if they could be incorporated into updated documentation. It would be especially helpful if the Sudoku driver had some of it's own tests so we driver authors have a realistic and working template to follow. I would offer to help with both but at this stage I don't understand how it should work. Getting the custom driver to build is preventing from being able to upgrade to v0.41+.

I tried again with Metabase tag "v1.42.0-preview1". I did not have a .lein-env file anywhere in my home dir or subfolders. I have used a fresh clone of Metabase, and cleared the .cpcache and target folders for the custom driver. I added the suggested ~/.clojure/deps.edn alias (with appropriate values/paths), and set up the same deps.edn structure as the Sudoku driver except with absolute paths for all values. Then ran clojure -X:deps prep as described in the Wiki.

The current issue is when running clojure -X:dev:build for the driver. It looks like there is some new validation on the resources/metabase-plugin.yaml file (maybe PR 19240)? For example the below errors don't like the "lazy-load" property which appears in built-in drivers as well.

      Plugin manifest found; validating it
-- Relevant specs -------

:build-drivers.lint-manifest-file/single-driver:
  (clojure.spec.alpha/keys
   :req-un
   [:build-drivers.lint-manifest-file/name :build-drivers.lint-manifest-file/lazy-load]
   :opt-un
   [:build-drivers.lint-manifest-file/parent
    :build-drivers.lint-manifest-file/display-name
    :build-drivers.lint-manifest-file/abstract
    :build-drivers.lint-manifest-file/connection-properties
    :build-drivers.lint-manifest-file/connection-properties-include-tunnel-config])
:build-drivers.lint-manifest-file/driver:
  (clojure.spec.alpha/or
   :single-driver
   :build-drivers.lint-manifest-file/single-driver
   :multiple-drivers
   (clojure.spec.alpha/coll-of :build-drivers.lint-manifest-file/single-driver))
:build-drivers.lint-manifest-file/plugin-manifest:
  (keys
   :req-un
   [:build-drivers.lint-manifest-file/info :build-drivers.lint-manifest-file/driver]
   :opt-un
   [:build-drivers.lint-manifest-file/init])

-- Spec failed --------------------

  {:info ..., :driver [:lazy-load true], :init ...}
                      ^^^^^^^^^^^^^^^^^

should satisfy

  map?

...
-------------------------
Detected 6 errors
Step failed: Driver verification failed: plugin manifest was invalid; see full explanation above
{:via
 [{:type clojure.lang.ExceptionInfo,
   :message "Driver verification failed: plugin manifest was invalid; see full explanation above",
   :data {:invalid-driver :custom-driver},
   :at [build_drivers.verify$verify_has_plugin_manifest$fn__18204 invoke "verify.clj" 55]}],
 :trace
 [[build_drivers.verify$verify_has_plugin_manifest$fn__18204 invoke "verify.clj" 55]
  [metabuild_common.steps$do_step invokeStatic "steps.clj" 85]
  [metabuild_common.steps$do_step invoke "steps.clj" 79]
...

I get the same kind of error message a few other settings as well:

  1. "Should satisfy: string?" for driver.connection-properties.custom-setting.placeholder: false
  2. "Should satisfy: map?" for driver.name: my-driver
  3. "Should satisfy: map?" for driver.display-name: My Driver
  4. "Should satisfy: map?" for driver.lazy-load: true
  5. "Should satisfy: map?" for driver.parent: postgres
  6. "Should satisfy: map?" for driver.connection-properties.custom-setting: as below
driver:
  connection-properties:
    - name: my-config
      display-name: My config
      placeholder: false

Thanks for any further help you can provide.

18

Hey @lindsay-stevens - I ran into this same error too. I'm fairly certain this was just a single syntax error in my plugin yaml file, but the verification step raised a bunch of other errors.

In my case, I had improper "merge" syntax.

Before:

    - merge:
        - password
        - display-name: Secret Key
        - name: secret_key

After:

    - merge:
        - password
        - display-name: Secret Key
          name: secret_key

This didn't entirely make sense to me...but it worked. ¯\(ツ)/¯

This is my current working verison of my Dockerfile for github.com/dacort/metabase-athena-driver - it builds, but I'm having trouble with my JDBC driver currently...

ARG METABASE_VERSION=v0.42.0-preview1

FROM clojure:openjdk-11-tools-deps-slim-buster AS build

# Requirements for building the driver
RUN apt-get update && \
    apt-get install -y \
    curl \
    make \
    unzip \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

# Set our base workdir
WORKDIR /build

# We need to retrieve metabase source
# Due to how ARG and FROM interact, we need to re-use the same ARG
# Ref: https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact
ARG METABASE_VERSION
RUN curl -Lo - https://github.com/metabase/metabase/archive/refs/tags/${METABASE_VERSION}.tar.gz | tar -xz \
    && mv metabase-* metabase

# Driver source goes inside metabase source so we can use their build scripts
WORKDIR /build/driver
COPY Makefile project.clj deps.edn ./
COPY src/ ./src
COPY resources/ ./resources

# Fetch our Athena JDBC driver
RUN make download-jar

# Now build the driver (this didn't work, but may have been the yaml syntax error...)
# WORKDIR /build/metabase
# RUN bin/build-driver.sh athena

# Now build Metabase dependencies first and then the driver
# We need to build java deps and then spark-sql deps
# Ref: https://github.com/metabase/metabase/wiki/Migrating-from-Leiningen-to-tools.deps#preparing-dependencies
WORKDIR /build/metabase
RUN clojure -X:deps prep
WORKDIR /build/metabase/modules/drivers
RUN clojure -X:deps prep

WORKDIR /build/driver
RUN clojure -X:dev:build

# Now we can run Metabase with our built driver
FROM metabase/metabase:${METABASE_VERSION}

COPY --chown=2000:2000 --from=build \
    /build/driver/target/athena.metabase-driver.jar \
    /plugins/athena.metabase-driver.jar