No method in multimethod 'connection-details->spec' for dispatch value snowflake

jtruong · April 16, 2020, 7:24pm

I just upgraded metabase from 31 to v0.35.2, essentially by switching branch to release-0.35.x. I ran the build script, bin/build no-translations, then got the error:

{
  "data": {
    "cols": [],
    "rows": []
  },
  "running_time": 0,
  "row_count": 0,
  "error": "No method in multimethod 'connection-details->spec' for dispatch value: :snowflake",
  "context": "question",
  "stacktrace": [
    "clojure.lang.MultiFn.getFn(MultiFn.java:156)",
    "clojure.lang.MultiFn.invoke(MultiFn.java:233)",
    "--> driver.sql_jdbc.connection$create_pool_BANG_.invokeStatic(connection.clj:88)",
    "driver.sql_jdbc.connection$create_pool_BANG_.invoke(connection.clj:82)",
    "driver.sql_jdbc.connection$db__GT_pooled_connection_spec.invokeStatic(connection.clj:141)",
    "driver.sql_jdbc.connection$db__GT_pooled_connection_spec.invoke(connection.clj:123)",
    "driver.sql_jdbc.execute$datasource.invokeStatic(execute.clj:111)",
    "driver.sql_jdbc.execute$datasource.invoke(execute.clj:107)",
    "driver.sql_jdbc.execute$eval80484$fn__80485.invoke(execute.clj:157)",
    "driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:386)",
    "driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:377)",
    "driver.sql_jdbc$eval83356$fn__83357.invoke(sql_jdbc.clj:50)",
    "query_processor.context$executef.invokeStatic(context.clj:59)",
    "query_processor.context$executef.invoke(context.clj:48)",
    "query_processor.context.default$default_runf.invokeStatic(default.clj:69)",
    "query_processor.context.default$default_runf.invoke(default.clj:67)",
    "query_processor.context$runf.invokeStatic(context.clj:45)",
    "query_processor.context$runf.invoke(context.clj:39)",
    "query_processor.reducible$pivot.invokeStatic(reducible.clj:34)",
    "query_processor.reducible$pivot.invoke(reducible.clj:31)",
    "query_processor.middleware.mbql_to_native$mbql__GT_native$fn__50697.invoke(mbql_to_native.clj:26)",
    "query_processor.middleware.check_features$check_features$fn__49931.invoke(check_features.clj:42)",
    "query_processor.middleware.optimize_datetime_filters$optimize_datetime_filters$fn__50896.invoke(optimize_datetime_filters.clj:133)",
    "query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__55276.invoke(wrap_value_literals.clj:137)",
    "query_processor.middleware.annotate$add_column_info$fn__47918.invoke(annotate.clj:577)",
    "query_processor.middleware.permissions$check_query_permissions$fn__49790.invoke(permissions.clj:64)",
    "query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__51433.invoke(pre_alias_aggregations.clj:40)",
    "query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__50020.invoke(cumulative_aggregations.clj:61)",
    "query_processor.middleware.resolve_joins$resolve_joins$fn__52007.invoke(resolve_joins.clj:183)",
    "query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__45845.invoke(add_implicit_joins.clj:245)",
    "query_processor.middleware.limit$limit$fn__50675.invoke(limit.clj:38)",
    "query_processor.middleware.format_rows$format_rows$fn__50648.invoke(format_rows.clj:81)",
    "query_processor.middleware.desugar$desugar$fn__50092.invoke(desugar.clj:22)",
    "query_processor.middleware.binning$update_binning_strategy$fn__48478.invoke(binning.clj:229)",
    "query_processor.middleware.resolve_fields$resolve_fields$fn__49584.invoke(resolve_fields.clj:24)",
    "query_processor.middleware.add_dimension_projections$add_remapping$fn__45357.invoke(add_dimension_projections.clj:270)",
    "query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__45593.invoke(add_implicit_clauses.clj:147)",
    "query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__46088.invoke(add_source_metadata.clj:105)",
    "query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__51658.invoke(reconcile_breakout_and_order_by_bucketing.clj:98)",
    "query_processor.middleware.auto_bucket_datetimes$auto_bucket_datetimes$fn__48111.invoke(auto_bucket_datetimes.clj:125)",
    "query_processor.middleware.resolve_source_table$resolve_source_tables$fn__49639.invoke(resolve_source_table.clj:46)",
    "query_processor.middleware.parameters$substitute_parameters$fn__51409.invoke(parameters.clj:97)",
    "query_processor.middleware.resolve_referenced$resolve_referenced_card_resources$fn__49693.invoke(resolve_referenced.clj:80)",
    "query_processor.middleware.expand_macros$expand_macros$fn__50356.invoke(expand_macros.clj:158)",
    "query_processor.middleware.add_timezone_info$add_timezone_info$fn__46135.invoke(add_timezone_info.clj:15)",
    "query_processor.middleware.splice_params_in_response$splice_params_in_response$fn__55108.invoke(splice_params_in_response.clj:32)",
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__51675$fn__51679.invoke(resolve_database_and_driver.clj:33)",
    "driver$do_with_driver.invokeStatic(driver.clj:61)",
    "driver$do_with_driver.invoke(driver.clj:57)",
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__51675.invoke(resolve_database_and_driver.clj:27)",
    "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__50570.invoke(fetch_source_query.clj:243)",
    "query_processor.middleware.store$initialize_store$fn__55125$fn__55126.invoke(store.clj:11)",
    "query_processor.store$do_with_store.invokeStatic(store.clj:46)",
    "query_processor.store$do_with_store.invoke(store.clj:40)",
    "query_processor.middleware.store$initialize_store$fn__55125.invoke(store.clj:10)",
    "query_processor.middleware.cache$run_query_with_cache.invokeStatic(cache.clj:181)",
    "query_processor.middleware.cache$run_query_with_cache.invoke(cache.clj:172)",
    "query_processor.middleware.cache$maybe_return_cached_results$fn__49012.invoke(cache.clj:207)",
    "query_processor.middleware.validate$validate_query$fn__55142.invoke(validate.clj:10)",
    "query_processor.middleware.normalize_query$normalize$fn__50718.invoke(normalize_query.clj:22)",
    "query_processor.middleware.add_rows_truncated$add_rows_truncated$fn__45951.invoke(add_rows_truncated.clj:36)",
    "query_processor.middleware.results_metadata$record_and_return_metadata_BANG_$fn__55087.invoke(results_metadata.clj:128)",
    "query_processor.middleware.constraints$add_default_userland_constraints$fn__49955.invoke(constraints.clj:42)",
    "query_processor.middleware.process_userland_query$process_userland_query$fn__51524.invoke(process_userland_query.clj:136)",
    "query_processor.middleware.catch_exceptions$catch_exceptions$fn__49884.invoke(catch_exceptions.clj:166)",
    "query_processor.reducible$async_qp$qp_STAR___30044$thunk__30045.invoke(reducible.clj:100)",
    "query_processor.reducible$async_qp$qp_STAR___30044$fn__30047.invoke(reducible.clj:105)"
  ],
  "class": "class java.lang.IllegalArgumentException",
  "status": "failed",

flamber · April 16, 2020, 7:43pm

Hi @jtruong
Try cleaning out old drivers that might causing some problem:
./bin/build-drivers.sh clean

Can you also post the output of the following:

# OS and version
 sed -nE 's/^PRETTY_NAME="([^"]+)"$/\1/p' /etc/os-release
# Git branch and commit
 echo `git status | head -1 && git log -1 --format="commit %H"`
# Java version and architecture
 java -version
# Java variable configuration
 echo $JAVA_HOME
# Java path configuration
 echo $PATH | grep --color=never -oe "/usr/lib/jvm/[^:]*"
# Lein version
 lein --version
# NodeJS version
 nodejs --version
# Yarn version
 yarn --version

jtruong · April 16, 2020, 8:00pm

i had already cleaned forgot to mention that.

here is the output of the versions:

java version "11.0.5" 2019-10-15 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.5+10-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.5+10-LTS, mixed mode)
/Library/Java/JavaVirtualMachines/jdk-11.0.5.jdk/Contents/Home
Leiningen 2.9.1 on Java 11.0.5 Java HotSpot(TM) 64-Bit Server VM
v10.14.2 #node
1.21.1 #yarn

i am on mac os Catalina 10.15.3 and this commit:

$ git status | head -1 && git log -1 --format="commit %H"
HEAD detached at upstream/release-0.35.x
commit f3b285701cd3ffe35a2ab4d7b28892407515512e

flamber · April 16, 2020, 8:23pm

@jtruong Hmm… I hoped it was just some old driver leftovers.
So you’re not having any problems if you run the official release?
What happens if you replace the Snowflake driver inside your build with the one inside the official release (JAR-files are just a ZIP archive)?

jtruong · April 16, 2020, 9:57pm

yes, me too i will try replacing the driver, but it will take me some time to get accustomed with the process. thank you for the suggestion.

do you start locally with lein ring server? i’m looking at the bin/start script and i wonder if i should use that instead. (been using the former command since 31.)

flamber · April 16, 2020, 10:35pm

@jtruong lein ring server is currently not working, use lein run
Running lein is not the same a building. And ./bin/start just starts an existing JAR-file, so that’s not going to change anything for you.

Are you making any changes to the source code or are you simply just building because you can? If latter, then just use the official release.

I don’t use Snowflake, so cannot test this, but I’m sure it’s just something in your environment that it causing something, so try building on a different environment.

jtruong · April 16, 2020, 10:49pm

ah good to know, re: lein. i added some custom code to the frontend, so yea i guess i don’t really need to build. i’ll try the official release after giving a lein run a go.

the official release is this jar? https://downloads.metabase.com/v0.35.2/metabase.jar

in release-0.31.3, the driver is at src/metabase/driver/snowflake.clj and contains

(defn- connection-details->spec

whereas in release-0.35.x, the driver is at modules/drivers/snowflake/src/metabase/driver/snowflake.clj and contains

(defmethod sql-jdbc.conn/connection-details->spec :snowflake

not sure if that matters but putting it here to see if the difference in function signature might hint at something…

flamber · April 17, 2020, 8:58am

@jtruong
Yes, that’s the official download JAR.
All drivers have changed a lot since 0.31 - drivers are now modular, so Metabase could support third-party drivers better.
Otherwise try enabling TRACE on all logging - I have a feeling that something is failing before the error you’re seeing:
https://gist.github.com/dacort/19d92572b669adc4437e3e30c7ca85a4

jtruong · April 17, 2020, 3:32pm

Thanks, will try that too...

Update with TRACE on query-processor:

Could not locate metabase/driver/snowflake__init.class, metabase/driver/snowflake.clj or metabase/driver/snowflake.cljc on classpath

i think i've seen that before, actually... but now more prominent.

flamber · April 17, 2020, 4:15pm

@jtruong
Have you tried to manually replace the driver in the official release with the one in your build? If it works, then it must be something in your environment and I don’t know what I could be, so I cannot help anymore. You need to debug more yourself.

jtruong · April 17, 2020, 7:55pm

thanks for your help. unzipping those Jars; they’re huge!

jtruong · April 29, 2020, 5:15pm

i tried with the official release driver but still no dice… if you have other thoughts or suggestions, please send them my way

flamber · April 29, 2020, 5:20pm

@jtruong
Are you saying that you’re getting that error with a pure release JAR? Then open an issue on that - that’s a bug we need to look into. But if it is custom builds, then the problem is (likely) somewhere in the environment.
You might get more information if you enable DEBUG or TRACE logging:
https://www.metabase.com/docs/latest/operations-guide/log-configuration.html

jtruong · April 29, 2020, 7:35pm

no, i tried by pulling the snowflake driver from the official release into my environment.

but i also just tried running the latest release-0.35.x by itself (ie. outside of my environment), and the same issue exists:

[backend] 04-29 14:30:06 INFO driver.impl :: Initializing driver :snowflake...
[backend] 04-29 14:30:06 DEBUG plugins.init-steps :: Loading plugin namespace metabase.driver.snowflake...
[backend] 04-29 14:30:06 ERROR driver.util :: Database connection error
[backend] clojure.lang.ExceptionInfo: Could not locate metabase/driver/snowflake__init.class, metabase/driver/snowflake.clj or metabase/driver/snowflake.cljc on classpath. {:classloader #object[clojure.lang.DynamicClassLoader 0x4768b95c "clojure.lang.DynamicClassLoader@4768b95c"], :classpath-urls ()}
.
...
... huge stack trace ...
...
.
[backend] 	... 1 more
[backend] 04-29 14:30:06 DEBUG middleware.log :: POST /api/setup/validate 400 5.6 ms (0 DB calls) 
[backend] {:errors {:dbname "No method in multimethod 'connection-details->spec' for dispatch value: :snowflake"}}
[backend]

this is by running yarn dev, btw. I will open an issue.

flamber · April 29, 2020, 7:44pm

@jtruong What do you mean with latest release-0.35.x ? Are you building from that branch or did you download a release from https://downloads.metabase.com/latest/metabase.jar ?

jtruong · April 29, 2020, 8:44pm

i did both:

the driver i used in my environment was taken from the downloaded jar,
and i built from the release-0.35.x branch to test its driver and in a clean environment

issue: https://github.com/metabase/metabase/issues/12426

flamber · April 29, 2020, 8:47pm

@jtruong Okay, I meant that you should try to run the official JAR release - if that works, then the problem must be somewhere in your environment.

jtruong · April 29, 2020, 11:56pm

hmm works fine with the official release https://www.metabase.com/start/jar.html

so something with the build?.. i’ll update the issue.

jtruong · May 1, 2020, 1:58am

@flamber Do you know why starting any Metabase jar from the repo directory would cause Metabase to fail looking for the Snowflake driver?

I have two jars:

the official downloaded one
the built one from the repo’s release-0.35.x branch

If i run either one outside of the repo, eg.:

~/Downloads:
$ java -jar ./metabase.jar

# or

/some/dir:
$ java -jar ~/git/metabase/target/uberjar/metabase.jar

These run fine, and the Snowflake driver is found, but if i run both within the repo directory, they both fail finding it. Ie.

~/git/metabase (on branch: release-0.35.x):
$ java -jar ~/Downloads/metabase.jar

# or

~/git/metabase (on branch: release-0.35.x):
$ java -jar ./target/uberjar/metabase.jar

~/Downloads/metabase.jar was downloaded from https://www.metabase.com/start/jar.html.

Anyway, seems there is something in my repo directory that overrides some paths, or something…

flamber · May 1, 2020, 8:26am

@jtruong Since you’re having problems with drivers, then I guess you have a corrupted driver in plugins directory in your repo, which for some unknown reason doesn’t get replaced, when starting the JAR. Perhaps something to do with permissions. Have a look at the startup log, which will tell you if it is extracting the drivers.