Hello, I am trying to upgrade my current metabase version 0.29.3 to latest running on AWS Elastic Beanstalk using a postgres db. I follow normal upgrade procedure and am getting a failed db migration. It looks like a qrtz.job issue, see logs below.
Are there any simple ways of clearing these jobs, etc?
Thanks in advance. Here is what I think is the key error.
09-13 07:43:00 e[1mERROR metabase.coree[0m :: Metabase Initialization FAILED
liquibase.exception.MigrationFailedException: Migration failed for change set migrations/000_migrations.yaml::89::camsaul:
Reason: liquibase.exception.DatabaseException: ERROR: relation "qrtz_job_details" already exists
Full logs
09-13 07:41:19 e[1mINFO metabase.utile[0m :: Loading Metabase...
09-13 07:41:20 e[1mINFO metabase.utile[0m :: Maximum memory available to JVM: 3.5 GB
09-13 07:41:22 e[1mINFO util.encryptione[0m :: Saved credentials encryption is DISABLED for this Metabase instance. 🔓
For more information, see https://www.metabase.com/docs/latest/operations-guide/start.html#encrypting-your-database-connection-details-at-rest
09-13 07:41:26 e[1mINFO metabase.drivere[0m :: e[34mRegistered abstract driver :sqle[0m 🚚
09-13 07:41:30 e[1mINFO metabase.coree[0m :: Starting Metabase in STANDALONE mode
09-13 07:41:30 e[1mINFO metabase.servere[0m :: Launching Embedded Jetty Webserver with config:
{:port 3000, :host "0.0.0.0"}
09-13 07:41:30 e[1mINFO metabase.coree[0m :: Starting Metabase version v0.32.0 (6970194 release-0.32.0) ...
09-13 07:41:30 e[1mINFO metabase.coree[0m :: System timezone is 'GMT' ...
09-13 07:41:30 e[1mINFO metabase.pluginse[0m :: e[35mLoading plugins in /plugins...e[0m
e[32mExtract file /modules/sparksql.metabase-driver.jar -> /plugins/sparksql.metabase-driver.jar took 418 mse[0m
e[32mExtract file /modules/bigquery.metabase-driver.jar -> /plugins/bigquery.metabase-driver.jar took 3 mse[0m
e[32mExtract file /modules/sqlserver.metabase-driver.jar -> /plugins/sqlserver.metabase-driver.jar took 8 mse[0m
e[32mExtract file /modules/sqlite.metabase-driver.jar -> /plugins/sqlite.metabase-driver.jar took 19 mse[0m
e[32mExtract file /modules/vertica.metabase-driver.jar -> /plugins/vertica.metabase-driver.jar took 470 µse[0m
e[32mExtract file /modules/redshift.metabase-driver.jar -> /plugins/redshift.metabase-driver.jar took 15 mse[0m
e[32mExtract file /modules/presto.metabase-driver.jar -> /plugins/presto.metabase-driver.jar took 1 mse[0m
e[32mExtract file /modules/druid.metabase-driver.jar -> /plugins/druid.metabase-driver.jar took 3 mse[0m
e[32mExtract file /modules/mongo.metabase-driver.jar -> /plugins/mongo.metabase-driver.jar took 41 mse[0m
e[32mExtract file /modules/google.metabase-driver.jar -> /plugins/google.metabase-driver.jar took 19 mse[0m
e[32mExtract file /modules/oracle.metabase-driver.jar -> /plugins/oracle.metabase-driver.jar took 674 µse[0m
e[32mExtract file /modules/snowflake.metabase-driver.jar -> /plugins/snowflake.metabase-driver.jar took 94 mse[0m
e[32mExtract file /modules/googleanalytics.metabase-driver.jar -> /plugins/googleanalytics.metabase-driver.jar took 4 mse[0m
09-13 07:41:31 e[1mDEBUG plugins.classloadere[0m :: Setting current thread context classloader to NEWLY CREATED classloader clojure.lang.DynamicClassLoader@167fdc7d...
09-13 07:41:31 e[1mINFO plugins.dependenciese[0m :: e[31mMetabase cannot initialize plugin Metabase Vertica Driver due to required dependencies.e[0m Metabase requires the Vertica JDBC driver in order to connect to Vertica databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/vertica.html for more details.
09-13 07:41:31 e[1mINFO plugins.dependenciese[0m :: Metabase Vertica Driver dependency {:class com.vertica.jdbc.Driver} satisfied? false
09-13 07:41:31 e[1mINFO plugins.dependenciese[0m :: e[33mPlugins with unsatisfied deps: ["Metabase Vertica Driver"]e[0m
09-13 07:41:31 e[1mINFO plugins.lazy-loaded-drivere[0m :: e[35mRegistering lazy loading driver :mongo...e[0m
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[34mRegistered driver :mongoe[0m 🚚
09-13 07:41:31 e[1mINFO plugins.lazy-loaded-drivere[0m :: e[35mRegistering lazy loading driver :google...e[0m
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[34mRegistered abstract driver :googlee[0m 🚚
09-13 07:41:31 e[1mINFO plugins.lazy-loaded-drivere[0m :: e[35mRegistering lazy loading driver :hive-like...e[0m
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[34mRegistered abstract driver :sql-jdbce[0m (parents: :sql) 🚚
e[32mLoad driver :sql-jdbc took 42 mse[0m
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[34mRegistered abstract driver :hive-likee[0m (parents: #{:sql-jdbc}) 🚚
09-13 07:41:31 e[1mINFO plugins.lazy-loaded-drivere[0m :: e[35mRegistering lazy loading driver :sparksql...e[0m
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[34mRegistered driver :sparksqle[0m (parents: #{:hive-like}) 🚚
09-13 07:41:31 e[1mINFO plugins.lazy-loaded-drivere[0m :: e[35mRegistering lazy loading driver :sqlite...e[0m
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[34mRegistered driver :sqlitee[0m (parents: #{:sql-jdbc}) 🚚
09-13 07:41:31 e[1mINFO plugins.dependenciese[0m :: Plugin 'Metabase Google Analytics Driver' depends on plugin 'Metabase Google Drivers Shared Dependencies'
09-13 07:41:31 e[1mINFO plugins.dependenciese[0m :: Metabase Google Analytics Driver dependency {:plugin Metabase Google Drivers Shared Dependencies} satisfied? true
09-13 07:41:31 e[1mINFO plugins.lazy-loaded-drivere[0m :: e[35mRegistering lazy loading driver :googleanalytics...e[0m
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[34mRegistered driver :googleanalyticse[0m (parents: #{:google}) 🚚
09-13 07:41:31 e[1mINFO plugins.lazy-loaded-drivere[0m :: e[35mRegistering lazy loading driver :snowflake...e[0m
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[34mRegistered driver :snowflakee[0m (parents: #{:sql-jdbc}) 🚚
09-13 07:41:31 e[1mINFO plugins.dependenciese[0m :: e[31mMetabase cannot initialize plugin Metabase Oracle Driver due to required dependencies.e[0m Metabase requires the Oracle JDBC driver in order to connect to Oracle databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/oracle.html for more details.
09-13 07:41:31 e[1mINFO plugins.dependenciese[0m :: Metabase Oracle Driver dependency {:class oracle.jdbc.OracleDriver} satisfied? false
09-13 07:41:31 e[1mINFO plugins.dependenciese[0m :: e[33mPlugins with unsatisfied deps: ["Metabase Oracle Driver" "Metabase Vertica Driver"]e[0m
09-13 07:41:31 e[1mINFO plugins.lazy-loaded-drivere[0m :: e[35mRegistering lazy loading driver :presto...e[0m
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[34mRegistered driver :prestoe[0m (parents: #{:sql}) 🚚
09-13 07:41:31 e[1mINFO plugins.lazy-loaded-drivere[0m :: e[35mRegistering lazy loading driver :druid...e[0m
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[34mRegistered driver :druide[0m 🚚
09-13 07:41:31 e[1mINFO plugins.dependenciese[0m :: Plugin 'Metabase BigQuery Driver' depends on plugin 'Metabase Google Drivers Shared Dependencies'
09-13 07:41:31 e[1mINFO plugins.dependenciese[0m :: Metabase BigQuery Driver dependency {:plugin Metabase Google Drivers Shared Dependencies} satisfied? true
09-13 07:41:31 e[1mINFO plugins.lazy-loaded-drivere[0m :: e[35mRegistering lazy loading driver :bigquery...e[0m
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[34mRegistered driver :bigquerye[0m (parents: #{:sql :google}) 🚚
09-13 07:41:31 e[1mINFO plugins.lazy-loaded-drivere[0m :: e[35mRegistering lazy loading driver :sqlserver...e[0m
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[34mRegistered driver :sqlservere[0m (parents: #{:sql-jdbc}) 🚚
09-13 07:41:31 e[1mINFO plugins.lazy-loaded-drivere[0m :: e[35mRegistering lazy loading driver :redshift...e[0m
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[34mRegistered driver :postgrese[0m (parents: :sql-jdbc) 🚚
e[32mLoad driver :postgres took 22 mse[0m
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[34mRegistered driver :redshifte[0m (parents: #{:postgres}) 🚚
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[34mRegistered driver :mysqle[0m (parents: :sql-jdbc) 🚚
e[32mLoad driver :mysql took 23 mse[0m
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[34mRegistered driver :h2e[0m (parents: :sql-jdbc) 🚚
e[32mLoad driver :h2 took 21 mse[0m
09-13 07:41:31 e[1mINFO metabase.coree[0m :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
09-13 07:41:31 e[1mINFO metabase.dbe[0m :: e[36mVerifying postgres Database Connection ...e[0m
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[33mInitializing driver :sql...e[0m
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[33mInitializing driver :sql-jdbc...e[0m
09-13 07:41:31 e[1mINFO metabase.drivere[0m :: e[33mInitializing driver :postgres...e[0m
09-13 07:41:31 e[1mINFO metabase.dbe[0m :: Verify Database Connection ... ✅
09-13 07:41:31 e[1mINFO metabase.dbe[0m :: Running Database Migrations...
09-13 07:41:32 e[1mINFO metabase.dbe[0m :: Setting up Liquibase...
09-13 07:41:32 e[1mINFO metabase.dbe[0m :: Liquibase is ready.
09-13 07:41:32 e[1mINFO metabase.dbe[0m :: Checking if Database has unrun migrations...
09-13 07:41:33 e[1mINFO metabase.dbe[0m :: Database has unrun migrations. Waiting for migration lock to be cleared...
09-13 07:41:33 e[1mINFO metabase.dbe[0m :: Migration lock is cleared. Running migrations...
09-13 07:41:34 e[1mERROR changelog.ChangeSete[0m :: Change Set migrations/000_migrations.yaml::89::camsaul failed. Error: ERROR: relation "qrtz_job_details" already exists [Failed SQL: CREATE TABLE public.QRTZ_JOB_DETAILS (SCHED_NAME VARCHAR(120) NOT NULL, JOB_NAME VARCHAR(200) NOT NULL, JOB_GROUP VARCHAR(200) NOT NULL, DESCRIPTION VARCHAR(250), JOB_CLASS_NAME VARCHAR(250) NOT NULL, IS_DURABLE BOOLEAN NOT NULL, IS_NONCONCURRENT BOOLEAN NOT NULL, IS_UPDATE_DATA BOOLEAN NOT NULL, REQUESTS_RECOVERY BOOLEAN NOT NULL, JOB_DATA BYTEA)]
09-13 07:41:34 e[1mWARN metabase.utile[0m :: e[31mauto-retry metabase.db$run_schema_migrations_BANG_$fn__14466@6ac3ef0b: Migration failed for change set migrations/000_migrations.yaml::89::camsaul:
Reason: liquibase.exception.DatabaseException: ERROR: relation "qrtz_job_details" already exists [Failed SQL: CREATE TABLE public.QRTZ_JOB_DETAILS (SCHED_NAME VARCHAR(120) NOT NULL, JOB_NAME VARCHAR(200) NOT NULL, JOB_GROUP VARCHAR(200) NOT NULL, DESCRIPTION VARCHAR(250), JOB_CLASS_NAME VARCHAR(250) NOT NULL, IS_DURABLE BOOLEAN NOT NULL, IS_NONCONCURRENT BOOLEAN NOT NULL, IS_UPDATE_DATA BOOLEAN NOT NULL, REQUESTS_RECOVERY BOOLEAN NOT NULL, JOB_DATA BYTEA)]e[0m
09-13 07:41:34 e[1mINFO metabase.dbe[0m :: Setting up Liquibase...
09-13 07:41:34 e[1mINFO metabase.dbe[0m :: Liquibase is ready.
09-13 07:41:34 e[1mINFO metabase.dbe[0m :: Checking if Database has unrun migrations...
09-13 07:41:34 e[1mINFO metabase.dbe[0m :: Database has unrun migrations. Waiting for migration lock to be cleared...
09-13 07:41:34 e[1mINFO metabase.dbe[0m :: Migration lock is cleared. Running migrations...
09-13 07:41:34 e[1mERROR changelog.ChangeSete[0m :: Change Set migrations/000_migrations.yaml::89::camsaul failed. Error: ERROR: relation "qrtz_job_details" already exists [Failed SQL: CREATE TABLE public.QRTZ_JOB_DETAILS (SCHED_NAME VARCHAR(120) NOT NULL, JOB_NAME VARCHAR(200) NOT NULL, JOB_GROUP VARCHAR(200) NOT NULL, DESCRIPTION VARCHAR(250), JOB_CLASS_NAME VARCHAR(250) NOT NULL, IS_DURABLE BOOLEAN NOT NULL, IS_NONCONCURRENT BOOLEAN NOT NULL, IS_UPDATE_DATA BOOLEAN NOT NULL, REQUESTS_RECOVERY BOOLEAN NOT NULL, JOB_DATA BYTEA)]
09-13 07:41:34 e[1mERROR metabase.coree[0m :: Metabase Initialization FAILED
liquibase.exception.MigrationFailedException: Migration failed for change set migrations/000_migrations.yaml::89::camsaul:
Reason: liquibase.exception.DatabaseException: ERROR: relation "qrtz_job_details" already exists [Failed SQL: CREATE TABLE public.QRTZ_JOB_DETAILS (SCHED_NAME VARCHAR(120) NOT NULL, JOB_NAME VARCHAR(200) NOT NULL, JOB_GROUP VARCHAR(200) NOT NULL, DESCRIPTION VARCHAR(250), JOB_CLASS_NAME VARCHAR(250) NOT NULL, IS_DURABLE BOOLEAN NOT NULL, IS_NONCONCURRENT BOOLEAN NOT NULL, IS_UPDATE_DATA BOOLEAN NOT NULL, REQUESTS_RECOVERY BOOLEAN NOT NULL, JOB_DATA BYTEA)]
at liquibase.changelog.ChangeSet.execute(ChangeSet.java:637)
at liquibase.changelog.visitor.UpdateVisitor.visit(UpdateVisitor.java:53)
at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:83)
at liquibase.Liquibase.update(Liquibase.java:202)
at liquibase.Liquibase.update(Liquibase.java:179)
at liquibase.Liquibase.update(Liquibase.java:175)
at metabase.db$migrate_up_if_needed_BANG_.invokeStatic(db.clj:209)
at metabase.db$migrate_up_if_needed_BANG_.invoke(db.clj:196)
at metabase.db$migrate_BANG_$fn__14448.invoke(db.clj:313)
at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:799)
at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:769)
at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:834)
at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:769)
at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:782)
at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:769)
at metabase.db$migrate_BANG_.invokeStatic(db.clj:301)
at metabase.db$migrate_BANG_.invoke(db.clj:282)
at metabase.db$run_schema_migrations_BANG_$fn__14466.invoke(db.clj:446)
at metabase.util$do_with_auto_retries.invokeStatic(util.clj:428)
at metabase.util$do_with_auto_retries.invoke(util.clj:420)
at metabase.util$do_with_auto_retries.invokeStatic(util.clj:432)
at metabase.util$do_with_auto_retries.invoke(util.clj:420)
at metabase.db$run_schema_migrations_BANG_.invokeStatic(db.clj:445)
at metabase.db$run_schema_migrations_BANG_.invoke(db.clj:428)
at metabase.db$setup_db_BANG_$fn__14475.invoke(db.clj:468)
at metabase.util$do_with_us_locale.invokeStatic(util.clj:683)
at metabase.util$do_with_us_locale.invoke(util.clj:669)
at metabase.db$setup_db_BANG_.invokeStatic(db.clj:466)
at metabase.db$setup_db_BANG_.doInvoke(db.clj:460)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at metabase.core$init_BANG_.invokeStatic(core.clj:72)
at metabase.core$init_BANG_.invoke(core.clj:51)
at metabase.core$start_normally.invokeStatic(core.clj:118)
at metabase.core$start_normally.invoke(core.clj:112)
at metabase.core$_main.invokeStatic(core.clj:138)
at metabase.core$_main.doInvoke(core.clj:133)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at metabase.core.main(Unknown Source)
Caused by: liquibase.exception.DatabaseException: ERROR: relation "qrtz_job_details" already exists [Failed SQL: CREATE TABLE public.QRTZ_JOB_DETAILS (SCHED_NAME VARCHAR(120) NOT NULL, JOB_NAME VARCHAR(200) NOT NULL, JOB_GROUP VARCHAR(200) NOT NULL, DESCRIPTION VARCHAR(250), JOB_CLASS_NAME VARCHAR(250) NOT NULL, IS_DURABLE BOOLEAN NOT NULL, IS_NONCONCURRENT BOOLEAN NOT NULL, IS_UPDATE_DATA BOOLEAN NOT NULL, REQUESTS_RECOVERY BOOLEAN NOT NULL, JOB_DATA BYTEA)]
at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:356)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:57)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:125)
at liquibase.database.AbstractJdbcDatabase.execute(AbstractJdbcDatabase.java:1229)
at liquibase.database.AbstractJdbcDatabase.executeStatements(AbstractJdbcDatabase.java:1211)
at liquibase.changelog.ChangeSet.execute(ChangeSet.java:600)
... 39 more
Caused by: org.postgresql.util.PSQLException: ERROR: relation "qrtz_job_details" already exists
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:266)
at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:352)
... 44 more
09-13 07:41:34 e[1mINFO metabase.coree[0m :: Metabase Shutting Down ...
09-13 07:41:34 e[1mINFO metabase.coree[0m :: Metabase Shutdown COMPLETE