Hi, Wondering if anyone is able to help troubleshoot an issue I'm having attempting to update metabase running on docker inside AWS ECS and a postgres database.
Since version 50.13 it has been crashing on startup, and from reading the logs it's an error with metabase database migration.
Logs below (truncated due to character limit):
| 1722034270421 | Warning: environ value jdk-11.0.24+8 for key :java-version has been overwritten with 11.0.24
| 1722034286046 | 2024-07-26 22:51:26,042 INFO metabase.core :: |
| 1722034286046 | Metabase v0.50.17 (afd6b17)
| 1722034286046 | Metabase Enterprise Edition extensions are NOT PRESENT. |
| 1722034286050 | 2024-07-26 22:51:26,050 INFO metabase.core :: Starting Metabase in STANDALONE mode |
| 1722034286096 | 2024-07-26 22:51:26,096 INFO metabase.server :: Launching Embedded Jetty Webserver with config:
| 1722034286165 | 2024-07-26 22:51:26,165 INFO metabase.core :: Starting Metabase version v0.50.17 (afd6b17) ... |
| 1722034286176 | 2024-07-26 22:51:26,176 INFO metabase.core :: System info: |
| 1722034286176 | {"file.encoding" "UTF-8", |
| 1722034286176 | "java.runtime.name" "OpenJDK Runtime Environment", |
| 1722034286176 | "java.runtime.version" "11.0.24+8", |
| 1722034286176 | "java.vendor" "Eclipse Adoptium", |
| 1722034286176 | "java.vendor.url" "https://adoptium.net/", |
| 1722034286176 | "java.version" "11.0.24", |
| 1722034286176 | "java.vm.name" "OpenJDK 64-Bit Server VM", |
| 1722034286176 | "java.vm.version" "11.0.24+8", |
| 1722034286176 | "os.name" "Linux", |
| 1722034286176 | "os.version" "5.10.220-209.867.amzn2.x86_64", |
| 1722034286176 | "user.language" "en",
| 1722034287520 | 2024-07-26 22:51:27,520 INFO driver.impl :: e[34mRegistered abstract driver :hive-likee[0m (parents: [:sql-jdbc]) 
|
| 1722034287520 | 2024-07-26 22:51:27,520 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sparksql...e[0m |
| 1722034287520 | 2024-07-26 22:51:27,520 INFO driver.impl :: e[34mRegistered driver :sparksqle[0m (parents: [:hive-like]) |
| 1722034287523 | 2024-07-26 22:51:27,522 INFO plugins.dependencies :: e[31mMetabase cannot initialize plugin Metabase Vertica Driver due to required dependencies.e[0m Metabase requires the Vertica JDBC driver in order to connect to Vertica databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/vertica.html for more details. |
| 1722034287523 | 2024-07-26 22:51:27,523 INFO plugins.dependencies :: Metabase Vertica Driver dependency {:class com.vertica.jdbc.Driver} satisfied? false |
| 1722034287523 | 2024-07-26 22:51:27,523 INFO plugins.dependencies :: e[33mPlugins with unsatisfied deps: ["Metabase Oracle Driver" "Metabase Vertica Driver"]e[0m |
| 1722034287533 | 2024-07-26 22:51:27,532 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :presto-jdbc...e[0m |
| 1722034287533 | 2024-07-26 22:51:27,533 INFO driver.impl :: e[34mRegistered driver :presto-jdbce[0m (parents: [:sql-jdbc]) |
| 1722034287543 | 2024-07-26 22:51:27,543 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute... |
| 1722034287549 | 2024-07-26 22:51:27,549 INFO db.setup :: e[36mVerifying postgres Database Connection ...e[0m |
| 1722034287985 | 2024-07-26 22:51:27,985 INFO db.setup :: Successfully verified PostgreSQL 11.22 application database connection. 
|
| 1722034287986 | 2024-07-26 22:51:27,986 INFO db.setup :: e[36mChecking if a database downgrade is required...e[0m |
| 1722034288997 | 2024-07-26 22:51:28,997 INFO db.setup :: Running Database Migrations... |
| 1722034288998 | 2024-07-26 22:51:28,997 INFO db.setup :: Setting up Liquibase... |
| 1722034289266 | 2024-07-26 22:51:29,266 INFO db.setup :: Liquibase is ready. |
| 1722034289268 | 2024-07-26 22:51:29,267 INFO db.liquibase :: Checking if Database has unrun migrations... |
| 1722034289959 | 2024-07-26 22:51:29,959 INFO db.liquibase :: Database has unrun migrations. Checking if migration lock is taken... |
| 1722034289994 | 2024-07-26 22:51:29,994 INFO db.liquibase :: No migration lock found. |
| 1722034289994 | 2024-07-26 22:51:29,994 INFO db.liquibase :: Migration lock acquired. |
| 1722034290419 | 2024-07-26 22:51:30,418 INFO db.liquibase :: Running 5 migrations ... |
| 1722034290939 | 2024-07-26 22:51:30,928 ERROR liquibase.changelog :: ChangeSet migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:07::calherries encountered an exception. |
| 1722034290939 | liquibase.exception.DatabaseException: ERROR: syntax error at or near "(" |
| 1722034290939 | Position: 87 [Failed SQL: (0) ALTER TABLE metabase_field ADD COLUMN unique_field_helper INTEGER GENERATED ALWAYS AS ( |
| 1722034290939 | CASE WHEN is_defective_duplicate = TRUE THEN NULL ELSE (CASE WHEN parent_id IS NULL THEN 0 ELSE parent_id END) END |
| 1722034290939 | ) STORED] |
| 1722034290939 | at metabase.db.liquibase$migrate_up_if_needed_BANG_$fn__44601.invoke(liquibase.clj:360) |
| 1722034290939 | at metabase.db.liquibase$run_in_scope_locked$reify__44597.run(liquibase.clj:325) |
| 1722034290939 | at liquibase.Scope.lambda$child$0(Scope.java:186) |
| 1722034290939 | at liquibase.Scope.child(Scope.java:195) |
| 1722034290939 | at liquibase.Scope.child(Scope.java:185) |
| 1722034290939 | at liquibase.Scope.child(Scope.java:164) |
| 1722034290939 | at metabase.db.liquibase$run_in_scope_locked.invokeStatic(liquibase.clj:318) |
| 1722034290939 | at metabase.db.liquibase$run_in_scope_locked.invoke(liquibase.clj:301) |
| 1722034290939 | at metabase.db.liquibase$migrate_up_if_needed_BANG_.invokeStatic(liquibase.clj:349) |
| 1722034290939 | at metabase.db.liquibase$migrate_up_if_needed_BANG_.invoke(liquibase.clj:342) |
| 1722034290939 | at metabase.db.setup$migrate_BANG_$fn__53537.invoke(setup.clj:84) |
| 1722034290939 | at metabase.db.liquibase$do_with_liquibase$f_STAR___44538.invoke(liquibase.clj:140) |
| 1722034290939 | at metabase.db.liquibase$do_with_liquibase.invokeStatic(liquibase.clj:143) |
| 1722034290939 | at metabase.db.liquibase$do_with_liquibase.invoke(liquibase.clj:131) |
| 1722034290939 | at metabase.db.setup$migrate_BANG_.invokeStatic(setup.clj:73) |
| 1722034290939 | at metabase.db.setup$migrate_BANG_.doInvoke(setup.clj:55) |
| 1722034290939 | at clojure.lang.RestFn.invoke(RestFn.java:425) |
| 1722034290939 | at metabase.db.setup$run_schema_migrations_BANG_.invokeStatic(setup.clj:149) |
| 1722034290939 | at metabase.db.setup$run_schema_migrations_BANG_.invoke(setup.clj:144) |
| 1722034290939 | at metabase.db.setup$setup_db_BANG_$fn__53565$fn__53566.invoke(setup.clj:167) |
| 1722034290939 | at metabase.util.jvm$do_with_us_locale.invokeStatic(jvm.clj:239) |
| 1722034290939 | at metabase.util.jvm$do_with_us_locale.invoke(jvm.clj:225) |
| 1722034290939 | at metabase.db.setup$setup_db_BANG_$fn__53565.invoke(setup.clj:161) |
| 1722034290939 | at metabase.db.setup$setup_db_BANG_.invokeStatic(setup.clj:160) |
| 1722034290939 | at metabase.db.setup$setup_db_BANG_.invoke(setup.clj:153) |
| 1722034290939 | at metabase.db$setup_db_BANG_$fn__53590.invoke(db.clj:86) |
| 1722034290939 | at metabase.db$setup_db_BANG_.invokeStatic(db.clj:81) |
| 1722034290939 | at metabase.db$setup_db_BANG_.doInvoke(db.clj:68) |
| 1722034290939 | at clojure.lang.RestFn.invoke(RestFn.java:421) |
| 1722034290939 | at metabase.core$init_BANG__STAR_.invokeStatic(core.clj:117) |
| 1722034290939 | at metabase.core$init_BANG__STAR_.invoke(core.clj:98) |
| 1722034290939 | at metabase.core$init_BANG_.invokeStatic(core.clj:170) |
| 1722034290939 | at metabase.core$init_BANG_.invoke(core.clj:165) |
| 1722034290939 | at metabase.core$start_normally.invokeStatic(core.clj:182) |
| 1722034290939 | at metabase.core$start_normally.invoke(core.clj:176) |
| 1722034290939 | at metabase.core$entrypoint.invokeStatic(core.clj:215) |
| 1722034290939 | Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near "(" |
| 1722034290939 | Position: 87 |
| 1722034290939 | at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:464) |
| 1722034290939 | ... 86 more |
| 1722034290974 | UPDATE SUMMARY |
| 1722034290974 | Run: 5 |
| 1722034290974 | Previously run: 347 |
| 1722034290974 | Filtered out: 50 |
| 1722034290974 | ------------------------------- |
| 1722034290974 | Total change sets: 402 |
| 1722034290974 | FILTERED CHANGE SETS SUMMARY |
| 1722034290974 | DBMS mismatch: 50 |
| 1722034291064 | 2024-07-26 22:51:31,062 ERROR metabase.core :: Metabase Initialization FAILED |
| 1722034291064 | liquibase.exception.CommandExecutionException: liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:07::calherries: |
| 1722034291064 | Reason: liquibase.exception.DatabaseException: ERROR: syntax error at or near "(" |
| 1722034291064 | Position: 87 [Failed SQL: (0) ALTER TABLE metabase_field ADD COLUMN unique_field_helper INTEGER GENERATED ALWAYS AS ( |
| 1722034291064 | CASE WHEN is_defective_duplicate = TRUE THEN NULL ELSE (CASE WHEN parent_id IS NULL THEN 0 ELSE parent_id END) END |
| 1722034291064 | ) STORED]
| 1722034291064 | at metabase.db.setup$run_schema_migrations_BANG_.invokeStatic(setup.clj:149) | |
| 1722034291064 | Caused by: liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:07::calherries: |
| 1722034291064 | Reason: liquibase.exception.DatabaseException: ERROR: syntax error at or near "(" |
| 1722034291064 | Position: 87 [Failed SQL: (0) ALTER TABLE metabase_field ADD COLUMN unique_field_helper INTEGER GENERATED ALWAYS AS ( |
| 1722034291064 | CASE WHEN is_defective_duplicate = TRUE THEN NULL ELSE (CASE WHEN parent_id IS NULL THEN 0 ELSE parent_id END) END |
| 1722034291064 | ) STORED] |
| 1722034291064 | Caused by: liquibase.exception.MigrationFailedException: Migration failed for changeset migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:07::calherries: |
| 1722034291064 | Reason: liquibase.exception.DatabaseException: ERROR: syntax error at or near "(" |
| 1722034291064 | Position: 87 [Failed SQL: (0) ALTER TABLE metabase_field ADD COLUMN unique_field_helper INTEGER GENERATED ALWAYS AS ( |
| 1722034291064 | CASE WHEN is_defective_duplicate = TRUE THEN NULL ELSE (CASE WHEN parent_id IS NULL THEN 0 ELSE parent_id END) END |
| 1722034291064 | ) STORED]
| 1722034291064 | Caused by: liquibase.exception.DatabaseException: ERROR: syntax error at or near "(" |
| 1722034291064 | Position: 87 [Failed SQL: (0) ALTER TABLE metabase_field ADD COLUMN unique_field_helper INTEGER GENERATED ALWAYS AS ( |
| 1722034291064 | CASE WHEN is_defective_duplicate = TRUE THEN NULL ELSE (CASE WHEN parent_id IS NULL THEN 0 ELSE parent_id END) END |
| 1722034291064 | ) STORED] | |
| 1722034291064 | Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near "(" |
| 1722034291064 | Position: 87
| 1722034291068 | 2024-07-26 22:51:31,067 INFO metabase.core :: Metabase Shutting Down ... |
| 1722034291068 | 2024-07-26 22:51:31,068 INFO metabase.server :: Shutting Down Embedded Jetty Webserver |
| 1722034291077 | 2024-07-26 22:51:31,077 WARN db.liquibase :: () |
| 1722034291077 | 2024-07-26 22:51:31,077 INFO metabase.core :: Metabase Shutdown COMPLETE | Unable to update Metabase due to migration error