At some point my docker container running metabase enterprise stopped working and won't restart properly due to the following error
2024-07-23 12:53:24,366 INFO metabase.util :: Maximum memory available to JVM: 1.9 GB
2024-07-23 12:53:28,672 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. 🔓
For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
2024-07-23 12:53:39,217 INFO driver.impl :: Registered abstract driver :sql 🚚
2024-07-23 12:53:39,228 INFO driver.impl :: Registered abstract driver :sql-jdbc (parents: [:sql]) 🚚
2024-07-23 12:53:39,239 INFO metabase.util :: Load driver :sql-jdbc took 67.6 ms
2024-07-23 12:53:39,240 INFO driver.impl :: Registered driver :h2 (parents: [:sql-jdbc]) 🚚
2024-07-23 12:53:39,558 INFO driver.impl :: Registered driver :mysql (parents: [:sql-jdbc]) 🚚
2024-07-23 12:53:39,617 INFO driver.impl :: Registered driver :postgres (parents: [:sql-jdbc]) 🚚
2024-07-23 12:53:45,219 INFO metabase.core ::
Metabase v1.50.14 (26d21b1)
Copyright © 2024 Metabase, Inc.
Metabase Enterprise Edition extensions are PRESENT.
Usage of Metabase Enterprise Edition features are subject to the Metabase Commercial License.See https://www.metabase.com/license/commercial/ for details.
2024-07-23 12:53:45,234 INFO metabase.core :: Starting Metabase in STANDALONE mode
2024-07-23 12:53:45,309 INFO metabase.server :: Launching Embedded Jetty Webserver with config:
{:port 3000}
2024-07-23 12:53:45,383 INFO metabase.core :: Starting Metabase version v1.50.14 (26d21b1) ...
2024-07-23 12:53:45,393 INFO metabase.core :: System info:
{"file.encoding" "UTF-8",
"java.runtime.name" "OpenJDK Runtime Environment",
"java.runtime.version" "11.0.24+8-LTS",
"java.vendor" "Amazon.com Inc.",
"java.vendor.url" "https://aws.amazon.com/corretto/",
"java.version" "11.0.24",
"java.vm.name" "OpenJDK 64-Bit Server VM",
"java.vm.version" "11.0.24+8-LTS",
"os.name" "Linux",
"os.version" "4.14.348-265.565.amzn2.x86_64",
"user.language" "en",
"user.timezone" "UTC"}
2024-07-23 12:53:45,396 INFO metabase.plugins :: Loading plugins in /home/ec2-user/plugins...
2024-07-23 12:53:46,314 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :snowflake...
2024-07-23 12:53:46,335 INFO driver.impl :: Registered driver :snowflake (parents: [:sql-jdbc]) 🚚
2024-07-23 12:53:46,355 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :sqlite...
2024-07-23 12:53:46,356 INFO driver.impl :: Registered driver :sqlite (parents: [:sql-jdbc]) 🚚
2024-07-23 12:53:46,391 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :bigquery-cloud-sdk...
2024-07-23 12:53:46,400 INFO driver.impl :: Registered driver :bigquery-cloud-sdk (parents: [:sql]) 🚚
2024-07-23 12:53:46,406 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :sqlserver...
2024-07-23 12:53:46,410 INFO driver.impl :: Registered driver :sqlserver (parents: [:sql-jdbc]) 🚚
2024-07-23 12:53:46,443 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :presto-jdbc...
2024-07-23 12:53:46,448 INFO driver.impl :: Registered driver :presto-jdbc (parents: [:sql-jdbc]) 🚚
2024-07-23 12:53:46,452 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :druid...
2024-07-23 12:53:46,455 INFO driver.impl :: Registered driver :druid 🚚
2024-07-23 12:53:46,468 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :hive-like...
2024-07-23 12:53:46,468 INFO driver.impl :: Registered abstract driver :hive-like (parents: [:sql-jdbc]) 🚚
2024-07-23 12:53:46,469 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :sparksql...
2024-07-23 12:53:46,470 INFO driver.impl :: Registered driver :sparksql (parents: [:hive-like]) 🚚
2024-07-23 12:53:46,488 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :oracle...
2024-07-23 12:53:46,489 INFO driver.impl :: Registered driver :oracle (parents: [:sql-jdbc]) 🚚
2024-07-23 12:53:46,502 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :mongo...
2024-07-23 12:53:46,504 INFO driver.impl :: Registered driver :mongo 🚚
2024-07-23 12:53:46,515 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :redshift...
2024-07-23 12:53:46,515 INFO driver.impl :: Registered driver :redshift (parents: [:postgres]) 🚚
2024-07-23 12:53:46,523 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :vertica...
2024-07-23 12:53:46,523 INFO driver.impl :: Registered driver :vertica (parents: [:sql-jdbc]) 🚚
2024-07-23 12:53:46,545 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :athena...
2024-07-23 12:53:46,546 INFO driver.impl :: Registered driver :athena (parents: [:sql-jdbc]) 🚚
2024-07-23 12:53:46,555 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :druid-jdbc...
2024-07-23 12:53:46,556 INFO driver.impl :: Registered driver :druid-jdbc (parents: [:sql-jdbc]) 🚚
2024-07-23 12:53:46,574 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
2024-07-23 12:53:46,577 INFO db.setup :: Verifying mysql Database Connection ...
2024-07-23 12:53:46,895 INFO db.setup :: Successfully verified MySQL 8.0.35 application database connection. ✅
2024-07-23 12:53:46,895 INFO db.setup :: Checking if a database downgrade is required...
2024-07-23 12:53:48,074 INFO db.setup :: Running Database Migrations...
2024-07-23 12:53:48,075 INFO db.setup :: Setting up Liquibase...
2024-07-23 12:53:48,554 INFO db.liquibase :: Updating liquibase table to reflect consolidated changeset filenames
2024-07-23 12:53:48,611 INFO db.liquibase :: No migration lock found.
2024-07-23 12:53:48,613 INFO db.liquibase :: Migration lock acquired.
2024-07-23 12:53:48,635 INFO db.setup :: Liquibase is ready.
2024-07-23 12:53:48,636 INFO db.liquibase :: Checking if Database has unrun migrations...
2024-07-23 12:53:49,764 ERROR metabase.core :: Metabase Initialization FAILED
liquibase.exception.ValidationFailedException: Validation Failed:
1 changesets check sum
migrations/001_update_migrations.yaml::v50.2024-02-26T22:15:54::noahmoss was: 9:838ed8f582c11e79f574db21b2dcb999 but is now: 9:29e840f2e163f8795c220faee8201c81
at liquibase.changelog.DatabaseChangeLog.validate(DatabaseChangeLog.java:358)
at liquibase.Liquibase.lambda$listUnrunChangeSets$20(Liquibase.java:1149)
at liquibase.Scope.lambda$child$0(Scope.java:186)
at liquibase.Scope.child(Scope.java:195)
at liquibase.Scope.child(Scope.java:185)
at liquibase.Scope.child(Scope.java:164)
at liquibase.Liquibase.runInScope(Liquibase.java:1419)
at liquibase.Liquibase.listUnrunChangeSets(Liquibase.java:1141)
at liquibase.Liquibase.listUnrunChangeSets(Liquibase.java:1131)
at metabase.db.liquibase$unrun_migrations$fn__44495.invoke(liquibase.clj:186)
at metabase.db.liquibase$do_with_liquibase$f_STAR___44480.invoke(liquibase.clj:140)
at metabase.db.liquibase$do_with_liquibase.invokeStatic(liquibase.clj:150)
at metabase.db.liquibase$do_with_liquibase.invoke(liquibase.clj:131)
at metabase.db.liquibase$unrun_migrations.invokeStatic(liquibase.clj:185)
at metabase.db.liquibase$unrun_migrations.invoke(liquibase.clj:176)
at metabase.db.liquibase$migrate_up_if_needed_BANG_.invokeStatic(liquibase.clj:346)
at metabase.db.liquibase$migrate_up_if_needed_BANG_.invoke(liquibase.clj:342)
at metabase.db.setup$migrate_BANG_$fn__53479.invoke(setup.clj:84)
at metabase.db.liquibase$do_with_liquibase$f_STAR___44480.invoke(liquibase.clj:140)
at metabase.db.liquibase$do_with_liquibase.invokeStatic(liquibase.clj:143)
at metabase.db.liquibase$do_with_liquibase.invoke(liquibase.clj:131)
at metabase.db.setup$migrate_BANG_.invokeStatic(setup.clj:73)
at metabase.db.setup$migrate_BANG_.doInvoke(setup.clj:55)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at metabase.db.setup$run_schema_migrations_BANG_.invokeStatic(setup.clj:149)
at metabase.db.setup$run_schema_migrations_BANG_.invoke(setup.clj:144)
at metabase.db.setup$setup_db_BANG_$fn__53507$fn__53508.invoke(setup.clj:167)
at metabase.util.jvm$do_with_us_locale.invokeStatic(jvm.clj:239)
at metabase.util.jvm$do_with_us_locale.invoke(jvm.clj:225)
at metabase.db.setup$setup_db_BANG_$fn__53507.invoke(setup.clj:161)
at metabase.db.setup$setup_db_BANG_.invokeStatic(setup.clj:160)
at metabase.db.setup$setup_db_BANG_.invoke(setup.clj:153)
at metabase.db$setup_db_BANG_$fn__53532.invoke(db.clj:86)
at metabase.db$setup_db_BANG_.invokeStatic(db.clj:81)
at metabase.db$setup_db_BANG_.doInvoke(db.clj:68)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at metabase.core$init_BANG__STAR_.invokeStatic(core.clj:117)
at metabase.core$init_BANG__STAR_.invoke(core.clj:98)
at metabase.core$init_BANG_.invokeStatic(core.clj:170)
at metabase.core$init_BANG_.invoke(core.clj:165)
at metabase.core$start_normally.invokeStatic(core.clj:182)
at metabase.core$start_normally.invoke(core.clj:176)
at metabase.core$entrypoint.invokeStatic(core.clj:215)
at metabase.core$entrypoint.doInvoke(core.clj:209)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.core$apply.invoke(core.clj:662)
at metabase.bootstrap$_main.invokeStatic(bootstrap.clj:31)
at metabase.bootstrap$_main.doInvoke(bootstrap.clj:28)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at metabase.bootstrap.main(Unknown Source)
2024-07-23 12:53:49,784 INFO metabase.core :: Metabase Shutting Down ...
2024-07-23 12:53:49,785 INFO metabase.server :: Shutting Down Embedded Jetty Webserver
2024-07-23 12:53:49,796 WARN db.liquibase :: ()
2024-07-23 12:53:49,797 INFO metabase.core :: Metabase Shutdown COMPLETE
It seems to be due to the metadata storage, which is mysql in my case. Can't figure out what's wrong here.
I tried to restore metastore from backups, but face the same issue.