Docker container fails with migration lock

Few details.
everything was fine till last week. due to some reasons we removed /deleted the pod and moved to another Aks.
DB is Postgress.
Now the pod is not up due to un run migrations.

logs:
2022-03-08 18:38:02,209 INFO plugins.dependencies :: Metabase Oracle Driver dependency {:class oracle.jdbc.OracleDriver} satisfied? false

2022-03-08 18:38:02,210 INFO plugins.dependencies :: Plugins with unsatisfied deps: ["Metabase Vertica Driver" "Metabase Oracle Driver"]

2022-03-08 18:38:02,211 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...

2022-03-08 18:38:02,216 INFO db.setup :: Verifying postgres Database Connection ...

2022-03-08 18:38:03,442 INFO db.setup :: Successfully verified PostgreSQL 10.17 application database connection. :white_check_mark:

2022-03-08 18:38:03,443 INFO db.setup :: Running Database Migrations...

2022-03-08 18:38:03,919 INFO db.setup :: Setting up Liquibase...

2022-03-08 18:38:04,182 INFO db.setup :: Liquibase is ready.

2022-03-08 18:38:04,183 INFO db.liquibase :: Checking if Database has unrun migrations...

2022-03-08 18:38:06,841 INFO db.liquibase :: Database has unrun migrations. Waiting for migration lock to be cleared...

2022-03-08 18:38:08,848 WARN metabase.util :: auto-retry metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__14364@3293288e: Database has migration lock; cannot run migrations. You can force-release these locks by running java -jar metabase.jar migrate release-locks.

2022-03-08 18:38:10,852 WARN metabase.util :: auto-retry metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__14364@3293288e: Database has migration lock; cannot run migrations. You can force-release these locks by running java -jar metabase.jar migrate release-locks.

2022-03-08 18:38:12,856 WARN metabase.util :: auto-retry metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__14364@3293288e: Database has migration lock; cannot run migrations. You can force-release these locks by running java -jar metabase.jar migrate release-locks.

2022-03-08 18:38:14,861 WARN metabase.util :: auto-retry metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__14364@3293288e: Database has migration lock; cannot run migrations. You can force-release these locks by running java -jar metabase.jar migrate release-locks.

2022-03-08 18:38:16,864 WARN metabase.util :: auto-retry metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__14364@3293288e: Database has migration lock; cannot run migrations. You can force-release these locks by running java -jar metabase.jar migrate release-locks.

2022-03-08 18:38:18,873 ERROR metabase.core :: Metabase Initialization FAILED

liquibase.exception.LockException: Database has migration lock; cannot run migrations. You can force-release these locks by running java -jar metabase.jar migrate release-locks.

at metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__14364.invoke(liquibase.clj:109)

Hi @gopalappu
Make sure you have backups. Go to the Postgres database table databasechangeloglock and truncate it or unlock the row and start Metabase again.
Make sure you're only starting a single pod during upgrades.

Hello there,

Old topic, I know, but Google send me there and the title is exactly what I'm searching for.

Slicy diference : newer version and not Postgres but good old metabase.db.mv.db.

This morning, I want to upgrade Metabase by docker pull / docker-compose up -d :

Logs :

2024-03-06 10:10:12,371 INFO metabase.util :: Maximum memory available to JVM: 960.0 MB
2024-03-06 10:10:15,430 WARN db.env :: WARNING: Using Metabase with an H2 application database is not recommended for production deployments. For production deployments, we highly recommend using Postgres, MySQL, or MariaDB instead. If you decide to continue to use H2, please be sure to back up the database file regularly. For more information, see https://metabase.com/docs/latest/operations-guide/migrating-from-h2.html
2024-03-06 10:10:15,732 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. 🔓
 For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
2024-03-06 10:10:24,651 INFO driver.impl :: Registered abstract driver :sql  🚚
2024-03-06 10:10:24,666 INFO driver.impl :: Registered abstract driver :sql-jdbc (parents: [:sql]) 🚚
2024-03-06 10:10:24,673 INFO metabase.util :: Load driver :sql-jdbc took 122.0 ms
2024-03-06 10:10:24,674 INFO driver.impl :: Registered driver :h2 (parents: [:sql-jdbc]) 🚚
2024-03-06 10:10:24,913 INFO driver.impl :: Registered driver :mysql (parents: [:sql-jdbc]) 🚚
2024-03-06 10:10:24,964 INFO driver.impl :: Registered driver :postgres (parents: [:sql-jdbc]) 🚚
2024-03-06 10:10:27,681 INFO metabase.core ::
Metabase v0.48.8 (a900c85)

Copyright © 2024 Metabase, Inc.

Metabase Enterprise Edition extensions are NOT PRESENT.
2024-03-06 10:10:27,695 INFO metabase.core :: Starting Metabase in STANDALONE mode
2024-03-06 10:10:27,844 INFO metabase.server :: Launching Embedded Jetty Webserver with config:
 {:port 3000, :host "0.0.0.0"}

2024-03-06 10:10:28,031 INFO metabase.core :: Starting Metabase version v0.48.8 (a900c85) ...
2024-03-06 10:10:28,056 INFO metabase.core :: System info:
 {"file.encoding" "UTF-8",
 "java.runtime.name" "OpenJDK Runtime Environment",
 "java.runtime.version" "11.0.22+7",
 "java.vendor" "Eclipse Adoptium",
 "java.vendor.url" "https://adoptium.net/",
 "java.version" "11.0.22",
 "java.vm.name" "OpenJDK 64-Bit Server VM",
 "java.vm.version" "11.0.22+7",
 "os.name" "Linux",
 "os.version" "5.4.0-99-generic",
 "user.language" "en",
 "user.timezone" "GMT"}

2024-03-06 10:10:28,066 INFO metabase.plugins :: Loading plugins in /plugins...
2024-03-06 10:10:28,947 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :sqlite...
2024-03-06 10:10:28,950 INFO driver.impl :: Registered driver :sqlite (parents: [:sql-jdbc]) 🚚
2024-03-06 10:10:28,975 INFO plugins.dependencies :: Metabase cannot initialize plugin Metabase Vertica Driver due to required dependencies. Metabase requires the Vertica JDBC driver in order to connect to Vertica databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/vertica.html for more details.

2024-03-06 10:10:28,978 INFO plugins.dependencies :: Metabase Vertica Driver dependency {:class com.vertica.jdbc.Driver} satisfied? false
2024-03-06 10:10:28,985 INFO plugins.dependencies :: Plugins with unsatisfied deps: ["Metabase Vertica Driver"]
2024-03-06 10:10:29,009 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :druid...
2024-03-06 10:10:29,011 INFO driver.impl :: Registered driver :druid  🚚
2024-03-06 10:10:29,044 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :bigquery-cloud-sdk...
2024-03-06 10:10:29,049 INFO driver.impl :: Registered driver :bigquery-cloud-sdk (parents: [:sql]) 🚚
2024-03-06 10:10:29,094 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :presto-jdbc...
2024-03-06 10:10:29,110 INFO driver.impl :: Registered driver :presto-jdbc (parents: [:sql-jdbc]) 🚚
2024-03-06 10:10:29,140 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :mongo...
2024-03-06 10:10:29,143 INFO driver.impl :: Registered driver :mongo  🚚
2024-03-06 10:10:29,159 INFO plugins.dependencies :: Metabase cannot initialize plugin Metabase Oracle Driver due to required dependencies. Metabase requires the Oracle JDBC driver in order to connect to Oracle databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/oracle.html for more details.

2024-03-06 10:10:29,165 INFO plugins.dependencies :: Metabase Oracle Driver dependency {:class oracle.jdbc.OracleDriver} satisfied? false
2024-03-06 10:10:29,171 INFO plugins.dependencies :: Plugins with unsatisfied deps: ["Metabase Vertica Driver" "Metabase Oracle Driver"]
2024-03-06 10:10:29,181 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :hive-like...
2024-03-06 10:10:29,182 INFO driver.impl :: Registered abstract driver :hive-like (parents: [:sql-jdbc]) 🚚
2024-03-06 10:10:29,182 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :sparksql...
2024-03-06 10:10:29,183 INFO driver.impl :: Registered driver :sparksql (parents: [:hive-like]) 🚚
2024-03-06 10:10:29,241 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :athena...
2024-03-06 10:10:29,242 INFO driver.impl :: Registered driver :athena (parents: [:sql-jdbc]) 🚚
2024-03-06 10:10:29,250 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :redshift...
2024-03-06 10:10:29,251 INFO driver.impl :: Registered driver :redshift (parents: [:postgres]) 🚚
2024-03-06 10:10:29,267 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :sqlserver...
2024-03-06 10:10:29,272 INFO driver.impl :: Registered driver :sqlserver (parents: [:sql-jdbc]) 🚚
2024-03-06 10:10:29,318 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :snowflake...
2024-03-06 10:10:29,325 INFO driver.impl :: Registered driver :snowflake (parents: [:sql-jdbc]) 🚚
2024-03-06 10:10:29,339 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :googleanalytics...
2024-03-06 10:10:29,340 INFO driver.impl :: Registered driver :googleanalytics  🚚
2024-03-06 10:10:29,355 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :csv...
2024-03-06 10:10:29,357 INFO driver.impl :: Registered driver :csv (parents: [:sql-jdbc]) 🚚
2024-03-06 10:10:29,387 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
2024-03-06 10:10:29,391 INFO db.setup :: Verifying h2 Database Connection ...
2024-03-06 10:10:30,380 INFO db.setup :: Successfully verified H2 2.1.214 (2022-06-13) application database connection. ✅
2024-03-06 10:10:30,385 INFO db.setup :: Checking if a database downgrade is required...
2024-03-06 10:10:31,209 INFO db.setup :: Running Database Migrations...
2024-03-06 10:10:31,210 INFO db.setup :: Setting up Liquibase...
2024-03-06 10:10:31,492 INFO db.setup :: Liquibase is ready.
2024-03-06 10:10:31,494 INFO db.liquibase :: Checking if Database has unrun migrations...
2024-03-06 10:10:32,025 INFO db.liquibase :: Database has unrun migrations. Checking if migraton lock is taken...
2024-03-06 10:10:34,034 WARN util.jvm :: auto-retry metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__47771@4169479d: Database has migration lock; cannot run migrations. You can force-release these locks by running `java -jar metabase.jar migrate release-locks`.
2024-03-06 10:10:36,043 WARN util.jvm :: auto-retry metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__47771@4169479d: Database has migration lock; cannot run migrations. You can force-release these locks by running `java -jar metabase.jar migrate release-locks`.
2024-03-06 10:10:38,049 WARN util.jvm :: auto-retry metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__47771@4169479d: Database has migration lock; cannot run migrations. You can force-release these locks by running `java -jar metabase.jar migrate release-locks`.
2024-03-06 10:10:40,057 WARN util.jvm :: auto-retry metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__47771@4169479d: Database has migration lock; cannot run migrations. You can force-release these locks by running `java -jar metabase.jar migrate release-locks`.
2024-03-06 10:10:42,062 WARN util.jvm :: auto-retry metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__47771@4169479d: Database has migration lock; cannot run migrations. You can force-release these locks by running `java -jar metabase.jar migrate release-locks`.
2024-03-06 10:10:44,068 ERROR metabase.core :: Metabase Initialization FAILED
liquibase.exception.LockException: Database has migration lock; cannot run migrations. You can force-release these locks by running `java -jar metabase.jar migrate release-locks`.
        at metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__47771.invoke(liquibase.clj:213)
        at metabase.util.jvm$do_with_auto_retries.invokeStatic(jvm.clj:176)
        at metabase.util.jvm$do_with_auto_retries.invoke(jvm.clj:165)
        at metabase.util.jvm$do_with_auto_retries.invokeStatic(jvm.clj:183)
        at metabase.util.jvm$do_with_auto_retries.invoke(jvm.clj:165)
        at metabase.util.jvm$do_with_auto_retries.invokeStatic(jvm.clj:183)
        at metabase.util.jvm$do_with_auto_retries.invoke(jvm.clj:165)
        at metabase.util.jvm$do_with_auto_retries.invokeStatic(jvm.clj:183)
        at metabase.util.jvm$do_with_auto_retries.invoke(jvm.clj:165)
        at metabase.util.jvm$do_with_auto_retries.invokeStatic(jvm.clj:183)
        at metabase.util.jvm$do_with_auto_retries.invoke(jvm.clj:165)
        at metabase.util.jvm$do_with_auto_retries.invokeStatic(jvm.clj:183)
        at metabase.util.jvm$do_with_auto_retries.invoke(jvm.clj:165)
        at metabase.db.liquibase$wait_for_migration_lock_to_be_cleared.invokeStatic(liquibase.clj:207)
        at metabase.db.liquibase$wait_for_migration_lock_to_be_cleared.invoke(liquibase.clj:202)
        at metabase.db.liquibase$migrate_up_if_needed_BANG_.invokeStatic(liquibase.clj:228)
        at metabase.db.liquibase$migrate_up_if_needed_BANG_.invoke(liquibase.clj:221)
        at metabase.db.setup$migrate_BANG_$fn__49980.invoke(setup.clj:80)
        at metabase.db.liquibase$do_with_liquibase$f_STAR___47748.invoke(liquibase.clj:134)
        at metabase.db.liquibase$do_with_liquibase.invokeStatic(liquibase.clj:137)
        at metabase.db.liquibase$do_with_liquibase.invoke(liquibase.clj:125)
        at metabase.db.setup$migrate_BANG_.invokeStatic(setup.clj:75)
        at metabase.db.setup$migrate_BANG_.doInvoke(setup.clj:56)
        at clojure.lang.RestFn.invoke(RestFn.java:445)
        at metabase.db.setup$run_schema_migrations_BANG_.invokeStatic(setup.clj:148)
        at metabase.db.setup$run_schema_migrations_BANG_.invoke(setup.clj:142)
        at metabase.db.setup$setup_db_BANG_$fn__50010$fn__50011.invoke(setup.clj:166)
        at metabase.util.jvm$do_with_us_locale.invokeStatic(jvm.clj:239)
        at metabase.util.jvm$do_with_us_locale.invoke(jvm.clj:225)
        at metabase.db.setup$setup_db_BANG_$fn__50010.invoke(setup.clj:161)
        at metabase.db.setup$setup_db_BANG_.invokeStatic(setup.clj:160)
        at metabase.db.setup$setup_db_BANG_.invoke(setup.clj:154)
        at metabase.db$setup_db_BANG_$fn__50024.invoke(db.clj:69)
        at metabase.db$setup_db_BANG_.invokeStatic(db.clj:64)
        at metabase.db$setup_db_BANG_.invoke(db.clj:55)
        at metabase.core$init_BANG__STAR_.invokeStatic(core.clj:113)
        at metabase.core$init_BANG__STAR_.invoke(core.clj:98)
        at metabase.core$init_BANG_.invokeStatic(core.clj:156)
        at metabase.core$init_BANG_.invoke(core.clj:151)
        at metabase.core$start_normally.invokeStatic(core.clj:168)
        at metabase.core$start_normally.invoke(core.clj:162)
        at metabase.core$entrypoint.invokeStatic(core.clj:201)
        at metabase.core$entrypoint.doInvoke(core.clj:195)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at clojure.lang.Var.applyTo(Var.java:705)
        at clojure.core$apply.invokeStatic(core.clj:667)
        at clojure.core$apply.invoke(core.clj:662)
        at metabase.bootstrap$_main.invokeStatic(bootstrap.clj:31)
        at metabase.bootstrap$_main.doInvoke(bootstrap.clj:28)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at metabase.bootstrap.main(Unknown Source)
2024-03-06 10:10:44,088 INFO metabase.core :: Metabase Shutting Down ...
2024-03-06 10:10:44,090 INFO metabase.server :: Shutting Down Embedded Jetty Webserver
2024-03-06 10:10:44,095 INFO metabase.core :: Metabase Shutdown COMPLETE

Main message is pretty clear : Database has migration lock; cannot run migrations. You can force-release these locks by running 'java -jar metabase.jar migrate release-locks'.

So here we go :

 docker-compose run --entrypoint /bin/bash metabase
974cc15af984:/# cd app/
974cc15af984:/app# java -jar metabase.jar migrate release-locks
Warning: environ value jdk-11.0.22+7 for key :java-version has been overwritten with 11.0.22
2024-03-06 10:11:26,683 INFO metabase.util :: Maximum memory available to JVM: 960.0 MB
2024-03-06 10:11:32,567 WARN db.env :: WARNING: Using Metabase with an H2 application database is not recommended for production deployments. For production deployments, we highly recommend using Postgres, MySQL, or MariaDB instead. If you decide to continue to use H2, please be sure to back up the database file regularly. For more information, see https://metabase.com/docs/latest/operations-guide/migrating-from-h2.html
2024-03-06 10:11:33,107 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. 🔓
 For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
2024-03-06 10:11:50,147 INFO driver.impl :: Registered abstract driver :sql  🚚
2024-03-06 10:11:50,171 INFO driver.impl :: Registered abstract driver :sql-jdbc (parents: [:sql]) 🚚
2024-03-06 10:11:50,182 INFO metabase.util :: Load driver :sql-jdbc took 239.8 ms
2024-03-06 10:11:50,183 INFO driver.impl :: Registered driver :h2 (parents: [:sql-jdbc]) 🚚
2024-03-06 10:11:50,636 INFO driver.impl :: Registered driver :mysql (parents: [:sql-jdbc]) 🚚
2024-03-06 10:11:50,800 INFO driver.impl :: Registered driver :postgres (parents: [:sql-jdbc]) 🚚
2024-03-06 10:11:56,545 INFO metabase.core ::
Metabase v0.48.8 (a900c85)

Copyright © 2024 Metabase, Inc.

Metabase Enterprise Edition extensions are NOT PRESENT.
2024-03-06 10:11:58,052 INFO db.setup :: Setting up Liquibase...
2024-03-06 10:12:00,182 INFO db.setup :: Liquibase is ready.

And that's it. No migration is running. If I re-run the command nothing more. If I'm trying to re-start the contener : same error about locking.

Do you know what can I do ?

Oh OK, I've found my error.

When I've run java -jar metabase.jar migrate release-locks without anything else, the script was run on a new db.

When the right environement variables it was better :

$docker-compose run --rm --entrypoint /bin/bash metabase

export MB_DB_TYPE=h2
export MB_DB_FILE=/app/metabase.db/metabase.db
cd app/
java -jar metabase.jar migrate release-locks
java -jar metabase.jar migrate up

After, my contener was able to start like always !

Hope that will help someone one day !

Bye !