Metabase Can't Startup because of `GET /api/health 503`

Get Help
1

Got this error, when starting up metabase after memory hang

2024-12-04 05:37:47,217 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
2024-12-04 05:37:47,227 INFO db.setup :: Verifying postgres Database Connection ...
2024-12-04 05:37:47,448 ERROR middleware.log :: GET /api/health 503 410.8 µs (0 DB calls) {:metabase-user-id nil} 
{:status "initializing", :progress 0.3}

Already did java -jar metabase.jar migrate release-locks still no help

image
image1378×162 9.79 KB

databasechangeloglock

Update, now stuck at

2024-12-04 06:38:05,941 INFO db.setup :: Verifying postgres Database Connection ...

after expanding java heap Xmx

2

Any help again @Luiggi ?
I saw you are also suggesting running release locks

3

Already resolved, got error max connection, error on memory heap

Solution, reset all process top release memory heap and reset DB to release connections

4

Hi mate, I'm meeting this same error, I'm using Postgres database on Digital Ocean and I don't have maximum connection error, memory is 44% ( I think this is not a high memory heap percentage ), could you think anything else will cause this error ?

Situation: I upgraded from 0.49.25.2 to 0.50.0 and it got stuck at this error for hours.

ERROR middleware.log :: GET /api/health 503 410.8 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}

5

Your Metabase can’t initialize, you should check why that is

6

It's deployed on a K8S pod, these are what I've checked:

  1. CPU and RAM of Metabase database and K8S pod are not high ( lower 50% )

  2. locked value = FALSE

I didn't run "java -jar metabase.jar migrate release-locks" yet, should I try ?

7

why would you do it? you need to identify what the logs say and where the initialization stops, never run commands "just in case"

8

This is the log before it got stuck in hanging of that progress 0.3 error ? I find some errors with Scheduler

Warning: environ value jdk-19.0.2+7 for key :java-version has been overwritten with 19.0.2
2025-05-26 07:27:08,950 INFO metabase.util :: Maximum memory available to JVM: 742.4 MB
2025-05-26 07:27:14,773 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. 🔓
 For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
2025-05-26 07:27:27,097 INFO driver.impl :: e[34mRegistered abstract driver :sqle[0m  🚚
2025-05-26 07:27:27,104 INFO driver.impl :: e[34mRegistered abstract driver :sql-jdbce[0m (parents: [:sql]) 🚚
2025-05-26 07:27:27,111 INFO metabase.util :: e[32mLoad driver :sql-jdbc took 45.8 mse[0m
2025-05-26 07:27:27,111 INFO driver.impl :: e[34mRegistered driver :h2e[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 07:27:27,444 INFO driver.impl :: e[34mRegistered driver :mysqle[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 07:27:27,540 INFO driver.impl :: e[34mRegistered driver :postgrese[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 07:27:30,504 INFO metabase.core ::
Metabase v0.50.0 (4fe42d3)

Copyright © 2025 Metabase, Inc.

Metabase Enterprise Edition extensions are NOT PRESENT.
2025-05-26 07:27:30,513 INFO metabase.core :: Starting Metabase in STANDALONE mode
2025-05-26 07:27:30,583 INFO metabase.server :: Launching Embedded Jetty Webserver with config:
 {:port 3000}

2025-05-26 07:27:30,654 INFO metabase.core :: Starting Metabase version v0.50.0 (4fe42d3) ...
2025-05-26 07:27:30,661 INFO metabase.core :: System info:
 {"file.encoding" "UTF-8",
 "java.runtime.name" "OpenJDK Runtime Environment",
 "java.runtime.version" "19.0.2+7",
 "java.vendor" "Eclipse Adoptium",
 "java.vendor.url" "https://adoptium.net/",
 "java.version" "19.0.2",
 "java.vm.name" "OpenJDK 64-Bit Server VM",
 "java.vm.version" "19.0.2+7",
 "os.name" "Linux",
 "os.version" "6.1.0-32-amd64",
 "user.language" "en",
 "user.timezone" "Etc/UTC"}

2025-05-26 07:27:30,663 INFO metabase.plugins :: Loading plugins in /home/metabase/plugins...
2025-05-26 07:27:31,261 INFO util.files :: Extract file /modules/sparksql.metabase-driver.jar -> /home/metabase/plugins/sparksql.metabase-driver.jar
2025-05-26 07:27:31,450 INFO util.files :: Extract file /modules/sqlserver.metabase-driver.jar -> /home/metabase/plugins/sqlserver.metabase-driver.jar
2025-05-26 07:27:31,460 INFO util.files :: Extract file /modules/snowflake.metabase-driver.jar -> /home/metabase/plugins/snowflake.metabase-driver.jar
2025-05-26 07:27:32,028 INFO util.files :: Extract file /modules/druid-jdbc.metabase-driver.jar -> /home/metabase/plugins/druid-jdbc.metabase-driver.jar
2025-05-26 07:27:32,072 INFO util.files :: Extract file /modules/mongo.metabase-driver.jar -> /home/metabase/plugins/mongo.metabase-driver.jar
2025-05-26 07:27:32,089 INFO util.files :: Extract file /modules/oracle.metabase-driver.jar -> /home/metabase/plugins/oracle.metabase-driver.jar
2025-05-26 07:27:32,091 INFO util.files :: Extract file /modules/athena.metabase-driver.jar -> /home/metabase/plugins/athena.metabase-driver.jar
2025-05-26 07:27:32,182 INFO util.files :: Extract file /modules/redshift.metabase-driver.jar -> /home/metabase/plugins/redshift.metabase-driver.jar
2025-05-26 07:27:32,189 INFO util.files :: Extract file /modules/presto-jdbc.metabase-driver.jar -> /home/metabase/plugins/presto-jdbc.metabase-driver.jar
2025-05-26 07:27:32,246 INFO util.files :: Extract file /modules/bigquery-cloud-sdk.metabase-driver.jar -> /home/metabase/plugins/bigquery-cloud-sdk.metabase-driver.jar
2025-05-26 07:27:32,494 INFO util.files :: Extract file /modules/druid.metabase-driver.jar -> /home/metabase/plugins/druid.metabase-driver.jar
2025-05-26 07:27:32,498 INFO util.files :: Extract file /modules/vertica.metabase-driver.jar -> /home/metabase/plugins/vertica.metabase-driver.jar
2025-05-26 07:27:32,500 INFO util.files :: Extract file /modules/sqlite.metabase-driver.jar -> /home/metabase/plugins/sqlite.metabase-driver.jar
2025-05-26 07:27:33,180 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sqlite...e[0m
2025-05-26 07:27:33,181 INFO driver.impl :: e[34mRegistered driver :sqlitee[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 07:27:33,187 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :druid...e[0m
2025-05-26 07:27:33,187 INFO driver.impl :: e[34mRegistered driver :druide[0m  🚚
2025-05-26 07:27:33,241 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :presto-jdbc...e[0m
2025-05-26 07:27:33,241 INFO driver.impl :: e[34mRegistered driver :presto-jdbce[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 07:27:33,245 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sqlserver...e[0m
2025-05-26 07:27:33,246 INFO driver.impl :: e[34mRegistered driver :sqlservere[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 07:27:33,256 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :athena...e[0m
2025-05-26 07:27:33,256 INFO driver.impl :: e[34mRegistered driver :athenae[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 07:27:33,272 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :bigquery-cloud-sdk...e[0m
2025-05-26 07:27:33,273 INFO driver.impl :: e[34mRegistered driver :bigquery-cloud-sdke[0m (parents: [:sql]) 🚚
2025-05-26 07:27:33,345 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :hive-like...e[0m
2025-05-26 07:27:33,346 INFO driver.impl :: e[34mRegistered abstract driver :hive-likee[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 07:27:33,346 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sparksql...e[0m
2025-05-26 07:27:33,346 INFO driver.impl :: e[34mRegistered driver :sparksqle[0m (parents: [:hive-like]) 🚚
2025-05-26 07:27:33,351 INFO plugins.dependencies :: e[31mMetabase cannot initialize plugin Metabase Oracle Driver due to required dependencies.e[0m Metabase requires the Oracle JDBC driver in order to connect to Oracle databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/oracle.html for more details.

2025-05-26 07:27:33,353 INFO plugins.dependencies :: Metabase Oracle Driver dependency {:class oracle.jdbc.OracleDriver} satisfied? false
2025-05-26 07:27:33,354 INFO plugins.dependencies :: e[33mPlugins with unsatisfied deps: ["Metabase Oracle Driver"]e[0m
2025-05-26 07:27:33,358 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :redshift...e[0m
2025-05-26 07:27:33,359 INFO driver.impl :: e[34mRegistered driver :redshifte[0m (parents: [:postgres]) 🚚
2025-05-26 07:27:33,362 INFO plugins.dependencies :: e[31mMetabase cannot initialize plugin Metabase Vertica Driver due to required dependencies.e[0m Metabase requires the Vertica JDBC driver in order to connect to Vertica databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/vertica.html for more details.

2025-05-26 07:27:33,363 INFO plugins.dependencies :: Metabase Vertica Driver dependency {:class com.vertica.jdbc.Driver} satisfied? false
2025-05-26 07:27:33,363 INFO plugins.dependencies :: e[33mPlugins with unsatisfied deps: ["Metabase Vertica Driver" "Metabase Oracle Driver"]e[0m
2025-05-26 07:27:33,367 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :druid-jdbc...e[0m
2025-05-26 07:27:33,368 INFO driver.impl :: e[34mRegistered driver :druid-jdbce[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 07:27:33,455 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :snowflake...e[0m
2025-05-26 07:27:33,456 INFO driver.impl :: e[34mRegistered driver :snowflakee[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 07:27:33,461 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :mongo...e[0m
2025-05-26 07:27:33,461 INFO driver.impl :: e[34mRegistered driver :mongoe[0m  🚚
2025-05-26 07:27:33,465 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :duckdb...e[0m
2025-05-26 07:27:33,466 INFO driver.impl :: e[34mRegistered driver :duckdbe[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 07:27:33,472 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
2025-05-26 07:27:33,474 INFO db.setup :: e[36mVerifying postgres Database Connection ...e[0m
2025-05-26 07:27:34,258 INFO db.setup :: Successfully verified PostgreSQL 16.9 application database connection. ✅
2025-05-26 07:27:34,258 INFO db.setup :: e[36mChecking if a database downgrade is required...e[0m
2025-05-26 07:27:36,079 INFO db.setup :: Running Database Migrations...
2025-05-26 07:27:36,080 INFO db.setup :: Setting up Liquibase...
2025-05-26 07:27:36,829 INFO db.setup :: Liquibase is ready.
2025-05-26 07:27:36,829 INFO db.liquibase :: Checking if Database has unrun migrations...
2025-05-26 07:27:38,540 INFO db.liquibase :: Database has unrun migrations. Checking if migration lock is taken...
2025-05-26 07:27:38,569 INFO db.liquibase :: No migration lock found.
2025-05-26 07:27:38,570 INFO db.liquibase :: Migration lock acquired.
2025-05-26 07:27:39,357 INFO db.liquibase :: Running 62 migrations ...
2025-05-26 07:28:22,775 INFO impl.StdSchedulerFactory :: Using default implementation for ThreadExecutor
2025-05-26 07:28:22,842 INFO core.SchedulerSignalerImpl :: Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
2025-05-26 07:28:22,843 INFO core.QuartzScheduler :: Quartz Scheduler v.2.3.2 created.
2025-05-26 07:28:22,844 INFO jdbcjobstore.JobStoreTX :: Using db table-based data access locking (synchronization).
2025-05-26 07:28:22,846 INFO jdbcjobstore.JobStoreTX :: JobStoreTX initialized.
2025-05-26 07:28:22,846 INFO core.QuartzScheduler :: Scheduler meta-data: Quartz Scheduler (v2.3.2) 'MetabaseScheduler' with instanceId 'data-metabase-deployment-test-5886d98dff-t6t551748244502777'
  Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
  NOT STARTED.
  Currently in standby mode.
  Number of jobs executed: 0
  Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
  Using job-store 'org.quartz.impl.jdbcjobstore.JobStoreTX' - which supports persistence. and is clustered.

2025-05-26 07:28:22,847 INFO impl.StdSchedulerFactory :: Quartz scheduler 'MetabaseScheduler' initialized from default resource file in Quartz package: 'quartz.properties'
2025-05-26 07:28:22,847 INFO impl.StdSchedulerFactory :: Quartz scheduler version: 2.3.2
2025-05-26 07:28:22,880 INFO jdbcjobstore.JobStoreTX :: ClusterManager: detected 1 failed or restarted instances.
2025-05-26 07:28:22,880 INFO jdbcjobstore.JobStoreTX :: ClusterManager: Scanning for instance "data-metabase-deployment-test-7cc65bd59d-tn9nt1748243039842"'s failed in-progress jobs.
2025-05-26 07:28:22,965 INFO jdbcjobstore.JobStoreTX :: ClusterManager: ......Freed 1 acquired trigger(s).
2025-05-26 07:28:22,965 INFO jdbcjobstore.JobStoreTX :: ClusterManager: ......Cleaned-up 1 other failed job(s).
2025-05-26 07:28:22,994 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_data-metabase-deployment-test-5886d98dff-t6t551748244502777 started.
2025-05-26 07:28:23,027 INFO jdbcjobstore.JobStoreTX :: Handling the first 20 triggers that missed their scheduled fire-time.  More misfired triggers remain to be processed.
2025-05-26 07:28:23,366 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_data-metabase-deployment-test-5886d98dff-t6t551748244502777 shutting down.
2025-05-26 07:28:23,366 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_data-metabase-deployment-test-5886d98dff-t6t551748244502777 paused.
2025-05-26 07:28:23,367 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_data-metabase-deployment-test-5886d98dff-t6t551748244502777 shutdown complete.
2025-05-26 07:28:23,379 INFO db.custom-migrations :: No forward migration for DeleteSendPulseTaskOnDowngrade
2025-05-26 07:28:23,388 INFO db.custom-migrations :: No forward migration for DeleteInitSendPulseTriggersOnDowngrade
2025-05-26 07:28:23,463 ERROR core.JobRunShell :: Job DEFAULT.metabase.task.sync-and-analyze.job threw an unhandled Exception:
java.lang.IllegalStateException: Attempting to call unbound fn: #'metabase.task.sync-databases/sync-and-analyze-database!
	at clojure.lang.Var$Unbound.throwArity(Var.java:45)
	at clojure.lang.AFn.invoke(AFn.java:32)
	at metabase.task.sync_databases.SyncAndAnalyzeDatabase.execute(sync_databases.clj:112)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
2025-05-26 07:28:23,468 ERROR core.ErrorLogger :: Job (DEFAULT.metabase.task.sync-and-analyze.job threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.IllegalStateException: Attempting to call unbound fn: #'metabase.task.sync-databases/sync-and-analyze-database!]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.lang.IllegalStateException: Attempting to call unbound fn: #'metabase.task.sync-databases/sync-and-analyze-database!
	at clojure.lang.Var$Unbound.throwArity(Var.java:45)
	at clojure.lang.AFn.invoke(AFn.java:32)
	at metabase.task.sync_databases.SyncAndAnalyzeDatabase.execute(sync_databases.clj:112)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
	... 1 more
2025-05-26 07:28:23,470 ERROR simpl.SimpleThreadPool :: Error while executing the Runnable:
java.lang.IllegalStateException: JobStore is shutdown - aborting retry
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.retryExecuteInNonManagedTXLock(JobStoreSupport.java:3833)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggeredJobComplete(JobStoreSupport.java:3116)
	at org.quartz.core.QuartzScheduler.notifyJobStoreJobComplete(QuartzScheduler.java:1786)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:269)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
2025-05-26 07:29:23,034 ERROR middleware.log :: e[31mGET /api/health 503 977.4 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
e[0m
2025-05-26 07:29:23,623 ERROR middleware.log :: e[31mGET /api/health 503 465.3 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
...
9

I just increased java heap size by setting -XmX to 3gb and re-deployed it, the same error still occurring

10

Hi @Luiggi, could you help me to take a look at this log ?

11

Please move directly to 50.36, don't use 50.0 which is a buggy minor version

12

I just tried to move to 50.36 also. It printed more logs about migration ( it looks like reading indexes/tables or something, I forgot to capture the log ), but it still got stuck at that progress 0.3 error at the end. Now, I'm testing on version 51.1

13

there's no logs here, please post the logs

14

This is the logs when I upgrade to 51.1, the progress 0.3 error is still hanging for hours

Picked up JAVA_TOOL_OPTIONS: -Xmx3G
Warning: environ value jdk-19.0.2+7 for key :java-version has been overwritten with 19.0.2
2025-05-26 10:55:53,794 INFO metabase.util :: Maximum memory available to JVM: 2.9 GB
2025-05-26 10:56:02,386 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. 🔓
 For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
WARNING: infinite? already refers to: #'kixi.stats.math/infinite? in namespace: kixi.stats.core, being replaced by: #'clojure.core/infinite?
2025-05-26 10:56:17,720 INFO driver.impl :: e[34mRegistered abstract driver :sqle[0m  🚚
2025-05-26 10:56:17,823 INFO driver.impl :: e[34mRegistered abstract driver :sql-jdbce[0m (parents: [:sql]) 🚚
2025-05-26 10:56:17,836 INFO metabase.util :: e[32mLoad driver :sql-jdbc took 36.1 mse[0m
2025-05-26 10:56:17,837 INFO driver.impl :: e[34mRegistered driver :h2e[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 10:56:18,701 INFO driver.impl :: e[34mRegistered driver :mysqle[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 10:56:18,822 INFO driver.impl :: e[34mRegistered driver :postgrese[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 10:56:24,307 INFO metabase.core ::
Metabase v0.51.1 (b8178a8)

Copyright © 2025 Metabase, Inc.

Metabase Enterprise Edition extensions are NOT PRESENT.
2025-05-26 10:56:24,330 INFO metabase.core :: Starting Metabase in STANDALONE mode
2025-05-26 10:56:24,514 INFO metabase.server :: Launching Embedded Jetty Webserver with config:
 {:port 3000}

2025-05-26 10:56:24,713 INFO metabase.core :: Starting Metabase version v0.51.1 (b8178a8) ...
2025-05-26 10:56:24,722 INFO metabase.core :: System info:
 {"file.encoding" "UTF-8",
 "java.runtime.name" "OpenJDK Runtime Environment",
 "java.runtime.version" "19.0.2+7",
 "java.vendor" "Eclipse Adoptium",
 "java.vendor.url" "https://adoptium.net/",
 "java.version" "19.0.2",
 "java.vm.name" "OpenJDK 64-Bit Server VM",
 "java.vm.version" "19.0.2+7",
 "os.name" "Linux",
 "os.version" "6.1.0-32-amd64",
 "user.language" "en",
 "user.timezone" "Etc/UTC"}

2025-05-26 10:56:24,725 INFO metabase.plugins :: Loading plugins in /home/metabase/plugins...
2025-05-26 10:56:25,406 INFO util.files :: Extract file /modules/sqlserver.metabase-driver.jar -> /home/metabase/plugins/sqlserver.metabase-driver.jar
2025-05-26 10:56:25,484 INFO util.files :: Extract file /modules/bigquery-cloud-sdk.metabase-driver.jar -> /home/metabase/plugins/bigquery-cloud-sdk.metabase-driver.jar
2025-05-26 10:56:26,194 INFO util.files :: Extract file /modules/athena.metabase-driver.jar -> /home/metabase/plugins/athena.metabase-driver.jar
2025-05-26 10:56:26,353 INFO util.files :: Extract file /modules/redshift.metabase-driver.jar -> /home/metabase/plugins/redshift.metabase-driver.jar
2025-05-26 10:56:26,362 INFO util.files :: Extract file /modules/snowflake.metabase-driver.jar -> /home/metabase/plugins/snowflake.metabase-driver.jar
2025-05-26 10:56:26,905 INFO util.files :: Extract file /modules/sparksql.metabase-driver.jar -> /home/metabase/plugins/sparksql.metabase-driver.jar
2025-05-26 10:56:26,907 INFO util.files :: Extract file /modules/druid.metabase-driver.jar -> /home/metabase/plugins/druid.metabase-driver.jar
2025-05-26 10:56:26,912 INFO util.files :: Extract file /modules/druid-jdbc.metabase-driver.jar -> /home/metabase/plugins/druid-jdbc.metabase-driver.jar
2025-05-26 10:56:26,975 INFO util.files :: Extract file /modules/vertica.metabase-driver.jar -> /home/metabase/plugins/vertica.metabase-driver.jar
2025-05-26 10:56:26,977 INFO util.files :: Extract file /modules/hive-like.metabase-driver.jar -> /home/metabase/plugins/hive-like.metabase-driver.jar
2025-05-26 10:56:27,066 INFO util.files :: Extract file /modules/oracle.metabase-driver.jar -> /home/metabase/plugins/oracle.metabase-driver.jar
2025-05-26 10:56:27,068 INFO util.files :: Extract file /modules/databricks.metabase-driver.jar -> /home/metabase/plugins/databricks.metabase-driver.jar
2025-05-26 10:56:27,240 INFO util.files :: Extract file /modules/presto-jdbc.metabase-driver.jar -> /home/metabase/plugins/presto-jdbc.metabase-driver.jar
2025-05-26 10:56:27,317 INFO util.files :: Extract file /modules/mongo.metabase-driver.jar -> /home/metabase/plugins/mongo.metabase-driver.jar
2025-05-26 10:56:27,342 INFO util.files :: Extract file /modules/sqlite.metabase-driver.jar -> /home/metabase/plugins/sqlite.metabase-driver.jar
2025-05-26 10:56:27,838 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sqlite...e[0m
2025-05-26 10:56:27,888 INFO driver.impl :: e[34mRegistered driver :sqlitee[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 10:56:27,898 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :druid...e[0m
2025-05-26 10:56:27,899 INFO driver.impl :: e[34mRegistered driver :druide[0m  🚚
2025-05-26 10:56:27,918 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :presto-jdbc...e[0m
2025-05-26 10:56:27,919 INFO driver.impl :: e[34mRegistered driver :presto-jdbce[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 10:56:27,992 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :hive-like...e[0m
2025-05-26 10:56:27,999 INFO driver.impl :: e[34mRegistered abstract driver :hive-likee[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 10:56:28,007 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sqlserver...e[0m
2025-05-26 10:56:28,019 INFO driver.impl :: e[34mRegistered driver :sqlservere[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 10:56:28,091 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :athena...e[0m
2025-05-26 10:56:28,091 INFO driver.impl :: e[34mRegistered driver :athenae[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 10:56:28,113 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :bigquery-cloud-sdk...e[0m
2025-05-26 10:56:28,114 INFO driver.impl :: e[34mRegistered driver :bigquery-cloud-sdke[0m (parents: [:sql]) 🚚
2025-05-26 10:56:28,184 INFO plugins.dependencies :: Plugin 'Metabase Spark SQL Driver' depends on plugin 'Metabase Hive Like Abstract Driver'
2025-05-26 10:56:28,187 INFO plugins.dependencies :: Metabase Spark SQL Driver dependency {:plugin Metabase Hive Like Abstract Driver} satisfied? true
2025-05-26 10:56:28,188 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sparksql...e[0m
2025-05-26 10:56:28,189 INFO driver.impl :: e[34mRegistered driver :sparksqle[0m (parents: [:hive-like]) 🚚
2025-05-26 10:56:28,197 INFO plugins.dependencies :: e[31mMetabase cannot initialize plugin Metabase Oracle Driver due to required dependencies.e[0m Metabase requires the Oracle JDBC driver in order to connect to Oracle databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/oracle.html for more details.

2025-05-26 10:56:28,199 INFO plugins.dependencies :: Metabase Oracle Driver dependency {:class oracle.jdbc.OracleDriver} satisfied? false
2025-05-26 10:56:28,200 INFO plugins.dependencies :: e[33mPlugins with unsatisfied deps: ["Metabase Oracle Driver"]e[0m
2025-05-26 10:56:28,205 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :redshift...e[0m
2025-05-26 10:56:28,205 INFO driver.impl :: e[34mRegistered driver :redshifte[0m (parents: [:postgres]) 🚚
2025-05-26 10:56:28,210 INFO plugins.dependencies :: e[31mMetabase cannot initialize plugin Metabase Vertica Driver due to required dependencies.e[0m Metabase requires the Vertica JDBC driver in order to connect to Vertica databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/vertica.html for more details.

2025-05-26 10:56:28,211 INFO plugins.dependencies :: Metabase Vertica Driver dependency {:class com.vertica.jdbc.Driver} satisfied? false
2025-05-26 10:56:28,211 INFO plugins.dependencies :: e[33mPlugins with unsatisfied deps: ["Metabase Vertica Driver" "Metabase Oracle Driver"]e[0m
2025-05-26 10:56:28,286 INFO plugins.dependencies :: Plugin 'Metabase Databricks Driver' depends on plugin 'Metabase Hive Like Abstract Driver'
2025-05-26 10:56:28,287 INFO plugins.dependencies :: Metabase Databricks Driver dependency {:plugin Metabase Hive Like Abstract Driver} satisfied? true
2025-05-26 10:56:28,288 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :databricks...e[0m
2025-05-26 10:56:28,288 INFO driver.impl :: e[34mRegistered driver :databrickse[0m (parents: [:hive-like]) 🚚
2025-05-26 10:56:28,385 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :druid-jdbc...e[0m
2025-05-26 10:56:28,386 INFO driver.impl :: e[34mRegistered driver :druid-jdbce[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 10:56:28,486 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :snowflake...e[0m
2025-05-26 10:56:28,487 INFO driver.impl :: e[34mRegistered driver :snowflakee[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 10:56:28,509 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :mongo...e[0m
2025-05-26 10:56:28,510 INFO driver.impl :: e[34mRegistered driver :mongoe[0m  🚚
2025-05-26 10:56:28,516 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :duckdb...e[0m
2025-05-26 10:56:28,517 INFO driver.impl :: e[34mRegistered driver :duckdbe[0m (parents: [:sql-jdbc]) 🚚
2025-05-26 10:56:28,588 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
2025-05-26 10:56:28,591 INFO db.setup :: e[36mVerifying postgres Database Connection ...e[0m
2025-05-26 10:56:29,812 INFO db.setup :: Successfully verified PostgreSQL 16.9 application database connection. ✅
2025-05-26 10:56:29,813 INFO db.setup :: e[36mChecking if a database downgrade is required...e[0m
2025-05-26 10:56:30,210 INFO liquibase.database :: Set default schema name to public
2025-05-26 10:56:32,196 INFO liquibase :: Parsed changelog file 'liquibase.yaml'
2025-05-26 10:56:32,584 INFO db.setup :: Running Database Migrations...
2025-05-26 10:56:32,585 INFO db.setup :: Setting up Liquibase...
2025-05-26 10:56:32,615 INFO liquibase.database :: Set default schema name to public
2025-05-26 10:56:33,202 INFO liquibase :: Parsed changelog file 'liquibase.yaml'
2025-05-26 10:56:33,484 INFO db.setup :: Liquibase is ready.
2025-05-26 10:56:33,485 INFO db.liquibase :: Checking if Database has unrun migrations...
2025-05-26 10:56:33,522 INFO liquibase.database :: Set default schema name to public
2025-05-26 10:56:33,991 INFO liquibase :: Parsed changelog file 'liquibase.yaml'
2025-05-26 10:56:34,119 INFO liquibase.changelog :: Reading from public.databasechangelog
2025-05-26 10:56:35,815 INFO db.liquibase :: Database has unrun migrations. Checking if migration lock is taken...
2025-05-26 10:56:35,902 INFO liquibase.lockservice :: Successfully acquired change log lock
2025-05-26 10:56:35,906 INFO db.liquibase :: No migration lock found.
2025-05-26 10:56:35,906 INFO db.liquibase :: Migration lock acquired.
2025-05-26 10:56:35,928 INFO liquibase.database :: Set default schema name to public
2025-05-26 10:56:36,596 INFO liquibase :: Parsed changelog file 'liquibase.yaml'
2025-05-26 10:56:36,630 INFO liquibase.changelog :: Reading from public.databasechangelog
2025-05-26 10:56:37,791 INFO db.liquibase :: Running 87 migrations ...
2025-05-26 10:56:38,099 INFO liquibase.changelog :: Reading from public.databasechangelog
2025-05-26 10:56:38,623 INFO liquibase.command :: Using deploymentId: 8256998622
2025-05-26 10:56:38,629 INFO liquibase.changelog :: Reading from public.databasechangelog
2025-05-26 11:10:22,790 ERROR middleware.log :: e[31mGET /api/health 503 2.0 ms (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
...
15

you're using java 19, we don't support java 19 in that version, only 11.
you're also using version 51.1, which is a very old version as well, in 51, the latest version is 51.14.

in that log, I only see the migration going well, you just need to wait

16

and please, ensure that your k8s cluster doesn't kill the instance while it's doing migrations

17

Yes, the instance run thoroughly for sure, it has run for 15 hours and get stuck at GET /api/health 503 218.2 µs (0 DB calls) {:metabase-user-id nil} {:status "initializing", :progress 0.3} as well
Now, I'm trying on 51.14 with java 11. I will get back to you soon.

20

Hi @Luiggi , I think I find out the problem, I have the public.view_log table with 16GB data, I check the migration process from version 49 to higher version need to run a query like below which stuck the migration

UPDATE report_card c SET view_count = (
    SELECT count(*)
    FROM view_log v
    WHERE v.model = 'card'
    AND v.model_id = c.id
)

The main reason is the public.view_log is too heavy to read