Summary
We self host metabase and the instance it was deployed in was rebooted. Upon reboot, the application is stuck at startup. One thing to note is that it was using the x.50.6 image but someone clicked the in-app upgrade button and it registered x.50.19 in the db (I see it in the settings).
In the past this happened but the application was explicit about the issues. Still, we reconfigured to declarative use x.50.19 now.
I have verified the credentials to the database are OK and the database is reachable from the instance. (The application claims it connected successfully but still I double checked)
Not sure if I should use other means of communicating the issue as we have a paid plan.
Dump
Setup: Docker
Image: metabase-enterprise
Tag: v1.50.19
Database: PostgreSQL
Engine version: 16.1 (RDS Aurora)
Log:
02:38:03.681 [main] INFO metabase.driver.impl - Registered abstract driver :hive-like (parents: [:sql-jdbc]) 🚚
02:38:03.681 [main] DEBUG metabase.plugins.lazy-loaded-driver - Registering lazy loading driver :sparksql...
02:38:03.682 [main] INFO metabase.driver.impl - Registered driver :sparksql (parents: [:hive-like]) 🚚
02:38:03.690 [main] INFO metabase.core - Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
02:38:03.693 [main] INFO metabase.db.setup - Verifying postgres Database Connection ...
02:38:03.693 [main] TRACE metabase.util.i18n - Translated "Unable to connect to Metabase {0} DB." for site locale "en" -> "Unable to connect to Metabase postgres DB."
02:38:05.377 [main] INFO metabase.db.setup - Successfully verified PostgreSQL 16.1 application database connection. ✅
02:38:05.377 [main] INFO metabase.db.setup - Checking if a database downgrade is required...
02:38:22.991 [qtp48337931-18] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "4.4 ms (0 DB calls)"
02:38:22.994 [qtp48337931-18] ERROR metabase.server.middleware.log - GET /api/health 503 4.4 ms (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:38:23.995 [qtp48337931-14] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "489.4 µs (0 DB calls)"
02:38:23.996 [qtp48337931-14] ERROR metabase.server.middleware.log - GET /api/health 503 489.4 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:38:37.828 [qtp48337931-17] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "509.2 µs (0 DB calls)"
02:38:37.829 [qtp48337931-17] ERROR metabase.server.middleware.log - GET /api/health 503 509.2 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:38:52.828 [qtp48337931-21] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "376.0 µs (0 DB calls)"
02:38:52.829 [qtp48337931-21] ERROR metabase.server.middleware.log - GET /api/health 503 376.0 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:39:07.828 [qtp48337931-16] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "412.2 µs (0 DB calls)"
02:39:07.829 [qtp48337931-16] ERROR metabase.server.middleware.log - GET /api/health 503 412.2 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:39:22.827 [qtp48337931-14] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "377.2 µs (0 DB calls)"
02:39:22.828 [qtp48337931-14] ERROR metabase.server.middleware.log - GET /api/health 503 377.2 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:39:30.994 [qtp48337931-21] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "462.9 µs (0 DB calls)"
02:39:30.995 [qtp48337931-21] ERROR metabase.server.middleware.log - GET /api/health 503 462.9 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:39:37.829 [qtp48337931-20] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "487.8 µs (0 DB calls)"
02:39:37.829 [qtp48337931-20] ERROR metabase.server.middleware.log - GET /api/health 503 487.8 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:39:52.828 [qtp48337931-18] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "489.6 µs (0 DB calls)"
02:39:52.829 [qtp48337931-18] ERROR metabase.server.middleware.log - GET /api/health 503 489.6 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:40:07.829 [qtp48337931-19] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "629.1 µs (0 DB calls)"
02:40:07.830 [qtp48337931-19] ERROR metabase.server.middleware.log - GET /api/health 503 629.1 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:40:17.829 [qtp48337931-14] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "542.3 µs (0 DB calls)"
02:40:17.830 [qtp48337931-14] ERROR metabase.server.middleware.log - GET /api/health 503 542.3 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:40:22.828 [qtp48337931-16] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "383.8 µs (0 DB calls)"
02:40:22.829 [qtp48337931-16] ERROR metabase.server.middleware.log - GET /api/health 503 383.8 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:40:27.829 [qtp48337931-18] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "531.9 µs (0 DB calls)"
02:40:27.830 [qtp48337931-18] ERROR metabase.server.middleware.log - GET /api/health 503 531.9 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:40:37.828 [qtp48337931-21] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "421.1 µs (0 DB calls)"
02:40:37.828 [qtp48337931-20] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "376.9 µs (0 DB calls)"
02:40:37.829 [qtp48337931-20] ERROR metabase.server.middleware.log - GET /api/health 503 376.9 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:40:37.829 [qtp48337931-21] ERROR metabase.server.middleware.log - GET /api/health 503 421.1 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:40:41.995 [qtp48337931-14] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "461.2 µs (0 DB calls)"
02:40:41.995 [qtp48337931-14] ERROR metabase.server.middleware.log - GET /api/health 503 461.2 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:40:47.829 [qtp48337931-18] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "541.5 µs (0 DB calls)"
02:40:47.830 [qtp48337931-18] ERROR metabase.server.middleware.log - GET /api/health 503 541.5 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:40:52.828 [qtp48337931-16] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "472.2 µs (0 DB calls)"
02:40:52.829 [qtp48337931-16] ERROR metabase.server.middleware.log - GET /api/health 503 472.2 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:40:57.829 [qtp48337931-19] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "456.2 µs (0 DB calls)"
02:40:57.829 [qtp48337931-19] ERROR metabase.server.middleware.log - GET /api/health 503 456.2 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:41:07.829 [qtp48337931-20] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "472.5 µs (0 DB calls)"
02:41:07.829 [qtp48337931-20] ERROR metabase.server.middleware.log - GET /api/health 503 472.5 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:41:07.831 [qtp48337931-16] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "457.5 µs (0 DB calls)"
02:41:07.834 [qtp48337931-16] ERROR metabase.server.middleware.log - GET /api/health 503 457.5 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:41:17.829 [qtp48337931-17] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "417.4 µs (0 DB calls)"
02:41:17.829 [qtp48337931-17] ERROR metabase.server.middleware.log - GET /api/health 503 417.4 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:41:22.829 [qtp48337931-19] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "422.6 µs (0 DB calls)"
02:41:22.829 [qtp48337931-19] ERROR metabase.server.middleware.log - GET /api/health 503 422.6 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}
02:41:27.829 [qtp48337931-14] TRACE metabase.util.i18n - Translated "{0} ({1} DB calls)" for site locale "en" -> "600.8 µs (0 DB calls)"
02:41:27.830 [qtp48337931-14] ERROR metabase.server.middleware.log - GET /api/health 503 600.8 µs (0 DB calls) {:metabase-user-id nil}
{:status "initializing", :progress 0.3}