Metabase HTTP ERROR 500 ERROR: relation "core_session" does not exist Position: 131

ss_vinoth22 · July 16, 2024, 1:45pm

I installed Metabase open-source v0.50.10 last week on a GKE cluster with PostgreSQL 16. I created several dashboards, but three days ago, the cluster restarted for a patch, causing Metabase to throw an error:

HTTP ERROR 500
ERROR: relation "core_session" does not exist
Position: 131

I restarted both PostgreSQL and Metabase, which triggered the whole migration process again, prompting me to start from scratch, including signing up again. Although I have data storage configured for PostgreSQL, do I also need data storage for Metabase? I’m puzzled as to why everything got erased, even though I specified a persistent volume for the PostgreSQL database. Am I missing something?

TonyC · July 16, 2024, 2:13pm

So you are saying that there was no Tables for the Metabase Application Database in your PostgreSQL?

ss_vinoth22 · July 16, 2024, 2:31pm

@TonyC I just noticed one thing whenever postgres DB is restarted, in DB i get this error

2024-07-16 14:24:57.568 UTC [68] ERROR:  relation "qrtz_triggers" does not exist at character 67
│ 2024-07-16 14:24:57.568 UTC [68] STATEMENT:  SELECT TRIGGER_NAME, TRIGGER_GROUP, NEXT_FIRE_TIME, PRIORITY FROM QRTZ_TRIGGERS WHERE SCHED_NAME = 'MetabaseScheduler' AND TRIGGER_STATE = $1 AND NEXT_FIRE_TIME <= $2 AND (MISFIRE_INSTR = -1 OR (MISFIRE_INS
│ 2024-07-16 14:24:57.570 UTC [68] ERROR:  current transaction is aborted, commands ignored until end of transaction block
│ 2024-07-16 14:24:57.570 UTC [68] STATEMENT:
│ 2024-07-16 14:24:57.573 UTC [68] ERROR:  relation "qrtz_triggers" does not exist at character 67
│ 2024-07-16 14:24:57.573 UTC [68] STATEMENT:  SELECT TRIGGER_NAME, TRIGGER_GROUP, NEXT_FIRE_TIME, PRIORITY FROM QRTZ_TRIGGERS WHERE SCHED_NAME = 'MetabaseScheduler' AND TRIGGER_STATE = $1 AND NEXT_FIRE_TIME <= $2 AND (MISFIRE_INSTR = -1 OR (MISFIRE_INS
│ 2024-07-16 14:24:57.574 UTC [68] ERROR:  current transaction is aborted, commands ignored until end of transaction block
│ 2024-07-16 14:24:57.574 UTC [68] STATEMENT:
│ 2024-07-16 14:25:01.498 UTC [68] ERROR:  relation "qrtz_scheduler_state" does not exist at character 15
│ 2024-07-16 14:25:01.498 UTC [68] STATEMENT:  SELECT * FROM QRTZ_SCHEDULER_STATE WHERE SCHED_NAME = 'MetabaseScheduler'
│ 2024-07-16 14:25:01.499 UTC [68] ERROR:  current transaction is aborted, commands ignored until end of transaction block
│ 2024-07-16 14:25:01.499 UTC [68] STATEMENT:
│ 2024-07-16 14:27:03.464 UTC [74] ERROR:  relation "public.databasechangeloglock" does not exist at character 22
│ 2024-07-16 14:27:03.464 UTC [74] STATEMENT:  SELECT COUNT(*) FROM public.databasechangeloglock
│ 2024-07-16 14:27:03.478 UTC [74] ERROR:  current transaction is aborted, commands ignored until end of transaction block
│ 2024-07-16 14:27:03.478 UTC [74] STATEMENT:
│ 2024-07-16 14:29:54.956 UTC [62] LOG:  checkpoint starting: time

In DB i can see the data, but it always prompt me to setup from beginning.

TonyC · July 16, 2024, 2:35pm

Are you sure you are properly connecting to this PostgreSQL? Are you passing the correct hosting values and are you sure the user you are using to connect to this App DB has full read/write permissions

What you are showing me are the logs that show up when you start metabase?

ss_vinoth22 · July 16, 2024, 2:36pm

@TonyC

 2024-07-16 14:34:02,248 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
│ 2024-07-16 14:34:02,253 INFO db.setup :: Verifying postgres Database Connection ...
│ 2024-07-16 14:34:07,890 INFO db.setup :: Successfully verified PostgreSQL 16.3 (Debian 16.3-1.pgdg120+1) application database connection. ✅
│ 2024-07-16 14:34:07,891 INFO db.setup :: Checking if a database downgrade is required...
│ 2024-07-16 14:34:13,450 INFO db.setup :: Running Database Migrations...
│ 2024-07-16 14:34:13,451 INFO db.setup :: Setting up Liquibase...
│ 2024-07-16 14:34:14,754 INFO db.liquibase :: Updating liquibase table to reflect consolidated changeset filenames
│ 2024-07-16 14:34:14,801 INFO db.liquibase :: No migration lock found.
│ 2024-07-16 14:34:14,810 INFO db.liquibase :: Migration lock acquired.
│ 2024-07-16 14:34:14,870 INFO db.setup :: Liquibase is ready.
│ 2024-07-16 14:34:14,870 INFO db.liquibase :: Checking if Database has unrun migrations...
│ 2024-07-16 14:34:16,802 INFO db.liquibase :: Database has unrun migrations. Checking if migration lock is taken...
│ 2024-07-16 14:34:16,819 INFO db.liquibase :: No migration lock found.
│ 2024-07-16 14:34:16,822 INFO db.liquibase :: Migration lock acquired.
│ 2024-07-16 14:34:17,498 INFO db.liquibase :: Running 350 migrations ...
│ 2024-07-16 14:34:22,587 INFO impl.StdSchedulerFactory :: Using default implementation for ThreadExecutor
│ 2024-07-16 14:34:22,616 INFO core.SchedulerSignalerImpl :: Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
│ 2024-07-16 14:34:22,618 INFO core.QuartzScheduler :: Quartz Scheduler v.2.3.2 created.
│ 2024-07-16 14:34:22,622 INFO jdbcjobstore.JobStoreTX :: Using db table-based data access locking (synchronization).
│ 2024-07-16 14:34:22,626 INFO jdbcjobstore.JobStoreTX :: JobStoreTX initialized.
│ 2024-07-16 14:34:22,628 INFO core.QuartzScheduler :: Scheduler meta-data: Quartz Scheduler (v2.3.2) 'MetabaseScheduler' with instanceId 'metabase-5b45b7d5d8-xbwnw1721140462592'
│   Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
│   NOT STARTED.
│   Currently in standby mode.
│   Number of jobs executed: 0
│   Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
│   Using job-store 'org.quartz.impl.jdbcjobstore.JobStoreTX' - which supports persistence. and is clustered.
│
│ 2024-07-16 14:34:22,629 INFO impl.StdSchedulerFactory :: Quartz scheduler 'MetabaseScheduler' initialized from default resource file in Quartz package: 'quartz.properties'
│ 2024-07-16 14:34:22,629 INFO impl.StdSchedulerFactory :: Quartz scheduler version: 2.3.2
│ 2024-07-16 14:34:22,716 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_metabase-5b45b7d5d8-xbwnw1721140462592 started.
│ 2024-07-16 14:34:22,753 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_metabase-5b45b7d5d8-xbwnw1721140462592 shutting down.
│ 2024-07-16 14:34:22,754 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_metabase-5b45b7d5d8-xbwnw1721140462592 paused.
│ 2024-07-16 14:34:22,757 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_metabase-5b45b7d5d8-xbwnw1721140462592 shutdown complete.
│ 2024-07-16 14:34:25,229 INFO db.custom-migrations :: No forward migration for DowngradeDashboardTab
│ 2024-07-16 14:34:29,587 INFO impl.StdSchedulerFactory :: Using default implementation for ThreadExecutor
│ 2024-07-16 14:34:29,617 INFO core.SchedulerSignalerImpl :: Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
│ 2024-07-16 14:34:29,628 INFO core.QuartzScheduler :: Quartz Scheduler v.2.3.2 created.
│ 2024-07-16 14:34:29,629 INFO jdbcjobstore.JobStoreTX :: Using db table-based data access locking (synchronization).
│ 2024-07-16 14:34:29,629 INFO jdbcjobstore.JobStoreTX :: JobStoreTX initialized.
│ 2024-07-16 14:34:29,629 INFO core.QuartzScheduler :: Scheduler meta-data: Quartz Scheduler (v2.3.2) 'MetabaseScheduler' with instanceId 'metabase-5b45b7d5d8-xbwnw1721140469589'
│   Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
│   NOT STARTED.
│   Currently in standby mode.
│   Number of jobs executed: 0
│   Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
│   Using job-store 'org.quartz.impl.jdbcjobstore.JobStoreTX' - which supports persistence. and is clustered.
2024-07-16 14:34:29,629 INFO jdbcjobstore.JobStoreTX :: JobStoreTX initialized.
│ 2024-07-16 14:34:29,629 INFO core.QuartzScheduler :: Scheduler meta-data: Quartz Scheduler (v2.3.2) 'MetabaseScheduler' with instanceId 'metabase-5b45b7d5d8-xbwnw1721140469589'
│   Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
│   NOT STARTED.
│   Currently in standby mode.
│   Number of jobs executed: 0
│   Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
│   Using job-store 'org.quartz.impl.jdbcjobstore.JobStoreTX' - which supports persistence. and is clustered.
│
│ 2024-07-16 14:34:29,629 INFO impl.StdSchedulerFactory :: Quartz scheduler 'MetabaseScheduler' initialized from default resource file in Quartz package: 'quartz.properties'
│ 2024-07-16 14:34:29,629 INFO impl.StdSchedulerFactory :: Quartz scheduler version: 2.3.2
│ 2024-07-16 14:34:29,665 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_metabase-5b45b7d5d8-xbwnw1721140469589 started.
│ 2024-07-16 14:34:29,701 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_metabase-5b45b7d5d8-xbwnw1721140469589 shutting down.
│ 2024-07-16 14:34:29,701 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_metabase-5b45b7d5d8-xbwnw1721140469589 paused.
│ 2024-07-16 14:34:29,705 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_metabase-5b45b7d5d8-xbwnw1721140469589 shutdown complete.
│ 2024-07-16 14:34:29,737 INFO db.custom-migrations :: No forward migration for DeleteSendPulseTaskOnDowngrade
│ 2024-07-16 14:34:29,751 INFO db.custom-migrations :: No forward migration for DeleteInitSendPulseTriggersOnDowngrade
│
│ UPDATE SUMMARY
│ Run:                        350
│ Previously run:               0
│ Filtered out:                50
│ -------------------------------
│ Total change sets:          400
│
│
│ FILTERED CHANGE SETS SUMMARY
│ DBMS mismatch:               50
│
│ 2024-07-16 14:34:31,374 INFO db.liquibase :: Migration complete in 13.9 s
│ 2024-07-16 14:34:31,453 INFO db.setup :: Database Migrations Current ... ✅
│ 2024-07-16 14:34:31,458 INFO metabase.util :: Database setup took 29.2 s
│ 2024-07-16 14:34:31,704 INFO metabase.core :: Looks like this is a new installation ... preparing setup wizard
│ 2024-07-16 14:34:32,114 INFO metabase.core :: Please use the following URL to setup your Metabase installation:
│
│ http://0.0.0.0:3000/setup/
│
│
│ 2024-07-16 14:34:32,305 INFO metabase.events :: Loading events namespace: metabase.events.audit-log 👂
│ 2024-07-16 14:34:32,505 INFO metabase.events :: Loading events namespace: metabase.events.driver-notifications 👂
│ 2024-07-16 14:34:32,526 INFO metabase.events :: Loading events namespace: metabase.events.last-login 👂
│ 2024-07-16 14:34:32,540 INFO metabase.events :: Loading events namespace: metabase.events.persisted-info 👂
│ 2024-07-16 14:34:32,561 INFO metabase.events :: Loading events namespace: metabase.events.recent-views 👂
│ 2024-07-16 14:34:32,590 INFO metabase.events :: Loading events namespace: metabase.events.revision 👂
│ 2024-07-16 14:34:32,648 INFO metabase.events :: Loading events namespace: metabase.events.schema 👂
│ 2024-07-16 14:34:32,649 INFO metabase.events :: Loading events namespace: metabase.events.sync-database 👂
│ 2024-07-16 14:34:32,666 INFO metabase.events :: Loading events namespace: metabase.events.view-log 👂
│ 2024-07-16 14:34:32,760 INFO metabase.sample-data :: Loading sample database
│ 2024-07-16 14:34:32,924 INFO util.files :: Extract file /sample-database.db.mv.db -> /plugins/sample-database.db.mv.db
│ 2024-07-16 14:34:33,100 INFO driver.impl :: Initializing driver :sql...
│ 2024-07-16 14:34:33,104 INFO driver.impl :: Initializing driver :sql-jdbc...
│ 2024-07-16 14:34:33,106 INFO driver.impl :: Initializing driver :h2...

This is log everytime i get after postgres restarts

TonyC · July 16, 2024, 2:38pm

Running 350 migrations that means the database you are connected to doesn't store the databasechangelog

if you go to the table databasechangelog, what entries do you find?

ss_vinoth22 · July 16, 2024, 2:41pm

ss_vinoth22 · July 16, 2024, 2:41pm

this is the entries that im seeing in changelog.

ss_vinoth22 · July 16, 2024, 2:47pm

this is the deployment yaml configuration

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metabase
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: metabase
  template:
    metadata:
      labels:
        app: metabase
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
        - name: metabase
          image: metabase/metabase:v0.50.12
          ports:
            - containerPort: 3000
          env:
            - name: MB_DB_TYPE
              value: "postgres"
            - name: MB_DB_DBNAME
              value: "postgresdb"
            - name: MB_DB_PORT
              value: "5432"
            - name: MB_DB_USER
              value: "admin"
              #valueFrom:
              #  secretKeyRef:
              #    name: metabase-secrets
              #    key: DB_USER
            - name: MB_DB_PASS
              value: "test"
            - name: MB_ENCRYPTION_SECRET_KEY
              value: "fsdfdsfdsfd="
            - name: MB_DB_HOST
            value: "postgres.default.svc.cluster.local"
          volumeMounts:
          - name: metabase-data
            mountPath: /metabase-data
      volumes:
      - name: metabase-data
        persistentVolumeClaim:
          claimName: metabase-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: metabase-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: Service
metadata:
  name: metabase-service
  namespace: analytics
spec:
  type: ClusterIP
  ports:
  - name: http-web
    protocol: TCP
    port: 80
    targetPort: 3000
  selector:
    app: metabase

ss_vinoth22 · July 16, 2024, 2:48pm

FYI im using postgre16.3, does metabase supports that? or is there any specific version of postgres is supported?

ss_vinoth22 · July 16, 2024, 2:57pm

@TonyC Am i missing something in a way postgres /metabase running?

TonyC · July 16, 2024, 3:13pm

Can you share a screenshot in desc order so i can confirm what is the last migration that ran

You can see via the date_executed that the migrations where run when you upgraded. It's like the Data in the DB was empty

Where are you hosting your PostgreSQL DB?

ss_vinoth22 · July 16, 2024, 3:15pm

Db is hosted is same namespace on kubernetes

ss_vinoth22 · July 16, 2024, 3:20pm

Here is the screenshot

ss_vinoth22 · July 16, 2024, 3:21pm

Why the data is empty,
I can see the DB connection is successful, so why is this not storing any data?

TonyC · July 17, 2024, 7:48am

WHAT! Are you sure you are properly persisting the postgreSQL data? ... I mean one explenation would be that kubernetes is wiping your application database data. Are you losing reports as well? when you perform a restart?

Like if you created a dashboard, is the dashboard still there or you have to recreate?

ss_vinoth22 · July 17, 2024, 2:31pm

mybad, I think i found out the issue, my Postgre DB persistent volume not properly configured .when postgres db is restarted it is wiping all data.

ss_vinoth22 · July 17, 2024, 2:32pm

yes it was deleting all stuffs including dashboard, i need to recreate all.

TonyC · July 17, 2024, 2:33pm

You need google more around kubernetes then or use a managed service for your Database

ss_vinoth22 · July 17, 2024, 2:48pm

Yeah, my bad. Actually, I had a Persistent volume setup, but it was pointing to the wrong directory.