Database has unrun migrations. Waiting for migration lock to

Endpoint to the hosted image just says connecting....in the logs it shows "Database has unrun migrations. Waiting for migration lock to..."

How to Reproduce

Followed tutorial: How to install metabase with aws ecs docker fargate | by Devran AYDIN | Medium

  1. Postgres DB
  2. AWS Fargate ECS
  3. Resources Snapshot
    58%20PM
  4. Image used: metabase/metabase
  5. Logs
    2020-12-09 04:15:23,256 INFO db.liquibase :: Database has unrun migrations. Waiting for migration lock to be cleared...
    2020-12-09 12:15:182020-12-09 04:15:18,424 INFO metabase.db :: Liquibase is ready.
    2020-12-09 12:15:182020-12-09 04:15:18,425 INFO db.liquibase :: Checking if Database has unrun migrations...
    2020-12-09 12:15:182020-12-09 04:15:18,267 INFO metabase.db :: Setting up Liquibase...
    2020-12-09 12:15:182020-12-09 04:15:18,219 INFO metabase.db :: Running Database Migrations...
    2020-12-09 12:15:182020-12-09 04:15:18,216 INFO metabase.db :: Successfully verified PostgreSQL 10.13 application database connection. :white_check_mark:
    2020-12-09 12:15:172020-12-09 04:15:17,332 INFO driver.impl :: e[33mInitializing driver :postgres...e[0m
    2020-12-09 12:15:172020-12-09 04:15:17,330 INFO driver.impl :: e[33mInitializing driver :sql...e[0m
    2020-12-09 12:15:172020-12-09 04:15:17,331 INFO driver.impl :: e[33mInitializing driver :sql-jdbc...e[0m
    2020-12-09 12:15:172020-12-09 04:15:17,320 INFO metabase.db :: e[36mVerifying postgres Database Connection ...e[0m
    2020-12-09 12:15:172020-12-09 04:15:17,258 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
    2020-12-09 12:15:172020-12-09 04:15:17,227 INFO driver.impl :: e[34mRegistered driver :mysqle[0m (parents: [:sql-jdbc]) :truck:
    2020-12-09 12:15:172020-12-09 04:15:17,162 INFO driver.impl :: e[34mRegistered driver :h2e[0m (parents: [:sql-jdbc]) :truck:
    2020-12-09 12:15:172020-12-09 04:15:17,141 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sqlite...e[0m
    2020-12-09 12:15:172020-12-09 04:15:17,146 INFO driver.impl :: e[34mRegistered driver :sqlitee[0m (parents: [:sql-jdbc]) :truck:
    2020-12-09 12:15:172020-12-09 04:15:17,138 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :druid...e[0m
    2020-12-09 12:15:172020-12-09 04:15:17,139 INFO driver.impl :: e[34mRegistered driver :druide[0m :truck:
    2020-12-09 12:15:172020-12-09 04:15:17,131 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sqlserver...e[0m

@flamber Is this happening because of these warnings

2020-12-09 18:02:56

2020-12-09 18:02:56,566 INFO metabase.util :: Maximum memory available to JVM: 1.0 GB

2020-12-09 18:02:53

WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.

2020-12-09 18:02:53

Warning: environ value jdk-11.0.7+10 for key :java-version has been overwritten with 11.0.7

Hi @sagun.garg
Wow it’s confusing, when the log is in reverse order.
If you are starting multiple instances at the same time, then perhaps one of them has locked the database.
You are providing very little information about what you are doing. Are you upgrading, are you setting up a new setup, are you …?

@flamber Thanks for the prompt response. I am using Metabase as a fresh user for the first time, setting it up by hosting(ECS using AWS fargate) and connecting it to my Postgres AWS RDS

I am setting up a new instance.

I am just running one instance and I believe there is no other instance running. I will attempt a complete fresh spawning of the instance.

2020-12-09 12:15:182020-12-09 04:15:18,216 INFO metabase.db :: Successfully verified PostgreSQL 10.13 application database connection. :white_check_mark:

Here are more detailed logs in the database

I am not sure if this snapshot is at all related to the above issue. But it seems suspicious

@sagun.garg Seems like it’s dropping packets between Metabase and Postgres, so that’s the problem. I have no idea why it does that. I have not seen this problem before, but try restarting both RDS and ECS.

@flamber I have tried both but but no relief.

Still stuck at the same place(since last 2 days). I am using metabase/metabase:latest as my base docker image. Can you please recommend me another previous version image which is stable like metabase/metabase:v0.37.0 will that do any magic ?

@flamber I managed to reboot again and check the DB logs again.

I am getting the following errors. Is this specific to me or are these a symptom of the attempt metabase making to connect to the PostgresDB in AWS RDS

@sagun.garg Try contacting AWS. I don’t know why you are seeing the error terminating connection due to administrator command, but there must clearly be a difference between your test setup and production.

@flamber Has anyone tested this latest docker image with AWS Fargate for a fresh self hosted solution with your own internal RDS Postgres DB version 10. This will help to confirm my 4th day struggle to install and self host in the first place.

@sagun.garg No, we have not tested against Fargate, but I know there are several other people using Metabase on Fargate.

@flamber I managed to run it with metabase/metabase:v0.35.3 and double horse power (CPU units, RAM = 2048, 8GB) on AWS Fargate. So definitely the latest image has some issues

@flamber But now the puzzling question is inspite of me passing Postgres Connection environment variables in the my task definition in AWS Fargate i.e Docker Environment variables as highlighted in this unofficial tutorial How to install metabase with aws ecs docker fargate | by Devran AYDIN | Medium why is metabase asking me to setup the connect to my DB Postgres once again?

Please see below

The GUI dosen't recognize the DB that supposedly it says in logs it has sucessfully connected to and had migration problems with :sweat_smile: (It's failing to recognize it's own big green tick in the logs)

:thinking:

@flamber I added the database once again using the GUI directions and now I am stuck with this error

I found these two tickets related to above issue

@flamber I tried again with latest version image i.e v0.37.3 instead of v0.35.3 and I am stuck with memory allocation issues i.e max 2 GB allocation to JVM, hence I need help with latest stable image which is better than v0.35.3. Please advice

@sagun.garg You are misunderstanding the database connections.

Metabase has it’s own application database, where it stores all it’s information. That’s what you are setting with the environment variable.

Then you have data sources, which are the databases you want to view, and they are setup via the GUI.

After you setup a data source, then Metabase needs to sync+scan the database before you can use it in Metabase GUI - otherwise you will see the error Distinct, non-empty sequence of Field clauses
https://github.com/metabase/metabase/issues/11354 - upvote by clicking :+1: on the first post

And I have a feeling that you are running out of entropy, which can cause very slow startup:
https://github.com/metabase/metabase/issues/10175

@flamber Thanks for that update, that literally threw me under the bus (but I managed to resolve this misunderstanding at my level) . I have sorted out the DB table issues as metabase level and data source level (separated the db concerns with two separate postgres DBs)

I ran the sync + scan

I am getting these errors in the Logs in Trouble Shooting Dashboard

and

FYI edited @flamber thanks

{
  "browser-info": {
    "language": "en-GB",
    "platform": "MacIntel",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36",
    "vendor": "Google Inc."
  },
  "system-info": {
    "file.encoding": "UTF-8",
    "java.runtime.name": "OpenJDK Runtime Environment",
    "java.runtime.version": "11.0.5+10",
    "java.vendor": "AdoptOpenJDK",
    "java.vendor.url": "https://adoptopenjdk.net/",
    "java.version": "11.0.5",
    "java.vm.name": "OpenJDK 64-Bit Server VM",
    "java.vm.version": "11.0.5+10",
    "os.name": "Linux",
    "os.version": "4.14.193-149.317.amzn2.x86_64",
    "user.language": "en",
    "user.timezone": "GMT"
  },
  "metabase-info": {
    "databases": [
      "h2",
      "postgres"
    ],
    "hosting-env": "unknown",
    "application-database": "postgres",
    "application-database-details": {
      "database": {
        "name": "PostgreSQL",
        "version": "12.4"
      },
      "jdbc-driver": {
        "name": "PostgreSQL JDBC Driver",
        "version": "42.2.8"
      }
    },
    "run-mode": "prod",
    "version": {
      "date": "2020-04-21",
      "tag": "v0.35.3",
      "branch": "release-0.35.x",
      "hash": "1d424cb"
    },
    "settings": {
      "report-timezone": null
    }
  }
}

@sagun.garg Post “Diagnostic Info” from Admin > Troubleshooting.

1 Like

Apologies for missing on it @flamber posting here once again for your persual

{
  "browser-info": {
    "language": "en-GB",
    "platform": "MacIntel",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36",
    "vendor": "Google Inc."
  },
  "system-info": {
    "file.encoding": "UTF-8",
    "java.runtime.name": "OpenJDK Runtime Environment",
    "java.runtime.version": "11.0.5+10",
    "java.vendor": "AdoptOpenJDK",
    "java.vendor.url": "https://adoptopenjdk.net/",
    "java.version": "11.0.5",
    "java.vm.name": "OpenJDK 64-Bit Server VM",
    "java.vm.version": "11.0.5+10",
    "os.name": "Linux",
    "os.version": "4.14.193-149.317.amzn2.x86_64",
    "user.language": "en",
    "user.timezone": "GMT"
  },
  "metabase-info": {
    "databases": [
      "h2",
      "postgres"
    ],
    "hosting-env": "unknown",
    "application-database": "postgres",
    "application-database-details": {
      "database": {
        "name": "PostgreSQL",
        "version": "12.4"
      },
      "jdbc-driver": {
        "name": "PostgreSQL JDBC Driver",
        "version": "42.2.8"
      }
    },
    "run-mode": "prod",
    "version": {
      "date": "2020-04-21",
      "tag": "v0.35.3",
      "branch": "release-0.35.x",
      "hash": "1d424cb"
    },
    "settings": {
      "report-timezone": null
    }
  }
}

@sagun.garg All those sync errors are fixed on later versions.

Hey @flamber Superthanks for the positive note

**I tried the latest version on metabase/metabase:latest as well on AWS Fargate **

but then it gets stuck at the same JVM memory allocation issue + unrun database migrations,

please can you recommend an alternate predecessor image version number better than v0.35.3 and less than v0.37.3 (i.e latest version) which covers these sync errors and also manages to run on self-hosted AWS Fargate