Migrating from heroku buildpacks to docker on heroku

jeanatfirstround.com · September 6, 2023, 10:14pm

Continuing the discussion from Support for Heroku:

Hi! We have metabase running on a heroku app that uses the old buildpacks. I am trying to upgrade to the docker configuration, so I can update metabase, so I can move to metabase cloud (version is too old to move to metabase cloud right now). Here's the steps I am taking:

clone down the heroku app (I don't have access to any github repo associated with the app):
heroku git:clone -a {my app}. The app is a fork / copy of GitHub - metabase/metabase-deploy: Metabase binary deployment
I added the Dockerfile and docker-entrypoint.sh as specified in related threads (they were not part of the repo before)
Ran heroku container:push web -a your-heroku-app and then heroku container:release web -a your-heroku-app
The app crashes:

Database has migration lock; cannot run migrations. You can force-release these locks by running `java -jar metabase.jar migrate release-locks`.e[0m
2023-09-06T21:58:01.471773+00:00 app[web.1]: 2023-09-06 21:58:01,471 WARN util.jvm :: e[31mauto-retry metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__51743@7170302a: Database has migration lock; cannot run migrations. You can force-release these locks by running `java -jar metabase.jar migrate release-locks`.e[0m
2023-09-06T21:58:03.502265+00:00 app[web.1]: 2023-09-06 21:58:03,502 WARN util.jvm :: e[31mauto-retry metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__51743@7170302a: Database has migration lock; cannot run migrations. You can force-release these locks by running `java -jar metabase.jar migrate release-locks`.e[0m
2023-09-06T21:58:05.535435+00:00 app[web.1]: 2023-09-06 21:58:05,535 WARN util.jvm :: e[31mauto-retry metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__51743@7170302a: Database has migration lock; cannot run migrations. You can force-release these locks by running `java -jar metabase.jar migrate release-locks`.e[0m
2023-09-06T21:58:07.595756+00:00 app[web.1]: 2023-09-06 21:58:07,578 ERROR metabase.core :: Metabase Initialization FAILED
2023-09-06T21:58:07.595777+00:00 app[web.1]: liquibase.exception.LockException: Database has migration lock; cannot run migrations. You can force-release these locks by running `java -jar metabase.jar migrate release-locks`.
2023-09-06T21:58:07.595778+00:00 app[web.1]: 	at metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__51743.invoke(liquibase.clj:139)
2023-09-06T21:58:07.595778+00:00 app[web.1]: 	at metabase.util.jvm$do_with_auto_retries.invokeStatic(jvm.clj:176)
2023-09-06T21:58:07.595779+00:00 app[web.1]: 	at metabase.util.jvm$do_with_auto_retries.invoke(jvm.clj:165)
2023-09-06T21:58:07.595779+00:00 app[web.1]: 	at metabase.util.jvm$do_with_auto_retries.invokeStatic(jvm.clj:183)
2023-09-06T21:58:07.595779+00:00 app[web.1]: 	at metabase.util.jvm$do_with_auto_retries.invoke(jvm.clj:165)
2023-09-06T21:58:07.595780+00:00 app[web.1]: 	at metabase.util.jvm$do_with_auto_retries.invokeStatic(jvm.clj:183)
2023-09-06T21:58:07.595780+00:00 app[web.1]: 	at metabase.util.jvm$do_with_auto_retries.invoke(jvm.clj:165)
2023-09-06T21:58:07.595780+00:00 app[web.1]: 	at metabase.util.jvm$do_with_auto_retries.invokeStatic(jvm.clj:183)
2023-09-06T21:58:07.595781+00:00 app[web.1]: 	at metabase.util.jvm$do_with_auto_retries.invoke(jvm.clj:165)
2023-09-06T21:58:07.595781+00:00 app[web.1]: 	at metabase.util.jvm$do_with_auto_retries.invokeStatic(jvm.clj:183)
2023-09-06T21:58:07.595781+00:00 app[web.1]: 	at metabase.util.jvm$do_with_auto_retries.invoke(jvm.clj:165)
2023-09-06T21:58:07.595782+00:00 app[web.1]: 	at metabase.util.jvm$do_with_auto_retries.invokeStatic(jvm.clj:183)
2023-09-06T21:58:07.595782+00:00 app[web.1]: 	at metabase.util.jvm$do_with_auto_retries.invoke(jvm.clj:165)
2023-09-06T21:58:07.595782+00:00 app[web.1]: 	at metabase.db.liquibase$wait_for_migration_lock_to_be_cleared.invokeStatic(liquibase.clj:134)
2023-09-06T21:58:07.595783+00:00 app[web.1]: 	at metabase.db.liquibase$wait_for_migration_lock_to_be_cleared.invoke(liquibase.clj:130)
2023-09-06T21:58:07.595783+00:00 app[web.1]: 	at metabase.db.liquibase$migrate_up_if_needed_BANG_.invokeStatic(liquibase.clj:150)
2023-09-06T21:58:07.595783+00:00 app[web.1]: 	at metabase.db.liquibase$migrate_up_if_needed_BANG_.invoke(liquibase.clj:144)
2023-09-06T21:58:07.595784+00:00 app[web.1]: 	at metabase.db.setup$fn__52038$migrate_BANG___52043$fn__52044$fn__52045.invoke(setup.clj:78)
2023-09-06T21:58:07.595784+00:00 app[web.1]: 	at metabase.db.liquibase$fn__51707$do_with_liquibase__51712$fn__51713.invoke(liquibase.clj:66)
2023-09-06T21:58:07.595784+00:00 app[web.1]: 	at metabase.db.liquibase$fn__51707$do_with_liquibase__51712.invoke(liquibase.clj:59)
2023-09-06T21:58:07.595784+00:00 app[web.1]: 	at metabase.db.setup$fn__52038$migrate_BANG___52043$fn__52044.invoke(setup.clj:73)
2023-09-06T21:58:07.595785+00:00 app[web.1]: 	at metabase.db.setup$fn__52038$migrate_BANG___52043.doInvoke(setup.clj:52)
2023-09-06T21:58:07.595785+00:00 app[web.1]: 	at clojure.lang.RestFn.invoke(RestFn.java:445)
2023-09-06T21:58:07.595785+00:00 app[web.1]: 	at metabase.db.setup$fn__52102$run_schema_migrations_BANG___52107$fn__52108.invoke(setup.clj:131)
2023-09-06T21:58:07.595786+00:00 app[web.1]: 	at metabase.db.setup$fn__52102$run_schema_migrations_BANG___52107.invoke(setup.clj:125)
2023-09-06T21:58:07.595786+00:00 app[web.1]: 	at metabase.db.setup$fn__52155$setup_db_BANG___52160$fn__52161$fn__52164$fn__52165.invoke(setup.clj:156)
2023-09-06T21:58:07.595786+00:00 app[web.1]: 	at metabase.util.jvm$do_with_us_locale.invokeStatic(jvm.clj:239)
2023-09-06T21:58:07.595786+00:00 app[web.1]: 	at metabase.util.jvm$do_with_us_locale.invoke(jvm.clj:225)
2023-09-06T21:58:07.595787+00:00 app[web.1]: 	at metabase.db.setup$fn__52155$setup_db_BANG___52160$fn__52161$fn__52164.invoke(setup.clj:152)
2023-09-06T21:58:07.595787+00:00 app[web.1]: 	at metabase.db.setup$fn__52155$setup_db_BANG___52160$fn__52161.invoke(setup.clj:151)
2023-09-06T21:58:07.595787+00:00 app[web.1]: 	at metabase.db.setup$fn__52155$setup_db_BANG___52160.invoke(setup.clj:145)
2023-09-06T21:58:07.595787+00:00 app[web.1]: 	at metabase.db$setup_db_BANG_$fn__52188.invoke(db.clj:66)
2023-09-06T21:58:07.595787+00:00 app[web.1]: 	at metabase.db$setup_db_BANG_.invokeStatic(db.clj:61)
2023-09-06T21:58:07.595787+00:00 app[web.1]: 	at metabase.db$setup_db_BANG_.invoke(db.clj:52)
2023-09-06T21:58:07.595788+00:00 app[web.1]: 	at metabase.core$init_BANG__STAR_.invokeStatic(core.clj:113)
2023-09-06T21:58:07.595788+00:00 app[web.1]: 	at metabase.core$init_BANG__STAR_.invoke(core.clj:99)
2023-09-06T21:58:07.595788+00:00 app[web.1]: 	at metabase.core$init_BANG_.invokeStatic(core.clj:159)
2023-09-06T21:58:07.595788+00:00 app[web.1]: 	at metabase.core$init_BANG_.invoke(core.clj:154)
2023-09-06T21:58:07.595788+00:00 app[web.1]: 	at metabase.core$start_normally.invokeStatic(core.clj:171)
2023-09-06T21:58:07.595788+00:00 app[web.1]: 	at metabase.core$start_normally.invoke(core.clj:165)
2023-09-06T21:58:07.595789+00:00 app[web.1]: 	at metabase.core$_main.invokeStatic(core.clj:204)
2023-09-06T21:58:07.595789+00:00 app[web.1]: 	at metabase.core$_main.doInvoke(core.clj:198)
2023-09-06T21:58:07.595789+00:00 app[web.1]: 	at clojure.lang.RestFn.invoke(RestFn.java:397)
2023-09-06T21:58:07.595790+00:00 app[web.1]: 	at clojure.lang.AFn.applyToHelper(AFn.java:152)
2023-09-06T21:58:07.595790+00:00 app[web.1]: 	at clojure.lang.RestFn.applyTo(RestFn.java:132)
2023-09-06T21:58:07.595790+00:00 app[web.1]: 	at clojure.lang.Var.applyTo(Var.java:705)
2023-09-06T21:58:07.595790+00:00 app[web.1]: 	at clojure.core$apply.invokeStatic(core.clj:667)
2023-09-06T21:58:07.595790+00:00 app[web.1]: 	at clojure.core$apply.invoke(core.clj:662)
2023-09-06T21:58:07.595791+00:00 app[web.1]: 	at metabase.bootstrap$_main.invokeStatic(bootstrap.clj:31)
2023-09-06T21:58:07.595791+00:00 app[web.1]: 	at metabase.bootstrap$_main.doInvoke(bootstrap.clj:28)
2023-09-06T21:58:07.595791+00:00 app[web.1]: 	at clojure.lang.RestFn.invoke(RestFn.java:397)
2023-09-06T21:58:07.595791+00:00 app[web.1]: 	at clojure.lang.AFn.applyToHelper(AFn.java:152)
2023-09-06T21:58:07.595791+00:00 app[web.1]: 	at clojure.lang.RestFn.applyTo(RestFn.java:132)
2023-09-06T21:58:07.595791+00:00 app[web.1]: 	at metabase.bootstrap.main(Unknown Source)
2023-09-06T21:58:07.607291+00:00 app[web.1]: 2023-09-06 21:58:07,604 INFO metabase.core :: Metabase Shutting Down ...
2023-09-06T21:58:07.609298+00:00 app[web.1]: 2023-09-06 21:58:07,609 INFO metabase.server :: Shutting Down Embedded Jetty Webserver
2023-09-06T21:58:07.625021+00:00 app[web.1]: 2023-09-06 21:58:07,624 INFO metabase.core :: Metabase Shutdown COMPLETE
2023-09-06T21:58:08.212454+00:00 heroku[web.1]: Process exited with status 1
2023-09-06T21:58:08.248158+00:00 heroku[web.1]: State changed from up to crashed

What am I missing?
my docker-entrypoint.sh:

#!/usr/bin/env bash

if [ "$PORT" ]; then
    export MB_JETTY_PORT="$PORT"
fi

if [ "$DATABASE_URL" ]; then
    export MB_DB_CONNECTION_URI="$DATABASE_URL"
fi

/app/run_metabase.sh

my Dockerfile:

FROM metabase/metabase:latest

COPY docker-entrypoint.sh /app/

RUN ["chmod", "+x", "/app/docker-entrypoint.sh"]

ENTRYPOINT [ "/app/docker-entrypoint.sh" ]

jeanatfirstround.com · September 6, 2023, 10:38pm

Do I need to upgrade my plan?

github.com/render-examples/metabase

Following install guide: Metabase crash loop, re: database migration lock, recommend starter-plus as min instance size?

opened 08:18AM - 11 Jan 21 UTC

sherbondy

Hey folks, I just tried setting up a Metabase instance alongside a fresh Rend…er Postgres DB instance per the install guide. (https://render.com/docs/deploy-metabase) I ran into the following deployment crash loop in the logs, where it seems like the Metabase server was trying to run its initial DB migrations on the first launch and complained it could not because the database was holding onto a migration lock: ``` Jan 11 03:53:05 PM 2021-01-11 07:53:05,573 INFO db.liquibase :: Checking if Database has unrun migrations... Jan 11 03:53:16 PM 2021-01-11 07:53:16,268 INFO db.liquibase :: Database has unrun migrations. Waiting for migration lock to be cleared... Jan 11 03:53:18 PM 2021-01-11 07:53:18,274 WARN metabase.util :: auto-retry metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__20958@4df6d39c: Database has migration lock; cannot run migrations. You can force-release these locks by running `java -jar metabase.jar migrate release-locks`. Jan 11 03:53:20 PM 2021-01-11 07:53:20,277 WARN metabase.util :: auto-retry metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__20958@4df6d39c: Database has migration lock; cannot run migrations. You can force-release these locks by running `java -jar metabase.jar migrate release-locks`. Jan 11 03:53:22 PM 2021-01-11 07:53:22,279 WARN metabase.util :: auto-retry metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__20958@4df6d39c: Database has migration lock; cannot run migrations. You can force-release these locks by running `java -jar metabase.jar migrate release-locks`. Jan 11 03:53:24 PM 2021-01-11 07:53:24,281 WARN metabase.util :: auto-retry metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__20958@4df6d39c: Database has migration lock; cannot run migrations. You can force-release these locks by running `java -jar metabase.jar migrate release-locks`. Jan 11 03:53:26 PM 2021-01-11 07:53:26,284 WARN metabase.util :: auto-retry metabase.db.liquibase$wait_for_migration_lock_to_be_cleared$fn__20958@4df6d39c: Database has migration lock; cannot run migrations. You can force-release these locks by running `java -jar metabase.jar migrate release-locks`. Jan 11 03:53:28 PM 2021-01-11 07:53:28,289 ERROR metabase.core :: Metabase Initialization FAILED Jan 11 03:53:28 PM liquibase.exception.LockException: Database has migration lock; cannot run migrations. You can force-release these locks by running `java -jar metabase.jar migrate release-locks`. ``` Has anyone else run into this issue on first deployment? I was able to jump into the Shell in the Render dashboard for the Metabase service and run the `java -jar metabase.jar migrate release-locks` command. But the migration lock seems to be a red herring, and the real underlying issue seems to be that the 512 MB of memory available for starter instances is simply not enough to feed the hungry, hungry JVM. It seems like Metabase crashes due to OOM in the middle of the initial migration process. Maybe the Deploy guide should mention `starter-plus` as the minimum advisable plan to deploy a Metabase instance?

If so, how do I do so? Is there a way to unblock the migrations if I truncate the database or something similar, is this a memory issue?

I seem to be locked out of upgrading, because we can’t log into our store account, because we need to create an instance there, which requires us to migrate our old instance, which requires an upgrade to finish migration from heroku buildpacks to docker, which requires a plan increase, which requires logging into our store account

jeanatfirstround.com · September 7, 2023, 3:26pm

Even when I use a fresh database with no data whatsoever, this issue occurs. How do I avoid Error R15 (Memory quota vastly exceeded)?

2023-09-07T15:24:09.441509+00:00 app[web.1]: ..instrumented #'metabase.util.honey-sql-2/cast
2023-09-07T15:24:09.442548+00:00 app[web.1]: ..instrumented #'metabase.util.honey-sql-2/quoted-cast
2023-09-07T15:24:09.443558+00:00 app[web.1]: ..instrumented #'metabase.util.honey-sql-2/maybe-cast
2023-09-07T15:24:10.726701+00:00 app[web.1]: ..instrumented #'metabase.models.permissions/classify-path
2023-09-07T15:24:10.735997+00:00 app[web.1]: ..instrumented #'metabase.models.permissions/classify-data-path
2023-09-07T15:24:10.922271+00:00 app[web.1]: ..instrumented #'metabase.models.permissions/generate-graph
2023-09-07T15:24:10.935337+00:00 app[web.1]: ..instrumented #'metabase.models.permissions/->v2-path
2023-09-07T15:24:10.982166+00:00 app[web.1]: ..instrumented #'metabase.models.permissions/update-db-data-access-permissions!
2023-09-07T15:24:10.988942+00:00 app[web.1]: ..instrumented #'metabase.models.permissions/update-group-permissions!
2023-09-07T15:24:11.023294+00:00 app[web.1]: ..instrumented #'metabase.models.permissions/update-data-perms-graph!
2023-09-07T15:24:18.440445+00:00 app[web.1]: ..instrumented #'metabase.models.parameter-card/upsert-or-delete-from-parameters!
2023-09-07T15:24:21.398439+00:00 heroku[web.1]: Process running mem=912M(178.3%)
2023-09-07T15:24:21.400409+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2023-09-07T15:24:22.893840+00:00 app[web.1]: WARNING: abs already refers to: #'clojure.core/abs in namespace: kixi.stats.math, being replaced by: #'kixi.stats.math/abs
2023-09-07T15:24:22.934849+00:00 app[web.1]: WARNING: abs already refers to: #'clojure.core/abs in namespace: kixi.stats.test, being replaced by: #'kixi.stats.math/abs
2023-09-07T15:24:23.056137+00:00 app[web.1]: WARNING: abs already refers to: #'clojure.core/abs in namespace: kixi.stats.distribution, being replaced by: #'kixi.stats.math/abs
2023-09-07T15:24:26.465930+00:00 app[web.1]: ..instrumented #'metabase.query-processor.middleware.permissions/check-query-permissions*
2023-09-07T15:24:34.366215+00:00 heroku[web.1]: Process running mem=1279M(250.0%)
2023-09-07T15:24:34.390695+00:00 heroku[web.1]: Error R15 (Memory quota vastly exceeded)
2023-09-07T15:24:34.393073+00:00 heroku[web.1]: Stopping process with SIGKILL
2023-09-07T15:24:34.591685+00:00 heroku[web.1]: Process exited with status 137
2023-09-07T15:24:34.617924+00:00 heroku[web.1]: State changed from starting to crashed
2023-09-07T15:24:46.000000+00:00 app[heroku-postgres]: source=DATABASE addon=postgresql-cubic-41212 sample#current_transaction=761 sample#db_size=7901999bytes sample#tables=0 sample#active-connections=13 sample#waiting-connections=0 sample#index-cache-hit-rate=0 sample#table-cache-hit-rate=0 sample#load-avg-1m=0.015 sample#load-avg-5m=0.145 sample#load-avg-15m=0.095 sample#read-iops=0 sample#write-iops=4.7284 sample#tmp-disk-used=543600640 sample#tmp-disk-available=72435191808 sample#memory-total=3944416kB sample#memory-free=2897688kB sample#memory-cached=776316kB sample#memory-postgres=20788kB sample#wal-percentage-used=0.04863773944905929

I've tried setting the JAVA_TOOL_OPTIONS as well.