Metabase 0.36 Upgrade

Kieran_m · July 24, 2020, 8:59am

We are currently running 0.35.4 and tried to roll out 0.36.

Ultimately we got:

2020-07-24T08:49:07.308350266Z 07-24 08:49:07 ERROR metabase.core :: Metabase Initialization FAILED

The errors we are seeing are:

2020-07-24T08:49:07.308401452Z liquibase.exception.MigrationFailedException: Migration failed for change set migrations/000_migrations.yaml::173::camsaul:

2020-07-24T08:49:07.308407244Z Reason: liquibase.exception.DatabaseException: ERROR: constraint "fk_activity_ref_user_id" of relation "activity" does not exist [Failed SQL: ALTER TABLE public.activity DROP CONSTRAINT fk_activity_ref_user_id]

Any idea why this FK constraint wouldn’t exist?

flamber · July 24, 2020, 9:35am

Hi @Kieran_m
Are you using Postgres for the backend?
Could it be that you have manually removed it?
Just add it, so the migration can complete, or set the environment variable MB_DB_AUTOMIGRATE=false and run the migration manually and skipping that query.

Kieran_m · July 24, 2020, 10:52am

Hi @flamber,

We are, sorry I should have stated. We wouldn’t have removed it manually, but I will give re-adding it manually.

Cheers
ps
Should it not a DROP IF EXISTS?

flamber · July 24, 2020, 10:59am

@Kieran_m Well it should exist, since it was added a long time ago (in changeset 13 probably like 5 years ago), so my best guess is that it was manually removed from your database for some reason.
We want the migration script to fail hard instead of allowing it to be open of corruption (I know in your specific case, if exists would probably be fine, but that wouldn’t be in many other cases).
This is the only way we can try to protect the integrity of the metadata.

Lourival · July 24, 2020, 1:54pm

Good morning everyone:

What’s new in this version?

flamber · July 24, 2020, 2:01pm

@Lourival
The changes are published the same place as always:
https://github.com/metabase/metabase/releases
And major releases usually also includes a blog post:
https://www.metabase.com/blog/

Lourival · July 24, 2020, 2:09pm

Hello @flamber, Thank you!

Wow, I was looking forward to this feature:
Better value labels for line, area, bar, and combo charts.

Congratulations to the whole team.

dirkjonker · July 28, 2020, 8:14am

We have a similar issue. I’m pretty sure nobody added/removed any foreign key constraints manually!

07-28 07:54:34 ERROR changelog.ChangeSet :: Change Set migrations/000_migrations.yaml::188::camsaul failed.  Error: ERROR: constraint "fk_session_ref_user_id" for relation "core_session" already exists [Failed SQL: ALTER TABLE public.core_session ADD CONSTRAINT fk_session_ref_user_id FOREIGN KEY (user_id) REFERENCES public.core_user (id) ON DELETE CASCADE]
07-28 07:54:34 WARN metabase.util :: auto-retry metabase.db.liquibase$migrate_up_if_needed_BANG_$fn__20939@6f9d3e6b: Migration failed for change set migrations/000_migrations.yaml::188::camsaul:
     Reason: liquibase.exception.DatabaseException: ERROR: constraint "fk_session_ref_user_id" for relation "core_session" already exists [Failed SQL: ALTER TABLE public.core_session ADD CONSTRAINT fk_session_ref_user_id FOREIGN KEY (user_id) REFERENCES public.core_user (id) ON DELETE CASCADE]
07-28 07:54:34 ERROR changelog.ChangeSet :: Change Set migrations/000_migrations.yaml::188::camsaul failed.  Error: ERROR: constraint "fk_session_ref_user_id" for relation "core_session" already exists [Failed SQL: ALTER TABLE public.core_session ADD CONSTRAINT fk_session_ref_user_id FOREIGN KEY (user_id) REFERENCES public.core_user (id) ON DELETE CASCADE]
07-28 07:54:34 WARN metabase.util :: auto-retry metabase.db.liquibase$migrate_up_if_needed_BANG_$fn__20939@6f9d3e6b: Migration failed for change set migrations/000_migrations.yaml::188::camsaul:
     Reason: liquibase.exception.DatabaseException: ERROR: constraint "fk_session_ref_user_id" for relation "core_session" already exists [Failed SQL: ALTER TABLE public.core_session ADD CONSTRAINT fk_session_ref_user_id FOREIGN KEY (user_id) REFERENCES public.core_user (id) ON DELETE CASCADE]
07-28 07:54:34 ERROR changelog.ChangeSet :: Change Set migrations/000_migrations.yaml::188::camsaul failed.  Error: ERROR: constraint "fk_session_ref_user_id" for relation "core_session" already exists [Failed SQL: ALTER TABLE public.core_session ADD CONSTRAINT fk_session_ref_user_id FOREIGN KEY (user_id) REFERENCES public.core_user (id) ON DELETE CASCADE]
07-28 07:54:34 WARN metabase.util :: auto-retry metabase.db.liquibase$migrate_up_if_needed_BANG_$fn__20939@6f9d3e6b: Migration failed for change set migrations/000_migrations.yaml::188::camsaul:
     Reason: liquibase.exception.DatabaseException: ERROR: constraint "fk_session_ref_user_id" for relation "core_session" already exists [Failed SQL: ALTER TABLE public.core_session ADD CONSTRAINT fk_session_ref_user_id FOREIGN KEY (user_id) REFERENCES public.core_user (id) ON DELETE CASCADE]
07-28 07:54:34 ERROR changelog.ChangeSet :: Change Set migrations/000_migrations.yaml::188::camsaul failed.  Error: ERROR: constraint "fk_session_ref_user_id" for relation "core_session" already exists [Failed SQL: ALTER TABLE public.core_session ADD CONSTRAINT fk_session_ref_user_id FOREIGN KEY (user_id) REFERENCES public.core_user (id) ON DELETE CASCADE]
07-28 07:54:34 ERROR metabase.core :: Metabase Initialization FAILED
liquibase.exception.MigrationFailedException: Migration failed for change set migrations/000_migrations.yaml::188::camsaul:
     Reason: liquibase.exception.DatabaseException: ERROR: constraint "fk_session_ref_user_id" for relation "core_session" already exists [Failed SQL: ALTER TABLE public.core_session ADD CONSTRAINT fk_session_ref_user_id FOREIGN KEY (user_id) REFERENCES public.core_user (id) ON DELETE CASCADE]
	at liquibase.changelog.ChangeSet.execute(ChangeSet.java:637)
	at liquibase.changelog.visitor.UpdateVisitor.visit(UpdateVisitor.java:53)
	at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:83)
	at liquibase.Liquibase.update(Liquibase.java:202)
	at liquibase.Liquibase.update(Liquibase.java:179)
	at liquibase.Liquibase.update(Liquibase.java:175)
	at metabase.db.liquibase$migrate_up_if_needed_BANG_$fn__20939.invoke(liquibase.clj:126)
	at metabase.util$do_with_auto_retries.invokeStatic(util.clj:423)
	at metabase.util$do_with_auto_retries.invoke(util.clj:415)
	at metabase.util$do_with_auto_retries.invokeStatic(util.clj:430)
	at metabase.util$do_with_auto_retries.invoke(util.clj:415)
	at metabase.util$do_with_auto_retries.invokeStatic(util.clj:430)
	at metabase.util$do_with_auto_retries.invoke(util.clj:415)
	at metabase.util$do_with_auto_retries.invokeStatic(util.clj:430)
	at metabase.util$do_with_auto_retries.invoke(util.clj:415)
	at metabase.db.liquibase$migrate_up_if_needed_BANG_.invokeStatic(liquibase.clj:126)
	at metabase.db.liquibase$migrate_up_if_needed_BANG_.invoke(liquibase.clj:114)
	at metabase.db$migrate_BANG_$fn__21719$fn__21720.invoke(db.clj:208)
	at metabase.db.liquibase$do_with_liquibase.invokeStatic(liquibase.clj:38)
	at metabase.db.liquibase$do_with_liquibase.invoke(liquibase.clj:29)
	at metabase.db$migrate_BANG_$fn__21719.invoke(db.clj:203)
	at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:807)
	at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:776)
	at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:852)
	at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:776)
	at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:789)
	at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:776)
	at metabase.db$migrate_BANG_.invokeStatic(db.clj:194)
	at metabase.db$migrate_BANG_.invoke(db.clj:173)
	at metabase.db$run_schema_migrations_BANG_.invokeStatic(db.clj:329)
	at metabase.db$run_schema_migrations_BANG_.invoke(db.clj:323)
	at metabase.db$setup_db_BANG__STAR_$fn__21801.invoke(db.clj:346)
	at metabase.util$do_with_us_locale.invokeStatic(util.clj:665)
	at metabase.util$do_with_us_locale.invoke(util.clj:651)
	at metabase.db$setup_db_BANG__STAR_.invokeStatic(db.clj:344)
	at metabase.db$setup_db_BANG__STAR_.invoke(db.clj:339)
	at metabase.db$setup_db_from_env_BANG_.invokeStatic(db.clj:358)
	at metabase.db$setup_db_from_env_BANG_.invoke(db.clj:351)
	at metabase.db$setup_db_BANG_$fn__21806.invoke(db.clj:368)
	at metabase.db$setup_db_BANG_.invokeStatic(db.clj:366)
	at metabase.db$setup_db_BANG_.invoke(db.clj:361)
	at metabase.core$init_BANG_.invokeStatic(core.clj:75)
	at metabase.core$init_BANG_.invoke(core.clj:54)
	at metabase.core$start_normally.invokeStatic(core.clj:119)
	at metabase.core$start_normally.invoke(core.clj:113)
	at metabase.core$_main.invokeStatic(core.clj:152)
	at metabase.core$_main.doInvoke(core.clj:146)
	at clojure.lang.RestFn.invoke(RestFn.java:397)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.RestFn.applyTo(RestFn.java:132)
	at metabase.core.main(Unknown Source)
Caused by: liquibase.exception.DatabaseException: ERROR: constraint "fk_session_ref_user_id" for relation "core_session" already exists [Failed SQL: ALTER TABLE public.core_session ADD CONSTRAINT fk_session_ref_user_id FOREIGN KEY (user_id) REFERENCES public.core_user (id) ON DELETE CASCADE]
	at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:356)
	at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:57)
	at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:125)
	at liquibase.database.AbstractJdbcDatabase.execute(AbstractJdbcDatabase.java:1229)
	at liquibase.database.AbstractJdbcDatabase.executeStatements(AbstractJdbcDatabase.java:1211)
	at liquibase.changelog.ChangeSet.execute(ChangeSet.java:600)
	... 50 more
Caused by: org.postgresql.util.PSQLException: ERROR: constraint "fk_session_ref_user_id" for relation "core_session" already exists
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2497)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2233)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:310)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370)
	at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:311)
	at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:297)
	at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:274)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:269)
	at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:352)
	... 55 more
07-28 07:54:35 INFO metabase.core :: Metabase Shutting Down ...
07-28 07:54:35 INFO metabase.server :: Shutting Down Embedded Jetty Webserver
07-28 07:54:35 INFO metabase.core :: Metabase Shutdown COMPLETE

Manually dropping the constraint did fix the issue it seems.

flamber · July 28, 2020, 8:53am

@dirkjonker You have just the opposite error. It’s trying to add fk_session_ref_user_id to core_session, but that already exists. That’s likely caused by a previous bad migration.

dirkjonker · July 28, 2020, 8:59am

Yes, manually dropping the constraint has fixed it luckily.

Kieran_m · July 28, 2020, 11:46am

Re-adding a lot of the FK constraints led to a successful migration .

We are seeing a bug though with the column order not reflecting the “database order”. This was our main reason for upgrading, so if anyone else has seen this problem or has any solutions that would be amazing!

Kieran_m · July 28, 2020, 1:34pm

Fixed.

We needed to cycle through all the different types of ordering in the UI and re-select “database” again in order to see the correct ordering.

flamber · July 28, 2020, 3:59pm

@Kieran_m Upvote this issue: https://github.com/metabase/metabase/issues/13024

Kieran_m · July 28, 2020, 4:18pm

Thanks you @flamber

flamber · July 31, 2020, 9:56am

@Kieran_m @dirkjonker Can you post the output of this query on your application database?

SELECT * FROM "public"."databasechangelog"

It would be interesting to see if the migration script actually succeeded on your step, but somehow the constraint was lost at a later time, or if the migration script failed and just continued.

There’s now been 3 reports of different old constraints that wasn’t applied or lost at some point.
https://github.com/metabase/metabase/issues/13043
And all relating to Postgres so far.