"changesets check sum" startup validation fails after failed upgrade

[tengammaps]

I'm running Metabase OSS directly on Ubuntu (no docker) against a Postgresql db. I recently tried to upgrade from v48 to v49, which failed. I think the failure was due to a way downrev PG instance, at 9.3, which does not have support for the "create index IF EXISTS..." syntax.

I have upgraded the DB to PG v11 so it should now have all the language support required. (and I am at Java v11.) Now the startup fails with:

Command failed with exception: liquibase.exception.ValidationFailedException: Validation Failed:
     1 changesets check sum
          migrations/001_update_migrations.yaml::v48.00-013::qnkhuat was: 9:8d9567c68982515a77ede7be26ebf112 but is now: 9:9478b5f507bc34df51aaa4d719fc1c37

I am quite tempted to edit the "databasechangelog" table and substitute the current checksum value with the old one, but I can't be sure that's a very smart idea since I don't even know what the checksum value is calculated against!

Any guidance very much appreciated

Here is the complete startup info to stdout:

mikem@rocinante:/opt/metabase$ sudo metabase.sh
2024-06-08 14:37:46,898 INFO metabase.util :: Maximum memory available to JVM: 7.3 GB
2024-06-08 14:37:48,697 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance.
For more information, see Redirecting…
2024-06-08 14:37:53,935 INFO driver.impl :: Registered abstract driver :sql
2024-06-08 14:37:53,966 INFO driver.impl :: Registered abstract driver :sql-jdbc (parents: [:sql])
2024-06-08 14:37:53,973 INFO metabase.util :: Load driver :sql-jdbc took 97.2 ms
2024-06-08 14:37:53,974 INFO driver.impl :: Registered driver :h2 (parents: [:sql-jdbc])
2024-06-08 14:37:54,143 INFO driver.impl :: Registered driver :mysql (parents: [:sql-jdbc])
2024-06-08 14:37:54,173 INFO driver.impl :: Registered driver :postgres (parents: [:sql-jdbc])
2024-06-08 14:37:55,675 INFO metabase.core ::
Metabase v0.49.13 (de28e83)

Copyright © 2024 Metabase, Inc.

Metabase Enterprise Edition extensions are NOT PRESENT.
2024-06-08 14:37:55,683 INFO metabase.core :: Starting Metabase in STANDALONE mode
2024-06-08 14:37:55,723 INFO metabase.server :: Launching Embedded Jetty Webserver with config:
{:port 2345}

2024-06-08 14:37:55,767 INFO metabase.core :: Starting Metabase version v0.49.13 (de28e83) ...
2024-06-08 14:37:55,771 INFO metabase.core :: System info:
{"file.encoding" "UTF-8",
"java.runtime.name" "OpenJDK Runtime Environment",
"java.runtime.version" "11.0.23+9-post-Ubuntu-1ubuntu122.04.1",
"java.vendor" "Ubuntu",
"java.vendor.url" "https://ubuntu.com/",
"java.version" "11.0.23",
"java.vm.name" "OpenJDK 64-Bit Server VM",
"java.vm.version" "11.0.23+9-post-Ubuntu-1ubuntu122.04.1",
"os.name" "Linux",
"os.version" "5.15.0-112-generic",
"user.language" "en",
"user.timezone" "America/Los_Angeles"}

2024-06-08 14:37:55,773 INFO metabase.plugins :: Loading plugins in /opt/metabase/plugins...
2024-06-08 14:37:56,124 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :googleanalytics...
2024-06-08 14:37:56,125 INFO driver.impl :: Registered driver :googleanalytics
2024-06-08 14:37:56,158 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :snowflake...
2024-06-08 14:37:56,159 INFO driver.impl :: Registered driver :snowflake (parents: [:sql-jdbc])
2024-06-08 14:37:56,167 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :hive-like...
2024-06-08 14:37:56,168 INFO driver.impl :: Registered abstract driver :hive-like (parents: [:sql-jdbc])
2024-06-08 14:37:56,168 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :sparksql...
2024-06-08 14:37:56,169 INFO driver.impl :: Registered driver :sparksql (parents: [:hive-like])
2024-06-08 14:37:56,177 INFO plugins.dependencies :: Metabase cannot initialize plugin Metabase Oracle Driver due to required dependencies. Metabase requires the Oracle JDBC driver in order to connect to Oracle databases, but we can't ship it as part of Metabase due to licensing restrictions. See Redirecting… for more details.

2024-06-08 14:37:56,178 INFO plugins.dependencies :: Metabase Oracle Driver dependency {:class oracle.jdbc.OracleDriver} satisfied? false
2024-06-08 14:37:56,179 INFO plugins.dependencies :: Plugins with unsatisfied deps: ["Metabase Oracle Driver"]
2024-06-08 14:37:56,183 INFO plugins.dependencies :: Metabase cannot initialize plugin Metabase Vertica Driver due to required dependencies. Metabase requires the Vertica JDBC driver in order to connect to Vertica databases, but we can't ship it as part of Metabase due to licensing restrictions. See Redirecting… for more details.

2024-06-08 14:37:56,183 INFO plugins.dependencies :: Metabase Vertica Driver dependency {:class com.vertica.jdbc.Driver} satisfied? false
2024-06-08 14:37:56,184 INFO plugins.dependencies :: Plugins with unsatisfied deps: ["Metabase Oracle Driver" "Metabase Vertica Driver"]
2024-06-08 14:37:56,198 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :presto-jdbc...
2024-06-08 14:37:56,199 INFO driver.impl :: Registered driver :presto-jdbc (parents: [:sql-jdbc])
2024-06-08 14:37:56,212 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :bigquery-cloud-sdk...
2024-06-08 14:37:56,213 INFO driver.impl :: Registered driver :bigquery-cloud-sdk (parents: [:sql])
2024-06-08 14:37:56,216 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :sqlite...
2024-06-08 14:37:56,217 INFO driver.impl :: Registered driver :sqlite (parents: [:sql-jdbc])
2024-06-08 14:37:56,225 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :mongo...
2024-06-08 14:37:56,226 INFO driver.impl :: Registered driver :mongo
2024-06-08 14:37:56,230 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :druid...
2024-06-08 14:37:56,231 INFO driver.impl :: Registered driver :druid
2024-06-08 14:37:56,240 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :athena...
2024-06-08 14:37:56,241 INFO driver.impl :: Registered driver :athena (parents: [:sql-jdbc])
2024-06-08 14:37:56,245 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :sqlserver...
2024-06-08 14:37:56,245 INFO driver.impl :: Registered driver :sqlserver (parents: [:sql-jdbc])
2024-06-08 14:37:56,249 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :redshift...
2024-06-08 14:37:56,250 INFO driver.impl :: Registered driver :redshift (parents: [:postgres])
2024-06-08 14:37:56,256 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
2024-06-08 14:37:56,258 INFO db.setup :: Verifying postgres Database Connection ...
2024-06-08 14:37:56,518 INFO db.setup :: Successfully verified PostgreSQL 14.12 (Ubuntu 14.12-0ubuntu0.22.04.1) application database connection.
2024-06-08 14:37:56,518 INFO db.setup :: Checking if a database downgrade is required...
2024-06-08 14:37:57,011 INFO db.setup :: Running Database Migrations...
2024-06-08 14:37:57,012 INFO db.setup :: Setting up Liquibase...
2024-06-08 14:37:57,156 INFO db.setup :: Liquibase is ready.
2024-06-08 14:37:57,156 INFO db.liquibase :: Checking if Database has unrun migrations...
2024-06-08 14:37:57,743 ERROR metabase.core :: Metabase Initialization FAILED
liquibase.exception.ValidationFailedException: Validation Failed:
1 changesets check sum
migrations/001_update_migrations.yaml::v48.00-013::qnkhuat was: 9:8d9567c68982515a77ede7be26ebf112 but is now: 9:9478b5f507bc34df51aaa4d719fc1c37

at liquibase.changelog.DatabaseChangeLog.validate(DatabaseChangeLog.java:358)
at liquibase.Liquibase.lambda$listUnrunChangeSets$20(Liquibase.java:1149)
at liquibase.Scope.lambda$child$0(Scope.java:186)
at liquibase.Scope.child(Scope.java:195)
at liquibase.Scope.child(Scope.java:185)
at liquibase.Scope.child(Scope.java:164)
at liquibase.Liquibase.runInScope(Liquibase.java:1419)
at liquibase.Liquibase.listUnrunChangeSets(Liquibase.java:1141)
at liquibase.Liquibase.listUnrunChangeSets(Liquibase.java:1131)
at metabase.db.liquibase$unrun_migrations$fn__48850.invoke(liquibase.clj:185)
at metabase.db.liquibase$do_with_liquibase$f_STAR___48835.invoke(liquibase.clj:139)
at metabase.db.liquibase$do_with_liquibase.invokeStatic(liquibase.clj:142)
at metabase.db.liquibase$do_with_liquibase.invoke(liquibase.clj:130)
at metabase.db.liquibase$unrun_migrations.invokeStatic(liquibase.clj:184)
at metabase.db.liquibase$unrun_migrations.invoke(liquibase.clj:175)
at metabase.db.liquibase$migrate_up_if_needed_BANG_.invokeStatic(liquibase.clj:291)
at metabase.db.liquibase$migrate_up_if_needed_BANG_.invoke(liquibase.clj:287)
at metabase.db.setup$migrate_BANG_$fn__51159.invoke(setup.clj:80)
at metabase.db.liquibase$do_with_liquibase$f_STAR___48835.invoke(liquibase.clj:139)
at metabase.db.liquibase$do_with_liquibase.invokeStatic(liquibase.clj:142)
at metabase.db.liquibase$do_with_liquibase.invoke(liquibase.clj:130)
at metabase.db.setup$migrate_BANG_.invokeStatic(setup.clj:75)
at metabase.db.setup$migrate_BANG_.doInvoke(setup.clj:56)
at clojure.lang.RestFn.invoke(RestFn.java:445)
at metabase.db.setup$run_schema_migrations_BANG_.invokeStatic(setup.clj:147)
at metabase.db.setup$run_schema_migrations_BANG_.invoke(setup.clj:141)
at metabase.db.setup$setup_db_BANG_$fn__51187$fn__51188.invoke(setup.clj:165)
at metabase.util.jvm$do_with_us_locale.invokeStatic(jvm.clj:239)
at metabase.util.jvm$do_with_us_locale.invoke(jvm.clj:225)
at metabase.db.setup$setup_db_BANG_$fn__51187.invoke(setup.clj:160)
at metabase.db.setup$setup_db_BANG_.invokeStatic(setup.clj:159)
at metabase.db.setup$setup_db_BANG_.invoke(setup.clj:153)
at metabase.db$setup_db_BANG_$fn__51207.invoke(db.clj:69)
at metabase.db$setup_db_BANG_.invokeStatic(db.clj:64)
at metabase.db$setup_db_BANG_.invoke(db.clj:55)
at metabase.core$init_BANG__STAR_.invokeStatic(core.clj:116)
at metabase.core$init_BANG__STAR_.invoke(core.clj:101)
at metabase.core$init_BANG_.invokeStatic(core.clj:159)
at metabase.core$init_BANG_.invoke(core.clj:154)
at metabase.core$start_normally.invokeStatic(core.clj:171)
at metabase.core$start_normally.invoke(core.clj:165)
at metabase.core$entrypoint.invokeStatic(core.clj:204)
at metabase.core$entrypoint.doInvoke(core.clj:198)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.core$apply.invoke(core.clj:662)
at metabase.bootstrap$_main.invokeStatic(bootstrap.clj:31)
at metabase.bootstrap$_main.doInvoke(bootstrap.clj:28)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at metabase.bootstrap.main(Unknown Source)

2024-06-08 14:37:57,749 INFO metabase.core :: Metabase Shutting Down ...
2024-06-08 14:37:57,750 INFO metabase.server :: Shutting Down Embedded Jetty Webserver
2024-06-08 14:37:57,756 WARN db.liquibase :: ()
2024-06-08 14:37:57,757 INFO metabase.core :: Metabase Shutdown COMPLETE
mikem@rocinante:/opt/metabase$

[tengammaps]

I said above I upgraded to PG 11, which was wrong. I am now running MB against PG v14.

Also, FWIW, I've tried "migrate down", which fails with the identical error message.

[TonyC]

Did you do the changes to PG and Java after you tried to upgrade? OR you restored a snapshot first and then you tried to upgrade again on a non problematic 48 version?

[tengammaps]

First, I appreciate very much your willingness to jump in and offer help!

Actually, I left out some details that I don't believe are relevant, but the chronology was:

1. Attempted 48.x (48.6.5, I think) to 49.13 upgrade, which failed; afterwards, MB would not start.
2. Tried to "migrate down", which also failed with "change set checksum was x is now y" error.
3. Installed PG14 on a DIFFERENT host on same network segment
4. Backed up original, failing, Metabase DB and restored backup to new PG 14 instance. Verified it had the same "change set checksum was x is now y" error.
5. Did some testing described below on new machine w/ PG 14.

Some new details, FYI:

• The good news is that having the mirrored DB gave me the courage (audacity? foolhardiness?) to twiddle the md5sum values in the 'databasechangelog' table. By doing this, I WAS able to get MB running again and upgraded to 49.13. So, I am back up in production, by just updating the MB "db host" field in all my connections to point to the new DB server. Crisis over (Just fyi, can't upgrade PG on original host for unrelated reasons.)

• I now suspect that the original upgrade failure could have been a memory problem. New host has 2x the RAM.

• The problem was originally a checksum failure, but when twiddling with downgrading, etc., the error sometimes switched to the PG "IF NOT EXISTS" unsupported error, so it's hard to tell when/if I fixed it on the original host...

Should I mark this issue Solved? I'm running again, but the only thing to learn is that editing md5 checksums works sometimes - but I hesitate to recommend that solution to others; I have no idea what the side effects might be...

I would still like to know what the checksum represents. Is it a checksum of something in the DB? Or of the SQL in the migration code? Etc?

Again, thank you very much TonyC!

[TonyC]

Thank you so much for the detailed overview!!

It's fine if you changed the md5sum values in the databasechangelog it shouldn't impact the process itself. It could be a case where liquibase checksums changed and the hash was not beign generated with the new checksum. In your case you should be fine though Some more reference around that can be found here:

add validated migration checksums to v45 v46 v47 and v48

[tengammaps]

Excellent! Thank you so much @TonyC ! I'll peruse the backgrounder you pointed me to, then I'm off to explore new features in v49...