Not able to update to 0.39 (from 0.37.8 )

Hi,
We tried to upgrade to 0.39.X a few times and all the time it freezes here:

We upgraded to 0.38.5, all was ok. And all further upgrades do not work no matter what version we try. 0.39.0, 0.39.0.1, 0.39.2 ...

Any advice will be greatly appreciated.

Thank you
Jaz

{
  "browser-info": {
    "language": "en-US",
    "platform": "Win32",
    "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36",
    "vendor": "Google Inc."
  },
  "system-info": {
    "file.encoding": "UTF-8",
    "java.runtime.name": "OpenJDK Runtime Environment",
    "java.runtime.version": "11.0.11+9",
    "java.vendor": "AdoptOpenJDK",
    "java.vendor.url": "https://adoptopenjdk.net/",
    "java.version": "11.0.11",
    "java.vm.name": "OpenJDK 64-Bit Server VM",
    "java.vm.version": "11.0.11+9",
    "os.name": "Linux",
    "os.version": "4.14.138-89.102.amzn1.x86_64",
    "user.language": "en",
    "user.timezone": "GMT"
  },
  "metabase-info": {
    "databases": [
      "googleanalytics",
      "postgres",
      "redshift",
      "mysql"
    ],
    "hosting-env": "unknown",
    "application-database": "h2",
    "application-database-details": {
      "database": {
        "name": "H2",
        "version": "1.4.197 (2018-03-18)"
      },
      "jdbc-driver": {
        "name": "H2 JDBC Driver",
        "version": "1.4.197 (2018-03-18)"
      }
    },
    "run-mode": "prod",
    "version": {
      "date": "2021-04-27",
      "tag": "v0.38.5",
      "branch": "release-x.38.x",
      "hash": "84eb12f"
    },
    "settings": {
      "report-timezone": null
    }
  }
}

Hi! Luis here. My 2 cents:

  1. get away from H2, it's not a database for production workloads, it's just for testing Metabase
  2. once you get away from H2, try giving Metabase a bit more RAM and try the upgrade again

good luck!

Thanks @Luiggi

Actually it is inconvenient for us to move away from H2 at this time. It's also unexpected (not in release notes) that we would need to leave H2 and add more ram to upgrade.

If anyone has upgraded to 0.39 without adding ram or ditching H2 I would really appreciate hearing it, and any tips you might have to get us past this screen freezing.

Alternatively if someone knows for a fact that upgrading is problematic while on H2 and same ram, then I would also like to hear it.

Thanks!

@jazz78
I'm sure that I have pointed out several times in previous topics, that H2 should not be used in production.
https://www.metabase.com/docs/latest/operations-guide/migrating-from-h2.html
Every single documentation page says that H2 is just an easy way to get started with Metabase, but it's recommended to use Postgres or MariaDB/MySQL in production.

It's not freezing - it's doing a migration, which can take a long time. On one of my systems with 30k fields, it took 4 minutes to complete that migration. So since I have no idea of the size of your H2 or the amount of fields, then it's difficult to say how long it takes.
If you close the process, while it's working, then the H2 database is likely corrupted and you'll need to restore from backup.
And slow upgrade is noted in the release notes, since I put it there: https://github.com/metabase/metabase/releases/tag/v0.39.0.1

Since you're mentioning limited RAM, then it would be helpful to know how little you're running with, and how much Metabase can use, which is noted in the first line during startup.

Hi @flamber thanks so much - we have 32 gb of ram getting some more stats. one moment.

@flamber update: We have 32gb ram and first line during start is 7.7gb

We since tried upgrading a test installation, and it worked but it took an hour. Does this feel right ?

@jazz78
If the instance is only running Metabase, then you're wasting RAM, since Metabase will maximum occupy ~8GB.

Since I have no idea how big your H2 file is or other CPU and disk IO performance, then yes, perhaps.
Or perhaps it's generating a lot of errors, which are written to the metabase.db.trace.db - and if that file is more than a few MB, then that usually indicates problems with your H2 (possible corruption). A large trace file can slow down everything. Shutdown Metabase, delete the trace file, start again.

Please read this:
https://www.metabase.com/learn/data-diet/analytics/metabase-at-scale.html

I'm not going to have a discussion about if H2 is viable to use in production - it's not, final. Search the forum to find people who were hit by corruption. H2 is also slower than external application databases, when it gets larger than 10MB (the db, not the trace).

Got it. We need to work on getting off H2. I know you have mentioned it to me before, we just have had a lot on our plate, and things have been working great (apart from this upgrade).

Getting file sizes etc in the meantime some info here (blurred out some sensitive info):

@flamber some more info:

Our test installation 'upgrade' logs:

 2021-05-26 09:22:44,296 INFO metabase.util :: Maximum memory available to JVM: 7.7 GB
 ...
 2021-05-26 09:23:00,996 INFO db.liquibase :: Migration lock is cleared. Running migrations...
 2021-05-26 10:18:15,517 INFO db.setup :: Database Migrations Current ...  ✅
 2021-05-26 10:18:15,535 INFO db.data-migrations :: Running all necessary data migrations, this may take a minute.
 2021-05-26 10:18:15,542 INFO db.data-migrations :: Running data migration 'migrate-click-through'...
 2021-05-26 10:18:15,581 INFO db.data-migrations :: Finished running data migrations.
 Database setup took 55.3 mins

metabase.db.mv.db - 51MB
8 CPU cores on a server.
no trace DB at all

And at the same time upgrade from 0.37.2 to 0.38.X takes couple of minutes.
Then any upgrade to 0.39 takes more than 1 hour.

Is this expected? Or hard to say?

ps. our DB is 51mb

@jazz78 So you're running multiple instances on the same host.

The migration still seems slow. I have an old computer and a 20MB H2 takes a handful of minutes. That's with 20k fields. So if you have a lot more fields, then it's going to be even slower, then times that by 2.5 (since mine is 20MB, yours is 50MB) and then it sounds kinda reasonable.
That migration process on Postgres takes a lot less time, since it has a better way of dropping columns for tables. MariaDB/MySQL should be faster too.

It's important to understand the different between migrations being done - normally only extra columns are added or other small changes, which also doesn't impact downgrade (which is officially unsupported), but 0.39.0 did a big migration by changing the table schema of one of the big tables.
Doing an upgrade from let's say 0.32 to 0.39 will run every migration between those versions, so you'll always have faster migrations if the upgrade is just to the next version.

We're trying to avoid making upgrades slow, but it's sometimes need to refactor the schema design.

@flamber thanks very much, very helpful. I don't know how many fields we have, I'm not sure a way to tell, but if i had to guess I would say like less than 100 (unless I'm thinking about fields the wrong way).

@jazz78 It's located in the application database table metabase_field.
"fields" are each column from each table from each database that Metabase has synced at some point. If you have 100 fields, then you have a really tiny data source, which I don't think. Then the H2 size seems way too big - unless you have created 100k questions (report_card) or have huge amount of user activity, which causes query_execution to explode. Or perhaps you have enabled way too much cache (Admin > Settings > Caching), which H2 definitely isn't the fastest to handle either.

@flamber thanks very much. I don't know where to find that table, or how, but yes i think my guess of 100 is probably wrong:

  1. caching is disabled
  2. less than 100k questions (probably like 1000 very max)
  3. i think the user activity is low, there are only 4 users on one and about 100 light users on the other biggest one, the rest have low users, low usage.

Given your most recent 'gentle' :slightly_smiling_face: encouragement, we are embarking on a transition to mysql. I will let you know if we continue to have upgrade issues after that, I am guessing not, given your feedback so far. Although, it's a bigger exercise overall so we probably won't have a clean understanding. But with the other benefits, it seems now is a good time to try, and maybe future 'upgrades' will go faster, and in the meantime we will enjoy the other benefits, and/or absence of corruption.