Hey team- looking for clarity in the Upgrading metabase running in a cluster docs. We run our instance in k8s. Is it safe to keep the previous pod running while the new major version pod initializes and runs migrations? I’m assuming no to be on the safe side but looking for clarification in case we need to optimize for HA.
No, it is not safe to keep a previous pod running while running an upgrade to a new version. All pods must be running the same version of Metabase. The instructions are clear that there can only be 1 pod running during upgrades and that is the one running the upgrade. Any other pods running will access an inconsistent app database and could either crash or damage the app database.
You will have to schedule downtime to perform upgrades. Rolling upgrades are not safe as database migration requirements are unpredictable. Most minor version upgrades should be very fast, maybe an extra 30-60 seconds depending on how large your install is and what migrations are required. Major upgrades should be carefully planned and tested.
Backup your app database before running any upgrade!
Be sure to disable all health checks on the pod running the upgrade to keep k8s from crashing the pod during the migration process and damaging the app database–the pod won’t respond to health checks while the migration is in process.
I figured, thank you for the clarification and detail! “The application won’t behave correctly” from the docs seems to be an understatement, then, hence the post. Much appreciated!