The documentation says to update: kill Metabse, replace the jar, start Metabase. This is simple enough and easy to automate but I can't find a clear answer on best practice for updating multiple Metabase instances behind a load balancer. Can instances be brought down and updated one at a time for zero downtime? Would that be problematic as updated instances might change the application database breaking instances that haven't been updated yet, or at worst corrupt the app database?
Currently my plan is to bring all the instances down and update them at the same time as it's the "safe" (but not ideal) option. These instances are self-hosted JAR releases of Metabase and the app database is a PostgreSQL DB self-hosted on another machine, but I imagine this question extends to the other release types and environments as well.
Hi @aneub
Correct, it could on a tiny chance cause a corruption, but very unlikely. More likely, while a migration update of the first instance might cause the other old instances to error if they are trying to do something to the database that has been changed/removed, but by the time that is happening, then you would already be upgrading those instances.
But you always have a backup before any upgrade, so in case it goes haywire, then you take down everything, revert to the backup, startup one instance and then the rest.
Thanks Flamber,
I think I will go the safe route here and take them all down for upgrades, it's only couple minutes of downtime that can be done when nobody is using Metabase, a lot less time than if something goes wrong. You mentioned when restoring an app database you would start up one instance before the rest, could upgrades on instances in parallel cause changes to the app database to be applied multiple times? When upgrading, should there be a "main instance" that updates first to ensure that doesn't happen?
@aneub When one instance has executed the migrations, then other instances will not run any migrations.
Only one instance can lock the database (see the table databasechangeloglock) for upgrade at the time, so the other instance will fails to start and likely be automatically restarted by your service and by the time they have started again, then the first instance would likely already have completed the migration, so the other instances would just skip that.
I would still recommend doing everything with delays instead of everything at once in parallel. Nothing good comes out of trying to start e.g. 5 large Java instances at the same time instead of spreading them over 5 minutes.