Metabase upgrade failed : v0.34.x to v35.x

We recently tried to upgrade Metabase to latest version and it failed. It worked on our dev and staging environments but failed for production. The only difference from the upgrade perspective between these was that dev and stage upgraded from v0.34.3 to v0.35.3 but production was upgraded from v0.34.1 to v0.35.3 directly.
When I first tried to perform upgrade on production (from v0.34.1 to v0.35.3), it failed with database timeout error :

05-07 15:39:14 \u001b[1mINFO driver.impl\u001b[0m :: \u001b[33mInitializing driver :postgres...\u001b[0m\n
05-07 15:39:18 \u001b[1mERROR driver.util\u001b[0m :: Database connection error\n
java.util.concurrent.TimeoutException: Timed out after 5.0 s\n\tat metabase.util$deref_with_timeout.invokeStatic(util.clj:301)\n\tat metabase.util$deref_with_timeout.invoke(util.clj:293)\n\tat metabase.util$do_with_timeout.invokeStatic(util.clj:307)\n\tat metabase.util$do_with_timeout.invoke(util.clj:304)\n\tat metabase.driver.util$can_connect_with_details_QMARK_.invokeStatic(util.clj:29)\n\tat metabase.driver.util$can_connect_with_details_QMARK_.doInvoke(util.clj:18)\n\tat clojure.lang.RestFn.invoke(RestFn.java:442)\n\tat clojure.lang.Var.invoke(Var.java:393)\n\tat metabase.db$fn__21294$verify_db_connection__21303$fn__21306$fn__21307.invoke(db.clj:309)\n\tat metabase.db$fn__21294$verify_db_connection__21303$fn__21306.invoke(db.clj:308)\n\tat metabase.db$fn__21294$verify_db_connection__21303.invoke(db.clj:300)\n\tat metabase.db$fn__21294$verify_db_connection__21303$fn__21304.invoke(db.clj:303)\n\tat metabase.db$fn__21294$verify_db_connection__21303.invoke(db.clj:300)\n\tat metabase.db$setup_db_BANG__STAR_$fn__21342.invoke(db.clj:345)\n\tat metabase.util$do_with_us_locale.invokeStatic(util.clj:649)\n\tat metabase.util$do_with_us_locale.invoke(util.clj:635)\n\tat metabase.db$setup_db_BANG__STAR_.invokeStatic(db.clj:344)\n\tat metabase.db$setup_db_BANG__STAR_.invoke(db.clj:339)\n\tat metabase.db$setup_db_from_env_BANG__STAR_.invokeStatic(db.clj:354)\n\tat metabase.db$setup_db_from_env_BANG__STAR_.invoke(db.clj:351)\n\tat metabase.db$setup_db_BANG_$fn__21353.invoke(db.clj:368)\n\tat metabase.db$setup_db_BANG_.invokeStatic(db.clj:366)\n\tat metabase.db$setup_db_BANG_.invoke(db.clj:361)\n\tat metabase.core$init_BANG_.invokeStatic(core.clj:77)\n\tat metabase.core$init_BANG_.invoke(core.clj:56)\n\tat metabase.core$start_normally.invokeStatic(core.clj:123)\n\tat metabase.core$start_normally.invoke(core.clj:117)\n\tat metabase.core$_main.invokeStatic(core.clj:156)\n\tat metabase.core$_main.doInvoke(core.clj:150)\n\tat clojure.lang.RestFn.invoke(RestFn.java:397)\n\tat clojure.lang.AFn.applyToHelper(AFn.java:152)\n\tat clojure.lang.RestFn.applyTo(RestFn.java:132)\n\tat metabase.core.main(Unknown Source)\n",
05-07 15:39:18 \u001b[1mERROR metabase.core\u001b[0m :: Metabase Initialization FAILED\n"

When it failed, I rolled back immediately to v0.34.1 and it was working fine. Then I tried again later to upgrade it to v0.34.3 (from v0.34.1) and it was successful. Later I tried for pushing v0.35.3 (from v0.34.3) and this time there was no error during upgrade. But, when I opened the metabase home page/login page, it was showing blank. I checked in private mode as well as with another browser but result was same. Then, I checked browser console and it was showing the below error :

TypeError: items is nullapp-main.bundle.js:5:2589175
    from self-hosted:505
    options https://<hostname>app/dist/app-main.bundle.js?f740cf3b1da47a41d2da:5
    <anonymous> https://<hostname>/app/dist/app-main.bundle.js?f740cf3b1da47a41d2da:5
    i https://<hostname>/app/dist/vendor.bundle.js?f740cf3b1da47a41d2da:1
    <anonymous> https://<hostname>/app/dist/app-main.bundle.js?f740cf3b1da47a41d2da:5
    i https://<hostname>/app/dist/vendor.bundle.js?f740cf3b1da47a41d2da:1
    <anonymous> https://<hostname>/app/dist/app-main.bundle.js?f740cf3b1da47a41d2da:5
    i https://<hostname>/app/dist/vendor.bundle.js?f740cf3b1da47a41d2da:1
    <anonymous> https://<hostname>/app/dist/app-main.bundle.js?f740cf3b1da47a41d2da:5
    i https://<hostname>/app/dist/vendor.bundle.js?f740cf3b1da47a41d2da:1
    <anonymous> https://<hostname>/app/dist/app-main.bundle.js?f740cf3b1da47a41d2da:5
    i https://<hostname>/app/dist/vendor.bundle.js?f740cf3b1da47a41d2da:1
    webpackJsonp https://<hostname>/app/dist/vendor.bundle.js?f740cf3b1da47a41d2da:1
    <anonymous> https://<hostname>/app/dist/app-main.bundle.js?f740cf3b1da47a41d2da:5

Now, I’ve reverted it back to v0.34.3. I would like to know how I can fix this issue.

Our metabase details :

{
  "browser-info": {
    "language": "en-US",
    "platform": "MacIntel",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:76.0) Gecko/20100101 Firefox/76.0",
    "vendor": ""
  },
  "system-info": {
    "java.runtime.name": "OpenJDK Runtime Environment",
    "java.runtime.version": "11.0.5+10",
    "java.vendor": "AdoptOpenJDK",
    "java.vendor.url": "https://adoptopenjdk.net/",
    "java.version": "11.0.5",
    "java.vm.name": "OpenJDK 64-Bit Server VM",
    "java.vm.version": "11.0.5+10",
    "os.name": "Linux",
    "os.version": "4.14.138+",
    "user.language": "en",
    "user.timezone": "GMT"
  },
  "metabase-info": {
    "databases": [
      "mongo",
      "postgres",
      "mysql"
    ],
    "hosting-env": "unknown",
    "application-database": "postgres",
    "application-database-details": {
      "database": {
        "name": "PostgreSQL",
        "version": "11.6"
      },
      "jdbc-driver": {
        "name": "PostgreSQL JDBC Driver",
        "version": "42.2.8"
      }
    },
    "run-mode": "prod",
    "version": {
      "date": "2020-02-25",
      "tag": "v0.34.3",
      "branch": "release-0.34.x",
      "hash": "1a83edb"
    },
    "settings": {
      "report-timezone": "Asia/Calcutta"
    }
  }
}

Hi @gauthamsuresh09

Latest release is 0.35.4

The timeout is likely caused by network problems between Metabase and Postgres. There’s no difference in the timeout between any of those versions. You can adjust the timeout if you know there might be more than 5 seconds initialization:
https://github.com/metabase/metabase/blob/master/docs/operations-guide/environment-variables.md#mb_db_connection_timeout_ms

Not sure why everything was blank after upgrade, but I’m sure there’s more errors than TypeError: items is null
My first guess is that you’re using a proxy and it’s doing something it shouldn’t:
https://www.metabase.com/docs/latest/troubleshooting-guide/proxies.html

Hi @flamber
I checked with 0.35.4, the issue is still there. The timeout error didn’t come up when I tried to upgrade second time, thanks for sharing the settings anyway. I’ll increase timeout so that this doesn’t happen again.
I’ll spend some time to debug it and see if the issue is with proxy, will update here with results.

Hi @flamber,
An update regarding the issue. Looks like it’s an issue related to proxy/cdn . I tried upgrade again and this time I waited for 30 minutes. At first, it was giving same error but later started working. Our metabase deployment is running in kubernetes cluster (GKE). Thanks a lot for the help

1 Like