Getting White Screen and Console Errors after Replacing JAR file during upgrade

My upgrade process:

  • Download the latest file using wget http://downloads.metabase.com/v0.41.4/metabase.jar
  • Stop the metabase service service metabase stop
  • Replace the old metabase.jar with the newly downloaded one, via cp
  • Restart metabase service metabase start
  • Restart nginx for good measure service nginx reload

Now if I look at my URL, I get a white-screen. It looks like all the front-end assets are cached, and there doesn't appear to be a way for me to get the new ones downloaded:

I get 404s on the following:

  • <base_url>/app/dist/vendor.bundle.css?545197817a30d4711623c9c68808deb4
  • <base_url>/app/dist/styles.bundle.css?11a344942a4866367832114531943739
  • <base_url>/app/dist/app-main.bundle.css?9bfad6329942536c7f51f47d163631c2
  • <base_url>/app/dist/vendor.bundle.js?8f423c47488b32b1f66b
  • <base_url>/app/dist/styles.bundle.js?8f423c47488b32b1f66b
  • <base_url>/app/dist/app-main.bundle.js?8f423c47488b32b1f66b
  • <base_url>/app/dist/app-main.bundle.js?8f423c47488b32b1f66b

Can someone help?

Hi @danhanly
Those file checksums (...?<checksum>) are not from Metabase.
You must be using caching or something like that, disable it:
https://www.metabase.com/docs/latest/troubleshooting-guide/proxies.html
Which version of Metabase did you upgrade from?

0.37.1 I believe.

I'll double-check my server configuration for caching systems.

Thanks

@danhanly Sounds like your Nginx might have caused files to be cached. Do a hard browser refresh, so you clear any cached files.
Otherwise use your browser developer Network-tab to see the headers, which should tell you why it's still referencing old files.

I looked into the 404 a little more, and I'm getting this console message:

The resource from “<base_url>/app/dist/app-main.bundle.css?9bfad6329942536c7f51f47d163631c2” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff)

Suggests that it's a normal 404, right? File can't be found with that checksum, so the MIME type is therefore null?

Just want to check there's no other issues going on here.

Apologise for the dumb questions, I inherited this system from a previous employee, so I'm not sure how it's set up.

@danhanly There's no /app/dist/app-main.bundle.css?9bfad6329942536c7f51f47d163631c2 reference in the Metabase page anymore. Guess you have cache somewhere.
The 0.41.4 reference is /app/dist/app-main.css?0bb19f8d32c8f260f8f3

So there's definitely some server-side caching going on then, since it's still requesting the old files.

I'll attempt the upgrade again and see what happens - and then I'll dig a little deeper for caching (I've checked the usual places and there's no cache software running on the server, and I've checked the obvious third-party caching configurations but found nothing yet).

@danhanly Nginx can do caching too - if configured to do that, so perhaps it's just bad Nginx configs?
It's hard to tell, but it's definitely something in your setup. And it can be tricky to figure that out, when you just took over someone else's setup.

1 Like

I think I'm getting somewhere now.

It appears that Jetty unable to initialise:

2021-12-13 12:03:08,135 INFO metabase.server :: Launching Embedded Jetty Webserver with config:
 {:port 3000}

2021-12-13 12:03:08,252 ERROR metabase.core :: Metabase Initialization FAILED
java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:3000

Finally have a new angle to explore :slight_smile:

@danhanly Likely because you have something else running on port 3000.

Looks to be an issue with Jetty.

After killing the process running on 3000, and restarting the metabase server, I'm getting this:

2021-12-13 12:09:01,110 INFO metabase.core :: Metabase Shutting Down ...
2021-12-13 12:09:01,111 INFO metabase.server :: Shutting Down Embedded Jetty Webserver
2021-12-13 12:09:16,120 WARN strategy.EatWhatYouKill ::
java.lang.NoClassDefFoundError: org/eclipse/jetty/util/IO
	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:599)
	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:509)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.util.IO
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 10 more

Updated a bunch of core packages, including the default-jre on my server and it seems to be all working now.

I want to thank you for your help @flamber this is one of the few tech communities that I really feel supports its users. I'm impressed with your assistance. Thanks a bunch!

1 Like

@danhanly Sounds like you took over an outdated setup. Make sure you're using Java 11: https://www.metabase.com/docs/latest/operations-guide/java-versions.html