Struggling to run Metabase as a service on Ubuntu

seang098 · December 28, 2020, 4:53pm

Hi all,

I’d be very grateful for any help in getting Metabase up and running as a service on Ubuntu with Nginx.

I’m following the guide at https://www.metabase.com/docs/latest/operations-guide/running-metabase-on-debian.html

I have NGINX, Java installed, and have a MySQL database and user ready. Domain name set up with letsencrypt.

First question, in the Metabase config file there are options for Jetty:

MB_JETTY_HOST=<0.0.0.0> MB_JETTY_PORT=<12345>

Where does Jetty fit in to this setup?

When I leave these options commented out, and try to start the service, I am getting the following printed to the syslog.

Dec 28 16:27:20 lemp-frontend systemd[1]: Started Metabase server.
Dec 28 16:27:24 lemp-frontend metabase[11717]: WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
Dec 28 16:27:28 lemp-frontend metabase[11717]: 2020-12-28 16:27:28,251 INFO metabase.util :: Maximum memory available to JVM: 237.8 MB
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' resumed (module 'builtin:omfile') [v8.2001.0 try https://www.rsyslog.com/e/2359 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2001.0 try https://www.rsyslog.com/e/2007 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' resumed (module 'builtin:omfile') [v8.2001.0 try https://www.rsyslog.com/e/2359 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2001.0 try https://www.rsyslog.com/e/2007 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' resumed (module 'builtin:omfile') [v8.2001.0 try https://www.rsyslog.com/e/2359 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2001.0 try https://www.rsyslog.com/e/2007 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' resumed (module 'builtin:omfile') [v8.2001.0 try https://www.rsyslog.com/e/2359 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2001.0 try https://www.rsyslog.com/e/2007 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' resumed (module 'builtin:omfile') [v8.2001.0 try https://www.rsyslog.com/e/2359 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2001.0 try https://www.rsyslog.com/e/2007 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' resumed (module 'builtin:omfile') [v8.2001.0 try https://www.rsyslog.com/e/2359 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2001.0 try https://www.rsyslog.com/e/2007 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' resumed (module 'builtin:omfile') [v8.2001.0 try https://www.rsyslog.com/e/2359 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2001.0 try https://www.rsyslog.com/e/2007 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' resumed (module 'builtin:omfile') [v8.2001.0 try https://www.rsyslog.com/e/2359 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2001.0 try https://www.rsyslog.com/e/2007 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' resumed (module 'builtin:omfile') [v8.2001.0 try https://www.rsyslog.com/e/2359 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2001.0 try https://www.rsyslog.com/e/2007 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' resumed (module 'builtin:omfile') [v8.2001.0 try https://www.rsyslog.com/e/2359 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2001.0 try https://www.rsyslog.com/e/2007 ]
Dec 28 16:27:28 lemp-frontend rsyslogd: action 'action-8-builtin:omfile' suspended (module 'builtin:omfile'), next retry is Mon Dec 28 16:27:58 2020, retry nbr 0. There should be messages before this one giving the reason for suspension. [v8.2001.0 try https://www.rsyslog.com/e/2007 ]
Dec 28 16:27:43 lemp-frontend systemd-resolved[601]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Dec 28 16:27:43 lemp-frontend systemd-resolved[601]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Dec 28 16:27:43 lemp-frontend metabase[11717]: Exception in thread "main" java.lang.ExceptionInInitializerError
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at java.base/java.lang.Class.forName0(Native Method)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at java.base/java.lang.Class.forName(Class.java:398)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.classForName(RT.java:2211)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.classForName(RT.java:2220)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.loadClassForName(RT.java:2239)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.load(RT.java:449)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.load(RT.java:424)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load$fn__6839.invoke(core.clj:6126)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load.invokeStatic(core.clj:6125)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load.doInvoke(core.clj:6109)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RestFn.invoke(RestFn.java:408)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_one.invokeStatic(core.clj:5908)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_one.invoke(core.clj:5903)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_lib$fn__6780.invoke(core.clj:5948)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_lib.invokeStatic(core.clj:5947)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_lib.doInvoke(core.clj:5928)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RestFn.applyTo(RestFn.java:142)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$apply.invokeStatic(core.clj:667)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_libs.invokeStatic(core.clj:5989)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_libs.doInvoke(core.clj:5969)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RestFn.applyTo(RestFn.java:137)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$apply.invokeStatic(core.clj:667)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$require.invokeStatic(core.clj:6007)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$require.doInvoke(core.clj:6007)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RestFn.invoke(RestFn.java:512)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at metabase.util.schema$loading__6721__auto____21020.invoke(schema.clj:1)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at metabase.util.schema__init.load(Unknown Source)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at metabase.util.schema__init.<clinit>(Unknown Source)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at java.base/java.lang.Class.forName0(Native Method)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at java.base/java.lang.Class.forName(Class.java:398)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.classForName(RT.java:2211)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.classForName(RT.java:2220)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.loadClassForName(RT.java:2239)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.load(RT.java:449)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.load(RT.java:424)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load$fn__6839.invoke(core.clj:6126)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load.invokeStatic(core.clj:6125)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load.doInvoke(core.clj:6109)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RestFn.invoke(RestFn.java:408)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_one.invokeStatic(core.clj:5908)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_one.invoke(core.clj:5903)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_lib$fn__6780.invoke(core.clj:5948)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_lib.invokeStatic(core.clj:5947)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_lib.doInvoke(core.clj:5928)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RestFn.applyTo(RestFn.java:142)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$apply.invokeStatic(core.clj:667)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_libs.invokeStatic(core.clj:5989)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_libs.doInvoke(core.clj:5969)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RestFn.applyTo(RestFn.java:137)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$apply.invokeStatic(core.clj:667)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$require.invokeStatic(core.clj:6007)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$require.doInvoke(core.clj:6007)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RestFn.invoke(RestFn.java:805)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at metabase.db$loading__6721__auto____20281.invoke(db.clj:1)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at metabase.db__init.load(Unknown Source)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at metabase.db__init.<clinit>(Unknown Source)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at java.base/java.lang.Class.forName0(Native Method)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at java.base/java.lang.Class.forName(Class.java:398)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.classForName(RT.java:2211)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.classForName(RT.java:2220)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.loadClassForName(RT.java:2239)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.load(RT.java:449)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.load(RT.java:424)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load$fn__6839.invoke(core.clj:6126)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load.invokeStatic(core.clj:6125)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load.doInvoke(core.clj:6109)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RestFn.invoke(RestFn.java:408)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_one.invokeStatic(core.clj:5908)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_one.invoke(core.clj:5903)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_lib$fn__6780.invoke(core.clj:5948)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_lib.invokeStatic(core.clj:5947)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_lib.doInvoke(core.clj:5928)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RestFn.applyTo(RestFn.java:142)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$apply.invokeStatic(core.clj:667)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_libs.invokeStatic(core.clj:5989)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load_libs.doInvoke(core.clj:5969)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RestFn.applyTo(RestFn.java:137)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$apply.invokeStatic(core.clj:667)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$require.invokeStatic(core.clj:6007)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$require.doInvoke(core.clj:6007)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RestFn.invoke(RestFn.java:619)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at metabase.core$loading__6721__auto____74093.invoke(core.clj:1)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at metabase.core__init.load(Unknown Source)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at metabase.core__init.<clinit>(Unknown Source)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at java.base/java.lang.Class.forName0(Native Method)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at java.base/java.lang.Class.forName(Class.java:398)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.classForName(RT.java:2211)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.classForName(RT.java:2220)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.loadClassForName(RT.java:2239)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.load(RT.java:449)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RT.load(RT.java:424)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load$fn__6839.invoke(core.clj:6126)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load.invokeStatic(core.clj:6125)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.core$load.doInvoke(core.clj:6109)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.RestFn.invoke(RestFn.java:408)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.Var.invoke(Var.java:384)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at clojure.lang.Util.loadWithClass(Util.java:251)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at metabase.core.<clinit>(Unknown Source)
Dec 28 16:27:43 lemp-frontend metabase[11717]: Caused by: java.lang.NumberFormatException: For input string: "<10"
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at java.base/java.lang.Integer.parseInt(Integer.java:652)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at java.base/java.lang.Integer.parseInt(Integer.java:770)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at metabase.config$config_int.invokeStatic(config.clj:60)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at metabase.config$config_int.invoke(config.clj:60)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at metabase.util.password$fn__21072.invokeStatic(password.clj:56)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at metabase.util.password$fn__21072.invoke(password.clj:56)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at metabase.util.password__init.load(Unknown Source)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011at metabase.util.password__init.<clinit>(Unknown Source)
Dec 28 16:27:43 lemp-frontend metabase[11717]: #011... 98 more
Dec 28 16:27:43 lemp-frontend systemd[1]: metabase.service: Main process exited, code=exited, status=1/FAILURE
Dec 28 16:27:43 lemp-frontend systemd[1]: metabase.service: Failed with result 'exit-code'.
Dec 28 16:27:43 lemp-frontend systemd[1]: metabase.service: Scheduled restart job, restart counter is at 28.

It would be really helpful if someone could help me decode these errors so I can start to troubleshoot!

Many thanks,
Sean

flamber · December 28, 2020, 5:04pm

Hi @seang098
Jetty is the internal webserver of Metabase.
I think you are misunderstanding the environment variables.
You should not setup the Jetty to listen on all interfaces, but just a local interface that Nginx can communicate with. Nginx, which is your reverse-proxy, should be the only way to communicate with Metabase from the outside.
And you don’t use < or > - the correct usage is something like MB_JETTY_PORT=12345 if you want it to listen on port 12345.

seang098 · December 28, 2020, 5:14pm

Hi @flamber

Many thanks for your explanation.
Ok, so I have set MB_JETTY_HOST=127.0.0.1 and MB_JETTY_PORT=3000 to match my Nginx config (I suspect these are the defaults anyway perhaps?)

I am still getting the syslog errors above unfortunately.

Any hints very much appreciated:)

flamber · December 28, 2020, 5:24pm

@seang098 Almost the defaults, yes: https://www.metabase.com/docs/latest/operations-guide/environment-variables.html
Something is telling me that something else isn’t correct - there are very few information, so try starting Metabase manually without a service involved.
I’m not sure if you have bad environment variables, wrong version of Java or something else. But I’m noticing that you are starting with 237MB RAM available for Metabase - that’s way too little.

Check envvars:

printenv | grep -e "^MB_"

And OS and Java info:

sed -nE 's/^PRETTY_NAME="([^"]+)"$/\1/p' /etc/os-release
java -version

seang098 · December 28, 2020, 6:42pm

Sucess!

My environment variables weren’t being read for some reason - moving them to etc/environment seemed to help!

Out of curiously - how much memory do you find you need to run Metabase? My database is small will end up 100MB at most. I’ve allocated 1GB for the JVM at the moment but would be good to save some if possible.

Many, many thanks for you fast replies and all your work on Metabase, it’s great to have such a powerful open source option.

All the best,
Sean

flamber · December 28, 2020, 6:54pm

@seang098 Just make sure that you test after restarting, so you know that it’s using the correct envvars - I have seen way too many people setup Metabase, who lost everything after a restart, since they didn’t know where the data was actually stored.
Do not use H2 as the application database in production:
https://www.metabase.com/docs/latest/operations-guide/migrating-from-h2.html

You shouldn’t need to have the envvars in /etc/environment - that will expose them everywhere. They should only be exposed to the Metabase service, but that has nothing to do with Metabase, but how to run services in general on Ubuntu, so I would recommend looking up some good resources on that - you can find a lot of excellent Linux help on stackoverflow.com

While the size of your data sources makes a difference, it depends more on the usage of Metabase - how much activity, which features being used (pulses, exports, …), etc, which will impact the RAM consumption.
You are likely fine with that amount - perhaps you can go a bit lower, but not a maximum of 237MB.
If you crash Metabase because of OutOfMemory, then you know you need to give more