Hey Team,
I'm getting 502 Bad Gateway error, Metabase is hosted on AWS EC2 instance and it was working fine, but since a day the application is down, the log isn't conclusive too. The t1.micro's utilization hasn't spiked up to.
Version - 0.33.2
Thanks,
Bharath.
Hi, You should check the EC2 instance status as well as your RDS database
Hi @bharathb
So that error comes from Nginx, not Metabase, so either Metabase is not running or the Nginx reverse-proxy configuration is incorrect.
Hi.
This error can be due to a failure in the initialization of the instances associated with the metabase environment, usually it is solved by restarting the metabase environment server.
@jireascos - Tried restarting the instance, post which Metabase was accessible showing setting up tables etc and again went back to 504 Bad Gateway.
Hi @flamber - any troubleshooting steps?
@bharathb
I guess the service/container crashes during startup?
What do you see in the log, when you try to start Metabase? Post the log from start to fail, if you don’t know what to look for.
Which version of Metabase? Which database are you using for internal metadata?
@bharathb
That’s your Nginx log - I’m looking for the Metabase log.
How are you running Metabase?
Does Metabase startup or does it crash instantly?
If it starts, can you copy the Diagnostic Info from Admin > Troubleshooting
@flamber- Its crashing instantly
@bharathb
How are you running Metabase - docker, JAR-file, …?
Without any crash traces or the log, it’s impossible to know what’s going on.
@flamber- Metabase is running using Jar File, The Instance was restarted. Post which it said loading and again crashed, PFA the screen
Metabase Log when starting
Sep 30 08:22:19 ip-172-31-4-153 systemd[1]: Stopped Metabase server.
Sep 30 08:22:19 ip-172-31-4-153 systemd[1]: Started Metabase server.
Sep 30 08:23:01 ip-172-31-4-153 metabase[6304]: 09-30 08:23:01 DEBUG plugins.classloader :: Using NEWLY CREATED classloader as shared context classloader: clojure.lang.DynamicClassLoader@377c68c6
Sep 30 08:23:01 ip-172-31-4-153 metabase[6304]: 09-30 08:23:01 DEBUG plugins.classloader :: Setting current thread context classloader to shared classloader clojure.lang.DynamicClassLoader@377c68c6...
Sep 30 08:23:09 ip-172-31-4-153 metabase[6304]: 09-30 08:23:09 INFO metabase.util :: Loading Metabase...
Sep 30 08:23:09 ip-172-31-4-153 metabase[6304]: 09-30 08:23:09 INFO metabase.util :: Maximum memory available to JVM: 239.8 MB
Output Command for - java -jar /var/metabase/metabase.jar migrate release-locks
Logs -
09-30 08:51:57 DEBUG plugins.classloader :: Using NEWLY CREATED classloader as shared context classloader: clojure.lang.DynamicClassLoader@377c68c6
09-30 08:51:57 DEBUG plugins.classloader :: Setting current thread context classloader to shared classloader clojure.lang.DynamicClassLoader@377c68c6...
09-30 08:52:10 INFO metabase.util :: Loading Metabase...
09-30 08:52:10 INFO metabase.util :: Maximum memory available to JVM: 239.8 MB
09-30 08:55:12 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance.
For more information, see Redirecting…
09-30 08:57:58 INFO metabase.driver :: Registered abstract driver :sql
09-30 09:00:12 WARN metabase.db :: WARNING: Using Metabase with an H2 application database is not recommended for production deployments. For production deployments, we highly recommend using Postgres, MySQL, or MariaDB instead. If you decide to continue to use H2, please be sure to back up the database file regularly. For more information, seehttps://metabase.com/docs/latest/operations-guide/migrating-from-h2.html
org.h2.jdbc.JdbcSQLException: IO Exception: null [90028-197]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
at org.h2.message.DbException.get(DbException.java:168)
at org.h2.mvstore.db.MVTableEngine$Store.convertIllegalStateException(MVTableEngine.java:192)
at org.h2.mvstore.db.MVTableEngine$Store.open(MVTableEngine.java:168)
at org.h2.mvstore.db.MVTableEngine.init(MVTableEngine.java:100)
at org.h2.engine.Database.getPageStore(Database.java:2538)
at org.h2.engine.Database.open(Database.java:709)
at org.h2.engine.Database.openDatabase(Database.java:286)
at org.h2.engine.Database.(Database.java:280)
at org.h2.engine.Engine.openSession(Engine.java:66)
at org.h2.engine.Engine.openSession(Engine.java:179)
at org.h2.engine.Engine.createSessionAndValidate(Engine.java:157)
at org.h2.engine.Engine.createSession(Engine.java:140)
at org.h2.engine.Engine.createSession(Engine.java:28)
at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:351)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:124)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:103)
at org.h2.Driver.connect(Driver.java:69)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at clojure.java.jdbc$get_driver_connection.invokeStatic(jdbc.clj:271)
at clojure.java.jdbc$get_driver_connection.invoke(jdbc.clj:250)
at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:411)
at clojure.java.jdbc$get_connection.invoke(jdbc.clj:274)
at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:833)
at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:769)
at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:782)
at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:769)
at metabase.db$migrate_BANG_.invokeStatic(db.clj:321)
at metabase.db$migrate_BANG_.invoke(db.clj:300)
at metabase.db$migrate_BANG_.invokeStatic(db.clj:318)
at metabase.db$migrate_BANG_.invoke(db.clj:300)
at metabase.cmd$migrate.invokeStatic(cmd.clj:29)
at metabase.cmd$migrate.invoke(cmd.clj:26)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:665)
at clojure.core$apply.invoke(core.clj:660)
at metabase.cmd$run_cmd$fn__64559.invoke(cmd.clj:128)
at metabase.cmd$run_cmd.invokeStatic(cmd.clj:128)
at metabase.cmd$run_cmd.invoke(cmd.clj:124)
at clojure.lang.Var.invoke(Var.java:388)
at metabase.core$run_cmd.invokeStatic(core.clj:133)
at metabase.core$run_cmd.invoke(core.clj:131)
at metabase.core$_main.invokeStatic(core.clj:142)
at metabase.core$_main.doInvoke(core.clj:138)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at metabase.core.main(Unknown Source)
Caused by: java.lang.IllegalStateException: Could not open file nio:/var/log/metabase.db.mv.db [1.4.197/1]
at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:870)
at org.h2.mvstore.FileStore.open(FileStore.java:179)
at org.h2.mvstore.MVStore.(MVStore.java:350)
at org.h2.mvstore.MVStore$Builder.open(MVStore.java:2934)
at org.h2.mvstore.db.MVTableEngine$Store.open(MVTableEngine.java:155)
... 44 more
Caused by: java.io.FileNotFoundException: /var/log/metabase.db.mv.db (Permission denied)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.(RandomAccessFile.java:243)
at java.io.RandomAccessFile.(RandomAccessFile.java:124)
at org.h2.store.fs.FileNio.(FilePathNio.java:43)
at org.h2.store.fs.FilePathNio.open(FilePathNio.java:23)
at org.h2.mvstore.FileStore.open(FileStore.java:155)
... 47 more
Command failed with exception: IO Exception: null [90028-197]
@bharathb
Okay, that’s very simple. You don’t have enough RAM to run Metabase. It looks like there’s only 239.8MB, which is way too little. You need to free up some RAM or get a larger server instance.
Metabase needs 512MB at a bare minimum, just to start - depending on your usage, I would recommend 1GB or 2GB.
Hi @flamber, thanks for responding.
I increased the memory to 2GB and it was running fine for couple of days. Now it's back to 502 Bad gateway error.
To experiment, I've increased the aws ec2 instance to t2.xlarge(16GB memory and 4CPU).
Regarding the metabase log, even after following all the steps in Redirecting…, log file is not updating. (if there's any other link to help setup log, please do share).
After running sudo java -jar /var/metabase/metabase.jar, everything went smoothly without any errors. But accessing the URL gives a fresh installation of metabase - showing setup page.
The metabase is setup is done as a service.
After running sudo java -jar /var/metabase/metabase.jar migrate release-locks, below is the output.
10-09 09:22:27 DEBUG plugins.classloader :: Using NEWLY CREATED classloader as shared context classloader: clojure.lang.DynamicClassLoader@377c68c6
10-09 09:22:27 DEBUG plugins.classloader :: Setting current thread context classloader to shared classloader clojure.lang.DynamicClassLoader@377c68c6...
10-09 09:22:28 INFO metabase.util :: Loading Metabase...
10-09 09:22:28 INFO metabase.util :: Maximum memory available to JVM: 3.5 GB
10-09 09:22:31 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance.
For more information, see Redirecting…
10-09 09:22:35 INFO metabase.driver :: Registered abstract driver :sql
10-09 09:22:42 WARN metabase.db :: WARNING: Using Metabase with an H2 application database is not recommended for production deployments. For production deployments, we highly recommend using Postgres, MySQL, or MariaDB instead. If you decide to continue to use H2, please be sure to back up the database file regularly. For more information, seehttps://metabase.com/docs/latest/operations-guide/migrating-from-h2.html
10-09 09:22:43 INFO metabase.db :: Setting up Liquibase...
10-09 09:22:43 INFO metabase.db :: Liquibase is ready.
You should not run Metabase with sudo.
If you’re running Metabase as a systemd service, then the log file is being handled by syslog by default, which might be redirected to a separate file, but that depends on your configuration.
The reason why it’s working, when you just start a new Metabase instance with a clean database is that it’s not doing anything - there’s no user activity, sync processes, pulses, or queries being executed. That instance will only use a few hundred MB of RAM.
I’m not sure why you’re migrate release-locks - ?
@flamber Thanks for replying. since I was not able to get a hold of log file, I was unclear about the root cause why metabase not starting properly. so migrate release-locks used to release if any locks present.
But finally I was able to get a hold of the log file and found the following database error.
Oct 10 05:44:42 ip-172-31-4-153 metabase[31430]: 10-10 05:44:42 #033[1mERROR driver.util#033[0m :: Database connection error
Oct 10 05:44:42 ip-172-31-4-153 metabase[31430]: org.postgresql.util.PSQLException: The connection attempt failed.
There seems to be some problem with the database configuration in the metabase.env file.
After correcting DB connection details, metabase is starting properly. But with no data - a fresh installation.
Is there's any way to get back the old data? If so it'd perfect, otherwise I think I'll go ahead and redo the dashboard.
Appreciate your help 
So all your metadata is in Postgres?
Are you using systemd for the service? Then you need to setup the environment variables, so Metabase reads the correct metadata. As noted in the docs:
https://www.metabase.com/docs/latest/operations-guide/running-metabase-on-debian.html#the-metabase-config-file
If you’re trying to start Metabase directly with java -jar ..., then you either need to define the environment variables first or add them as parameters.
Metadata is in Postgres (remote DB).
Using systemd to start and stop the service (following this tutorial https://github.com/metabase/metabase/issues/5785).
Previously the DB host was pointing to a different host which is no longer available. May be that’s why connection problem occurred. Also the metadata is in old DB host I presume?
@bharathb
That’s a fairly old guide - the one in the documentation is newer and more complete.
If you have changed the host of the metadata database, and now Metabase doesn’t have any content, then yes, the metadata was stored in the previous database.
