I’m getting 502 Bad Gateway error, Metabase is hosted on AWS EC2 instance and it was working fine, but since a day the application is down, the log isn’t conclusive too. The t1.micro’s utilization hasn’t spiked up to.
Hi.
This error can be due to a failure in the initialization of the instances associated with the metabase environment, usually it is solved by restarting the metabase environment server.
@bharathb
I guess the service/container crashes during startup?
What do you see in the log, when you try to start Metabase? Post the log from start to fail, if you don’t know what to look for.
Which version of Metabase? Which database are you using for internal metadata?
@bharathb
That’s your Nginx log - I’m looking for the Metabase log.
How are you running Metabase?
Does Metabase startup or does it crash instantly?
If it starts, can you copy the Diagnostic Info from Admin > Troubleshooting
Output Command for - java -jar /var/metabase/metabase.jar migrate release-locks
Logs -
09-30 08:51:57 DEBUG plugins.classloader :: Using NEWLY CREATED classloader as shared context classloader: clojure.lang.DynamicClassLoader@377c68c6
09-30 08:51:57 DEBUG plugins.classloader :: Setting current thread context classloader to shared classloader clojure.lang.DynamicClassLoader@377c68c6…
09-30 08:52:10 INFO metabase.util :: Loading Metabase…
09-30 08:52:10 INFO metabase.util :: Maximum memory available to JVM: 239.8 MB
09-30 08:55:12 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance.
For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
09-30 08:57:58 INFO metabase.driver :: Registered abstract driver :sql
09-30 09:00:12 WARN metabase.db :: WARNING: Using Metabase with an H2 application database is not recommended for production deployments. For production deployments, we highly recommend using Postgres, MySQL, or MariaDB instead. If you decide to continue to use H2, please be sure to back up the database file regularly. For more information, seehttps://metabase.com/docs/latest/operations-guide/migrating-from-h2.html
org.h2.jdbc.JdbcSQLException: IO Exception: null [90028-197]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
at org.h2.message.DbException.get(DbException.java:168)
at org.h2.mvstore.db.MVTableEngine$Store.convertIllegalStateException(MVTableEngine.java:192)
at org.h2.mvstore.db.MVTableEngine$Store.open(MVTableEngine.java:168)
at org.h2.mvstore.db.MVTableEngine.init(MVTableEngine.java:100)
at org.h2.engine.Database.getPageStore(Database.java:2538)
at org.h2.engine.Database.open(Database.java:709)
at org.h2.engine.Database.openDatabase(Database.java:286)
at org.h2.engine.Database.(Database.java:280)
at org.h2.engine.Engine.openSession(Engine.java:66)
at org.h2.engine.Engine.openSession(Engine.java:179)
at org.h2.engine.Engine.createSessionAndValidate(Engine.java:157)
at org.h2.engine.Engine.createSession(Engine.java:140)
at org.h2.engine.Engine.createSession(Engine.java:28)
at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:351)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:124)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:103)
at org.h2.Driver.connect(Driver.java:69)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at clojure.java.jdbc$get_driver_connection.invokeStatic(jdbc.clj:271)
at clojure.java.jdbc$get_driver_connection.invoke(jdbc.clj:250)
at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:411)
at clojure.java.jdbc$get_connection.invoke(jdbc.clj:274)
at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:833)
at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:769)
at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:782)
at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:769)
at metabase.db$migrate_BANG_.invokeStatic(db.clj:321)
at metabase.db$migrate_BANG_.invoke(db.clj:300)
at metabase.db$migrate_BANG_.invokeStatic(db.clj:318)
at metabase.db$migrate_BANG_.invoke(db.clj:300)
at metabase.cmd$migrate.invokeStatic(cmd.clj:29)
at metabase.cmd$migrate.invoke(cmd.clj:26)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:665)
at clojure.core$apply.invoke(core.clj:660)
at metabase.cmd$run_cmd$fn__64559.invoke(cmd.clj:128)
at metabase.cmd$run_cmd.invokeStatic(cmd.clj:128)
at metabase.cmd$run_cmd.invoke(cmd.clj:124)
at clojure.lang.Var.invoke(Var.java:388)
at metabase.core$run_cmd.invokeStatic(core.clj:133)
at metabase.core$run_cmd.invoke(core.clj:131)
at metabase.core$_main.invokeStatic(core.clj:142)
at metabase.core$_main.doInvoke(core.clj:138)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at metabase.core.main(Unknown Source)
Caused by: java.lang.IllegalStateException: Could not open file nio:/var/log/metabase.db.mv.db [1.4.197/1]
at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:870)
at org.h2.mvstore.FileStore.open(FileStore.java:179)
at org.h2.mvstore.MVStore.(MVStore.java:350)
at org.h2.mvstore.MVStore$Builder.open(MVStore.java:2934)
at org.h2.mvstore.db.MVTableEngine$Store.open(MVTableEngine.java:155)
… 44 more
Caused by: java.io.FileNotFoundException: /var/log/metabase.db.mv.db (Permission denied)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.(RandomAccessFile.java:243)
at java.io.RandomAccessFile.(RandomAccessFile.java:124)
at org.h2.store.fs.FileNio.(FilePathNio.java:43)
at org.h2.store.fs.FilePathNio.open(FilePathNio.java:23)
at org.h2.mvstore.FileStore.open(FileStore.java:155)
… 47 more
Command failed with exception: IO Exception: null [90028-197]
@bharathb
Okay, that’s very simple. You don’t have enough RAM to run Metabase. It looks like there’s only 239.8MB, which is way too little. You need to free up some RAM or get a larger server instance.
Metabase needs 512MB at a bare minimum, just to start - depending on your usage, I would recommend 1GB or 2GB.
Hi @flamber, thanks for responding.
I increased the memory to 2GB and it was running fine for couple of days. Now it’s back to 502 Bad gateway error.
To experiment, I’ve increased the aws ec2 instance to t2.xlarge(16GB memory and 4CPU).
Regarding the metabase log, even after following all the steps in https://www.metabase.com/docs/latest/operations-guide/running-metabase-on-debian.html, log file is not updating. (if there’s any other link to help setup log, please do share).
After running sudo java -jar /var/metabase/metabase.jar, everything went smoothly without any errors. But accessing the URL gives a fresh installation of metabase - showing setup page.
The metabase is setup is done as a service.
After running sudo java -jar /var/metabase/metabase.jar migrate release-locks, below is the output.
10-09 09:22:27 DEBUG plugins.classloader :: Using NEWLY CREATED classloader as shared context classloader: clojure.lang.DynamicClassLoader@377c68c6
10-09 09:22:27 DEBUG plugins.classloader :: Setting current thread context classloader to shared classloader clojure.lang.DynamicClassLoader@377c68c6…
10-09 09:22:28 INFO metabase.util :: Loading Metabase…
10-09 09:22:28 INFO metabase.util :: Maximum memory available to JVM: 3.5 GB
10-09 09:22:31 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance.
10-09 09:22:35 INFO metabase.driver :: Registered abstract driver :sql
10-09 09:22:42 WARN metabase.db :: WARNING: Using Metabase with an H2 application database is not recommended for production deployments. For production deployments, we highly recommend using Postgres, MySQL, or MariaDB instead. If you decide to continue to use H2, please be sure to back up the database file regularly. For more information, seehttps://metabase.com/docs/latest/operations-guide/migrating-from-h2.html
10-09 09:22:43 INFO metabase.db :: Setting up Liquibase…
10-09 09:22:43 INFO metabase.db :: Liquibase is ready.
If you’re running Metabase as a systemd service, then the log file is being handled by syslog by default, which might be redirected to a separate file, but that depends on your configuration.
The reason why it’s working, when you just start a new Metabase instance with a clean database is that it’s not doing anything - there’s no user activity, sync processes, pulses, or queries being executed. That instance will only use a few hundred MB of RAM.
@flamber Thanks for replying. since I was not able to get a hold of log file, I was unclear about the root cause why metabase not starting properly. so migrate release-locks used to release if any locks present.
But finally I was able to get a hold of the log file and found the following database error.
Oct 10 05:44:42 ip-172-31-4-153 metabase[31430]: 10-10 05:44:42 #033[1mERROR driver.util#033[0m :: Database connection error
Oct 10 05:44:42 ip-172-31-4-153 metabase[31430]: org.postgresql.util.PSQLException: The connection attempt failed.
There seems to be some problem with the database configuration in the metabase.env file.
After correcting DB connection details, metabase is starting properly. But with no data - a fresh installation.
Is there’s any way to get back the old data? If so it’d perfect, otherwise I think I’ll go ahead and redo the dashboard.
If you’re trying to start Metabase directly with java -jar ..., then you either need to define the environment variables first or add them as parameters.
Previously the DB host was pointing to a different host which is no longer available. May be that’s why connection problem occurred. Also the metadata is in old DB host I presume?
@bharathb
That’s a fairly old guide - the one in the documentation is newer and more complete.
If you have changed the host of the metadata database, and now Metabase doesn’t have any content, then yes, the metadata was stored in the previous database.