502 Gateway Error (Nginx) Metabase

Bug reports
1

Hi,

My server provider installed Metabase and SSL on a VPS with Debian 9 and 2GB Ram following this instruccions Install Metabase - SSL with Nginx

Everything works fine except that sometimes when I try to access to a Dashboard or Browse the database (Postgresql) it crashes and gives me a "502 Gateway error" But when I wait about 10 -15 sec and reload the page, it works again (the time that metabase starts again)

000134
0001341418×252 33.1 KB

I followed this post https://discourse.metabase.com/t/502-bad-gateway/7501 without success.

For now I'm using the H2 database, but I plan to migrate to Postgresql. Can this issue be related to actual H2 database?

My Metabase info

{
"browser-info": {
"language": "en-US",
"platform": "Win32",
"userAgent": "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36",
"vendor": "Google Inc."
},
"system-info": {
"java.runtime.name": "OpenJDK Runtime Environment",
"java.runtime.version": "1.8.0_242-8u242-b08-1~deb9u1-b08",
"java.vendor": "Oracle Corporation",
"java.vendor.url": "http://java.oracle.com/",
"java.version": "1.8.0_242",
"java.vm.name": "OpenJDK 64-Bit Server VM",
"java.vm.version": "25.242-b08",
"os.name": "Linux",
"os.version": "4.15.18-19-pve",
"user.language": "en",
"user.timezone": "Europe/Madrid"
},
"metabase-info": {
"databases": [
"postgres"
],
"hosting-env": "unknown",
"application-database": "h2",
"application-database-details": {
"database": {
"name": "H2",
"version": "1.4.197 (2018-03-18)"
},
"jdbc-driver": {
"name": "H2 JDBC Driver",
"version": "1.4.197 (2018-03-18)"
}
},
"run-mode": "prod",
"version": {
"tag": "v0.34.3",
"date": "2020-02-25",
"branch": "release-0.34.x",
"hash": "1a83edb"
},
"settings": {
"report-timezone": "Europe/Berlin"
}
}
}

My nginx.conf located in /etc/nginx/sites-available/metabase.conf :

server {
listen [::]:80;
listen 80;
server_name mywebsite.com;
return 301 https://mywebsite.com$request_uri;
}
server {
listen [::]:443 ssl;
listen 443 ssl;
server_name mywebsite.com;
ssl_certificate /etc/letsencrypt/live/mywebsite.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mywebsite.com/privkey.pem;
location / {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://localhost:3000;
client_max_body_size 100M;
}
}

Checking the Metabase log when it crashes, I see that it starts again loading all from the beginning

[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:55+01:00 INFO metabase.core Starting Metabase in STANDALONE mode
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:55+01:00 INFO metabase.server Launching Embedded Jetty Webserver with config:
{:port 3000}
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:55+01:00 INFO metabase.core Starting Metabase version v0.34.3 (1a83edb release-0.34.x) ...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:55+01:00 INFO metabase.core System info:
{"java.runtime.name" "OpenJDK Runtime Environment",
"java.runtime.version" "1.8.0_242-8u242-b08-1~deb9u1-b08",
"java.vendor" "Oracle Corporation",
"java.vendor.url" "http://java.oracle.com/",
"java.version" "1.8.0_242",
"java.vm.name" "OpenJDK 64-Bit Server VM",
"java.vm.version" "25.242-b08",
"os.name" "Linux",
"os.version" "4.15.18-19-pve",
"user.language" "en",
"user.timezone" "Europe/Madrid"}
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:55+01:00 INFO metabase.plugins Loading plugins in /apps/java/metabase/plugins...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.plugins.classloader Added URL file:/apps/java/metabase/plugins/ojdbc8.jar to classpath
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.plugins.classloader Added URL file:/apps/java/metabase/plugins/vertica-jdbc-8.0.1-0.jar to classpath
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 DEBUG metabase.plugins.lazy-loaded-driver Registering lazy loading driver :snowflake...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered abstract driver :sql :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered abstract driver :sql-jdbc (parents: [:sql]) :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered driver :snowflake (parents: [:sql-jdbc]) :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.plugins.dependencies Metabase Oracle Driver dependency {:class oracle.jdbc.OracleDriver} satisfied? true
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 DEBUG metabase.plugins.lazy-loaded-driver Registering lazy loading driver :oracle...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered driver :oracle (parents: [:sql-jdbc]) :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 DEBUG metabase.plugins.lazy-loaded-driver Registering lazy loading driver :google...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered abstract driver :google :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 DEBUG metabase.plugins.lazy-loaded-driver Registering lazy loading driver :sqlite...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered driver :sqlite (parents: [:sql-jdbc]) :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 DEBUG metabase.plugins.lazy-loaded-driver Registering lazy loading driver :redshift...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered driver :postgres (parents: [:sql-jdbc]) :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered driver :redshift (parents: [:postgres]) :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 DEBUG metabase.plugins.lazy-loaded-driver Registering lazy loading driver :mongo...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered driver :mongo :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 DEBUG metabase.plugins.lazy-loaded-driver Registering lazy loading driver :presto...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered driver :presto (parents: [:sql]) :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.plugins.dependencies Metabase Vertica Driver dependency {:class com.vertica.jdbc.Driver} satisfied? true
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 DEBUG metabase.plugins.lazy-loaded-driver Registering lazy loading driver :vertica...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered driver :vertica (parents: [:sql-jdbc]) :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.plugins.dependencies Metabase Google Analytics Driver dependency {:plugin Metabase Google Drivers Shared Dependencies} satisfied? true
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.plugins.dependencies Plugin 'Metabase Google Analytics Driver' depends on plugin 'Metabase Google Drivers Shared Dependencies'
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 DEBUG metabase.plugins.lazy-loaded-driver Registering lazy loading driver :googleanalytics...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered driver :googleanalytics (parents: [:google]) :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered driver :druid :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 DEBUG metabase.plugins.lazy-loaded-driver Registering lazy loading driver :druid...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 DEBUG metabase.plugins.lazy-loaded-driver Registering lazy loading driver :hive-like...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered abstract driver :hive-like (parents: [:sql-jdbc]) :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 DEBUG metabase.plugins.lazy-loaded-driver Registering lazy loading driver :sparksql...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered driver :sparksql (parents: [:hive-like]) :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.plugins.dependencies Plugin 'Metabase BigQuery Driver' depends on plugin 'Metabase Google Drivers Shared Dependencies'
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 DEBUG metabase.plugins.lazy-loaded-driver Registering lazy loading driver :bigquery...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.plugins.dependencies Metabase BigQuery Driver dependency {:plugin Metabase Google Drivers Shared Dependencies} satisfied? true
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered driver :bigquery (parents: [:sql :google]) :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 DEBUG metabase.plugins.lazy-loaded-driver Registering lazy loading driver :sqlserver...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered driver :sqlserver (parents: [:sql-jdbc]) :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered driver :h2 (parents: [:sql-jdbc]) :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Registered driver :mysql (parents: [:sql-jdbc]) :truck:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.core Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 WARN metabase.db WARNING: Using Metabase with an H2 application database is not recommended for production deployments. For production deployments, we highly recommend using Postgres, MySQL, or MariaDB instead. If you decide to continue to use H2, please be sure to back up the database file regularly. For more information, seehttps://metabase.com/docs/latest/operations-guide/migrating-from-h2.html
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.db Verifying h2 Database Connection ...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Initializing driver :sql...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Initializing driver :sql-jdbc...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:56+01:00 INFO metabase.driver.impl Initializing driver :h2...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:57+01:00 INFO metabase.db Successfully verified H2 1.4.197 (2018-03-18) application database connection. :white_check_mark:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:57+01:00 INFO metabase.db Running Database Migrations...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:57+01:00 INFO metabase.db Setting up Liquibase...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:57+01:00 INFO metabase.db Liquibase is ready.
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:57+01:00 INFO metabase.db.liquibase Checking if Database has unrun migrations...
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.db Database Migrations Current ... :white_check_mark:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.db.migrations Running all necessary data migrations, this may take a minute.
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.db.migrations Finished running data migrations.
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.events Starting events listener: metabase.events.activity-feed :ear:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.events Starting events listener: metabase.events.dependencies :ear:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.events Starting events listener: metabase.events.driver-notifications :ear:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.events Starting events listener: metabase.events.last-login :ear:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.events Starting events listener: metabase.events.metabot-lifecycle :ear:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.events Starting events listener: metabase.events.notifications :ear:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.events Starting events listener: metabase.events.revision :ear:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.events Starting events listener: metabase.events.sync-database :ear:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.events Starting events listener: metabase.events.view-log :ear:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.task Initializing task SyncDatabases :calendar:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.task Initializing task CheckForNewVersions :calendar:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.task Initializing task SendAnonymousUsageStats :calendar:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.task Initializing task SendAbandomentEmails :calendar:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.task Initializing task SendPulses :calendar:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.task Initializing task SendFollowUpEmails :calendar:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.task Initializing task TaskHistoryCleanup :calendar:
[ee458fa9-52c3-4dc2-ac5b-93837b571be4] 2020-03-20T19:41:59+01:00 INFO metabase.core Metabase Initialization COMPLETE`

Checking the Nginx error.log file I see this:

2020/03/20 19:41:40 [error] 158#158: *3050 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://127.0.0.1:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:40 [error] 158#158: *3050 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://[::1]:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:40 [error] 161#161: *2950 upstream prematurely closed connection while reading response header from upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/database?include_tables=true&include_cards=true HTTP/1.1", upstream: "http://[::1]:3000/api/database?include_tables=true&include_cards=true", host: "mywebsite.com", referrer: "https://mywebsite.com/question"
2020/03/20 19:41:40 [error] 161#161: *2950 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/database?include_tables=true&include_cards=true HTTP/1.1", upstream: "http://127.0.0.1:3000/api/database?include_tables=true&include_cards=true", host: "mywebsite.com", referrer: "https://mywebsite.com/question"
2020/03/20 19:41:40 [error] 161#161: *2950 no live upstreams while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://localhost/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:41 [error] 158#158: *3050 no live upstreams while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://localhost/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:41 [error] 161#161: *2950 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://127.0.0.1:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:41 [error] 161#161: *2950 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://[::1]:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:41 [error] 158#158: *3050 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://[::1]:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:41 [error] 158#158: *3050 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://127.0.0.1:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:41 [error] 158#158: *3056 no live upstreams while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://localhost/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:41 [error] 158#158: *3050 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://[::1]:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:41 [error] 158#158: *3050 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://127.0.0.1:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:41 [error] 158#158: *3056 no live upstreams while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://localhost/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:41 [error] 158#158: *3050 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://[::1]:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:41 [error] 158#158: *3050 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://127.0.0.1:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:41 [error] 158#158: *3056 no live upstreams while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://localhost/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:42 [error] 158#158: *3056 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://[::1]:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:42 [error] 158#158: *3056 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://127.0.0.1:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:43 [error] 158#158: *3056 no live upstreams while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://localhost/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:44 [error] 158#158: *3056 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://[::1]:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:44 [error] 158#158: *3056 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://127.0.0.1:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:45 [error] 158#158: *3056 no live upstreams while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://localhost/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:46 [error] 158#158: *3056 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://[::1]:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:46 [error] 158#158: *3056 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://127.0.0.1:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:47 [error] 158#158: *3056 no live upstreams while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://localhost/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:48 [error] 158#158: *3056 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://[::1]:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:48 [error] 158#158: *3056 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://127.0.0.1:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:49 [error] 158#158: *3056 no live upstreams while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://localhost/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:50 [error] 158#158: *3056 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://[::1]:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:50 [error] 158#158: *3056 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://127.0.0.1:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:51 [error] 158#158: *3056 no live upstreams while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://localhost/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:52 [error] 158#158: *3056 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://[::1]:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:52 [error] 158#158: *3056 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://127.0.0.1:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:53 [error] 158#158: *3056 no live upstreams while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://localhost/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:54 [error] 158#158: *3056 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://[::1]:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:54 [error] 158#158: *3056 connect() failed (111: Connection refused) while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://127.0.0.1:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:41:55 [error] 158#158: *3056 no live upstreams while connecting to upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://localhost/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"
2020/03/20 19:42:56 [error] 158#158: *3056 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 31.4.000.00,, server: mywebsite.com, request: "GET /api/util/logs HTTP/1.1", upstream: "http://[::1]:3000/api/util/logs", host: "mywebsite.com", referrer: "mywebsite.com is available for purchase - Sedo.com"

Thanks for any help in advance

2

Hi @comb
Ignore the Nginx right now, since you need to focus on why the Metabase process is crashing.
Check your syslog - or sudo journalctl -u metabase if you’ve setup systemd and called it “metabase”

3

HI @flamber

Thanks for the quick reply.

using this command sudo journalctl -u metabase I get his info on the time that it crashed.

Mar 20 19:38:17 vl18400 metabase[2699]: 03-20 19:38:17 DEBUG middleware.log :: POST /api/card/289/query 202 [ASYNC: completed] 183.7 ms (19 DB calls) Jetty threads: 2/50 (6 idle
Mar 20 19:38:17 vl18400 metabase[2699]: 03-20 19:38:17 DEBUG middleware.log :: POST /api/card/385/query 202 [ASYNC: completed] 174.6 ms (14 DB calls) Jetty threads: 2/50 (6 idle
Mar 20 19:38:17 vl18400 metabase[2699]: 03-20 19:38:17 DEBUG middleware.log :: POST /api/card/321/query 202 [ASYNC: completed] 219.6 ms (19 DB calls) Jetty threads: 2/50 (6 idle
Mar 20 19:38:17 vl18400 metabase[2699]: 03-20 19:38:17 DEBUG middleware.log :: POST /api/card/449/query 202 [ASYNC: completed] 204.6 ms (14 DB calls) Jetty threads: 2/50 (6 idle
Mar 20 19:38:17 vl18400 metabase[2699]: 03-20 19:38:17 DEBUG middleware.log :: POST /api/card/545/query 202 [ASYNC: completed] 221.6 ms (14 DB calls) Jetty threads: 2/50 (6 idle
Mar 20 19:38:21 vl18400 metabase[2699]: 03-20 19:38:21 DEBUG middleware.log :: GET /api/database/2/schemas 200 52.2 ms (4 DB calls) Jetty threads: 3/50 (6 idle, 0 queued) (65 to
Mar 20 19:38:25 vl18400 metabase[2699]: 03-20 19:38:25 DEBUG middleware.log :: GET /api/database/2/metadata 200 4.5 s (9 DB calls) Jetty threads: 3/50 (5 idle, 0 queued) (65 tot
Mar 20 19:41:40 vl18400 systemd[1]: metabase.service: Main process exited, code=killed, status=9/KILL
Mar 20 19:41:40 vl18400 systemd[1]: metabase.service: Unit entered failed state.
Mar 20 19:41:40 vl18400 systemd[1]: metabase.service: Failed with result 'signal'.
Mar 20 19:41:41 vl18400 systemd[1]: metabase.service: Service hold-off time over, scheduling restart.
Mar 20 19:41:41 vl18400 systemd[1]: Stopped Metabase server.
Mar 20 19:41:41 vl18400 systemd[1]: metabase.service: Failed to reset devices.list: Operation not permitted
Mar 20 19:41:41 vl18400 systemd[1]: Started Metabase server.
Mar 20 19:41:43 vl18400 metabase[3205]: 03-20 19:41:43 DEBUG plugins.classloader :: Using NEWLY CREATED classloader as shared context classloader: clojure.lang.DynamicClassLoade
Mar 20 19:41:43 vl18400 metabase[3205]: 03-20 19:41:43 INFO metabase.util :: Loading Metabase...
Mar 20 19:41:43 vl18400 metabase[3205]: 03-20 19:41:43 INFO metabase.util :: Maximum memory available to JVM: 13.9 GB
Mar 20 19:41:50 vl18400 metabase[3205]: 03-20 19:41:50 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. :unlock:
Mar 20 19:41:50 vl18400 metabase[3205]: For more information, see Redirecting…
Mar 20 19:41:55 vl18400 metabase[3205]: 03-20 19:41:55 INFO metabase.core :: Starting Metabase in STANDALONE mode
Mar 20 19:41:55 vl18400 metabase[3205]: 03-20 19:41:55 INFO metabase.server :: Launching Embedded Jetty Webserver with config:
Mar 20 19:41:55 vl18400 metabase[3205]: {:port 3000}
Mar 20 19:41:55 vl18400 metabase[3205]: 03-20 19:41:55 INFO metabase.core :: Starting Metabase version v0.34.3 (1a83edb release-0.34.x) ...
Mar 20 19:41:55 vl18400 metabase[3205]: 03-20 19:41:55 INFO metabase.core :: System info:
Mar 20 19:41:55 vl18400 metabase[3205]: {"java.runtime.name" "OpenJDK Runtime Environment",
Mar 20 19:41:55 vl18400 metabase[3205]: "java.runtime.version" "1.8.0_242-8u242-b08-1~deb9u1-b08",
Mar 20 19:41:55 vl18400 metabase[3205]: "java.vendor" "Oracle Corporation",
Mar 20 19:41:55 vl18400 metabase[3205]: "java.vendor.url" "http://java.oracle.com/",
Mar 20 19:41:55 vl18400 metabase[3205]: "java.version" "1.8.0_242",
Mar 20 19:41:55 vl18400 metabase[3205]: "java.vm.name" "OpenJDK 64-Bit Server VM",
Mar 20 19:41:55 vl18400 metabase[3205]: "java.vm.version" "25.242-b08",
Mar 20 19:41:55 vl18400 metabase[3205]: "os.name" "Linux",
Mar 20 19:41:55 vl18400 metabase[3205]: "os.version" "4.15.18-19-pve",
Mar 20 19:41:55 vl18400 metabase[3205]: "user.language" "en",
Mar 20 19:41:55 vl18400 metabase[3205]: "user.timezone" "Europe/Madrid"}
Mar 20 19:41:55 vl18400 metabase[3205]: 03-20 19:41:55 INFO metabase.plugins :: Loading plugins in /apps/java/metabase/plugins...
Mar 20 19:41:56 vl18400 metabase[3205]: 03-20 19:41:56 INFO plugins.classloader :: Added URL file:/apps/java/metabase/plugins/ojdbc8.jar to classpath

I see on 19:41:40 where all starts it says Mar 20 19:41:40 vl18400 systemd[1]: metabase.service: Main process exited, code=killed, status=9/KILL

My /etc/systemd/system/metabase.service contains:

[Unit]
Description=Metabase server
After=syslog.target
After=network.target
[Service]
WorkingDirectory=/apps/java/metabase
ExecStart=/usr/bin/java -jar /apps/java/metabase/metabase.jar
EnvironmentFile=/etc/default/metabase
User=metabase
Type=simple
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=metabase
SuccessExitStatus=143
TimeoutStopSec=120
Restart=always
[Install]
WantedBy=multi-user.target

4

@comb Something is wrong. You say it’s a 2GB VPS, but it is setting heap much higher:
Mar 20 19:41:43 vl18400 metabase[3205]: 03-20 19:41:43 INFO metabase.util :: Maximum memory available to JVM: 13.9 GB
https://www.metabase.com/docs/latest/troubleshooting-guide/running.html#metabase-fails-to-start-due-to-heap-space-outofmemoryerrors
I would recommend that you use AdoptOpenJRE:
https://adoptopenjdk.net/releases.html?variant=openjdk11&jvmVariant=hotspot

5

@flamber thanks for the info.

I purchased the server with 2GB and this are the stats

000136

000137
000137775×501 31.4 KB
000138
000138781×500 33.1 KB

Do I have to use the JVM option to limit the memory usage? Because I see any error on the log of memory.

Tried to remove the OpenJDK 64-Bit Server VM that I have installed right now and to install the AdoptOpenJRE, but with no success... Will ask the server provider if they can install it..

Do you think if I upgrade to 4GB Ram this issue can be resolved on that way?

6

@comb I would highly recommend that you use the Docker container instead. Then you don’t have to deal with any extra software or services, it’s one small little container.
There’s thousands of Metabase installations, and very few with problems like this, so I guess it has something to do with the server/setup.

7

Hi @flamber, sorry for the late replay, had no time to try it till today. I was able to configure it with the docker container and now it works correctly, without that previous error. Thanks for the help!

Only one more question, is 2GB of memory enough to run the docker? Because sometimes it takes too long to access a question to edit it, or to access to the dashboard, even sometimes it crashes giving an memory error. Any way, we can close this thread as solved with the Docker installation. And I will try it this way, and open a new thread with the memory issue if needed.

8

@comb If Metabase is crashing because of out-of-memory, then you need to give it more memory:
https://www.metabase.com/docs/latest/troubleshooting-guide/running.html#metabase-fails-to-start-due-to-heap-space-outofmemoryerrors

9

Yes, thanks for the link, I already configured it with 3GB and created the path for the " Diagnosing memory issues causing OutOfMemoryErrors" to check if I get any errors. If there is any issue, I will open a new thread with all the information. Thanks a lot for your help!