I have metabase installed on centos. Metabase crashes silently and doesn’t leave any traces in the log.
Just upgraded to the latest version, hoping it would stop silent crashing, but no luck. Any suggestions?
Do other java/jvm programs run successfully? It’s really strange that it wouldn’t spit anything out to stdout/stderr. We’re typically very verbose on the logging front.
Are you running the Jar or the docker image?
Hey Sameer - Using the jar, running through systemctl. The metabase logs stop after the system dies, and nothing is logged there, until the service is restarted.
What happens if you run it from the command line?
Same thing. I have one piece of new info, the latest crashes have come after adding a mongodb connection. Based on system monitor, process died at 3:10 UTC. Last logs below
06-28 02:50:27 ESC[1mWARN metabase.driverESC[0m :: Don’t know how to map class ‘class org.bson.types.ObjectId’ to a Field base_type, falling back to :UnknownField.
06-28 02:50:27 ESC[1mWARN metabase.driverESC[0m :: Don’t know how to map class ‘null’ to a Field base_type, falling back to :UnknownField.
06-28 02:50:27 ESC[1mWARN metabase.driverESC[0m :: Don’t know how to map class ‘null’ to a Field base_type, falling back to :UnknownField.
06-28 02:50:27 ESC[1mWARN metabase.driverESC[0m :: Don’t know how to map class ‘null’ to a Field base_type, falling back to :UnknownField.
06-28 02:50:29 ESC[1mWARN metabase.driverESC[0m :: Don’t know how to map class ‘class org.bson.types.ObjectId’ to a Field base_type, falling back to :UnknownField.
06-28 02:50:29 ESC[1mWARN metabase.driverESC[0m :: Don’t know how to map class ‘null’ to a Field base_type, falling back to :UnknownField.
06-28 02:50:29 ESC[1mWARN metabase.driverESC[0m :: Don’t know how to map class ‘null’ to a Field base_type, falling back to :UnknownField.
06-28 02:50:29 ESC[1mWARN metabase.driverESC[0m :: Don’t know how to map class ‘null’ to a Field base_type, falling back to :UnknownField.
06-28 02:50:30 ESC[1mWARN metabase.driverESC[0m :: Don’t know how to map class ‘class org.bson.types.ObjectId’ to a Field base_type, falling back to :UnknownField.
06-28 02:50:30 ESC[1mINFO metabase.sync-databaseESC[0m :: ESC[35mFinished syncing MongoDB database ‘mongo’. (19 s)ESC[0m
I’m going to remove Mongo for now and track down the previous crashes.
Hmm … there’s nothing in what you’ve pasted that would cause a crash.
It looks like the sync succeeded. Was that the last line before it died? It seems like the server was up for another 30 minutes without any other output if so. That suggests the process was killed somehow.
I agree. No, there was nothing else in the log or the system log that would suggest the application being terminated by anything else. I’m at a loss.
System crashed again this morning, even though Mongo was removed.
Last line in log
06-28 15:50:15 ESC[1mINFO metabase.sync-databaseESC[0m :: ESC[35mFinished syncing MySQL database ‘prod-replica’. (15 s)ESC[0m
Do you have any monitoring on that instance? Without access to the entire logs, there’s not much for me to go by in trying to help you.
Is this on a machine you’re running or a cloud instance? Is there anything else running?
EC2 instance, dedicated to metabase. Have some monitoring setup but, again, not providing any useful info. We’ll add some more and see if there is anything that I can share with you.
Thanks.
Closing the loop here. Did some troubleshooting, and still couldn’t find anything. Upgrade instance size last week and no longer seeing any crashes. Will let you know if that changes.
Thanks for the help
No worries. I’m a little worried we can’t re-produce it. But hoping it was either something to do with your instance or that we can narrow it down somehow in the future.
We’re seeing something similar and suspect it is OOME. Having a small instance 2cores 8GB ram, trying to download 800k rows with about 20 columns causes metabase to crash. Again no errors just the last log line being the massive select statement being printed.
Version is 0.23
Is there any recommendations on instance sizes? Feels like the serdes for CSV and json are both CPU and memory intensive.
what is the result size if you limit it to 80k rows?
that sounds out of wack
Works with a really small data set.
Can you share how big the download file is, and how many rows? I’m guessing this is happening due to jvm memory settings but curious how big the actual file is.
yup I suspected the same after some further research. I have upgraded my instance and given it 20GB via -Xmx option. no issues so far.
Is there a way to turn on warning or error logging? i.e. change the logging level?
thanks
Our logs are pretty chatty, and OOM errors definitely show up in the server logs.
I’m having the same issue with the 0.29. The process is still running, no info on logs. Happens pretty constantly (twice a day, I’d say), we even created a custom command on our Debian remote reserver to kill the process and restart our metabase instance. Should the problem be related to java, would the process still be running?
We also use jenkins on that server, and it doesnt present any problem, even when metabase crashes.
We are having the same issue… I decided to try the manual start of the Jar process and wait till it crashes, here is what I got in the console log (this was after +/- of 1 hour, I was the only user. I also typed ‘ls’ to make sure the console was not disconnected), the ‘Killed’ isn’t saying much unfortunately, no error, nothing…:
05-27 03:00:02 INFO sync.util :: FINISHED: Sync metadata for postgres Database 4 ‘confetti-history’ (161 ms)
05-27 03:02:29 DEBUG middleware.log :: GET /api/session/properties 200 4 ms (1 DB calls) Jetty threads: 5/50 (2 idle, 0 queued) (85 total active threads) Queries in flight: 0
05-27 03:02:29 DEBUG middleware.log :: GET /api/setup/admin_checklist 200 10 ms (10 DB calls) Jetty threads: 5/50 (2 idle, 0 queued) (87 total active threads) Queries in flight: 0
05-27 03:02:29 DEBUG middleware.log :: GET /api/setting 200 540 µs (0 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (89 total active threads) Queries in flight: 0
05-27 03:02:30 DEBUG middleware.log :: GET /api/database 200 4 ms (3 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (89 total active threads) Queries in flight: 0
05-27 03:02:30 DEBUG middleware.log :: GET /api/database/1/metadata 200 13 ms (9 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (89 total active threads) Queries in flight: 0
05-27 03:02:30 DEBUG middleware.log :: GET /api/database/1/idfields 200 7 ms (6 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (89 total active threads) Queries in flight: 0
05-27 03:02:32 DEBUG middleware.log :: GET /api/database/2/metadata 200 81 ms (9 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (89 total active threads) Queries in flight: 0
05-27 03:02:33 DEBUG middleware.log :: GET /api/database/2/idfields 200 18 ms (6 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (89 total active threads) Queries in flight: 0
05-27 03:02:57 DEBUG middleware.log :: GET /api/database 200 5 ms (3 DB calls) Jetty threads: 5/50 (2 idle, 0 queued) (89 total active threads) Queries in flight: 0
05-27 03:02:58 DEBUG middleware.log :: GET /api/user/current 200 3 ms (3 DB calls) Jetty threads: 6/50 (3 idle, 0 queued) (91 total active threads) Queries in flight: 0
05-27 03:02:58 DEBUG middleware.log :: GET /api/collection/root/items 200 8 ms (7 DB calls) Jetty threads: 5/50 (3 idle, 0 queued) (91 total active threads) Queries in flight: 0
05-27 03:02:58 DEBUG middleware.log :: GET /api/automagic-dashboards/database/4/candidates 200 15 ms (6 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (91 total active threads) Queries in flight: 0
05-27 03:02:58 DEBUG middleware.log :: GET /api/session/properties 200 3 ms (1 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (91 total active threads) Queries in flight: 0
05-27 03:02:58 DEBUG middleware.log :: GET /api/collection/root 200 2 ms (2 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (91 total active threads) Queries in flight: 0
05-27 03:02:58 DEBUG middleware.log :: GET /api/setting 200 942 µs (0 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (91 total active threads) Queries in flight: 0
05-27 03:02:59 DEBUG middleware.log :: GET /api/database 200 5 ms (3 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (91 total active threads) Queries in flight: 0
05-27 03:03:01 DEBUG middleware.log :: GET /api/metric 200 3 ms (3 DB calls) Jetty threads: 6/50 (3 idle, 0 queued) (91 total active threads) Queries in flight: 0
05-27 03:03:01 DEBUG middleware.log :: GET /api/segment 200 10 ms (6 DB calls) Jetty threads: 7/50 (2 idle, 0 queued) (91 total active threads) Queries in flight: 0
05-27 03:03:01 DEBUG middleware.log :: GET /api/database 200 27 ms (5 DB calls) Jetty threads: 4/50 (2 idle, 0 queued) (91 total active threads) Queries in flight: 0
05-27 03:03:06 DEBUG middleware.log :: GET /api/table/21/query_metadata 200 15 ms (10 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (91 total active threads) Queries in flight: 0
05-27 03:03:06 DEBUG middleware.log :: GET /api/table/32/query_metadata 200 16 ms (10 DB calls) Jetty threads: 5/50 (3 idle, 0 queued) (91 total active threads) Queries in flight: 0
05-27 03:03:06 DEBUG middleware.log :: GET /api/table/11/query_metadata 200 17 ms (8 DB calls) Jetty threads: 5/50 (3 idle, 0 queued) (91 total active threads) Queries in flight: 0
05-27 03:03:06 DEBUG middleware.log :: GET /api/table/21/fks 200 6 ms (5 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (91 total active threads) Queries in flight: 0
05-27 03:03:08 DEBUG middleware.log :: POST /api/dataset 200 [ASYNC: completed] 396 ms (16 DB calls) Jetty threads: 3/50 (3 idle, 0 queued) (91 total active threads) Queries in flight: 0
05-27 03:03:52 DEBUG middleware.log :: PUT /api/field/445 200 8 ms (5 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (91 total active threads) Queries in flight: 0
05-27 03:03:52 DEBUG middleware.log :: GET /api/database/2/idfields 200 20 ms (6 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (91 total active threads) Queries in flight: 0
05-27 03:04:00 DEBUG middleware.log :: PUT /api/field/432 200 8 ms (5 DB calls) Jetty threads: 4/50 (2 idle, 0 queued) (90 total active threads) Queries in flight: 0
05-27 03:04:00 DEBUG middleware.log :: GET /api/database/2/idfields 200 15 ms (6 DB calls) Jetty threads: 4/50 (2 idle, 0 queued) (90 total active threads) Queries in flight: 0
05-27 03:04:46 DEBUG middleware.log :: PUT /api/field/444 200 6 ms (5 DB calls) Jetty threads: 4/50 (4 idle, 0 queued) (88 total active threads) Queries in flight: 0
05-27 03:04:46 DEBUG middleware.log :: GET /api/database/2/idfields 200 15 ms (6 DB calls) Jetty threads: 4/50 (4 idle, 0 queued) (88 total active threads) Queries in flight: 0
05-27 03:04:54 DEBUG middleware.log :: POST /api/dataset 200 [ASYNC: completed] 407 ms (15 DB calls) Jetty threads: 3/50 (4 idle, 0 queued) (88 total active threads) Queries in flight: 0
05-27 03:05:40 DEBUG middleware.log :: GET /api/table/21/query_metadata 200 14 ms (10 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (87 total active threads) Queries in flight: 0
05-27 03:05:40 DEBUG middleware.log :: GET /api/table/32/query_metadata 200 17 ms (10 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (87 total active threads) Queries in flight: 0
05-27 03:05:40 DEBUG middleware.log :: GET /api/table/11/query_metadata 200 13 ms (8 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (87 total active threads) Queries in flight: 0
05-27 03:05:41 DEBUG middleware.log :: GET /api/database/2/idfields 200 17 ms (6 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (87 total active threads) Queries in flight: 0
05-27 03:05:53 DEBUG middleware.log :: PUT /api/field/432 200 5 ms (5 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (87 total active threads) Queries in flight: 0
05-27 03:05:55 DEBUG middleware.log :: PUT /api/field/432 200 5 ms (5 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (87 total active threads) Queries in flight: 0
05-27 03:06:23 DEBUG middleware.log :: GET /api/database 200 5 ms (3 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (87 total active threads) Queries in flight: 0
05-27 03:06:23 DEBUG middleware.log :: GET /api/database/2/metadata 200 90 ms (9 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (87 total active threads) Queries in flight: 0
05-27 03:06:24 DEBUG middleware.log :: GET /api/database/2/idfields 200 13 ms (6 DB calls) Jetty threads: 4/50 (3 idle, 0 queued) (87 total active threads) Queries in flight: 0
05-27 03:06:31 DEBUG middleware.log :: GET /api/database 200 4 ms (3 DB calls) Jetty threads: 5/50 (2 idle, 0 queued) (87 total active threads) Queries in flight: 0
05-27 03:06:31 DEBUG middleware.log :: GET /api/automagic-dashboards/database/4/candidates 200 9 ms (6 DB calls) Jetty threads: 4/50 (2 idle, 0 queued) (88 total active threads) Queries in flight: 0
05-27 03:06:33 DEBUG middleware.log :: GET /api/database/2/schemas 200 6 ms (4 DB calls) Jetty threads: 4/50 (2 idle, 0 queued) (88 total active threads) Queries in flight: 0
05-27 03:06:34 DEBUG middleware.log :: GET /api/database/2/schema/public 200 8 ms (4 DB calls) Jetty threads: 4/50 (2 idle, 0 queued) (88 total active threads) Queries in flight: 0
05-27 03:06:38 DEBUG middleware.log :: GET /api/table/37/query_metadata 200 14 ms (10 DB calls) Jetty threads: 5/50 (2 idle, 0 queued) (88 total active threads) Queries in flight: 0
05-27 03:06:38 DEBUG middleware.log :: GET /api/table/32/query_metadata 200 14 ms (10 DB calls) Jetty threads: 4/50 (2 idle, 0 queued) (88 total active threads) Queries in flight: 1
05-27 03:06:39 DEBUG middleware.log :: GET /api/table/37/fks 200 5 ms (5 DB calls) Jetty threads: 4/50 (2 idle, 0 queued) (88 total active threads) Queries in flight: 1
05-27 03:06:40 DEBUG middleware.log :: POST /api/dataset 200 [ASYNC: completed] 2 s (15 DB calls) Jetty threads: 3/50 (2 idle, 0 queued) (88 total active threads) Queries in flight: 0
05-27 03:07:23 DEBUG middleware.log :: PUT /api/field/910 200 7 ms (5 DB calls) Jetty threads: 4/50 (2 idle, 0 queued) (88 total active threads) Queries in flight: 0
05-27 03:07:23 DEBUG middleware.log :: GET /api/database/2/idfields 200 14 ms (6 DB calls) Jetty threads: 4/50 (2 idle, 0 queued) (88 total active threads) Queries in flight: 0
05-27 03:07:30 DEBUG middleware.log :: PUT /api/field/910 200 7 ms (6 DB calls) Jetty threads: 4/50 (2 idle, 0 queued) (88 total active threads) Queries in flight: 0
05-27 03:07:30 DEBUG middleware.log :: GET /api/database/2/idfields 200 15 ms (6 DB calls) Jetty threads: 4/50 (2 idle, 0 queued) (88 total active threads) Queries in flight: 0
05-27 03:14:01 DEBUG middleware.log :: GET /api/database 200 4 ms (3 DB calls) Jetty threads: 4/50 (2 idle, 0 queued) (87 total active threads) Queries in flight: 0
05-27 03:14:01 DEBUG middleware.log :: GET /api/automagic-dashboards/database/4/candidates 200 12 ms (6 DB calls) Jetty threads: 4/50 (2 idle, 0 queued) (87 total active threads) Queries in flight: 0
05-27 03:14:16 DEBUG middleware.log :: GET /api/database 200 6 ms (3 DB calls) Jetty threads: 5/50 (1 idle, 0 queued) (88 total active threads) Queries in flight: 0
05-27 03:14:16 DEBUG middleware.log :: GET /api/automagic-dashboards/database/4/candidates 200 12 ms (6 DB calls) Jetty threads: 4/50 (1 idle, 0 queued) (88 total active threads) Queries in flight: 0
05-27 03:14:24 DEBUG middleware.log :: GET /api/table/11/query_metadata 200 14 ms (8 DB calls) Jetty threads: 4/50 (1 idle, 0 queued) (88 total active threads) Queries in flight: 0
05-27 03:14:24 DEBUG middleware.log :: GET /api/table/11/fks 200 7 ms (5 DB calls) Jetty threads: 4/50 (1 idle, 0 queued) (88 total active threads) Queries in flight: 1
05-27 03:14:26 DEBUG middleware.log :: POST /api/dataset 200 [ASYNC: completed] 2 s (16 DB calls) Jetty threads: 3/50 (1 idle, 0 queued) (88 total active threads) Queries in flight: 0
05-27 03:14:58 DEBUG middleware.log :: POST /api/dataset 200 [ASYNC: completed] 698 ms (14 DB calls) Jetty threads: 3/50 (1 idle, 0 queued) (88 total active threads) Queries in flight: 0
05-27 03:15:45 DEBUG middleware.log :: POST /api/dataset 200 [ASYNC: completed] 647 ms (14 DB calls) Jetty threads: 3/50 (4 idle, 0 queued) (87 total active threads) Queries in flight: 0
Killed
bash-4.2$ ls
metabase.jar plugins
bash-4.2$
Hi @naxels
Without knowing more about your environment, it’s difficult to know.
Which version of OS, Java and Metabase?
When the crash happens, do you see anything in other system logs?
Are you running Metabase manually or as a service?
Could you be running out of memory? Check by running free -m