Metabase crashing and no info in Logs


#1

I have metabase installed on centos. Metabase crashes silently and doesn’t leave any traces in the log.

Just upgraded to the latest version, hoping it would stop silent crashing, but no luck. Any suggestions?


Crashes / Can't save questions
#2

Do other java/jvm programs run successfully? It’s really strange that it wouldn’t spit anything out to stdout/stderr. We’re typically very verbose on the logging front.

Are you running the Jar or the docker image?


#3

Hey Sameer - Using the jar, running through systemctl. The metabase logs stop after the system dies, and nothing is logged there, until the service is restarted.


#4

What happens if you run it from the command line?


#5

Same thing. I have one piece of new info, the latest crashes have come after adding a mongodb connection. Based on system monitor, process died at 3:10 UTC. Last logs below

06-28 02:50:27 ESC[1mWARN metabase.driverESC[0m :: Don’t know how to map class ‘class org.bson.types.ObjectId’ to a Field base_type, falling back to :UnknownField.
06-28 02:50:27 ESC[1mWARN metabase.driverESC[0m :: Don’t know how to map class ‘null’ to a Field base_type, falling back to :UnknownField.
06-28 02:50:27 ESC[1mWARN metabase.driverESC[0m :: Don’t know how to map class ‘null’ to a Field base_type, falling back to :UnknownField.
06-28 02:50:27 ESC[1mWARN metabase.driverESC[0m :: Don’t know how to map class ‘null’ to a Field base_type, falling back to :UnknownField.
06-28 02:50:29 ESC[1mWARN metabase.driverESC[0m :: Don’t know how to map class ‘class org.bson.types.ObjectId’ to a Field base_type, falling back to :UnknownField.
06-28 02:50:29 ESC[1mWARN metabase.driverESC[0m :: Don’t know how to map class ‘null’ to a Field base_type, falling back to :UnknownField.
06-28 02:50:29 ESC[1mWARN metabase.driverESC[0m :: Don’t know how to map class ‘null’ to a Field base_type, falling back to :UnknownField.
06-28 02:50:29 ESC[1mWARN metabase.driverESC[0m :: Don’t know how to map class ‘null’ to a Field base_type, falling back to :UnknownField.
06-28 02:50:30 ESC[1mWARN metabase.driverESC[0m :: Don’t know how to map class ‘class org.bson.types.ObjectId’ to a Field base_type, falling back to :UnknownField.
06-28 02:50:30 ESC[1mINFO metabase.sync-databaseESC[0m :: ESC[35mFinished syncing MongoDB database ‘mongo’. (19 s)ESC[0m

I’m going to remove Mongo for now and track down the previous crashes.


#6

Hmm … there’s nothing in what you’ve pasted that would cause a crash.

It looks like the sync succeeded. Was that the last line before it died? It seems like the server was up for another 30 minutes without any other output if so. That suggests the process was killed somehow.


#7

I agree. No, there was nothing else in the log or the system log that would suggest the application being terminated by anything else. I’m at a loss.
System crashed again this morning, even though Mongo was removed.

Last line in log
06-28 15:50:15 ESC[1mINFO metabase.sync-databaseESC[0m :: ESC[35mFinished syncing MySQL database ‘prod-replica’. (15 s)ESC[0m


#8

Do you have any monitoring on that instance? Without access to the entire logs, there’s not much for me to go by in trying to help you.

Is this on a machine you’re running or a cloud instance? Is there anything else running?


#9

EC2 instance, dedicated to metabase. Have some monitoring setup but, again, not providing any useful info. We’ll add some more and see if there is anything that I can share with you.

Thanks.


#10

Closing the loop here. Did some troubleshooting, and still couldn’t find anything. Upgrade instance size last week and no longer seeing any crashes. Will let you know if that changes.

Thanks for the help


#11

No worries. I’m a little worried we can’t re-produce it. But hoping it was either something to do with your instance or that we can narrow it down somehow in the future.


#12

We’re seeing something similar and suspect it is OOME. Having a small instance 2cores 8GB ram, trying to download 800k rows with about 20 columns causes metabase to crash. Again no errors just the last log line being the massive select statement being printed.

Version is 0.23

Is there any recommendations on instance sizes? Feels like the serdes for CSV and json are both CPU and memory intensive.


#13

what is the result size if you limit it to 80k rows?

that sounds out of wack


#14

Works with a really small data set.


#15

Can you share how big the download file is, and how many rows? I’m guessing this is happening due to jvm memory settings but curious how big the actual file is.


#16

yup I suspected the same after some further research. I have upgraded my instance and given it 20GB via -Xmx option. no issues so far.

Is there a way to turn on warning or error logging? i.e. change the logging level?

thanks


#17

Our logs are pretty chatty, and OOM errors definitely show up in the server logs.


#18

I’m having the same issue with the 0.29. The process is still running, no info on logs. Happens pretty constantly (twice a day, I’d say), we even created a custom command on our Debian remote reserver to kill the process and restart our metabase instance. Should the problem be related to java, would the process still be running?

We also use jenkins on that server, and it doesnt present any problem, even when metabase crashes.