java.lang.OutOfMemoryError: unable to create new native thread

Get Help
1

Hallo!
I’m new to metabase so please be patient with me.

We are facing issues with our first metabase installation. And we can’t get it working.
The error we see in the logs a lot is:
java.lang.OutOfMemoryError: unable to create new native thread

Our Setup:

When starting metabase the log file shows that it is working. e.g. the sample DB is analyzed etc.
Then when you are asked to use the webinterface with localhost:3000/setup I can call that and start registering.
So far so good but then we start to get the errors. This is so heavy that simple commands on the server like “top” or “ls” are not working anymore (system echos: bash: fork: retry: Resource temporarily unavailable)
The java process is using a lot of memory and a bit of CPU and is not responding.
And then the web interface is also not reacting anymore.

I have some log data with comments here: https://pastebin.com/GT7dQVfH
I found a similar issue for a previous version of metabase (0.28.3) But no solution there: https:// github. com/metabase/metabase/issues/7243 (can’t put more then 2 links in as newbie)

2

Have you added your database? Could it be doing the initial read of the database?

3

Which version of Java are you on? If it's Java 7 you probably want to up it to Java 8 as 7 will be deprecated in Metabase v0.30 AND with Java 7 you need a command line setting to give it access to more memory.

What's the output of free -h before/after starting Metabase?

4

Hallo,
thanks for your interest.

java -version

openjdk version “1.8.0_171”
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~deb9u1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

free -h
BEFORE START: 
              total        used        free      shared  buff/cache   available
Mem:           2.0G         55M        1.8G         20M        126M          0B
Swap:            0B          0B          0B

AFTER START: (swap is 0)

                  total        used        free      shared  buff/cache   available
Mem:           2.0G        632M        1.2G         20M        185M          0B
Mem:           2.0G        660M        1.1G         20M        227M          0B

then log stopped with: 

 ERROR metabase.core :: Metabase Initialization FAILED:  unable to create new native thread
and free shows this: since then: 
    Mem:           2.0G         55M        1.7G         20M        231M          0B
5

hallo,
thanks for asking:
I think that the DB read did not happen. I’m not sure. The DB was available in the drop down with the sample data base. But I could not sync it successfully. I was not able to see the tables etc.

Since then I wasn’t successful with accessing the webinterface on port 3000. (always the same error displayed)

6

Hallo,
I’m still playing around with this.
I want to add some findings, maybe they help you to help us.

After a reboot of the server it looks always a bit better for some time.

I was able go into the webIF.
I found out that under Admin settings --> X-Rays. The default is set to “enable + HiFi performance”. I seem to have been able to disable it last time. I got no feedback that is it off last time. But now it is set to: off.
It is a bit better to work with the webIF since that… but always stops reacting after some time.
Sometimes a browser refresh helps. Sometimes I just left it all (and went away to handle my frustation :slight_smile: ) and when coming back it worked again for some time.

I was able to connect my external DB. Which ended in a freeze…

But at least it is there now and I could also navigate to the tables or get a simple chart until freeze again…

There are regular jobs running in metabase to refresh the database. This seems to happen according the log.

Typical number on jetty threads looks ok to me: 

 Jetty threads: 8/50 (4 busy, 4 idle, 0 queued)

Java “hangs” between 30-60% of CPU usage.
This is so severe that it destabilises the whole server. Sometimes I can’t ssh in. Then I can’t call simple commands… then it works again…

I have a “feeling” that we have something set wrong with the way java works… but I’m not familiar with Java stuff…

Can anyone provide with tools to check more?

7

I agree - and I’m not super familiar with it either. I just know that Java tries to build its own memory management on top of OS’s which comes with its own peculiarities.

Turns out googling for your exact error given in the title of this topic spits out “interesting” stuff (here’s just one example):

I also tried spinning up a Debian 9 on my local machine (Win 10 with Debian under WSL). “Unfortunately” it worked fine but I had more free memory - so in comparison to your free -h stats my Metabase ate around 2 gigs (sorry lost the exact numbers due to a reboot before I could write this). Seeing your numbers Java clearly tries to be clever based on what memory there’s available.

If you’re not able to easily feed your Metabase environment some more memory I think you have to go down the path of investigating/googling “unable to create new native thread” which is most likely to be the resource being depleted (first). It seems there are both OS and Java knobs to be twiddled with to see if you can get it running in your environment.

Please share your findings I’m sure they’ll be useful for others trying to get going with least amount of memory possible.

8

Hallo,
we gave up on this one and moved to another server. This solved our problem but, of course, not the root cause of this topic. It seems that the environment of the VM of the first server couldn’t handle…
I suggest to keep the topic open for now. Maybe others come across the same issue and can continue investigation.
Thanks a gain for your interest!