I’m running metabase (v57) on AWS ECS. Over time (currently a few days) the VM runs out of memory and dies, being automatically restarted by the ECS scheduler:
I am running a 3GB/1vCPU fargate runner with the env var JAVA_OPTS set to-Xmx3g. Can anyone suggest better settings? Is this normal/expected?
You need to leave some memory for the system. 3GB is too small for Metabase, especially if you use dashboard notifications. GraalVM is a massive memory pig. You really want more like 5-6GB of Java heap on a 8GB host. There is ~1GB in non-heap overhead you have to account for.
Increase the runner memory size and Java heap limit (-Xmx) to match.
On a platform where you don’t/can’t use GraalVM, you can get away with a smaller heap. I ran Metabase in 256MB of heap until I recently had to bump it to 512MB.
Thanks for the reply. I’m just using metabase/metabase:latest in the ECS task definition - I cant really see anything in the docs about using GraalVM? Whats the easiest way of running Metabase with a low memory overhead? 6GB seems kind of crazy?!
Java apps are memory beasts.
GraalVM is a component of the JVM. It’s used by Metabase’s notification code to create the notification email/Slack post content for dashboard notifications (Metabase running frontend JavaScript code in the backend).
As I said, I can run Metabase lean, but I’m running on FreeBSD where GraalVM isn’t supported, and I don’t use dashboard notifications. I suppose you could build an OS image & task def to use FreeBSD instead of a Linux distro for your task, but that sounds like a lot of work just to pinch pennies on AWS.
If we’re talking about AWS costs, are you running on Fargate? The markup on Fargate far outstrips the minor cost increase from using a larger instance on ECS EC2.
Ah understood - thanks thats v informative. The docs are pretty light on “minimum” memory requirements. Our install is pretty simple and only connects to 1 DB. I’ve no idea what to set memory at in that case…
The guidance in the docs seems out of date from my experience. It suggests 2GB RAM and 1 CPU core as a minimum. I would suggest, not as a minimum but as a starter server for general purpose use, 8GB RAM with a 5-6GB heap and 2-4 cores to give room to grow.
I’ve followed that but the memory just gets consumed then the docker container dies along with the Fargate instanec
send us the heap dump when the container dies. Also, as previously mentioned: you need to assign to Metabase 80-85% of the total memory of the host, since it needs that for the non-heap space and host stuff
I’ve got it running at 2.25GB on a 3GB instance now. Will see how that works out. Not sure I can store a heap dump as Fargate instances are ephemeral.
Is a 3GB host/2.25GB to the VM not sufficient for a fairly small install with < 10 users and no more then 2 or 3 concurrent users?
This is what top looks like running on a Rocky Linux 9 VM with 3GB of physical RAM, 2 CPU cores, and a JVM with a 2250m heap limit, after mailing myself the Ecommerce Insights dashboard from the Sample Database. It’s a little tight, but no OOM exceptions yet.
top - 21:03:12 up 52 min, 2 users, load average: 0.20, 0.26, 0.23
Tasks: 186 total, 1 running, 185 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.2 us, 0.0 sy, 0.0 ni, 99.5 id, 0.0 wa, 0.2 hi, 0.0 si, 0.0 st
MiB Mem : 2773.7 total, 111.7 free, 2395.8 used, 452.9 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 377.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
65996 metabase 20 0 5269272 2.0g 30508 S 0.0 72.8 2:02.28 java