Hello, everyone, didn’t know about this forum so posted about my problem on Github, can you please take a look: Scheduled notifications not being sent to email · Issue #66386 · metabase/metabase · GitHub
Repeating my reply on GitHub for the record,
"java.vendor": "OpenJDK BSD Porting Team",
"os.name": "FreeBSD",
Notification emails that include dashboard contents require GraalVM. GraalVM is a binary blob component that is only supported on Windows and Linux. GraalVM is not supported on BSD OpenJDK.
You can install a Linux-compiled JDK (like Eclipse Temurin) and it works fine under modern FreeBSD versions with Linux compatibility enabled.
1 Like
Thanks, @dwhitemv Would you also happen to know why notifications (including non-dashboard ones) start working fine generally for 30-60 minutes and then stop until Java is SIGKILL-ed and Metabase restarted? This is more urgent than the dashboard sending.
An error similar to this can be seen before restarting:
2025-11-30 21:23:25,085 INFO notification.send :: Gracefully shutting down notification dispatcher with 209 pending notifications to process {mb-dispatcher-count=2}
Wild guess is the JVM is running out of memory. Are there any OutOfMemory exceptions being thrown?
Nope, there aren’t. Here’s how java is run (inside a FreeBSD jail):
java -Dhttp.proxyHost=127.0.0.2 -Dhttp.proxyPort=8080 \
-Dhttps.proxyHost=127.0.0.2 -Dhttps.proxyPort=8080 \
-Xss8m -Xms1g -Xmx8g \
-jar /path/to/metabase.jar
I too thought it was about memory and doubled the settings as shown. (they were 4m-512m-8g).
The java process currently uses 15G 6384M (SIZE RES) as per top and refuses to send mail, so memory doesn’t look like an issue.
Mail sending itself worked without a hitch before the upgrade.
I’m experimenting with Linux jails under FreeBSD using its Linux Binary Compatibility as described here: Chapter 12. Linux Binary Compatibility | FreeBSD Documentation Portal
The idea is to run Metabase there.
Long story short: I now have a decent Debian/Ubuntu distribution after running this on the host machine:
sudo jexec metabase-linux chroot /compat/ubuntu /bin/bash
root@metabase-linux:/#
Could someone explain how to use GraalVM there?
I downloaded OpenJRE 21 & GraalVM 21:
https://download.oracle.com/graalvm/21/latest/graalvm-jdk-21_linux-x64_bin.tar.gz
and unpacked them. How do I tell Metabase where GraalVM is? Does it use it as a binary?
Oh, it already includes JRE. Nice)
Ok, managed to get it up & running, but…
2025-12-01 15:02:58,221 WARN notification.send :: Error sending to channel channel/email {mb-channel_type=:channel/email, mb-handler_id=, mb-notification_id=466, mb-payload_type=:notification/dashboard, mb-quartz-job-type=SendPulse}
clojure.lang.ExceptionInfo: Error registering fonts: Metabase will not be able to send Pulses. This is a known issue with certain JVMs. See ``https://github.com/metabase/metabase/issues/7986`` and for more details.
Installing fontconfig stopped that.
I think you got everything else worked out, but one comment on your memory settings:
-Xss8m -Xms1g -Xmx8g
I would not change the stack settings (-Xss, -Xms) unless you are getting crashes due to out of stack. The default stack size is larger than those. Otherwise you can cause out of stack crashes.
1 Like
What the hell is this?) Can it be ignored? I don’t think we configured or need any kind of a SSH connection for MB to use.
2025-12-01 21:48:00,004 INFO task.sync-databases :: Starting sync task for Database 92. {mb-quartz-job-type=SyncAndAnalyzeDatabase}
2025-12-01 21:48:00,058 WARN session.ClientSessionImpl :: exceptionCaught(ClientSessionImpl[ec2-user@/127.0.0.2:22])[state=Opened] NoSuchProviderException: JCE cannot authenticate the provider BC
2025-12-01 21:48:00,058 ERROR driver.util :: Failed to connect to Database {mb-quartz-job-type=SyncAndAnalyzeDatabase}
org.apache.sshd.common.SshException: [ssh-connection]: Failed (NoSuchProviderException) to execute: JCE cannot authenticate the provider BC
Caused by: java.util.jar.JarException: The JCE Provider file:/home/metabase/metabase.jar is not signed.
Whatever Database ID 92 is has SSH tunneling enabled. Double check your database configurations. You should be able to go to the path /admin/databases/92 to find it.
1 Like
This database cannot be browsed. The error is a bit misleading, can it be attributed to SSH connection problems?
There was a problem with your question
Most of the time this is caused by an invalid selection or bad input value. Double check your inputs and retry your query.
Hide error details
Here's the full error message
The JCE Provider file:/home/metabase/metabase.jar is not signed.
It’s related, something is using an encryption level not allowed by the installed policy so Java Security is demanding the JAR be signed by the JCE CA to allow it. Nothing in Metabase should require this, though, otherwise it would ship signed.
I’ve had this error appear once, but I believe it was due to running the wrong JVM version. You’re on the right version (if your GitHub post is accurate) so I’m not sure what in there is needing protected java encryption. You appear to be outside the US so I’m not sure what restrictions are in place for your country, or what restrictions Java thinks it needs to apply for your location. You might check if there is a ~/.java directory for the user running Metabase, and if it has any files called java.security or java.policy in it, which could change the JVM behavior from the shipping defaults.
If you don’t need the ssh tunneling (looks like it was for accessing redshift with that ‘ec2-user’ on there?) then turn it off.
Digging into this a bit more, its possible multiple versions of the BouncyCastle java security provider has ended up in the classpath. One of them may be contaminated in a way that breaks the signature on it. I would check if any unusual jar files are in the default classpath and remove them.
Run java -XshowSettings:properties -version and check of java.class.path is set, and if so, what files are in that directory. Also double check that java.home points to the correct location.
It’s an important tunnel to Amazon AFAIK. Just checked this Linuxy version on the same host we run the FreeBSD Metabase (the one having problems) in case it was due to IP address of the other host not being whitelisted - but it’s the same error. There’s just ~/.java/fonts folder, nothing else. It’s the Linuxy GraalVM version in use, no other jdk installed. And this problem isn’t seen when running Metabase under pure FreeBSD. v0.57.4 in both cases.
@40000000692f1a7139058e9c 2025-12-02 20:57:11,956 ERROR models.database :: Health check: failure with error Digit Uid Staging {:id 93 :reason exception :message [ssh-connection]: Failed (NoSuchProviderException) to execute: JCE cannot authenticate the provider BC}
@40000000692f1a7139059284 org.apache.sshd.common.SshException: [ssh-connection]: Failed (NoSuchProviderException) to execute: JCE cannot authenticate the provider BC
@40000000692f1a7139059284 at org.apache.sshd.common.future.AbstractSshFuture.lambda$verifyResult$2(AbstractSshFuture.java:146)
@40000000692f1a713905a224 at org.apache.sshd.common.future.AbstractSshFuture.formatExceptionMessage(AbstractSshFuture.java:206)
@40000000692f1a713905a60c at org.apache.sshd.common.future.AbstractSshFuture.verifyResult(AbstractSshFuture.java:145)
@40000000692f1a713905a60c at org.apache.sshd.client.future.DefaultAuthFuture.verify(DefaultAuthFuture.java:56)
@40000000692f1a713905a60c at org.apache.sshd.client.future.DefaultAuthFuture.verify(DefaultAuthFuture.java:35)
@40000000692f1a713905addc at metabase.driver.sql_jdbc.connection.ssh_tunnel$start_ssh_tunnel_BANG_.invokeStatic(ssh_tunnel.clj:77)
@40000000692f1a713905addc at metabase.driver.sql_jdbc.connection.ssh_tunnel$start_ssh_tunnel_BANG_.invoke(ssh_tunnel.clj:61)
@40000000692f1a713905bd7c at metabase.driver.sql_jdbc.connection.ssh_tunnel$include_ssh_tunnel_BANG_.invokeStatic(ssh_tunnel.clj:103)
@40000000692f1a713905bd7c at metabase.driver.sql_jdbc.connection.ssh_tunnel$include_ssh_tunnel_BANG_.invoke(ssh_tunnel.clj:97)
@40000000692f1a713905c164 at metabase.driver.sql_jdbc.connection.ssh_tunnel$do_with_ssh_tunnel.invokeStatic(ssh_tunnel.clj:145)
@40000000692f1a713905c54c at metabase.driver.sql_jdbc.connection.ssh_tunnel$do_with_ssh_tunnel.invoke(ssh_tunnel.clj:141)
@40000000692f1a713905c934 at metabase.driver.sql_jdbc.connection$do_with_connection_spec_for_testing_connection.invokeStatic(connection.clj:350)
@40000000692f1a713905c934 at metabase.driver.sql_jdbc.connection$do_with_connection_spec_for_testing_connection.invoke(connection.clj:346)
@40000000692f1a713905d104 at metabase.driver.sql_jdbc.connection$can_connect_QMARK_.invokeStatic(connection.clj:378)
@40000000692f1a713905d104 at metabase.driver.sql_jdbc.connection$can_connect_QMARK_.invoke(connection.clj:374)
@40000000692f1a713905d104 at metabase.driver.sql_jdbc$fn__97987.invokeStatic(sql_jdbc.clj:56)
@40000000692f1a713905d4ec at metabase.driver.sql_jdbc$fn__97987.invoke(sql_jdbc.clj:54)
@40000000692f1a713905d8d4 at clojure.lang.MultiFn.invoke(MultiFn.java:234)
@40000000692f1a713905ec5c at metabase.warehouses.models.database$health_check_database_BANG_$fn__65004$fn__65007.invoke(database.clj:211)
@40000000692f1a713905ec5c at clojure.core$binding_conveyor_fn$fn__5844.invoke(core.clj:2047)
@40000000692f1a713905ec5c at clojure.lang.AFn.call(AFn.java:18)
@40000000692f1a713905ec5c at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
@40000000692f1a713905ec5c at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
@40000000692f1a713905f42c at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
@40000000692f1a713905f814 at java.base/java.lang.Thread.run(Thread.java:1583)
@40000000692f1a713905f814 Caused by: java.util.jar.JarException: The JCE Provider file:/home/metabase/metabase.jar is not signed.
@40000000692f1a713905f814 at java.base/javax.crypto.JarVerifier.verifySingleJar(JarVerifier.java:440)
@40000000692f1a713905ffe4 at java.base/javax.crypto.JarVerifier.verifyJars(JarVerifier.java:319)
@40000000692f1a713905ffe4 at java.base/javax.crypto.JarVerifier.verify(JarVerifier.java:263)
@40000000692f1a713905ffe4 at java.base/javax.crypto.ProviderVerifier.verify(ProviderVerifier.java:130)
@40000000692f1a713905ffe4 at java.base/javax.crypto.JceSecurity.verifyProvider(JceSecurity.java:196)
@40000000692f1a7139060f84 at java.base/javax.crypto.JceSecurity$2.apply(JceSecurity.java:222)
@40000000692f1a7139060f84 at java.base/javax.crypto.JceSecurity$2.apply(JceSecurity.java:211)
@40000000692f1a7139060f84 at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1740)
@40000000692f1a7139060f84 at java.base/javax.crypto.JceSecurity.getVerificationResult(JceSecurity.java:211)
I just unpacked the graalvm tarball, I believe java deduces its JAVA_HOME automatically. And MB is run like this (please excuse the -Xss, no time for that yet)):
/usr/local/graalvm-jdk-21.0.9+7.1/bin/java -Dhttp.proxyHost=127.0.0.2 -Dhttp.proxyPort=8080 \
-Dhttps.proxyHost=127.0.0.2 -Dhttps.proxyPort=8080 \
-Duser.timezone="Asia/Baku" \
-Xss8m -Xms1g -Xmx8g \
-jar ${metabase_dir}/metabase.jar
Digging into this a bit more, its possible multiple versions of the BouncyCastle java security provider has ended up in the classpath. One of them may be contaminated in a way that breaks the signature on it. I would check if any unusual jar files are in the default classpath and remove them.
Run
java -XshowSettings:properties -versionand check ofjava.class.pathis set, and if so, what files are in that directory. Also double check thatjava.homepoints to the correct location.
Ok will do now.
class.path is empty.
java.home points to the unpacked tarball and was deduced.
metabase@metabase-linux:/home/metabase$ /usr/local/graalvm-jdk-21.0.9+7.1/bin/java -XshowSettings:properties -version
Property settings:
file.encoding = UTF-8
file.separator = /
java.class.path =
java.class.version = 65.0
java.home = /usr/local/graalvm-jdk-21.0.9+7.1
java.io.tmpdir = /tmp
java.library.path = /usr/java/packages/lib
/usr/lib64
/lib64
/lib
/usr/lib
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 21.0.9+7-LTS-jvmci-23.1-b79
java.specification.name = Java Platform API Specification
java.specification.vendor = Oracle Corporation
java.specification.version = 21
java.vendor = Oracle Corporation
java.vendor.url = https://java.oracle.com/
java.vendor.url.bug = https://bugreport.java.com/bugreport/
java.vendor.version = Oracle GraalVM 21.0.9+7.1
java.version = 21.0.9
java.version.date = 2025-10-21
java.vm.compressedOopsMode = Non-zero disjoint base
java.vm.info = mixed mode, sharing
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name = Java Virtual Machine Specification
java.vm.specification.vendor = Oracle Corporation
java.vm.specification.version = 21
java.vm.vendor = Oracle Corporation
java.vm.version = 21.0.9+7-LTS-jvmci-23.1-b79
jdk.debug = release
jdk.internal.vm.ci.enabled = true
line.separator = \n
native.encoding = ANSI_X3.4-1968
os.arch = amd64
os.name = Linux
os.version = 5.15.0
path.separator = :
stderr.encoding = ANSI_X3.4-1968
stdout.encoding = ANSI_X3.4-1968
sun.arch.data.model = 64
sun.boot.library.path = /usr/local/graalvm-jdk-21.0.9+7.1/lib
sun.cpu.endian = little
sun.io.unicode.encoding = UnicodeLittle
sun.java.launcher = SUN_STANDARD
sun.jnu.encoding = ANSI_X3.4-1968
sun.management.compiler = HotSpot 64-Bit Tiered Compilers
user.country = US
user.dir = /home/metabase
user.home = /home/metabase
user.language = en
user.name = metabase
java version "21.0.9" 2025-10-21 LTS
Java(TM) SE Runtime Environment Oracle GraalVM 21.0.9+7.1 (build 21.0.9+7-LTS-jvmci-23.1-b79)
Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 21.0.9+7.1 (build 21.0.9+7-LTS-jvmci-23.1-b79, mixed mode, sharing)
metabase@metabase-linux:/home/metabase$
user.country = US
Speaking of which, should I change it? I didn't set it. To the actual country name (Azerbaijan)?
I tested that PostgreSQL over SSH works on BSD OpenJDK, so its not a fundamental bug like that. Even tested with a FreeBSD 15 relay host which is using very fresh OpenSSH.
I don’t see any bouncycastle files in the redshift driver.
Have you put anything in the plugins/ directory? Like say, any database drivers you didn’t end up using, or anything from previous versions?
Also, just for funsies, can you try Temurin instead of Oracle Java? In case Oracle implanted something?