Hello Metabase community,
I'm currently having a bit of trouble making Metabase work since changing SSL CA certificates on RDS.
My setup :
- Metabase docker image : metabase/metabase:v0.49.11
- An RDS MySQL 8.0.35 as the application DB
- A switch from Certificate authority
rds-ca-2019tords-ca-rsa2048-g1
Since I changed to the new certificate authority, it seems that Metabase can't validate the SSL Handshake.
When I'm starting the Metabase docker image on a brand new RDS instance with rds-ca-2019, it works just fine, and when I use the newer rds-ca-rsa2048-g1 I get this error :
2024-05-22 16:27:26,778 ERROR metabase.core :: Metabase Initialization FAILED
clojure.lang.ExceptionInfo: Unable to connect to Metabase mysql DB. {}
at metabase.db.setup$verify_db_connection$fn__51167.invoke(setup.clj:113)
at metabase.db.setup$verify_db_connection.invokeStatic(setup.clj:111)
at metabase.db.setup$verify_db_connection.invoke(setup.clj:103)
at metabase.db.setup$setup_db_BANG_$fn__51187$fn__51188.invoke(setup.clj:163)
at metabase.util.jvm$do_with_us_locale.invokeStatic(jvm.clj:239)
at metabase.util.jvm$do_with_us_locale.invoke(jvm.clj:225)
at metabase.db.setup$setup_db_BANG_$fn__51187.invoke(setup.clj:160)
at metabase.db.setup$setup_db_BANG_.invokeStatic(setup.clj:159)
at metabase.db.setup$setup_db_BANG_.invoke(setup.clj:153)
at metabase.db$setup_db_BANG_$fn__51207.invoke(db.clj:69)
at metabase.db$setup_db_BANG_.invokeStatic(db.clj:64)
at metabase.db$setup_db_BANG_.invoke(db.clj:55)
at metabase.core$init_BANG__STAR_.invokeStatic(core.clj:116)
at metabase.core$init_BANG__STAR_.invoke(core.clj:101)
at metabase.core$init_BANG_.invokeStatic(core.clj:159)
at metabase.core$init_BANG_.invoke(core.clj:154)
at metabase.core$start_normally.invokeStatic(core.clj:171)
at metabase.core$start_normally.invoke(core.clj:165)
at metabase.core$entrypoint.invokeStatic(core.clj:204)
at metabase.core$entrypoint.doInvoke(core.clj:198)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.core$apply.invoke(core.clj:662)
at metabase.bootstrap$_main.invokeStatic(bootstrap.clj:31)
at metabase.bootstrap$_main.doInvoke(bootstrap.clj:28)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at metabase.bootstrap.main(Unknown Source)
Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:692)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)
at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:372)
at clojure.java.jdbc$get_connection.invoke(jdbc.clj:274)
at clojure.java.jdbc$db_query_with_resultset_STAR_.invokeStatic(jdbc.clj:1111)
at clojure.java.jdbc$db_query_with_resultset_STAR_.invoke(jdbc.clj:1093)
at clojure.java.jdbc$query.invokeStatic(jdbc.clj:1182)
at clojure.java.jdbc$query.invoke(jdbc.clj:1144)
at clojure.java.jdbc$query.invokeStatic(jdbc.clj:1160)
at clojure.java.jdbc$query.invoke(jdbc.clj:1144)
at metabase.driver.sql_jdbc.connection$can_connect_with_spec_QMARK_.invokeStatic(connection.clj:331)
at metabase.driver.sql_jdbc.connection$can_connect_with_spec_QMARK_.invoke(connection.clj:328)
at metabase.db.setup$verify_db_connection$fn__51167.invoke(setup.clj:111)
... 31 more
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1507)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)
... 43 more
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to HostAddress{host='test-mysql.cvfsdxgcvaax.eu-west-1.rds.amazonaws.com', port=3306, type='master'}. Host '52.51.172.48' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:73)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:188)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1403)
at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:635)
at org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:150)
at org.mariadb.jdbc.Driver.connect(Driver.java:89)
at java.sql/java.sql.DriverManager.getConnection(Unknown Source)
at java.sql/java.sql.DriverManager.getConnection(Unknown Source)
at metabase.db.data_source.DataSource.getConnection(data_source.clj:33)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:161)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:213)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1176)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1163)
at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1908)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
Caused by: java.sql.SQLException: Host '52.51.172.48' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'
at org.mariadb.jdbc.internal.com.read.ReadInitialHandShakePacket.<init>(ReadInitialHandShakePacket.java:92)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:543)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1390)
... 13 more
The log in the RDS database tells me that the host (docker image) trying to connect to the database is getting HANDSHAKE_ERRORS
I have a setup before I launch Metabase which looks like this :
#!/usr/bin/env bash
set -e
# Import RDS certificate
echo "Import New RDS certificate"
wget -O /app/eu-west-1-bundle.pem -q https://truststore.pki.rds.amazonaws.com/eu-west-1/eu-west-1-bundle.pem
keytool -importcert -alias 'Amazon RDS Bundle Certificate' -cacerts -storepass changeit -noprompt -file /app/eu-west-1-bundle.pem
keytool -importcert -alias 'Amazon RDS CA Certificate' -cacerts -storepass changeit -noprompt -file /app/certs/rds-combined-ca-bundle.pem
#Prepare env variables
export MB_JETTY_PORT=9000
export MB_DB_TYPE=mysql
export MB_DB_DBNAME=metabase
export MB_DB_USER=admin
export MB_DB_PASS="password"
export MB_DB_HOST=HOST
export MB_DB_PORT=3306
export MB_DB_CONNECTION_URI="mysql://${MB_DB_HOST}/${MB_DB_DBNAME}?useSSL=true&user=${MB_DB_USER}&password=${MB_DB_PASS}"
export JAVA_TOOL_OPTIONS="-Xmx512m"
echo $MB_DB_CONNECTION_URI
/app/run_metabase.sh
You can see that I import the CA certs using keytool, and that worked fine for the previous CA.
If I run the same docker image on a brand new RDS instance with the old CA, it works fine.
Is anything I'm doing wrong ?
I will add that I can connect to the RDS instance with SSL and the certificates imported just fine using DBeaver so the instance doesn't seem to be the problem.
Thanks a lot for the help and for the Metabase product which is great !