I often have my dashboards crash for seemingly no reason. Going into the individual query, however, you can see that the connection to MySQL (v. 8.0.34 - Percona Server) is stuck and you need to do a FLUSH HOSTS to remedy it.
This is a problem because my users do not see the detail of the problem and cannot take action. Until I intervene, Metabase is unusable.
Is there any setting I can configure in the server or on Metabase to avoid this?
Thanks
Please post the error you’re seeing, first time we hear about this
Hi Luiggi,
I only hided the server IP with XXX.XXX.XXX.XXX
[9dcbaff2-5e8b-4551-a0e8-52b3435c4004] 2023-10-10T03:01:15+02:00 ERROR org.quartz.core.ErrorLogger Job (DEFAULT.metabase.task.sync-and-analyze.job threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.sql.SQLException: Connections could not be acquired from the underlying database!]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:692)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)
at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:372)
at clojure.java.jdbc$get_connection.invoke(jdbc.clj:274)
at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:343)
at clojure.java.jdbc$get_connection.invoke(jdbc.clj:274)
at metabase.driver.sql_jdbc.sync.describe_database$describe_database.invokeStatic(describe_database.clj:163)
at metabase.driver.sql_jdbc.sync.describe_database$describe_database.invoke(describe_database.clj:160)
at metabase.driver.sql_jdbc$fn__105192.invokeStatic(sql_jdbc.clj:75)
at metabase.driver.sql_jdbc$fn__105192.invoke(sql_jdbc.clj:73)
at clojure.lang.MultiFn.invoke(MultiFn.java:234)
at metabase.sync.fetch_metadata$fn__83477$db_metadata__83482$fn__83483.invoke(fetch_metadata.clj:15)
at metabase.sync.fetch_metadata$fn__83477$db_metadata__83482.invoke(fetch_metadata.clj:12)
at metabase.sync.sync_metadata$fn__85891$sync_db_metadata_BANG___85896$fn__85897.invoke(sync_metadata.clj:58)
at metabase.sync.sync_metadata$fn__85891$sync_db_metadata_BANG___85896.invoke(sync_metadata.clj:55)
at metabase.task.sync_databases$sync_and_analyze_database_BANG_.invokeStatic(sync_databases.clj:76)
at metabase.task.sync_databases$sync_and_analyze_database_BANG_.invoke(sync_databases.clj:67)
at metabase.task.sync_databases.SyncAndAnalyzeDatabase.execute(sync_databases.clj:86)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
... 1 more
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1507)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)
... 19 more
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to HostAddress{host='XXX.XXX.XXX.XXX', port=3306, type='master'}. Host 'XXX.XXX.XXX.XXX' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:73)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:188)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1402)
at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:635)
at org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:150)
at org.mariadb.jdbc.Driver.connect(Driver.java:89)
at java.sql/java.sql.DriverManager.getConnection(Unknown Source)
at java.sql/java.sql.DriverManager.getConnection(Unknown Source)
at metabase.connection_pool$proxy_data_source$reify__18089.getConnection(connection_pool.clj:29)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:161)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:147)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:202)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1176)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1163)
at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1908)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
Caused by: java.sql.SQLException: Host 'XXX.XXX.XXX.XXX' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'
at org.mariadb.jdbc.internal.com.read.ReadInitialHandShakePacket.<init>(ReadInitialHandShakePacket.java:92)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:530)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1389)
... 14 more
Please note that this occurs with two different installation of metabase, but with two DB with the same version (Percona MySQL 8.0.34.x)
is it possible that you're trying to connect to that DB with other system in the same server and failing to do so? we haven't seen this error ever. If Metabase can connect to MySQL then it does not error at all and therefore it shouldn't get blocked