Queries Timing Out After Upgrading to v0.45

gauri · March 7, 2023, 7:09pm

Hi,
I'm a total beginner with both AWS and Metabase, but yesterday I was asked to upgrade metabase on behalf of one of our clients.
We're hosting it on Elastic Beanstalk.
The steps I followed:

I took a snapshot of the application database
I followed these steps: deploying-new-versions-of-metabase-on-elastic-beanstalk

Everything worked and now I have the latest version of metabase. However, I'm having issues loading my queries and dashboards. Initially, when I opened a dashboard, some queries would work and others would time out. Now, either they all time out or they throw a connection error. But I haven't changed any configurations.

I'm pretty stumped on this as I'm very new and don't know what I need to search for.
Here's the diagnostic info.

{
  "browser-info": {
    "language": "en-US",
    "platform": "Win32",
    "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36",
    "vendor": "Google Inc."
  },
  "system-info": {
    "file.encoding": "UTF-8",
    "java.runtime.name": "OpenJDK Runtime Environment",
    "java.runtime.version": "11.0.18+10",
    "java.vendor": "Eclipse Adoptium",
    "java.vendor.url": "https://adoptium.net/",
    "java.version": "11.0.18",
    "java.vm.name": "OpenJDK 64-Bit Server VM",
    "java.vm.version": "11.0.18+10",
    "os.name": "Linux",
    "os.version": "4.14.252-195.483.amzn2.x86_64",
    "user.language": "en",
    "user.timezone": "GMT"
  },
  "metabase-info": {
    "databases": [
      "postgres"
    ],
    "hosting-env": "unknown",
    "application-database": "postgres",
    "application-database-details": {
      "database": {
        "name": "PostgreSQL",
        "version": "13.7"
      },
      "jdbc-driver": {
        "name": "PostgreSQL JDBC Driver",
        "version": "42.5.0"
      }
    },
    "run-mode": "prod",
    "version": {
      "date": "2023-02-19",
      "tag": "v0.45.3",
      "branch": "release-x.45.x",
      "hash": "070f57b"
    },
    "settings": {
      "report-timezone": "Asia/Calcutta"
    }
  }
}

Luiggi · March 7, 2023, 8:22pm

Can you check the logs of Metabase to see if there are any errors? if you see any errors in the queries they should be reflected in the logs

gauri · March 7, 2023, 9:18pm

Luiggi, thank you for responding.

I went into Admin > Torubleshooting > Logs and found this. This seems to imply that the connection to the source db is improperly setup but I doublechecked this and the credentials are correct. Should I post the full logs?

[9d2e2e09-93d6-4d6d-88c2-d20751f4cc5a] 2023-03-07T16:17:23-05:00 ERROR metabase.query-processor.middleware.catch-exceptions Error processing query: The connection attempt failed.
{:database_id 2,
 :started_at #t "2023-03-07T21:12:13.149225Z[GMT]",
 :via
 [{:status :failed,
   :class org.postgresql.util.PSQLException,
   :error "The connection attempt failed.",
   :stacktrace
   ["org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:331)"
    "org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)"
    "org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:247)"
    "org.postgresql.Driver.makeConnection(Driver.java:434)"
    "org.postgresql.Driver.connect(Driver.java:291)"
    "java.sql/java.sql.DriverManager.getConnection(Unknown Source)"
    "java.sql/java.sql.DriverManager.getConnection(Unknown Source)"
    "--> connection_pool$proxy_data_source$reify__12282.getConnection(connection_pool.clj:29)"],
   :state "08001"}
  {:status :failed,
   :class com.mchange.v2.resourcepool.CannotAcquireResourceException,
   :error "A ResourcePool could not acquire a resource from its primary factory or source.",
   :stacktrace
   ["com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1507)"
    "com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)"
    "com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)"
    "com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)"
    "com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)"
    "com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)"
    "--> driver.sql_jdbc.execute$fn__55892.invokeStatic(execute.clj:191)"
    "driver.sql_jdbc.execute$fn__55892.invoke(execute.clj:189)"
    "driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:496)"
    "driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:485)"
    "driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:493)"
    "driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:485)"
    "driver.sql_jdbc$fn__88601.invokeStatic(sql_jdbc.clj:58)"
    "driver.sql_jdbc$fn__88601.invoke(sql_jdbc.clj:56)"
    "query_processor.context$executef.invokeStatic(context.clj:59)"
    "query_processor.context$executef.invoke(context.clj:48)"
    "query_processor.context.default$default_runf.invokeStatic(default.clj:67)"
    "query_processor.context.default$default_runf.invoke(default.clj:65)"
    "query_processor.context$runf.invokeStatic(context.clj:45)"
    "query_processor.context$runf.invoke(context.clj:39)"
    "query_processor.reducible$identity_qp.invokeStatic(reducible.clj:12)"
    "query_processor.reducible$identity_qp.invoke(reducible.clj:9)"
    "query_processor.middleware.cache$maybe_return_cached_results$maybe_return_cached_results_STAR___53600.invoke(cache.clj:220)"
    "query_processor.middleware.permissions$check_query_permissions$fn__49259.invoke(permissions.clj:109)"
    "query_processor.middleware.mbql_to_native$mbql__GT_native$fn__52545.invoke(mbql_to_native.clj:23)"
    "query_processor$fn__55076$combined_post_process__55081$combined_post_process_STAR___55082.invoke(query_processor.clj:212)"
    "query_processor$fn__55076$combined_pre_process__55077$combined_pre_process_STAR___55078.invoke(query_processor.clj:209)"
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__53521$fn__53526.invoke(resolve_database_and_driver.clj:35)"
    "driver$do_with_driver.invokeStatic(driver.clj:76)"
    "driver$do_with_driver.invoke(driver.clj:72)"
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__53521.invoke(resolve_database_and_driver.clj:34)"
    "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__49525.invoke(fetch_source_query.clj:314)"
    "query_processor.middleware.store$initialize_store$fn__49715$fn__49716.invoke(store.clj:11)"
    "query_processor.store$do_with_store.invokeStatic(store.clj:45)"
    "query_processor.store$do_with_store.invoke(store.clj:39)"
    "query_processor.middleware.store$initialize_store$fn__49715.invoke(store.clj:10)"
    "query_processor.middleware.normalize_query$normalize$fn__53793.invoke(normalize_query.clj:22)"
    "query_processor.middleware.constraints$add_default_userland_constraints$fn__50803.invoke(constraints.clj:53)"
    "query_processor.middleware.process_userland_query$process_userland_query$fn__53732.invoke(process_userland_query.clj:145)"
    "query_processor.middleware.catch_exceptions$catch_exceptions$fn__54104.invoke(catch_exceptions.clj:167)"
    "query_processor.reducible$async_qp$qp_STAR___45514$thunk__45516.invoke(reducible.clj:100)"
    "query_processor.reducible$async_qp$qp_STAR___45514$fn__45518.invoke(reducible.clj:105)"]}

Luiggi · March 8, 2023, 1:33pm

yes please, does this happen after a reboot?