V0.43.0- Search Bar not working and Home screen not loading

pa101 · May 16, 2022, 8:58pm

Hi , We have upgraded from v0.41.7 to v0.43. We are facing following issues post upgrade:

Search bar not working. Attached screenshot for the error
Home screen not loading
Database takes 50s to 60s to load when writing a new query

pa101 · May 17, 2022, 4:54am

2022-05-17 09:46:40,826 ERROR middleware.log :: GET /api/setup/admin_checklist 500 1.5 mins (8 DB calls)
{:via
[{:type java.sql.SQLException,
:message "An SQLException was provoked by the following failure: java.lang.InterruptedException",
:at [com.mchange.v2.sql.SqlUtils toSQLException "SqlUtils.java" 118]}
{:type java.lang.InterruptedException, :at [java.lang.Object wait nil -2]}],
:trace
[[java.lang.Object wait nil -2]
[com.mchange.v2.resourcepool.BasicResourcePool awaitAvailable "BasicResourcePool.java" 1503]
[com.mchange.v2.resourcepool.BasicResourcePool prelimCheckoutResource "BasicResourcePool.java" 644]
[com.mchange.v2.resourcepool.BasicResourcePool checkoutResource "BasicResourcePool.java" 554]
[com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool
checkoutAndMarkConnectionInUse
"C3P0PooledConnectionPool.java"
758]
[com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool checkoutPooledConnection "C3P0PooledConnectionPool.java" 685]
[com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource getConnection "AbstractPoolBackedDataSource.java" 140]
[metabase.db.setup$reify__34959 getConnection "setup.clj" 170]
[clojure.java.jdbc$get_connection invokeStatic "jdbc.clj" 372]
[clojure.java.jdbc$get_connection invoke "jdbc.clj" 274]
[clojure.java.jdbc$db_query_with_resultset_STAR_ invokeStatic "jdbc.clj" 1111]
[clojure.java.jdbc$db_query_with_resultset_STAR_ invoke "jdbc.clj" 1093]
[clojure.java.jdbc$query invokeStatic "jdbc.clj" 1182]

pa101 · May 17, 2022, 4:55am

022-05-17 09:46:40,854 ERROR metabase.server :: Unexpected Exception in API request handler
org.eclipse.jetty.io.EofException
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:825)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:555)
at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:922)
at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:999)

metabase-gr8 · May 17, 2022, 5:05am

I work in the same team where we are facing this issue. I have also noticed a strange thing after upgrade. This is a k8 setup and the pods are restarting once in an hour with the above error logs. Have you noticed this behaviour before? How do we fix this?

metabase-gr8 · May 17, 2022, 5:08am

We have noticed one more error after upgrade. We upgraded bigquery-cloud-sdk before upgrading to metabase v0.43 with json file authentication for datasets. But still we see below logs for 'bigquery-cloud-sdk' having oauth style auth. I am confused by this statement - "this database must therefore be updated". What is meant by this update? Do we need to re-add the service account json? (edited)

[737c95a8-409c-48cd-987b-6363470cf750] 2022-05-17T10:12:44+05:30 ERROR metabase.driver.bigquery-cloud-sdk Database ID 6, which was migrated from the legacy :bigquery driver to :bigquery-cloud-sdk, has one or more OAuth style authentication scheme parameters saved to db-details, which cannot be automatically migrated to the newer driver (since it *requires* service-account-json intead); this database must therefore be updated by an administrator (by adding a service-account-json) before sync and queries will work again
[737c95a8-409c-48cd-987b-6363470cf750] 2022-05-17T10:12:44+05:30 ERROR metabase.driver.bigquery-cloud-sdk Database ID 6, which was migrated from the legacy :bigquery driver to :bigquery-cloud-sdk, has one or more OAuth style authentication scheme parameters saved to db-details, which cannot be automatically migrated to the newer driver (since it *requires* service-account-json intead); this database must therefore be updated by an administrator (by adding a service-account-json) before sync and queries will work again
[737c95a8-409c-48cd-987b-6363470cf750] 2022-05-17T10:12:44+05:30 ERROR metabase.driver.bigquery-cloud-sdk Database ID 6, which was migrated from the legacy :bigquery driver to :bigquery-cloud-sdk, has one or more OAuth style authentication scheme parameters saved to db-details, which cannot be automatically migrated to the newer driver (since it *requires* service-account-json intead); this database must therefore be updated by an administrator (by adding a service-account-json) before sync and queries will work again
[737c95a8-409c-48cd-987b-6363470cf750] 2022-05-17T10:12:45+05:30 ERROR metabase.query-processor.middleware.catch-exceptions Error processing query: null
{:database_id 6,
 :started_at #t "2022-05-17T10:12:44.493939+05:30[Asia/Kolkata]",
 :via
 [{:status :failed,
   :class com.google.cloud.bigquery.BigQueryException,
   :error "Table \"grouptag_post_activity\" must be qualified with a dataset (e.g. dataset.table).",
   :stacktrace
   ["com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)"
    "com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.queryRpc(HttpBigQueryRpc.java:652)"
    "com.google.cloud.bigquery.BigQueryImpl$35.call(BigQueryImpl.java:1278)"
    "com.google.cloud.bigquery.BigQueryImpl$35.call(BigQueryImpl.java:1275)"
    "com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)"
    "com.google.cloud.RetryHelper.run(RetryHelper.java:76)"
    "com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)"
    "com.google.cloud.bigquery.BigQueryImpl.queryRpc(BigQueryImpl.java:1274)"
    "com.google.cloud.bigquery.BigQueryImpl.query(BigQueryImpl.java:1262)"
    "--> driver.bigquery_cloud_sdk$execute_bigquery$fn__81034.invoke(bigquery_cloud_sdk.clj:193)"]}
  {:status :failed,
   :class java.util.concurrent.ExecutionException,
   :error
   "com.google.cloud.bigquery.BigQueryException: Table \"grouptag_post_activity\" must be qualified with a dataset (e.g. dataset.table).",
   :stacktrace

flamber · May 17, 2022, 8:36am

Hi @pa101
Upgrade to 0.43.1: https://github.com/metabase/metabase/releases/latest
The search error is from your reverse-proxy, not Metabase.
You are like seeing https://github.com/metabase/metabase/issues/22636, which was mitigated in 0.43.1

@metabase-gr8 Yes, update the service account. Seems like something wasn't done correctly.

pa101 · May 17, 2022, 5:05pm

Hi @flamber- We upgraded to v0.43.1 and also updated the service account

Search bar is working now
Database is loading quickly when writing new query

However-

pods restarting frequently is still an issue
Home page is still not loading
38%20AM1558×786 18.5 KB

pa101 · May 17, 2022, 5:06pm

[11b3e05a-08c9-4cee-8770-5915acea7ed4] 2022-05-17T22:16:26+05:30 ERROR metabase.driver.bigquery-cloud-sdk Database ID 7, which was migrated from the legacy :bigquery driver to :bigquery-cloud-sdk, has one or more OAuth style authentication scheme parameters saved to db-details, which cannot be automatically migrated to the newer driver (since it *requires* service-account-json intead); this database must therefore be updated by an administrator (by adding a service-account-json) before sync and queries will work again
[11b3e05a-08c9-4cee-8770-5915acea7ed4] 2022-05-17T22:16:26+05:30 ERROR metabase.driver.bigquery-cloud-sdk Database ID 7, which was migrated from the legacy :bigquery driver to :bigquery-cloud-sdk, has one or more OAuth style authentication scheme parameters saved to db-details, which cannot be automatically migrated to the newer driver (since it *requires* service-account-json intead); this database must therefore be updated by an administrator (by adding a service-account-json) before sync and queries will work again
[11b3e05a-08c9-4cee-8770-5915acea7ed4] 2022-05-17T22:16:26+05:30 ERROR metabase.driver.bigquery-cloud-sdk Database ID 7, which was migrated from the legacy :bigquery driver to :bigquery-cloud-sdk, has one or more OAuth style authentication scheme parameters saved to db-details, which cannot be automatically migrated to the newer driver (since it *requires* service-account-json intead); this database must therefore be updated by an administrator (by adding a service-account-json) before sync and queries will work again
[11b3e05a-08c9-4cee-8770-5915acea7ed4] 2022-05-17T22:16:27+05:30 ERROR metabase.query-processor.middleware.catch-exceptions Error processing query: null
{:database_id 7,
 :started_at #t "2022-05-17T22:16:26.834814+05:30[Asia/Kolkata]",
 :via
 [{:status :failed,
   :class com.google.cloud.bigquery.BigQueryException,
   :error "Table \"report_profile_actions\" must be qualified with a dataset (e.g. dataset.table).",
   :stacktrace
   ["com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)"
    "com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.queryRpc(HttpBigQueryRpc.java:652)"
    "com.google.cloud.bigquery.BigQueryImpl$35.call(BigQueryImpl.java:1278)"
    "com.google.cloud.bigquery.BigQueryImpl$35.call(BigQueryImpl.java:1275)"
    "com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)"
    "com.google.cloud.RetryHelper.run(RetryHelper.java:76)"
    "com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)"
    "com.google.cloud.bigquery.BigQueryImpl.queryRpc(BigQueryImpl.java:1274)"
    "com.google.cloud.bigquery.BigQueryImpl.query(BigQueryImpl.java:1262)"
    "--> driver.bigquery_cloud_sdk$execute_bigquery$fn__81053.invoke(bigquery_cloud_sdk.clj:193)"]}
  {:status :failed,
   :class java.util.concurrent.ExecutionException,
   :error
   "com.google.cloud.bigquery.BigQueryException: Table \"report_profile_actions\" must be qualified with a dataset (e.g. dataset.table).",
   :stacktrace

flamber · May 17, 2022, 5:58pm

@pa101 Did you perhaps switch back and forth between the old and the new BigQuery driver, before you upgraded to 0.43.1?

Seems like you might have messed up the metabase_database.details

metabase-gr8 · May 17, 2022, 6:28pm

Yup, We did that. How do we fix this metabase_database.details? We didn't change anything in db.

flamber · May 17, 2022, 6:36pm

@metabase-gr8 Since I don't know which state you have gotten into, then it's likely easier to create a new database connection and then check the differences in metabase_database.details and fix the old existing database details, and then delete the new unused database. Make sure you have backups first.

metabase-gr8 · May 17, 2022, 6:38pm

I don't see this new database option in admin panel of metabase.

flamber · May 17, 2022, 6:39pm

@metabase-gr8 Admin > Databases > Create databases.

metabase-gr8 · May 17, 2022, 6:47pm

Are we talking about big query database or the default mysql db that we update for metabase internal usage?

metabase-gr8 · May 17, 2022, 6:48pm

Unfortunately, I can't see this option in Admin Panel.

flamber · May 17, 2022, 6:49pm

@metabase-gr8 Click the big blue "Add database" and create the BigQuery again. Then check the application database table metabase_database.details and figure out what the difference is, and then fix the old connection, so it starts working again.

metabase-gr8 · May 17, 2022, 6:51pm

Cool. Let us give it a try and see how it works out.

Thanks Flamber