Metabase 0.43.3 taking too much time to submit queries to database

Hi,

In the last couple of weeks we have made two changes to our metabase instance.

  1. We've moved away from elastic beanstack and installed metabse on a ECS with fargate
  2. We've upgraded the metabase instance to 0.43.3 from 0.42.X at the same time, when we've done this migration.

The initial couple of days were smooth, from the last couple of days we are facing issues with the Middleware taking too much time to submit request on downstream databases.

Till yesterday this issue was mainly with redshift connections, But from today, the issue started with MySQL as well.

Do you think this issue is because of the upgrade to 0.43.3 or because of the change in the backend from elastic beanstalk to ECS?

I've also checked the application database to see if there are any tables with huge loads, there is still 85% free memory on my postgres instance. below is the screenshot of the output of this query

with tbl as (select table_schema, table_name from information_schema.tables where table_name not like 'pg_%' and table_schema != 'information_schema' and table_schema in ('public')) --change 'public' if Metabase appdb is in another schema
select
  table_schema
  , table_name
  , pg_total_relation_size('"'||table_schema||'"."'||table_name||'"') size_total
  , pg_relation_size('"'||table_schema||'"."'||table_name||'"') size_tbl
  , (xpath('/row/c/text()', query_to_xml(format('select count(*) as c from %I.%I', table_schema, table_name), false, true, '')))[1]::text::bigint count_rows
from tbl
order by 5 desc nulls last;

Below is my diagnostic info.
{
"browser-info": {
"language": "en-IN",
"platform": "MacIntel",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36",
"vendor": "Google Inc."
},
"system-info": {
"file.encoding": "UTF-8",
"java.runtime.name": "OpenJDK Runtime Environment",
"java.runtime.version": "11.0.15+10",
"java.vendor": "Eclipse Adoptium",
"java.vendor.url": "https://adoptium.net/",
"java.version": "11.0.15",
"java.vm.name": "OpenJDK 64-Bit Server VM",
"java.vm.version": "11.0.15+10",
"os.name": "Linux",
"os.version": "4.14.281-212.502.amzn2.x86_64",
"user.language": "en",
"user.timezone": "GMT"
},
"metabase-info": {
"databases": [
"mysql",
"postgres",
"redshift",
"googleanalytics",
"bigquery-cloud-sdk"
],
"hosting-env": "unknown",
"application-database": "postgres",
"application-database-details": {
"database": {
"name": "PostgreSQL",
"version": "12.8"
},
"jdbc-driver": {
"name": "PostgreSQL JDBC Driver",
"version": "42.3.2"
}
},
"run-mode": "prod",
"version": {
"date": "2022-06-13",
"tag": "v0.43.3",
"branch": "release-x.43.x",
"hash": "c9c7ef0"
},
"settings": {
"report-timezone": "Asia/Kolkata"
}
}
}

this is one of the log, that seems to be breaking the public dashboards as well.

This specific dashboard is running on redshift

[4fbaecb2-bce8-40df-8c81-e5973b854cbc] 2022-07-08T13:51:21+05:30 DEBUG metabase.server.middleware.log GET /api/public/dashboard/088097b7-1e2b-4317-8410-0ccb56810ebc/dashcard/214/card/425 202 [ASYNC: completed] 30.5 s (16 DB calls) App DB connections: 2/15 Jetty threads: 2/50 (12 idle, 0 queued) (264 total active threads) Queries in flight: 50 (22 queued); redshift DB 11 connections: 15/15 (98 threads blocked)

hi @pldeepesh, it seems that your Metabase instance is queuing queries. This might be because of https://github.com/metabase/metabase/issues/23452, but please let us know if that's not the case