Metabase gets stuck with queries in flight

Hello there!

We have been trying to use metabase for the last month. It gets stuck from time to time and it does not load any question, query or dashboard.
We are using v0.43.4, but the same error happened with the older versions.

Queries in flight normally get resolved in seconds and the counter goes to 0, but regularly every hour it seems to get stuck with queries in flight with not a clear error. It will reach a top number of 50 queries in flight.. I have been trying to debug it for the last couple of weeks and I cannot find an answer.

[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T11:59:30+01:00 INFO metabase.query-processor.middleware.cache Query took 1.8 s to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T11:59:30+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/24/card/26/query 202 [ASYNC: completed] 2.0 s (27 DB calls) App DB connections: 1/15 Jetty threads: 2/50 (6 idle, 0 queued) (119 total active threads) Queries in flight: 5 (0 queued); redshift DB 2 connections: 12/12 (0 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T11:59:30+01:00 INFO metabase.query-processor.card Question's average execution duration is 582.0 ms; using 'magic' TTL of 6.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T11:59:31+01:00 INFO metabase.query-processor.middleware.cache Query took 1.4 s to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T11:59:31+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/25/card/27/query 202 [ASYNC: completed] 1.5 s (27 DB calls) App DB connections: 1/15 Jetty threads: 2/50 (6 idle, 0 queued) (119 total active threads) Queries in flight: 5 (0 queued); redshift DB 2 connections: 12/13 (0 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T11:59:31+01:00 INFO metabase.query-processor.middleware.cache Query took 64.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T11:59:31+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/17/card/19/query 202 [ASYNC: completed] 195.0 ms (36 DB calls) App DB connections: 1/15 Jetty threads: 2/50 (6 idle, 0 queued) (118 total active threads) Queries in flight: 4 (0 queued); redshift DB 2 connections: 12/13 (0 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T11:59:31+01:00 INFO metabase.query-processor.middleware.cache Query took 1.0 s to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T11:59:31+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/74/card/66/query 202 [ASYNC: completed] 1.2 s (36 DB calls) App DB connections: 1/15 Jetty threads: 2/50 (6 idle, 0 queued) (119 total active threads) Queries in flight: 3 (0 queued); redshift DB 2 connections: 11/13 (0 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T11:59:31+01:00 INFO metabase.query-processor.middleware.cache Query took 1.3 s to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T11:59:31+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/36/card/40/query 202 [ASYNC: completed] 1.4 s (33 DB calls) App DB connections: 1/15 Jetty threads: 2/50 (6 idle, 0 queued) (119 total active threads) Queries in flight: 2 (0 queued); redshift DB 2 connections: 12/13 (0 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T11:59:42+01:00 INFO metabase.query-processor.middleware.cache Query took 22.2 s to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T11:59:42+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/37/card/41/query 202 [ASYNC: completed] 22.3 s (36 DB calls) App DB connections: 1/15 Jetty threads: 2/50 (6 idle, 0 queued) (118 total active threads) Queries in flight: 1 (0 queued); redshift DB 2 connections: 10/12 (0 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T11:59:46+01:00 INFO metabase.query-processor.middleware.cache Query took 16.9 s to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T11:59:46+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/37/card/41/query 202 [ASYNC: completed] 17.1 s (36 DB calls) App DB connections: 1/15 Jetty threads: 2/50 (6 idle, 0 queued) (118 total active threads) Queries in flight: 0 (0 queued); redshift DB 2 connections: 11/13 (0 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:00:00+01:00 INFO metabase.task.send-pulses Sending scheduled pulses...
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:00+01:00 DEBUG metabase.server.middleware.log GET /api/user/current 200 20.3 ms (9 DB calls) App DB connections: 0/15 Jetty threads: 4/50 (3 idle, 0 queued) (111 total active threads) Queries in flight: 0 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:00+01:00 DEBUG metabase.server.middleware.log GET /api/session/properties 200 26.4 ms (4 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (3 idle, 0 queued) (111 total active threads) Queries in flight: 0 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:00+01:00 DEBUG metabase.server.middleware.log GET /api/collection/root 200 1.9 ms (2 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (111 total active threads) Queries in flight: 0 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:00+01:00 DEBUG metabase.server.middleware.log GET /api/collection/tree 200 17.6 ms (13 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (111 total active threads) Queries in flight: 0 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:00+01:00 DEBUG metabase.server.middleware.log GET /api/bookmark 200 2.5 ms (1 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (111 total active threads) Queries in flight: 0 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:00+01:00 DEBUG metabase.server.middleware.log GET /api/database 200 10.2 ms (3 DB calls) App DB connections: 2/15 Jetty threads: 6/50 (1 idle, 0 queued) (111 total active threads) Queries in flight: 0 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:00+01:00 DEBUG metabase.server.middleware.log GET /api/activity/recent_views 200 40.2 ms (8 DB calls) App DB connections: 1/15 Jetty threads: 4/50 (2 idle, 0 queued) (111 total active threads) Queries in flight: 0 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:00+01:00 DEBUG metabase.server.middleware.log GET /api/activity/popular_items 200 53.6 ms (10 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (3 idle, 0 queued) (111 total active threads) Queries in flight: 0 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:02+01:00 DEBUG metabase.server.middleware.log GET /api/dashboard/2 200 124.3 ms (28 DB calls) App DB connections: 3/15 Jetty threads: 3/50 (4 idle, 0 queued) (111 total active threads) Queries in flight: 0 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:02+01:00 DEBUG metabase.server.middleware.log GET /api/field/2277 200 6.3 ms (6 DB calls) App DB connections: 2/15 Jetty threads: 7/50 (0 idle, 0 queued) (111 total active threads) Queries in flight: 0 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:02+01:00 DEBUG metabase.server.middleware.log GET /api/field/2258 200 6.8 ms (6 DB calls) App DB connections: 2/15 Jetty threads: 7/50 (0 idle, 0 queued) (111 total active threads) Queries in flight: 0 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:02+01:00 DEBUG metabase.server.middleware.log GET /api/field/2264 200 11.7 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 7/50 (0 idle, 0 queued) (111 total active threads) Queries in flight: 0 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:02+01:00 DEBUG metabase.server.middleware.log GET /api/field/2278 200 9.9 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 7/50 (0 idle, 0 queued) (111 total active threads) Queries in flight: 0 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:02+01:00 DEBUG metabase.server.middleware.log GET /api/field/2262 200 19.1 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 6/50 (0 idle, 0 queued) (115 total active threads) Queries in flight: 0 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:02+01:00 DEBUG metabase.server.middleware.log GET /api/collection/2 200 4.0 ms (3 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (115 total active threads) Queries in flight: 0 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 INFO metabase.query-processor.card Question's average execution duration is 765.0 ms; using 'magic' TTL of 8.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 INFO metabase.query-processor.card Question's average execution duration is 739.0 ms; using 'magic' TTL of 7.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 INFO metabase.query-processor.card Question's average execution duration is 316.0 ms; using 'magic' TTL of 3.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 INFO metabase.query-processor.card Question's average execution duration is 873.0 ms; using 'magic' TTL of 9.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 INFO metabase.query-processor.card Question's average execution duration is 730.0 ms; using 'magic' TTL of 7.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 INFO metabase.query-processor.card Question's average execution duration is 785.0 ms; using 'magic' TTL of 8.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 INFO metabase.query-processor.middleware.cache Query took 225.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/26/card/28/query 202 [ASYNC: completed] 507.0 ms (18 DB calls) App DB connections: 2/15 Jetty threads: 2/50 (6 idle, 0 queued) (117 total active threads) Queries in flight: 5 (0 queued); redshift DB 2 connections: 7/7 (0 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 INFO metabase.query-processor.middleware.cache Query took 281.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 INFO metabase.query-processor.middleware.cache Query took 170.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/21/card/23/query 202 [ASYNC: completed] 672.5 ms (36 DB calls) App DB connections: 1/15 Jetty threads: 3/50 (6 idle, 0 queued) (119 total active threads) Queries in flight: 4 (0 queued); redshift DB 2 connections: 9/9 (3 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/22/card/24/query 202 [ASYNC: completed] 772.0 ms (36 DB calls) App DB connections: 1/15 Jetty threads: 3/50 (6 idle, 0 queued) (118 total active threads) Queries in flight: 3 (0 queued); redshift DB 2 connections: 8/8 (0 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 INFO metabase.query-processor.middleware.cache Query took 318.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 INFO metabase.query-processor.card Question's average execution duration is 409.0 ms; using 'magic' TTL of 4.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/23/card/25/query 202 [ASYNC: completed] 861.8 ms (36 DB calls) App DB connections: 1/15 Jetty threads: 5/50 (3 idle, 0 queued) (117 total active threads) Queries in flight: 2 (0 queued); redshift DB 2 connections: 8/8 (3 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 INFO metabase.query-processor.middleware.cache Query took 488.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/19/card/21/query 202 [ASYNC: completed] 989.3 ms (36 DB calls) App DB connections: 0/15 Jetty threads: 5/50 (4 idle, 0 queued) (117 total active threads) Queries in flight: 2 (0 queued); redshift DB 2 connections: 8/8 (2 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 INFO metabase.query-processor.middleware.cache Query took 615.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/20/card/22/query 202 [ASYNC: completed] 1.1 s (36 DB calls) App DB connections: 2/15 Jetty threads: 6/50 (3 idle, 0 queued) (117 total active threads) Queries in flight: 1 (0 queued); redshift DB 2 connections: 8/8 (1 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 INFO metabase.query-processor.card Question's average execution duration is 460.0 ms; using 'magic' TTL of 5.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 INFO metabase.query-processor.middleware.cache Query took 76.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 INFO metabase.query-processor.card Question's average execution duration is 469.0 ms; using 'magic' TTL of 5.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:03+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/38/card/43/query 202 [ASYNC: completed] 437.1 ms (36 DB calls) App DB connections: 2/15 Jetty threads: 4/50 (4 idle, 0 queued) (118 total active threads) Queries in flight: 2 (0 queued); redshift DB 2 connections: 8/13 (0 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 INFO metabase.query-processor.card Question's average execution duration is 11.9 s; using 'magic' TTL of 2.0 mins 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 INFO metabase.query-processor.card Question's average execution duration is 229.0 ms; using 'magic' TTL of 2.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 INFO metabase.query-processor.middleware.cache Query took 86.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/25/card/27/query 202 [ASYNC: completed] 426.9 ms (27 DB calls) App DB connections: 2/15 Jetty threads: 4/50 (4 idle, 0 queued) (117 total active threads) Queries in flight: 3 (0 queued); redshift DB 2 connections: 7/13 (0 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 INFO metabase.query-processor.middleware.cache Query took 105.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 INFO metabase.query-processor.card Question's average execution duration is 513.0 ms; using 'magic' TTL of 5.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/24/card/26/query 202 [ASYNC: completed] 517.8 ms (27 DB calls) App DB connections: 5/15 Jetty threads: 4/50 (5 idle, 0 queued) (117 total active threads) Queries in flight: 3 (0 queued); redshift DB 2 connections: 8/13 (0 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 INFO metabase.query-processor.card Question's average execution duration is 685.0 ms; using 'magic' TTL of 7.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 INFO metabase.query-processor.card Question's average execution duration is 229.0 ms; using 'magic' TTL of 2.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 INFO metabase.query-processor.middleware.cache Query took 68.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/18/card/20/query 202 [ASYNC: completed] 477.2 ms (36 DB calls) App DB connections: 1/15 Jetty threads: 2/50 (6 idle, 0 queued) (119 total active threads) Queries in flight: 4 (0 queued); redshift DB 2 connections: 8/13 (0 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 INFO metabase.query-processor.middleware.cache Query took 70.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/74/card/66/query 202 [ASYNC: completed] 398.1 ms (36 DB calls) App DB connections: 2/15 Jetty threads: 2/50 (6 idle, 0 queued) (119 total active threads) Queries in flight: 3 (0 queued); redshift DB 2 connections: 10/13 (0 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 INFO metabase.query-processor.middleware.cache Query took 29.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/17/card/19/query 202 [ASYNC: completed] 258.9 ms (36 DB calls) App DB connections: 2/15 Jetty threads: 4/50 (5 idle, 0 queued) (118 total active threads) Queries in flight: 2 (0 queued); redshift DB 2 connections: 10/13 (0 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 INFO metabase.query-processor.card Question's average execution duration is 3.4 s; using 'magic' TTL of 34.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 INFO metabase.query-processor.middleware.cache Query took 300.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 DEBUG metabase.server.middleware.log POST /api/dashboard/2/dashcard/36/card/40/query 202 [ASYNC: completed] 589.7 ms (33 DB calls) App DB connections: 2/15 Jetty threads: 5/50 (3 idle, 0 queued) (117 total active threads) Queries in flight: 2 (0 queued); redshift DB 2 connections: 9/13 (0 threads blocked)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 INFO metabase.query-processor.card Question's average execution duration is 3.4 s; using 'magic' TTL of 34.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 INFO metabase.query-processor.card Question's average execution duration is 3.8 s; using 'magic' TTL of 38.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 INFO metabase.query-processor.card Question's average execution duration is 3.4 s; using 'magic' TTL of 34.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 INFO metabase.query-processor.card Question's average execution duration is 3.5 s; using 'magic' TTL of 35.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:04+01:00 INFO metabase.query-processor.card Question's average execution duration is 3.4 s; using 'magic' TTL of 34.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:05+01:00 INFO metabase.query-processor.middleware.cache Query took 1.0 s to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:05+01:00 INFO metabase.query-processor.middleware.cache Query took 1.1 s to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:05+01:00 INFO metabase.query-processor.middleware.cache Query took 1.2 s to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:06+01:00 INFO metabase.query-processor.middleware.cache Query took 1.3 s to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:06+01:00 INFO metabase.query-processor.middleware.cache Query took 1.4 s to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:06+01:00 INFO metabase.query-processor.middleware.cache Query took 1.2 s to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:23+01:00 INFO metabase.query-processor.card Question's average execution duration is 1.6 s; using 'magic' TTL of 16.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:24+01:00 INFO metabase.query-processor.middleware.cache Query took 229.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:55+01:00 DEBUG metabase.server.middleware.log GET /api/user/current 200 9.1 ms (9 DB calls) App DB connections: 0/15 Jetty threads: 4/50 (5 idle, 0 queued) (119 total active threads) Queries in flight: 8 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:55+01:00 DEBUG metabase.server.middleware.log GET /api/session/properties 200 16.1 ms (4 DB calls) App DB connections: 0/15 Jetty threads: 4/50 (5 idle, 0 queued) (119 total active threads) Queries in flight: 8 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:55+01:00 DEBUG metabase.server.middleware.log GET /api/collection/root 200 2.1 ms (2 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (6 idle, 0 queued) (119 total active threads) Queries in flight: 8 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:55+01:00 DEBUG metabase.server.middleware.log GET /api/bookmark 200 4.2 ms (1 DB calls) App DB connections: 3/15 Jetty threads: 6/50 (3 idle, 0 queued) (119 total active threads) Queries in flight: 8 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:55+01:00 DEBUG metabase.server.middleware.log GET /api/database 200 19.4 ms (3 DB calls) App DB connections: 0/15 Jetty threads: 6/50 (3 idle, 0 queued) (119 total active threads) Queries in flight: 8 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:55+01:00 DEBUG metabase.server.middleware.log GET /api/collection/tree 200 23.3 ms (13 DB calls) App DB connections: 0/15 Jetty threads: 5/50 (3 idle, 0 queued) (119 total active threads) Queries in flight: 8 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:55+01:00 DEBUG metabase.server.middleware.log GET /api/activity/popular_items 200 54.9 ms (10 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (5 idle, 0 queued) (119 total active threads) Queries in flight: 8 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:55+01:00 DEBUG metabase.server.middleware.log GET /api/activity/recent_views 200 43.1 ms (8 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (5 idle, 0 queued) (119 total active threads) Queries in flight: 8 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:57+01:00 DEBUG metabase.server.middleware.log GET /api/dashboard/2 200 108.1 ms (28 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (6 idle, 0 queued) (119 total active threads) Queries in flight: 8 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:57+01:00 DEBUG metabase.server.middleware.log GET /api/field/2277 200 25.9 ms (6 DB calls) App DB connections: 2/15 Jetty threads: 7/50 (2 idle, 0 queued) (119 total active threads) Queries in flight: 8 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:57+01:00 DEBUG metabase.server.middleware.log GET /api/field/2278 200 25.3 ms (6 DB calls) App DB connections: 2/15 Jetty threads: 7/50 (2 idle, 0 queued) (119 total active threads) Queries in flight: 8 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:57+01:00 DEBUG metabase.server.middleware.log GET /api/field/2262 200 27.9 ms (6 DB calls) App DB connections: 1/15 Jetty threads: 7/50 (2 idle, 0 queued) (119 total active threads) Queries in flight: 8 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:57+01:00 DEBUG metabase.server.middleware.log GET /api/field/2258 200 28.0 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 7/50 (2 idle, 0 queued) (119 total active threads) Queries in flight: 8 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:57+01:00 DEBUG metabase.server.middleware.log GET /api/field/2264 200 24.6 ms (6 DB calls) App DB connections: 1/15 Jetty threads: 7/50 (2 idle, 0 queued) (119 total active threads) Queries in flight: 8 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:57+01:00 DEBUG metabase.server.middleware.log GET /api/collection/2 200 3.1 ms (3 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (6 idle, 0 queued) (120 total active threads) Queries in flight: 8 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:57+01:00 INFO metabase.query-processor.card Question's average execution duration is 310.0 ms; using 'magic' TTL of 3.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:57+01:00 INFO metabase.query-processor.card Question's average execution duration is 727.0 ms; using 'magic' TTL of 7.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:57+01:00 INFO metabase.query-processor.card Question's average execution duration is 855.0 ms; using 'magic' TTL of 9.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:57+01:00 INFO metabase.query-processor.card Question's average execution duration is 733.0 ms; using 'magic' TTL of 7.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:57+01:00 INFO metabase.query-processor.middleware.cache Query took 48.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:57+01:00 INFO metabase.query-processor.card Question's average execution duration is 732.0 ms; using 'magic' TTL of 7.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:57+01:00 INFO metabase.query-processor.card Question's average execution duration is 696.0 ms; using 'magic' TTL of 7.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:57+01:00 INFO metabase.query-processor.middleware.cache Query took 52.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:57+01:00 INFO metabase.query-processor.middleware.cache Query took 56.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:57+01:00 INFO metabase.query-processor.middleware.cache Query took 51.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:58+01:00 INFO metabase.query-processor.middleware.cache Query took 147.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:06:58+01:00 INFO metabase.query-processor.middleware.cache Query took 125.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:07:00+01:00 INFO metabase.query-processor.card Question's average execution duration is 3.2 s; using 'magic' TTL of 32.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:07:00+01:00 INFO metabase.query-processor.card Question's average execution duration is 3.2 s; using 'magic' TTL of 32.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:07:00+01:00 INFO metabase.query-processor.card Question's average execution duration is 3.5 s; using 'magic' TTL of 35.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:07:00+01:00 INFO metabase.query-processor.card Question's average execution duration is 3.2 s; using 'magic' TTL of 32.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:07:00+01:00 INFO metabase.query-processor.card Question's average execution duration is 3.2 s; using 'magic' TTL of 32.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:07:00+01:00 INFO metabase.query-processor.card Question's average execution duration is 3.2 s; using 'magic' TTL of 32.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:07:00+01:00 INFO metabase.query-processor.middleware.cache Query took 64.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:07:00+01:00 INFO metabase.query-processor.middleware.cache Query took 105.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:07:00+01:00 INFO metabase.query-processor.middleware.cache Query took 112.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:07:00+01:00 INFO metabase.query-processor.middleware.cache Query took 163.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:07:00+01:00 INFO metabase.query-processor.middleware.cache Query took 88.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:07:00+01:00 INFO metabase.query-processor.middleware.cache Query took 205.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:16:04+01:00 WARN org.eclipse.jetty.util.thread.QueuedThreadPool 
java.lang.IllegalStateException: COMPLETED
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:915)
	at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:502)
	at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:421)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:567)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:352)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:16:04+01:00 WARN org.eclipse.jetty.util.thread.QueuedThreadPool 
java.lang.IllegalStateException: COMPLETED
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:915)
	at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:502)
	at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:421)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:567)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:352)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:16:04+01:00 WARN org.eclipse.jetty.util.thread.QueuedThreadPool 
java.lang.IllegalStateException: COMPLETED
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:915)
	at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:502)
	at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:421)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:567)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:352)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:16:04+01:00 WARN org.eclipse.jetty.util.thread.QueuedThreadPool 
java.lang.IllegalStateException: COMPLETED
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:915)
	at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:502)
	at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:421)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:567)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:352)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:16:04+01:00 WARN org.eclipse.jetty.util.thread.QueuedThreadPool 
java.lang.IllegalStateException: COMPLETED
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:915)
	at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:502)
	at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:421)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:567)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:352)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:16:04+01:00 WARN org.eclipse.jetty.util.thread.QueuedThreadPool 
java.lang.IllegalStateException: COMPLETED
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:915)
	at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:502)
	at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:421)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:567)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:352)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:16:23+01:00 WARN org.eclipse.jetty.util.thread.QueuedThreadPool 
java.lang.IllegalStateException: COMPLETED
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:915)
	at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:502)
	at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:421)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:567)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:352)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:16:57+01:00 WARN org.eclipse.jetty.util.thread.QueuedThreadPool 
java.lang.IllegalStateException: COMPLETED
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:915)
	at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:502)
	at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:421)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:567)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:352)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:16:57+01:00 WARN org.eclipse.jetty.util.thread.QueuedThreadPool 
java.lang.IllegalStateException: COMPLETED
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:915)
	at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:502)
	at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:421)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:567)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:352)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:16:57+01:00 WARN org.eclipse.jetty.util.thread.QueuedThreadPool 
java.lang.IllegalStateException: COMPLETED
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:915)
	at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:502)
	at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:421)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:567)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:352)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:16:57+01:00 WARN org.eclipse.jetty.util.thread.QueuedThreadPool 
java.lang.IllegalStateException: COMPLETED
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:915)
	at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:502)
	at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:421)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:567)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:352)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:16:57+01:00 WARN org.eclipse.jetty.util.thread.QueuedThreadPool 
java.lang.IllegalStateException: COMPLETED
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:915)
	at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:502)
	at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:421)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:567)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:352)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:16:57+01:00 WARN org.eclipse.jetty.util.thread.QueuedThreadPool 
java.lang.IllegalStateException: COMPLETED
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:915)
	at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:502)
	at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:421)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:567)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:352)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:17:00+01:00 WARN org.eclipse.jetty.util.thread.QueuedThreadPool 
java.lang.IllegalStateException: COMPLETED
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:915)
	at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:502)
	at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:421)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:567)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:352)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:17:00+01:00 WARN org.eclipse.jetty.util.thread.QueuedThreadPool 
java.lang.IllegalStateException: COMPLETED
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:915)
	at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:502)
	at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:421)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:567)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:352)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:17:00+01:00 WARN org.eclipse.jetty.util.thread.QueuedThreadPool 
java.lang.IllegalStateException: COMPLETED
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:915)
	at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:502)
	at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:421)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:567)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:352)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:17:00+01:00 WARN org.eclipse.jetty.util.thread.QueuedThreadPool 
java.lang.IllegalStateException: COMPLETED
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:915)
	at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:502)
	at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:421)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:567)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:352)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:07+01:00 DEBUG metabase.server.middleware.log GET /api/user/current 200 13.6 ms (10 DB calls) App DB connections: 0/15 Jetty threads: 4/50 (3 idle, 0 queued) (112 total active threads) Queries in flight: 20 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:07+01:00 DEBUG metabase.server.middleware.log GET /api/session/properties 200 19.1 ms (4 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (3 idle, 0 queued) (112 total active threads) Queries in flight: 20 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:07+01:00 DEBUG metabase.server.middleware.log GET /api/database 200 7.1 ms (3 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (112 total active threads) Queries in flight: 20 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:32+01:00 DEBUG metabase.server.middleware.log GET /api/user/current 200 22.9 ms (9 DB calls) App DB connections: 0/15 Jetty threads: 5/50 (2 idle, 0 queued) (112 total active threads) Queries in flight: 20 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:32+01:00 DEBUG metabase.server.middleware.log GET /api/session/properties 200 33.8 ms (4 DB calls) App DB connections: 0/15 Jetty threads: 4/50 (2 idle, 0 queued) (114 total active threads) Queries in flight: 20 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:32+01:00 DEBUG metabase.server.middleware.log GET /api/collection/root 200 3.1 ms (2 DB calls) App DB connections: 1/15 Jetty threads: 4/50 (3 idle, 0 queued) (114 total active threads) Queries in flight: 20 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:32+01:00 DEBUG metabase.server.middleware.log GET /api/collection/tree 200 23.4 ms (13 DB calls) App DB connections: 1/15 Jetty threads: 4/50 (3 idle, 0 queued) (114 total active threads) Queries in flight: 20 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:32+01:00 DEBUG metabase.server.middleware.log GET /api/bookmark 200 7.4 ms (1 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (3 idle, 0 queued) (114 total active threads) Queries in flight: 20 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:32+01:00 DEBUG metabase.server.middleware.log GET /api/database 200 7.1 ms (3 DB calls) App DB connections: 0/15 Jetty threads: 4/50 (3 idle, 0 queued) (114 total active threads) Queries in flight: 20 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:32+01:00 DEBUG metabase.server.middleware.log GET /api/dashboard/2 200 100.5 ms (28 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (3 idle, 0 queued) (114 total active threads) Queries in flight: 20 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:32+01:00 DEBUG metabase.server.middleware.log GET /api/field/2262 200 12.8 ms (6 DB calls) App DB connections: 5/15 Jetty threads: 7/50 (0 idle, 0 queued) (115 total active threads) Queries in flight: 20 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:32+01:00 DEBUG metabase.server.middleware.log GET /api/field/2277 200 12.8 ms (6 DB calls) App DB connections: 2/15 Jetty threads: 7/50 (0 idle, 0 queued) (115 total active threads) Queries in flight: 20 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:32+01:00 DEBUG metabase.server.middleware.log GET /api/field/2264 200 24.0 ms (6 DB calls) App DB connections: 1/15 Jetty threads: 6/50 (1 idle, 0 queued) (115 total active threads) Queries in flight: 20 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:32+01:00 DEBUG metabase.server.middleware.log GET /api/field/2258 200 19.6 ms (6 DB calls) App DB connections: 1/15 Jetty threads: 6/50 (1 idle, 0 queued) (115 total active threads) Queries in flight: 20 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:32+01:00 DEBUG metabase.server.middleware.log GET /api/field/2278 200 12.2 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (115 total active threads) Queries in flight: 20 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:32+01:00 DEBUG metabase.server.middleware.log GET /api/collection/2 200 2.9 ms (3 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (5 idle, 0 queued) (115 total active threads) Queries in flight: 20 (0 queued)
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:33+01:00 INFO metabase.query-processor.card Question's average execution duration is 1.3 s; using 'magic' TTL of 13.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:33+01:00 INFO metabase.query-processor.card Question's average execution duration is 1.0 s; using 'magic' TTL of 10.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:33+01:00 INFO metabase.query-processor.card Question's average execution duration is 1.2 s; using 'magic' TTL of 12.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:33+01:00 INFO metabase.query-processor.card Question's average execution duration is 286.0 ms; using 'magic' TTL of 3.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:33+01:00 INFO metabase.query-processor.card Question's average execution duration is 1.2 s; using 'magic' TTL of 12.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:33+01:00 INFO metabase.query-processor.card Question's average execution duration is 891.0 ms; using 'magic' TTL of 9.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:33+01:00 INFO metabase.query-processor.middleware.cache Query took 247.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:33+01:00 INFO metabase.query-processor.middleware.cache Query took 406.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:33+01:00 INFO metabase.query-processor.middleware.cache Query took 406.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:33+01:00 INFO metabase.query-processor.middleware.cache Query took 411.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:34+01:00 INFO metabase.query-processor.middleware.cache Query took 716.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:18:34+01:00 INFO metabase.query-processor.middleware.cache Query took 783.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:19:33+01:00 INFO metabase.query-processor.card Question's average execution duration is 1.6 s; using 'magic' TTL of 16.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:19:33+01:00 INFO metabase.query-processor.card Question's average execution duration is 1.1 s; using 'magic' TTL of 11.0 s 💾
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:19:34+01:00 INFO metabase.query-processor.middleware.cache Query took 327.0 ms to run; minimum for cache eligibility is 60.0 s
[53e44685-abeb-4439-9222-096ca135ca29] 2022-08-12T12:19:34+01:00 INFO metabase.query-processor.middleware.cache Query took 224.0 ms to run; minimum for cache eligibility is 60.0 s

Hi @jorge_qv
Post "Diagnostic Info" from Admin > Troubleshooting.
Are you using SSH to connect to the database? Try 0.43.4.2
Try looking on the database to see current connections.
Have a look at these topics: https://discourse.metabase.com/search?q="https%3A%2F%2Fgithub.com%2Fmetabase%2Fmetabase%2Fissues%2F8679"

Hi @flamber !
Thank you for being so fast.
We are using SSH to connect. I have just deployed v0.43.4.2.
7 active connections, it does not look like to exceed those 7.

I will let you know how it perfoms next week.
If there is something else you want me to try, let me know :slight_smile:

@jorge_qv You shouldn't see more problems then after 0.43.4.2. If you are limited to 7 connections on the database, then you should set that in Metabase too, just to avoid Metabase having to try to ask for more connections:
https://www.metabase.com/docs/latest/operations-guide/environment-variables.html#mb_jdbc_data_warehouse_max_connection_pool_size

Hi @flamber,
Metabase has been working with no issues for 2 days, that is better than before. But Metabase got stuck today again. My guess is again the query in flight.
Thank you for your previous help.

[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:06:55+01:00 DEBUG metabase.server.middleware.log GET /api/card/78 200 10.7 ms (10 DB calls) App DB connections: 1/15 Jetty threads: 3/50 (4 idle, 0 queued) (113 total active threads) Queries in flight: 0 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:06:55+01:00 DEBUG metabase.server.middleware.log GET /api/alert/question/78 200 1.3 ms (1 DB calls) App DB connections: 1/15 Jetty threads: 3/50 (4 idle, 0 queued) (113 total active threads) Queries in flight: 0 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:06:55+01:00 INFO metabase.query-processor.card Question's average execution duration is 338.0 ms; using 'magic' TTL of 3.0 s 💾
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:07:02+01:00 DEBUG metabase.server.middleware.log GET /api/database/2/autocomplete_suggestions 200 26.6 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (114 total active threads) Queries in flight: 1 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:07:02+01:00 DEBUG metabase.server.middleware.log GET /api/database/2/autocomplete_suggestions 200 29.5 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (115 total active threads) Queries in flight: 1 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:07:02+01:00 DEBUG metabase.server.middleware.log GET /api/database/2/autocomplete_suggestions 200 26.6 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (115 total active threads) Queries in flight: 1 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:07:02+01:00 DEBUG metabase.server.middleware.log GET /api/database/2/autocomplete_suggestions 200 27.6 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (115 total active threads) Queries in flight: 1 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:07:02+01:00 DEBUG metabase.server.middleware.log GET /api/database/2/autocomplete_suggestions 200 27.2 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (115 total active threads) Queries in flight: 1 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:07:03+01:00 DEBUG metabase.server.middleware.log GET /api/database/2/autocomplete_suggestions 200 27.4 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (115 total active threads) Queries in flight: 1 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:07:03+01:00 DEBUG metabase.server.middleware.log GET /api/database/2/autocomplete_suggestions 200 26.4 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (115 total active threads) Queries in flight: 1 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:07:03+01:00 DEBUG metabase.server.middleware.log GET /api/database/2/autocomplete_suggestions 200 58.5 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (115 total active threads) Queries in flight: 1 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:07:03+01:00 DEBUG metabase.server.middleware.log GET /api/database/2/autocomplete_suggestions 200 40.4 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (115 total active threads) Queries in flight: 1 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:07:03+01:00 DEBUG metabase.server.middleware.log GET /api/database/2/autocomplete_suggestions 200 48.8 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (3 idle, 0 queued) (115 total active threads) Queries in flight: 1 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:07:03+01:00 DEBUG metabase.server.middleware.log GET /api/database/2/autocomplete_suggestions 200 33.4 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (115 total active threads) Queries in flight: 1 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:07:03+01:00 DEBUG metabase.server.middleware.log GET /api/database/2/autocomplete_suggestions 200 25.9 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (115 total active threads) Queries in flight: 1 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:07:04+01:00 DEBUG metabase.server.middleware.log GET /api/database/2/autocomplete_suggestions 200 28.2 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (115 total active threads) Queries in flight: 1 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:07:04+01:00 DEBUG metabase.server.middleware.log GET /api/database/2/autocomplete_suggestions 200 41.4 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (115 total active threads) Queries in flight: 1 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:07:04+01:00 DEBUG metabase.server.middleware.log GET /api/database/2/autocomplete_suggestions 200 24.8 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (115 total active threads) Queries in flight: 1 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:07:04+01:00 DEBUG metabase.server.middleware.log GET /api/database/2/autocomplete_suggestions 200 26.5 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (115 total active threads) Queries in flight: 1 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:07:04+01:00 DEBUG metabase.server.middleware.log GET /api/database/2/autocomplete_suggestions 200 36.9 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (115 total active threads) Queries in flight: 1 (0 queued)
[5b2def86-c51d-4f83-9aaf-8d9d9937b587] 2022-08-17T17:07:06+01:00 DEBUG metabase.server.middleware.log GET /api/database/2/autocomplete_suggestions 200 18.2 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (115 total active threads) Queries in flight: 1 (0 queued)

@jorge_qv Your logs doesn't indicate the problem. It takes a lot of logs to figure out where problems originated.
Disable caching temporarily.

Hi @flamber !

I have just disabled caching. I will post any updates here.

Thank you for your help.

Hey Flamber!
I hope you are doing well,
We are still using 0.43.4.2. Is there any newer version that support correctly the SSH tunnel?

Have a good day!

@jorge_qv Upgrade to 44.4 or newer: https://github.com/metabase/metabase/issues/25112