I have several questions and put them on a dashboard. Every time I open/refresh the dashboards, it run all queries. But for some questions, they get error
Error processing query: Error running query
from time to time. And if I enter that question, it can give answer once, and then get the same error; unstable results/failure.
Here are some error logs:
2023-07-14 18:03:45,685 DEBUG middleware.log :: GET /api/session/properties 200 21.4 ms (6 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (218 total active threads) Queries in flight: 1 (0 queued)
2023-07-14 18:03:45,814 DEBUG middleware.log :: GET /api/database 200 73.0 ms (3 DB calls) App DB connections: 2/15 Jetty threads: 5/50 (2 idle, 0 queued) (218 total active threads) Queries in flight: 1 (0 queued)
2023-07-14 18:03:45,940 DEBUG middleware.log :: GET /api/dashboard/300 200 198.5 ms (19 DB calls) App DB connections: 2/15 Jetty threads: 4/50 (3 idle, 0 queued) (218 total active threads) Queries in flight: 1 (0 queued)
2023-07-14 18:03:46,021 DEBUG middleware.log :: GET /api/collection/tree 200 273.6 ms (5 DB calls) App DB connections: 1/15 Jetty threads: 3/50 (4 idle, 0 queued) (218 total active threads) Queries in flight: 1 (0 queued)
2023-07-14 18:03:46,225 WARN sql-jdbc.connection :: Hash of database 79 details changed; marking pool invalid to reopen it
2023-07-14 18:03:46,313 WARN sql-jdbc.connection :: Hash of database 79 details changed; marking pool invalid to reopen it
2023-07-14 18:03:46,323 WARN sql-jdbc.connection :: Hash of database 83 details changed; marking pool invalid to reopen it
2023-07-14 18:03:46,524 ERROR middleware.catch-exceptions :: Error processing query: Error running query
{:database_id 81,
:started_at #t "2023-07-14T18:03:46.469208960Z[GMT]",
:json_query
{:constraints {:max-results 10000, :max-results-bare-rows 2000},
:type :native,
:middleware {:js-int-to-string? true, :ignore-cached-results? false},
:native
{:query
"SELECT \"public\".\"debezium_signals\".\"id\" AS \"id\", \"public\".\"debezium_signals\".\"type\" AS \"type\", \"public\".\"debezium_signals\".\"data\" AS \"data\"\nFROM \"public\".\"debezium_signals\"\nLIMIT 20",
:template-tags {},
:collection "debezium_signals"},
:database 81,
:parameters [],
:async? true,
:cache-ttl nil},
:status :failed,
:class java.lang.NullPointerException,
:stacktrace [nil],
:card_id 2404,
:context :dashboard,
:error nil,
:row_count 0,
:running_time 0,
:data {:rows [], :cols []}}
2023-07-14 18:03:46,526 DEBUG middleware.log :: POST /api/dashboard/300/dashcard/2310/card/2404/query 202 [ASYNC: completed] 402.1 ms (12 DB calls) App DB connections: 1/15 Jetty threads: 3/50 (5 idle, 0 queued) (225 total active threads) Queries in flight: 6 (0 queued)
2023-07-14 18:03:46,844 DEBUG middleware.log :: POST /api/dashboard/300/dashcard/2306/card/2397/query 202 [ASYNC: completed] 739.4 ms (18 DB calls) App DB connections: 1/15 Jetty threads: 2/50 (6 idle, 0 queued) (225 total active threads) Queries in flight: 5 (0 queued); postgres DB 79 connections: 0/0 (1 threads blocked)
2023-07-14 18:03:46,926 DEBUG middleware.log :: POST /api/dashboard/300/dashcard/2313/card/2407/query 202 [ASYNC: completed] 818.6 ms (20 DB calls) App DB connections: 2/15 Jetty threads: 2/50 (6 idle, 0 queued) (225 total active threads) Queries in flight: 4 (0 queued); postgres DB 83 connections: 0/0 (0 threads blocked)
2023-07-14 18:03:47,022 DEBUG middleware.log :: POST /api/dashboard/300/dashcard/2305/card/2394/query 202 [ASYNC: completed] 925.3 ms (20 DB calls) App DB connections: 2/15 Jetty threads: 2/50 (6 idle, 0 queued) (225 total active threads) Queries in flight: 3 (0 queued); postgres DB 79 connections: 0/0 (0 threads blocked)
2023-07-14 18:03:47,725 DEBUG middleware.log :: POST /api/dashboard/300/dashcard/2307/card/2396/query 202 [ASYNC: completed] 1.6 s (18 DB calls) App DB connections: 2/15 Jetty threads: 2/50 (6 idle, 0 queued) (226 total active threads) Queries in flight: 2 (0 queued); postgres DB 79 connections: 0/0 (1 threads blocked)
2023-07-14 18:03:47,869 DEBUG middleware.log :: POST /api/dashboard/300/dashcard/2303/card/2392/query 202 [ASYNC: completed] 1.8 s (19 DB calls) App DB connections: 2/15 Jetty threads: 2/50 (6 idle, 0 queued) (226 total active threads) Queries in flight: 1 (0 queued); postgres DB 79 connections: 0/0 (3 threads blocked)
Basically, I'm using AWS IAM-based access to connect to a RDS proxy on AWS. Metabase didn't provide that option, so I added that feature in my customized version. Previously, the connection to the same RDS proxy with username + password worked well. And my IAM-based access to other databases are working fine. This error message didn't provide much information on how I could debug the case.