Queries time out at 15 seconds despite settings

Get Help
1

We are running Metabase v0.54.6.x on a Kubernetes cluster and have no problems with short running queries to an MSSQL database.
With more complex queries that take longer to run, we are getting the "Your question took too long" around the 15 second mark, regardless of the query.

DB QUERY TIMEOUT is set to 20 min
DB CONNECTION TIMEOUT is set to 2 minutes
JETTY_IDLE_TIMEOUT set to 2 minutes
DB_CONNECTION_IDLE_TIMEOUT set to 2 minutes

Error from logs (truncated):
[{:status :failed,
:class clojure.lang.ExceptionInfo,
:error "Error executing query: The query was canceled.",
:stacktrace
["--> driver.sql_jdbc.execute$execute_reducible_query$fn__89272$fn__89273.invoke(execute.clj:747)"
"driver.sql_jdbc.execute$execute_reducible_query$fn__89272.invoke(execute.clj:744)"
"driver.sql_jdbc.execute$fn__89061$fn__89062.invoke(execute.clj:399)"
"driver.sql_jdbc.execute$do_with_resolved_connection89031__89032.invokeStatic(execute.clj:339)"
"driver.sql_jdbc.execute$do_with_resolved_connection89031__89032.invoke(execute.clj:322)"
"driver.sql_jdbc.execute$fn__89061.invokeStatic(execute.clj:393)"
"driver.sql_jdbc.execute$fn__89061.invoke(execute.clj:391)"
"driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:738)"
"driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:724)"
"driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:735)"
"driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:724)"
"driver.sql_jdbc$fn__91385.invokeStatic(sql_jdbc.clj:79)"
"driver.sql_jdbc$fn__91385.invoke(sql_jdbc.clj:77)"
"query_processor.pipeline$STAR_execute_STAR.invokeStatic(pipeline.clj:52)"

We have tried the following:

  1. Standing up a pod in the same cluster with mssql-tools loaded to run the same query. The query ran over a minute but succeeded. The same network egress exists for both pods, so I'm doubtful that the problem is a downstream NLB
  2. Connecting Metabase to another database in a different location and running other complex queries. We also saw the same 15 second limitation when connecting to the different database.

What other setting could be adjusted to resolve this? Is this a bug?

2

We discovered the problem, and it was not Metabase. We are using a Gloo gateway to manage traffic to data sources and the gateway had the 15 second timeout.

1 Like