After much troubleshooting we found out there is a default 20m query timeout built into the code. If you would like to see a feature to make this configurable, please go vote here: Create an ENV Variable that allows you to setup when a query would timeout · Issue #41685 · metabase/metabase (github.com)
(def ^:dynamic ^Long *query-timeout-ms*
"Maximum amount of time query is allowed to run, in ms."
;; I don't know if these numbers make sense, but my thinking is we want to enable (somewhat) long-running queries on
;; prod but for test and dev purposes we want to fail faster because it usually means I broke something in the QP
;; code
(u/minutes->ms
(if config/is-prod?
20
3)))