"Client closed connection prematurely" error

Get Help
1

I am running Metabase inside a GKE cluster using the official Docker image and am running into an error with a question on BigQuery taking too long to load. The ingress for the pod is an nginx reverse proxy and a global external application load balancer on GCP, both of which have already been configured to have 300 second timeouts (proxy_connect_timeout, proxy_send_timeout, and proxy_read_timeout for nginx; network endpoint group timeout on the load balancer). The logs in the nginx reverse proxy are showing an error code of 499. The web client does not appear to ever receive a response so there is no error code in the network tab. The query has been tested on BigQuery itself and takes about 50 seconds to run, far shorter than I would expect this issue to happen for.

Log excerpt:

[d4186a3a-8312-4d11-9c14-0d636b459902] 2024-06-21T12:25:55-05:00 ERROR metabase.query-processor.streaming Client closed connection prematurely
org.eclipse.jetty.io.EofException
   at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:116 undefined)
   at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422 undefined)
   at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:275 undefined)
   at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:254 undefined)
   at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:386 undefined)
   at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:832 undefined)
   at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:243 undefined)
   at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224 undefined)
   at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:589 undefined)
   at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:1051 undefined)
   at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:1123 undefined)
   at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:270 undefined)
   at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:254 undefined)
   at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:736 undefined)
   at java.base/java.util.zip.DeflaterOutputStream.flush(Unknown Source)
   at metabase.async.streaming_response$delay_output_stream$fn__52170.invoke(streaming_response.clj:122 undefined)
   at metabase.async.streaming_response.proxy$java.io.OutputStream$ff19274a.flush(Unknown Source)
   at java.base/sun.nio.cs.StreamEncoder.implFlush(Unknown Source)
   at java.base/sun.nio.cs.StreamEncoder.flush(Unknown Source)
   at java.base/java.io.OutputStreamWriter.flush(Unknown Source)
   at java.base/java.io.BufferedWriter.flush(Unknown Source)
   at metabase.query_processor.streaming.json$fn$reify__67245.finish_BANG_(json.clj:115 undefined)
   at metabase.query_processor.streaming$streaming_result_fn$result__68031$fn__68032.invoke(streaming.clj:144 undefined)
   at metabase.query_processor.streaming$streaming_result_fn$result__68031.invoke(streaming.clj:143 undefined)
   at metabase.query_processor.pipeline$_STAR_reduce_STAR_.invokeStatic(pipeline.clj:72 undefined)
   at metabase.query_processor.pipeline$_STAR_reduce_STAR_.invoke(pipeline.clj:49 undefined)
   at metabase.query_processor.pipeline$_STAR_run_STAR_$respond__56626.invoke(pipeline.clj:86 undefined)
   at metabase.driver.bigquery_cloud_sdk$post_process_native.invokeStatic(bigquery_cloud_sdk.clj:425 undefined)
   at metabase.driver.bigquery_cloud_sdk$post_process_native.invoke(bigquery_cloud_sdk.clj:408 undefined)
   at metabase.driver.bigquery_cloud_sdk$_STAR_process_native_STAR_$thunk__123670.invoke(bigquery_cloud_sdk.clj:441 undefined)
   at metabase.driver.bigquery_cloud_sdk$_STAR_process_native_STAR_.invokeStatic(bigquery_cloud_sdk.clj:450 undefined)
   at metabase.driver.bigquery_cloud_sdk$_STAR_process_native_STAR_.invoke(bigquery_cloud_sdk.clj:430 undefined)
   at metabase.driver.bigquery_cloud_sdk$fn__123677.invokeStatic(bigquery_cloud_sdk.clj:471 undefined)
   at metabase.driver.bigquery_cloud_sdk$fn__123677.invoke(bigquery_cloud_sdk.clj:463 undefined)
   at clojure.lang.MultiFn.invoke(MultiFn.java:244 undefined)
   at metabase.query_processor.pipeline$_STAR_execute_STAR_.invokeStatic(pipeline.clj:47 undefined)
   at metabase.query_processor.pipeline$_STAR_execute_STAR_.invoke(pipeline.clj:34 undefined)
   at metabase.query_processor.pipeline$_STAR_run_STAR_.invokeStatic(pipeline.clj:88 undefined)
   at metabase.query_processor.pipeline$_STAR_run_STAR_.invoke(pipeline.clj:81 undefined)
   at metabase.query_processor.execute$run.invokeStatic(execute.clj:60 undefined)
   at metabase.query_processor.execute$run.invoke(execute.clj:54 undefined)
   at metabase.query_processor.middleware.update_used_cards$update_used_cards_BANG_$fn__69571.invoke(update_used_cards.clj:40 undefined)
   at metabase.query_processor.execute$add_native_form_to_result_metadata$fn__69580.invoke(execute.clj:23 undefined)
   at metabase.query_processor.execute$add_preprocessed_query_to_result_metadata_for_userland_query$fn__69585.invoke(execute.clj:34 undefined)
   at metabase.query_processor.middleware.cache$maybe_return_cached_results$maybe_return_cached_results_STAR___69557.invoke(cache.clj:242 undefined)
   at metabase.query_processor.middleware.permissions$check_query_permissions$fn__63714.invoke(permissions.clj:118 undefined)
   at metabase.query_processor.middleware.enterprise$check_download_permissions_middleware$fn__64290.invoke(enterprise.clj:51 undefined)
   at metabase.query_processor.middleware.enterprise$maybe_apply_column_level_perms_check_middleware$fn__64300.invoke(enterprise.clj:64 undefined)
   at metabase.query_processor.execute$execute$fn__69612.invoke(execute.clj:92 undefined)
   at metabase.query_processor.setup$do_with_qp_setup.invokeStatic(setup.clj:225 undefined)
   at metabase.query_processor.setup$do_with_qp_setup.invoke(setup.clj:216 undefined)
   at metabase.query_processor.execute$execute.invokeStatic(execute.clj:91 undefined)
   at metabase.query_processor.execute$execute.invoke(execute.clj:87 undefined)
   at metabase.query_processor$process_query_STAR__STAR_.invokeStatic(query_processor.clj:47 undefined)
   at metabase.query_processor$process_query_STAR__STAR_.invoke(query_processor.clj:43 undefined)
   at metabase.query_processor.middleware.enterprise$fn__64317$handle_audit_app_internal_queries__64318$fn__64320.invoke(enterprise.clj:96 undefined)
   at metabase.query_processor.middleware.enterprise$handle_audit_app_internal_queries_middleware$fn__64328.invoke(enterprise.clj:103 undefined)
   at metabase.query_processor.middleware.process_userland_query$process_userland_query_middleware$fn__75430.invoke(process_userland_query.clj:182 undefined)
   at metabase.query_processor.middleware.catch_exceptions$catch_exceptions$fn__75499.invoke(catch_exceptions.clj:128 undefined)
   at metabase.query_processor$process_query$fn__75536.invoke(query_processor.clj:78 undefined)
   at metabase.query_processor.setup$do_with_canceled_chan$fn__64732.invoke(setup.clj:189 undefined)
   at metabase.query_processor.setup$do_with_database_local_settings$fn__64727.invoke(setup.clj:181 undefined)
   at metabase.query_processor.setup$do_with_driver$fn__64722$fn__64723.invoke(setup.clj:166 undefined)
   at metabase.driver$do_with_driver.invokeStatic(driver.clj:104 undefined)
   at metabase.driver$do_with_driver.invoke(driver.clj:99 undefined)
   at metabase.query_processor.setup$do_with_driver$fn__64722.invoke(setup.clj:165 undefined)
   at metabase.query_processor.setup$do_with_metadata_provider$fn__64715$fn__64718.invoke(setup.clj:151 undefined)
   at metabase.query_processor.store$do_with_metadata_provider.invokeStatic(store.clj:171 undefined)
   at metabase.query_processor.store$do_with_metadata_provider.invoke(store.clj:151 undefined)
   at metabase.query_processor.store$do_with_metadata_provider.invokeStatic(store.clj:160 undefined)
   at metabase.query_processor.store$do_with_metadata_provider.invoke(store.clj:151 undefined)
   at metabase.query_processor.setup$do_with_metadata_provider$fn__64715.invoke(setup.clj:150 undefined)
   at metabase.query_processor.setup$do_with_resolved_database$fn__64709.invoke(setup.clj:128 undefined)
   at metabase.query_processor.setup$do_with_qp_setup.invokeStatic(setup.clj:232 undefined)
   at metabase.query_processor.setup$do_with_qp_setup.invoke(setup.clj:216 undefined)
   at metabase.query_processor$process_query.invokeStatic(query_processor.clj:76 undefined)
   at metabase.query_processor$process_query.invoke(query_processor.clj:69 undefined)
   at metabase.query_processor.card$process_query_for_card_default_qp.invokeStatic(card.clj:170 undefined)
   at metabase.query_processor.card$process_query_for_card_default_qp.invoke(card.clj:166 undefined)
   at metabase.query_processor.card$process_query_for_card_default_run_fn$fn__81495$fn__81496.invoke(card.clj:177 undefined)
   at metabase.query_processor.streaming$_streaming_response$fn__68043$fn__68044$fn__68045.invoke(streaming.clj:175 undefined)
   at metabase.query_processor.streaming$_streaming_response$fn__68043$fn__68044.invoke(streaming.clj:174 undefined)
   at metabase.query_processor.streaming$do_with_streaming_rff.invokeStatic(streaming.clj:165 undefined)
   at metabase.query_processor.streaming$do_with_streaming_rff.invoke(streaming.clj:152 undefined)
   at metabase.query_processor.streaming$_streaming_response$fn__68043.invoke(streaming.clj:171 undefined)
   at clojure.lang.AFn.applyToHelper(AFn.java:156 undefined)
   at clojure.lang.AFn.applyTo(AFn.java:144 undefined)
   at clojure.core$apply.invokeStatic(core.clj:667 undefined)
   at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1990 undefined)
   at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1990 undefined)
   at clojure.lang.RestFn.applyTo(RestFn.java:142 undefined)
   at clojure.core$apply.invokeStatic(core.clj:671 undefined)
   at clojure.core$bound_fn_STAR_$fn__5818.doInvoke(core.clj:2020 undefined)
   at clojure.lang.RestFn.invoke(RestFn.java:421 undefined)
   at metabase.async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:68 undefined)
   at metabase.async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:66 undefined)
   at metabase.async.streaming_response$do_f_async$task__52153.invoke(streaming_response.clj:87 undefined)
   at clojure.lang.AFn.run(AFn.java:22 undefined)
   at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
   at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Broken pipe,
   at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
   at java.base/sun.nio.ch.SocketDispatcher.writev(Unknown Source)
   at java.base/sun.nio.ch.IOUtil.write(Unknown Source)
   at java.base/sun.nio.ch.IOUtil.write(Unknown Source)
   at java.base/sun.nio.ch.SocketChannelImpl.write(Unknown Source)
   at java.base/java.nio.channels.SocketChannel.write(Unknown Source)
   at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:110 undefined)
   ... 102 more
[d4186a3a-8312-4d11-9c14-0d636b459902] 2024-06-21T12:25:55-05:00 DEBUG metabase.server.middleware.log POST /api/card/2/query 202 [ASYNC: canceled] 59.3 s (26 DB calls) App DB connections: 1/13 Jetty threads: 2/50 (5 idle, 0 queued) (72 total active threads) Queries in flight: 0 (0 queued) {:metabase-user-id 11}

Diagnostic Info:

{
  "browser-info": {
    "language": "en-US",
    "platform": "Linux x86_64",
    "userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0",
    "vendor": ""
  },
  "system-info": {
    "file.encoding": "UTF-8",
    "java.runtime.name": "OpenJDK Runtime Environment",
    "java.runtime.version": "11.0.23+9",
    "java.vendor": "Eclipse Adoptium",
    "java.vendor.url": "https://adoptium.net/",
    "java.version": "11.0.23",
    "java.vm.name": "OpenJDK 64-Bit Server VM",
    "java.vm.version": "11.0.23+9",
    "os.name": "Linux",
    "os.version": "5.15.146+",
    "user.language": "en",
    "user.timezone": "GMT"
  },
  "metabase-info": {
    "databases": [
      "h2",
      "bigquery-cloud-sdk"
    ],
    "hosting-env": "unknown",
    "application-database": "mysql",
    "application-database-details": {
      "database": {
        "name": "MySQL",
        "version": "8.0.31-google"
      },
      "jdbc-driver": {
        "name": "MariaDB Connector/J",
        "version": "2.7.10"
      }
    },
    "run-mode": "prod",
    "version": {
      "date": "2024-06-19",
      "tag": "v0.50.6",
      "hash": "a5fbebf"
    },
    "settings": {
      "report-timezone": null
    }
  }
}

EDIT: Formatted log excerpt for easier viewing

2

Is it the same question? Like are you using some filter on metabase which you are not passing to the bigquery?

Is it possible there are some network delays when retrieving data from bigquery to metabase? So it's taking more time on metabase

3

I'm having the issue with the same collection of questions which are more complex and take longer to execute that my other ones. I have other collections of BigQuery questions that are in dashboards which have no issues executing. There are filters on this question not in use for my testing purposes at the moment, and the query preview shows that they are not being inserted into the query being sent to BQ.

I suppose so? The collection of questions I'm working with right now each take 50-90 seconds when executed on BQ directly so if the API calls from Metabase to BQ are timing out then that would make sense. However the "client closed connection prematurely" error and the 499 error code in nginx console point to the web client terminating the connection, not the BQ request failing. Is this a configurable value for the frontend client perhaps? I can't find it in the admin dashboard.

The most difficult part about diagnosing this issue is reproducing it. Last week and earlier today I was seeing this issue happen, but within the last couple of hours and later on the original post date it's disappeared with no configuration changes made to the GKE cluster or load balancer. My attempts to reproduce it by running the Docker image locally while connected to the same backend database did not show the issue either which leads me to believe that this is an issue with how I'm hosting it, but I'm stumped. I've checked all of the networking configurations on the GCP end and everything is set at a limit high enough that this should get through with more than enough breathing room. But this isn't a forum for GCP so I don't expect much help there, so unless anyone has any insight as to what the cause may be I might have to take my issue elsewhere.

4

Chiming in to mention that it may be worth looking at BigQuery execution logs as well. For example, depending on how may queries you had in flight and what tuning you have in place it's possible to hit a slot quota and for the queries to be placed in queue (thus extending the execution time). If you have any sort of budget, spending or resource limitations enabled on BQ that could have an impact, too - but I'd expect a more explicit error in that case (though I've never tested it).

When you tested your local docker image were you testing it at the same time you were actively observing this behavior in the GKE cluster?

5

I don't have any slot reservations/commitments currently, so I think(?) that shouldn't impact it.

Yes.

6

You might be hitting this issue (Metabase 0.50.5 is unavailable) +CrashOnOutOfMemoryError cpu problem · Issue #44359 · metabase/metabase · GitHub

Which is causing question to take longer to run ... cna you increase the timeout and see what happens?

7

Sorry for the late reply, I was at a conference.

The timeouts are already at 300 seconds, do you want me to increase them further for some reason? The questions are not taking nearly that long to execute, the longest that I'm seeing in the Metabase console log is about 100 seconds. Not sure how increasing it even further would help, but if you can let me know I'll do it to see.

That's an interesting issue that you've shared, I've been reading through it. I'm not necessarily sure if it's the same issue though. I haven't been seeing many CPU spike issues with the pods, which I have set to a limit of 1 CPU unit in Kubernetes. I did see that v0.50.8 contained a BQ enhancement and updated, but the same issue is still occurring. Here is a screenshot of the CPU and RAM metrics from the pod after the update and after I've tried running several of the questions from the collection where I'm experiencing this issue.

image
image1060×256 12.9 KB

In my attempts to fix and reproduce the exact conditions of this issue I have noticed a few more things which may be helpful in understanding the issue but I'm not sure.

  • Sometimes these big questions do execute, the frontend will continue spinning for the full duration and display the result. The backend will show that the request completed ("ASYNC: completed" on the log for that request in the console).
    Question still executing
    2024-07-01_13-02
    Question completed
    image
    2024-07-01_13-02_1
  • When the error occurs, the frontend client stops updating the timer in the page title and begins to display the "Still Here..." title and displays the "your question took too long" screen at 30 seconds, not sure if that's related to a networking issue or some kind of Metabase configuration.
    image
    image
  • Sometimes an entirly different error occurs, but I haven't been able to reproduce it consistently. I don't have a log for it on-hand, but I remember that it looked like it consisted of two errors, with a top level one from Metabase with the message "Closed" and a related one with the message "Deflater is closed". I'm not sure exactly what these might be related to. If I see the error occur again I'll post the log in this thread.
8

Yes please if you get the logs!

I mean the error seems to point to connection being closed. So who and what is closing that is the real question

9

Not sure if this will be helpful at all, I tried downgrading to v0.49.19 to see if the issue was still occuring and it is. However, a different error is shown in the logs. Same behavior as before on the frontend side. The query and template variables have been redacted.

2024-07-02 19:01:01,688 DEBUG middleware.log :: e[32mPOST /api/card/2/query 202 [ASYNC: canceled] 33.1 s (15 DB calls) App DB connections: 2/15 Jetty threads: 2/50 (9 idle, 0 queued) (63 total active threads) Queries in flight: 0 (0 queued)e[0m
2024-07-02 19:01:01,916 ERROR middleware.catch-exceptions :: Error processing query: Query cancelled
{:database_id 2,
 :started_at #t "2024-07-02T19:00:28.652038Z[GMT]",
 :action_id nil,
 :json_query
 {:constraints {:max-results 10000, :max-results-bare-rows 2000},
  :type :native,
  :middleware {:js-int-to-string? true, :ignore-cached-results? true, :process-viz-settings? false},
  :native
  {:template-tags
   {"redacted_variable"
    {:id "f273fada-03e1-b3d7-e1b9-68fe67a40406", :name "redacted_variable", :display-name "Redacted Variable", :type :text}},
   :query
   "SELECT 1;/* redacted by eksmith */"},
  :viz-settings nil,
  :database 2,
  :parameters
  [{:id "f273fada-03e1-b3d7-e1b9-68fe67a40406",
    :type "category",
    :value nil,
    :target ["variable" ["template-tag" "redacted_variable"]]}],
  :async? true,
  :cache-ttl nil},
 :status :failed,
 :class clojure.lang.ExceptionInfo,
 :stacktrace
 ["--> driver.bigquery_cloud_sdk$execute_bigquery.invokeStatic(bigquery_cloud_sdk.clj:378)"
  "driver.bigquery_cloud_sdk$execute_bigquery.invoke(bigquery_cloud_sdk.clj:346)"
  "driver.bigquery_cloud_sdk$execute_bigquery_on_db.invokeStatic(bigquery_cloud_sdk.clj:390)"
  "driver.bigquery_cloud_sdk$execute_bigquery_on_db.invoke(bigquery_cloud_sdk.clj:388)"
  "driver.bigquery_cloud_sdk$_STAR_process_native_STAR_$thunk__117163.invoke(bigquery_cloud_sdk.clj:438)"
  "driver.bigquery_cloud_sdk$_STAR_process_native_STAR_.invokeStatic(bigquery_cloud_sdk.clj:446)"
  "driver.bigquery_cloud_sdk$_STAR_process_native_STAR_.invoke(bigquery_cloud_sdk.clj:431)"
  "driver.bigquery_cloud_sdk$fn__117170.invokeStatic(bigquery_cloud_sdk.clj:467)"
  "driver.bigquery_cloud_sdk$fn__117170.invoke(bigquery_cloud_sdk.clj:459)"
  "query_processor.context$executef.invokeStatic(context.clj:60)"
  "query_processor.context$executef.invoke(context.clj:49)"
  "query_processor.context.default$default_runf.invokeStatic(default.clj:44)"
  "query_processor.context.default$default_runf.invoke(default.clj:42)"
  "query_processor.context$runf.invokeStatic(context.clj:46)"
  "query_processor.context$runf.invoke(context.clj:40)"
  "query_processor.reducible$identity_qp.invokeStatic(reducible.clj:39)"
  "query_processor.reducible$identity_qp.invoke(reducible.clj:36)"
  "query_processor.middleware.cache$maybe_return_cached_results$maybe_return_cached_results_STAR___72656.invoke(cache.clj:229)"
  "query_processor.middleware.permissions$check_query_permissions$fn__67003.invoke(permissions.clj:140)"
  "query_processor.middleware.enterprise$check_download_permissions_middleware$fn__72477.invoke(enterprise.clj:51)"
  "query_processor.middleware.enterprise$maybe_apply_column_level_perms_check_middleware$fn__72487.invoke(enterprise.clj:64)"
  "query_processor.middleware.mbql_to_native$mbql__GT_native$fn__71919.invoke(mbql_to_native.clj:24)"
  "query_processor$fn__73824$combined_post_process__73829$combined_post_process_STAR___73830.invoke(query_processor.clj:262)"
  "query_processor$fn__73824$combined_pre_process__73825$combined_pre_process_STAR___73826.invoke(query_processor.clj:259)"
  "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__67100.invoke(fetch_source_query.clj:303)"
  "query_processor.middleware.resolve_database_and_driver$resolve_driver_and_database_local_values$fn__72567$fn__72571.invoke(resolve_database_and_driver.clj:77)"
  "driver$do_with_driver.invokeStatic(driver.clj:97)"
  "driver$do_with_driver.invoke(driver.clj:92)"
  "query_processor.middleware.resolve_database_and_driver$resolve_driver_and_database_local_values$fn__72567.invoke(resolve_database_and_driver.clj:76)"
  "query_processor.middleware.store$initialize_store$fn__67727$fn__67728.invoke(store.clj:14)"
  "query_processor.store$do_with_metadata_provider.invokeStatic(store.clj:169)"
  "query_processor.store$do_with_metadata_provider.invoke(store.clj:150)"
  "query_processor.store$do_with_metadata_provider.invokeStatic(store.clj:158)"
  "query_processor.store$do_with_metadata_provider.invoke(store.clj:150)"
  "query_processor.middleware.store$initialize_store$fn__67727.invoke(store.clj:13)"
  "query_processor.middleware.resolve_database_and_driver$resolve_database$fn__72564.invoke(resolve_database_and_driver.clj:60)"
  "query_processor.middleware.normalize_query$normalize$fn__72869.invoke(normalize_query.clj:38)"
  "query_processor.middleware.enterprise$fn__72504$handle_audit_app_internal_queries__72505$fn__72507.invoke(enterprise.clj:96)"
  "query_processor.middleware.enterprise$handle_audit_app_internal_queries_middleware$fn__72515.invoke(enterprise.clj:103)"
  "query_processor.middleware.constraints$mark_needs_default_userland_constraints$fn__71630.invoke(constraints.clj:104)"
  "query_processor.middleware.process_userland_query$process_userland_query$fn__72800.invoke(process_userland_query.clj:156)"
  "query_processor.middleware.catch_exceptions$catch_exceptions$fn__73401.invoke(catch_exceptions.clj:171)"
  "query_processor.reducible$async_qp$qp_STAR___63219$thunk__63221.invoke(reducible.clj:126)"
  "query_processor.reducible$async_qp$qp_STAR___63219$fn__63223.invoke(reducible.clj:131)"],
 :card_id 2,
 :context :question,
 :error "Query cancelled",
 :row_count 0,
 :running_time 0,
 :ex-data
 {:sql
  "SELECT 1;/* redacted by eksmith */",
  :parameters nil,
  :metabase.driver.bigquery-cloud-sdk/cancelled? true},
 :data {:rows [], :cols []}}
2024-07-02T19:01:01.918781885Z
10

I was finally able to capture this error (query and template variables redacted):

2024-07-09 22:52:06,937 ERROR middleware.catch-exceptions :: Error processing query: Closed
{:database_id 2,
 :started_at #t "2024-07-09T22:50:58.498927Z[Etc/UTC]",
 :via
 [{:status :failed,
   :class clojure.lang.ExceptionInfo,
   :error "Error reducing result rows: Closed",
   :stacktrace
   ["--> query_processor.pipeline$_STAR_reduce_STAR_$fn__56641.invoke(pipeline.clj:74)"
    "query_processor.pipeline$_STAR_reduce_STAR_.invokeStatic(pipeline.clj:62)"
    "query_processor.pipeline$_STAR_reduce_STAR_.invoke(pipeline.clj:49)"
    "query_processor.pipeline$_STAR_run_STAR_$respond__56653.invoke(pipeline.clj:95)"
    "driver.bigquery_cloud_sdk$post_process_native.invokeStatic(bigquery_cloud_sdk.clj:425)"
    "driver.bigquery_cloud_sdk$post_process_native.invoke(bigquery_cloud_sdk.clj:408)"
    "driver.bigquery_cloud_sdk$_STAR_process_native_STAR_$thunk__119376.invoke(bigquery_cloud_sdk.clj:441)"
    "driver.bigquery_cloud_sdk$_STAR_process_native_STAR_.invokeStatic(bigquery_cloud_sdk.clj:455)"
    "driver.bigquery_cloud_sdk$_STAR_process_native_STAR_.invoke(bigquery_cloud_sdk.clj:430)"
    "driver.bigquery_cloud_sdk$fn__119383.invokeStatic(bigquery_cloud_sdk.clj:471)"
    "driver.bigquery_cloud_sdk$fn__119383.invoke(bigquery_cloud_sdk.clj:463)"
    "query_processor.pipeline$_STAR_execute_STAR_.invokeStatic(pipeline.clj:47)"
    "query_processor.pipeline$_STAR_execute_STAR_.invoke(pipeline.clj:34)"
    "query_processor.pipeline$_STAR_run_STAR_.invokeStatic(pipeline.clj:97)"
    "query_processor.pipeline$_STAR_run_STAR_.invoke(pipeline.clj:90)"
    "query_processor.execute$run.invokeStatic(execute.clj:61)"
    "query_processor.execute$run.invoke(execute.clj:55)"
    "query_processor.execute$add_native_form_to_result_metadata$fn__69763.invoke(execute.clj:24)"
    "query_processor.execute$add_preprocessed_query_to_result_metadata_for_userland_query$fn__69768.invoke(execute.clj:35)"
    "query_processor.middleware.cache$maybe_return_cached_results$maybe_return_cached_results_STAR___69754.invoke(cache.clj:242)"
    "query_processor.middleware.permissions$check_query_permissions$fn__63764.invoke(permissions.clj:118)"
    "query_processor.middleware.enterprise$check_download_permissions_middleware$fn__64362.invoke(enterprise.clj:51)"
    "query_processor.middleware.enterprise$maybe_apply_column_level_perms_check_middleware$fn__64372.invoke(enterprise.clj:64)"
    "query_processor.execute$execute$fn__69795.invoke(execute.clj:93)"
    "query_processor.setup$do_with_qp_setup.invokeStatic(setup.clj:225)"
    "query_processor.setup$do_with_qp_setup.invoke(setup.clj:216)"
    "query_processor.execute$execute.invokeStatic(execute.clj:92)"
    "query_processor.execute$execute.invoke(execute.clj:88)"
    "query_processor$process_query_STAR__STAR_.invokeStatic(query_processor.clj:47)"
    "query_processor$process_query_STAR__STAR_.invoke(query_processor.clj:43)"
    "query_processor.middleware.enterprise$fn__64389$handle_audit_app_internal_queries__64390$fn__64392.invoke(enterprise.clj:96)"
    "query_processor.middleware.enterprise$handle_audit_app_internal_queries_middleware$fn__64400.invoke(enterprise.clj:103)"
    "query_processor.middleware.process_userland_query$process_userland_query_middleware$fn__75630.invoke(process_userland_query.clj:182)"
    "query_processor.middleware.catch_exceptions$catch_exceptions$fn__75699.invoke(catch_exceptions.clj:128)"
    "query_processor$process_query$fn__75736.invoke(query_processor.clj:78)"
    "query_processor.setup$do_with_canceled_chan$fn__64804.invoke(setup.clj:189)"
    "query_processor.setup$do_with_database_local_settings$fn__64799.invoke(setup.clj:181)"
    "query_processor.setup$do_with_driver$fn__64794$fn__64795.invoke(setup.clj:166)"
    "driver$do_with_driver.invokeStatic(driver.clj:104)"
    "driver$do_with_driver.invoke(driver.clj:99)"
    "query_processor.setup$do_with_driver$fn__64794.invoke(setup.clj:165)"
    "query_processor.setup$do_with_metadata_provider$fn__64787$fn__64790.invoke(setup.clj:151)"
    "query_processor.store$do_with_metadata_provider.invokeStatic(store.clj:171)"
    "query_processor.store$do_with_metadata_provider.invoke(store.clj:151)"
    "query_processor.store$do_with_metadata_provider.invokeStatic(store.clj:160)"
    "query_processor.store$do_with_metadata_provider.invoke(store.clj:151)"
    "query_processor.setup$do_with_metadata_provider$fn__64787.invoke(setup.clj:150)"
    "query_processor.setup$do_with_resolved_database$fn__64781.invoke(setup.clj:128)"
    "query_processor.setup$do_with_qp_setup.invokeStatic(setup.clj:232)"
    "query_processor.setup$do_with_qp_setup.invoke(setup.clj:216)"
    "query_processor$process_query.invokeStatic(query_processor.clj:76)"
    "query_processor$process_query.invoke(query_processor.clj:69)"
    "query_processor.card$process_query_for_card_default_qp.invokeStatic(card.clj:170)"
    "query_processor.card$process_query_for_card_default_qp.invoke(card.clj:166)"
    "query_processor.card$process_query_for_card_default_run_fn$fn__81707$fn__81708.invoke(card.clj:177)"
    "query_processor.streaming$_streaming_response$fn__68238$fn__68239$fn__68240.invoke(streaming.clj:175)"
    "query_processor.streaming$_streaming_response$fn__68238$fn__68239.invoke(streaming.clj:174)"
    "query_processor.streaming$do_with_streaming_rff.invokeStatic(streaming.clj:165)"
    "query_processor.streaming$do_with_streaming_rff.invoke(streaming.clj:152)"
    "query_processor.streaming$_streaming_response$fn__68238.invoke(streaming.clj:171)"
    "async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:68)"
    "async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:66)"
    "async.streaming_response$do_f_async$task__52165.invoke(streaming_response.clj:87)"],
   :error_type :qp,
   :ex-data {:type :qp}}],
 :action_id nil,
 :error_type :qp,
 :json_query
 {:constraints {:max-results 10000, :max-results-bare-rows 2000},
  :type :native,
  :middleware
  {:js-int-to-string? true, :ignore-cached-results? false, :process-viz-settings? false, :userland-query? true},
  :cache-strategy nil,
  :native
  {:template-tags
   {"redacted_variable"
    {:id "f273fada-03e1-b3d7-e1b9-68fe67a40406", :name "redacted_variable", :display-name "Redacted Variable", :type :text}},
   :query
   "SELECT 1;/* redacted by eksmith */"},
  :viz-settings nil,
  :database 2,
  :parameters
  [{:id "f273fada-03e1-b3d7-e1b9-68fe67a40406",
    :type "category",
    :value nil,
    :target ["variable" ["template-tag" "redacted_variable"]]}]},
 :status :failed,
 :class org.eclipse.jetty.io.EofException,
 :stacktrace
 ["org.eclipse.jetty.server.HttpOutput.checkWritable(HttpOutput.java:756)"
  "org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:780)"
  "java.base/java.util.zip.DeflaterOutputStream.flush(Unknown Source)"
  "--> async.streaming_response$delay_output_stream$fn__52182.invoke(streaming_response.clj:122)"
  "async.streaming_response.proxy$java.io.OutputStream$ff19274a.flush(Unknown Source)"
  "query_processor.streaming.json$fn$reify__67361.write_row_BANG_(json.clj:95)"
  "query_processor.streaming$streaming_rff$fn__68213$fn__68218.invoke(streaming.clj:133)"
  "query_processor.middleware.process_userland_query$add_and_save_execution_metadata_xform_BANG_$execution_info_rf_STAR___75622.invoke(process_userland_query.clj:123)"
  "query_processor.reducible$combine_additional_reducing_fns$combine_additional_reducing_fns_rf_STAR___56675.invoke(reducible.clj:113)"
  "query_processor.middleware.add_rows_truncated$add_rows_truncated_xform$fn__71518.invoke(add_rows_truncated.clj:34)"
  "query_processor.middleware.format_rows$format_rows_xform$fn__71891.invoke(format_rows.clj:73)"
  "query_processor.reducible$combine_additional_reducing_fns$combine_additional_reducing_fns_rf_STAR___56675.invoke(reducible.clj:113)"
  "query_processor.pipeline$_STAR_reduce_STAR_$fn__56641$fn__56642$wrapper__56643.invoke(pipeline.clj:70)"
  "query_processor.pipeline$_STAR_reduce_STAR_$fn__56641.invoke(pipeline.clj:63)"
  "query_processor.pipeline$_STAR_reduce_STAR_.invokeStatic(pipeline.clj:62)"
  "query_processor.pipeline$_STAR_reduce_STAR_.invoke(pipeline.clj:49)"
  "query_processor.pipeline$_STAR_run_STAR_$respond__56653.invoke(pipeline.clj:95)"
  "driver.bigquery_cloud_sdk$post_process_native.invokeStatic(bigquery_cloud_sdk.clj:425)"
  "driver.bigquery_cloud_sdk$post_process_native.invoke(bigquery_cloud_sdk.clj:408)"
  "driver.bigquery_cloud_sdk$_STAR_process_native_STAR_$thunk__119376.invoke(bigquery_cloud_sdk.clj:441)"
  "driver.bigquery_cloud_sdk$_STAR_process_native_STAR_.invokeStatic(bigquery_cloud_sdk.clj:455)"
  "driver.bigquery_cloud_sdk$_STAR_process_native_STAR_.invoke(bigquery_cloud_sdk.clj:430)"
  "driver.bigquery_cloud_sdk$fn__119383.invokeStatic(bigquery_cloud_sdk.clj:471)"
  "driver.bigquery_cloud_sdk$fn__119383.invoke(bigquery_cloud_sdk.clj:463)"
  "query_processor.pipeline$_STAR_execute_STAR_.invokeStatic(pipeline.clj:47)"
  "query_processor.pipeline$_STAR_execute_STAR_.invoke(pipeline.clj:34)"
  "query_processor.pipeline$_STAR_run_STAR_.invokeStatic(pipeline.clj:97)"
  "query_processor.pipeline$_STAR_run_STAR_.invoke(pipeline.clj:90)"
  "query_processor.execute$run.invokeStatic(execute.clj:61)"
  "query_processor.execute$run.invoke(execute.clj:55)"
  "query_processor.execute$add_native_form_to_result_metadata$fn__69763.invoke(execute.clj:24)"
  "query_processor.execute$add_preprocessed_query_to_result_metadata_for_userland_query$fn__69768.invoke(execute.clj:35)"
  "query_processor.middleware.cache$maybe_return_cached_results$maybe_return_cached_results_STAR___69754.invoke(cache.clj:242)"
  "query_processor.middleware.permissions$check_query_permissions$fn__63764.invoke(permissions.clj:118)"
  "query_processor.middleware.enterprise$check_download_permissions_middleware$fn__64362.invoke(enterprise.clj:51)"
  "query_processor.middleware.enterprise$maybe_apply_column_level_perms_check_middleware$fn__64372.invoke(enterprise.clj:64)"
  "query_processor.execute$execute$fn__69795.invoke(execute.clj:93)"
  "query_processor.setup$do_with_qp_setup.invokeStatic(setup.clj:225)"
  "query_processor.setup$do_with_qp_setup.invoke(setup.clj:216)"
  "query_processor.execute$execute.invokeStatic(execute.clj:92)"
  "query_processor.execute$execute.invoke(execute.clj:88)"
  "query_processor$process_query_STAR__STAR_.invokeStatic(query_processor.clj:47)"
  "query_processor$process_query_STAR__STAR_.invoke(query_processor.clj:43)"
  "query_processor.middleware.enterprise$fn__64389$handle_audit_app_internal_queries__64390$fn__64392.invoke(enterprise.clj:96)"
  "query_processor.middleware.enterprise$handle_audit_app_internal_queries_middleware$fn__64400.invoke(enterprise.clj:103)"
  "query_processor.middleware.process_userland_query$process_userland_query_middleware$fn__75630.invoke(process_userland_query.clj:182)"
  "query_processor.middleware.catch_exceptions$catch_exceptions$fn__75699.invoke(catch_exceptions.clj:128)"
  "query_processor$process_query$fn__75736.invoke(query_processor.clj:78)"
  "query_processor.setup$do_with_canceled_chan$fn__64804.invoke(setup.clj:189)"
  "query_processor.setup$do_with_database_local_settings$fn__64799.invoke(setup.clj:181)"
  "query_processor.setup$do_with_driver$fn__64794$fn__64795.invoke(setup.clj:166)"
  "driver$do_with_driver.invokeStatic(driver.clj:104)"
  "driver$do_with_driver.invoke(driver.clj:99)"
  "query_processor.setup$do_with_driver$fn__64794.invoke(setup.clj:165)"
  "query_processor.setup$do_with_metadata_provider$fn__64787$fn__64790.invoke(setup.clj:151)"
  "query_processor.store$do_with_metadata_provider.invokeStatic(store.clj:171)"
  "query_processor.store$do_with_metadata_provider.invoke(store.clj:151)"
  "query_processor.store$do_with_metadata_provider.invokeStatic(store.clj:160)"
  "query_processor.store$do_with_metadata_provider.invoke(store.clj:151)"
  "query_processor.setup$do_with_metadata_provider$fn__64787.invoke(setup.clj:150)"
  "query_processor.setup$do_with_resolved_database$fn__64781.invoke(setup.clj:128)"
  "query_processor.setup$do_with_qp_setup.invokeStatic(setup.clj:232)"
  "query_processor.setup$do_with_qp_setup.invoke(setup.clj:216)"
  "query_processor$process_query.invokeStatic(query_processor.clj:76)"
  "query_processor$process_query.invoke(query_processor.clj:69)"
  "query_processor.card$process_query_for_card_default_qp.invokeStatic(card.clj:170)"
  "query_processor.card$process_query_for_card_default_qp.invoke(card.clj:166)"
  "query_processor.card$process_query_for_card_default_run_fn$fn__81707$fn__81708.invoke(card.clj:177)"
  "query_processor.streaming$_streaming_response$fn__68238$fn__68239$fn__68240.invoke(streaming.clj:175)"
  "query_processor.streaming$_streaming_response$fn__68238$fn__68239.invoke(streaming.clj:174)"
  "query_processor.streaming$do_with_streaming_rff.invokeStatic(streaming.clj:165)"
  "query_processor.streaming$do_with_streaming_rff.invoke(streaming.clj:152)"
  "query_processor.streaming$_streaming_response$fn__68238.invoke(streaming.clj:171)"
  "async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:68)"
  "async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:66)"
  "async.streaming_response$do_f_async$task__52165.invoke(streaming_response.clj:87)"],
 :card_id 17,
 :context :question,
 :error "Closed",
 :row_count 0,
 :running_time 0,
 :data {:rows [], :cols []}}
2024-07-09 22:52:06,956 ERROR async.streaming-response :: Caught unexpected Exception in streaming response body
java.lang.NullPointerException: Deflater has been closed
#011at java.base/java.util.zip.Deflater.ensureOpen(Unknown Source)
#011at java.base/java.util.zip.Deflater.deflate(Unknown Source)
#011at java.base/java.util.zip.Deflater.deflate(Unknown Source)
#011at java.base/java.util.zip.GZIPOutputStream.finish(Unknown Source)
#011at java.base/java.util.zip.DeflaterOutputStream.close(Unknown Source)
#011at metabase.async.streaming_response$delay_output_stream$fn__52180.invoke(streaming_response.clj:120)
#011at metabase.async.streaming_response.proxy$java.io.OutputStream$ff19274a.close(Unknown Source)
#011at metabase.async.streaming_response$write_error_BANG_.invokeStatic(streaming_response.clj:57)
#011at metabase.async.streaming_response$write_error_BANG_.invoke(streaming_response.clj:45)
#011at metabase.query_processor.streaming$_streaming_response$fn__68238$fn__68239.invoke(streaming.clj:184)
#011at metabase.query_processor.streaming$do_with_streaming_rff.invokeStatic(streaming.clj:165)
#011at metabase.query_processor.streaming$do_with_streaming_rff.invoke(streaming.clj:152)
#011at metabase.query_processor.streaming$_streaming_response$fn__68238.invoke(streaming.clj:171)
#011at clojure.lang.AFn.applyToHelper(AFn.java:156)
#011at clojure.lang.AFn.applyTo(AFn.java:144)
#011at clojure.core$apply.invokeStatic(core.clj:667)
#011at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1990)
#011at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1990)
#011at clojure.lang.RestFn.applyTo(RestFn.java:142)
#011at clojure.core$apply.invokeStatic(core.clj:671)
#011at clojure.core$bound_fn_STAR_$fn__5818.doInvoke(core.clj:2020)
#011at clojure.lang.RestFn.invoke(RestFn.java:421)
#011at metabase.async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:68)
#011at metabase.async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:66)
#011at metabase.async.streaming_response$do_f_async$task__52165.invoke(streaming_response.clj:87)
#011at clojure.lang.AFn.run(AFn.java:22)
#011at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
#011at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
#011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
#011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
#011at java.base/java.lang.Thread.run(Unknown Source)
2024-07-09 22:52:06,979 ERROR async.streaming-response :: Error writing error to output stream {:via [{:type java.lang.NullPointerException, :message Deflater has been closed, :at [java.util.zip.Deflater ensureOpen nil -1]}], :trace [[java.util.zip.Deflater ensureOpen nil -1] [java.util.zip.Deflater deflate nil -1] [java.util.zip.Deflater deflate nil -1] [java.util.zip.GZIPOutputStream finish nil -1] [java.util.zip.DeflaterOutputStream close nil -1] [metabase.async.streaming_response$delay_output_stream$fn__52180 invoke streaming_response.clj 120] [metabase.async.streaming_response.proxy$java.io.OutputStream$ff19274a close nil -1] [metabase.async.streaming_response$write_error_BANG_ invokeStatic streaming_response.clj 57] [metabase.async.streaming_response$write_error_BANG_ invoke streaming_response.clj 45] [metabase.query_processor.streaming$_streaming_response$fn__68238$fn__68239 invoke streaming.clj 184] [metabase.query_processor.streaming$do_with_streaming_rff invokeStatic streaming.clj 165] [metabase.query_processor.streaming$do_with_streaming_rff invoke streaming.clj 152] [metabase.query_processor.streaming$_streaming_response$fn__68238 invoke streaming.clj 171] [clojure.lang.AFn applyToHelper AFn.java 156] [clojure.lang.AFn applyTo AFn.java 144] [clojure.core$apply invokeStatic core.clj 667] [clojure.core$with_bindings_STAR_ invokeStatic core.clj 1990] [clojure.core$with_bindings_STAR_ doInvoke core.clj 1990] [clojure.lang.RestFn applyTo RestFn.java 142] [clojure.core$apply invokeStatic core.clj 671] [clojure.core$bound_fn_STAR_$fn__5818 doInvoke core.clj 2020] [clojure.lang.RestFn invoke RestFn.java 421] [metabase.async.streaming_response$do_f_STAR_ invokeStatic streaming_response.clj 68] [metabase.async.streaming_response$do_f_STAR_ invoke streaming_response.clj 66] [metabase.async.streaming_response$do_f_async$task__52165 invoke streaming_response.clj 87] [clojure.lang.AFn run AFn.java 22] [java.util.concurrent.Executors$RunnableAdapter call nil -1] [java.util.concurrent.FutureTask run nil -1] [java.util.concurrent.ThreadPoolExecutor runWorker nil -1] [java.util.concurrent.ThreadPoolExecutor$Worker run nil -1] [java.lang.Thread run nil -1]], :cause Deflater has been closed, :_status 500}
java.lang.NullPointerException: Deflater has been closed
#011at java.base/java.util.zip.Deflater.ensureOpen(Unknown Source)
#011at java.base/java.util.zip.Deflater.deflate(Unknown Source)
#011at java.base/java.util.zip.Deflater.deflate(Unknown Source)
#011at java.base/java.util.zip.GZIPOutputStream.finish(Unknown Source)
#011at java.base/java.util.zip.DeflaterOutputStream.close(Unknown Source)
#011at metabase.async.streaming_response$delay_output_stream$fn__52180.invoke(streaming_response.clj:120)
#011at metabase.async.streaming_response.proxy$java.io.OutputStream$ff19274a.close(Unknown Source)
#011at java.base/sun.nio.cs.StreamEncoder.implClose(Unknown Source)
#011at java.base/sun.nio.cs.StreamEncoder.close(Unknown Source)
#011at java.base/java.io.OutputStreamWriter.close(Unknown Source)
#011at java.base/java.io.BufferedWriter.close(Unknown Source)
#011at metabase.async.streaming_response$write_error_BANG_.invokeStatic(streaming_response.clj:60)
#011at metabase.async.streaming_response$write_error_BANG_.invoke(streaming_response.clj:45)
#011at metabase.async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:77)
#011at metabase.async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:66)
#011at metabase.async.streaming_response$do_f_async$task__52165.invoke(streaming_response.clj:87)
#011at clojure.lang.AFn.run(AFn.java:22)
#011at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
#011at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
#011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
#011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
#011at java.base/java.lang.Thread.run(Unknown Source)
2024-07-09 22:52:06,983 ERROR async.streaming-response :: bound-fn caught unexpected Exception
java.lang.NullPointerException: Deflater has been closed
#011at java.base/java.util.zip.Deflater.ensureOpen(Unknown Source)
#011at java.base/java.util.zip.Deflater.deflate(Unknown Source)
#011at java.base/java.util.zip.Deflater.deflate(Unknown Source)
#011at java.base/java.util.zip.GZIPOutputStream.finish(Unknown Source)
#011at java.base/java.util.zip.DeflaterOutputStream.close(Unknown Source)
#011at metabase.async.streaming_response$delay_output_stream$fn__52180.invoke(streaming_response.clj:120)
#011at metabase.async.streaming_response.proxy$java.io.OutputStream$ff19274a.close(Unknown Source)
#011at metabase.async.streaming_response$write_error_BANG_.invokeStatic(streaming_response.clj:57)
#011at metabase.async.streaming_response$write_error_BANG_.invoke(streaming_response.clj:45)
#011at metabase.async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:77)
#011at metabase.async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:66)
#011at metabase.async.streaming_response$do_f_async$task__52165.invoke(streaming_response.clj:87)
#011at clojure.lang.AFn.run(AFn.java:22)
#011at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
#011at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
#011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
#011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
#011at java.base/java.lang.Thread.run(Unknown Source)
2024-07-09 22:52:06,985 DEBUG middleware.log :: #033[32mPOST /api/card/17/query 202 [ASYNC: unexpected-error] 1.1 mins (10 DB calls) App DB connections: 0/7 Jetty threads: 2/50 (6 idle, 0 queued) (55 total active threads) Queries in flight: 1 (0 queued) {:metabase-user-id 11}#033[0m

I should mention that this error was captured in my testing of Metabase off Kubernetes and on a Debian VM, but this was the same error that I was seeing in Kubernetes. I will post back with my findings on my testing as I attempt to figure out if my Kubernetes setup is the cause of this issue.

11

So I'm back now after a week of using Metabase on a Debian VM, and I have to say my experience is a lot better. I'm seeing this issue much less frequently (although it still happens sometimes)and it's much more reliable, so for the forseeable future I'll be sticking with that solution as I don't have a lot of time to continue debugging the Kubernetes setup unless there are any quick checks/fixes to find out what's wrong. Thanks to everyone who provided input on this issue.