Metabase slowdown seems to stem from broken pipe/ Error reducing result rows Errors

Bug reports
1

Loading the results of questions was exceptionally slow, requiring minutes for simple table scans of relatively small tables.

Looking at the log, I observed a number of clojure errors as per below

[6865e7cb-206a-4ce8-9b00-e9869cad918b] 2021-06-02T10:47:21+08:00 DEBUG metabase.server.middleware.log GET /api/util/bug_report_details 200 107.1 ms (1 DB calls) App DB connections: 1/15 Jetty threads: 3/50 (25 idle, 0 queued) (170 total active threads) Queries in flight: 14 (0 queued)
[6865e7cb-206a-4ce8-9b00-e9869cad918b] 2021-06-02T10:47:25+08:00 ERROR metabase.query-processor.middleware.catch-exceptions Error processing query: null
{:database_id 5,
 :started_at #t "2021-06-02T02:46:49.112886Z[GMT]",
 :via
 [{:status :failed,
   :class org.eclipse.jetty.io.EofException,
   :error nil,
   :stacktrace
   ["org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)"
    "org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)"
    "org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)"
    "org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)"
    "org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:820)"
    "org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)"
    "org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)"
    "org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:544)"
    "org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:838)"
    "org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:910)"
    "org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:284)"
    "org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:268)"
    "org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:713)"
    "java.base/java.util.zip.DeflaterOutputStream.flush(Unknown Source)"
    "--> async.streaming_response$delay_output_stream$fn__17621.invoke(streaming_response.clj:119)"
    "async.streaming_response.proxy$java.io.OutputStream$ff19274a.flush(Unknown Source)"
    "query_processor.streaming.json$fn$reify__65061.write_row_BANG_(json.clj:64)"
    "query_processor.streaming$streaming_rff$fn__65556$fn__65557.invoke(streaming.clj:34)"
    "query_processor.middleware.process_userland_query$add_and_save_execution_info_xform_BANG_$execution_info_rf_STAR___49333.invoke(process_userland_query.clj:94)"
    "query_processor.reducible$combine_additional_reducing_fns$fn__33728.invoke(reducible.clj:216)"
    "query_processor.middleware.add_rows_truncated$add_rows_truncated_xform$fn__45569.invoke(add_rows_truncated.clj:27)"
    "query_processor.middleware.format_rows$format_rows_xform$fn__48091.invoke(format_rows.clj:68)"
    "query_processor.reducible$combine_additional_reducing_fns$fn__33728.invoke(reducible.clj:216)"
    "query_processor.middleware.cache$save_results_xform$fn__46819.invoke(cache.clj:114)"
    "query_processor.middleware.limit$limit_xform$fn__48150.invoke(limit.clj:24)"
    "query_processor.context.default$default_reducef$fn__33619.invoke(default.clj:58)"
    "query_processor.context.default$default_reducef.invokeStatic(default.clj:57)"
    "query_processor.context.default$default_reducef.invoke(default.clj:48)"
    "query_processor.context$reducef.invokeStatic(context.clj:69)"
    "query_processor.context$reducef.invoke(context.clj:62)"
    "query_processor.context.default$default_runf$respond_STAR___33623.invoke(default.clj:69)"
    "driver.bigquery$post_process_native$fn__1478.invoke(bigquery.clj:251)"
    "driver.bigquery$do_with_finished_response.invokeStatic(bigquery.clj:171)"
    "driver.bigquery$do_with_finished_response.invoke(bigquery.clj:162)"
    "driver.bigquery$post_process_native.invokeStatic(bigquery.clj:234)"
    "driver.bigquery$post_process_native.invoke(bigquery.clj:228)"
    "driver.bigquery$process_native_STAR_$thunk__1576.invoke(bigquery.clj:277)"
    "driver.bigquery$process_native_STAR_.invokeStatic(bigquery.clj:279)"
    "driver.bigquery$process_native_STAR_.invoke(bigquery.clj:272)"
    "driver.bigquery$eval1580$fn__1582.invoke(bigquery.clj:299)"
    "query_processor.context$executef.invokeStatic(context.clj:59)"
    "query_processor.context$executef.invoke(context.clj:48)"
    "query_processor.context.default$default_runf.invokeStatic(default.clj:68)"
    "query_processor.context.default$default_runf.invoke(default.clj:66)"
    "query_processor.context$runf.invokeStatic(context.clj:45)"
    "query_processor.context$runf.invoke(context.clj:39)"
    "query_processor.reducible$pivot.invokeStatic(reducible.clj:34)"
    "query_processor.reducible$pivot.invoke(reducible.clj:31)"
    "query_processor.middleware.mbql_to_native$mbql__GT_native$fn__48167.invoke(mbql_to_native.clj:25)"
    "query_processor.middleware.check_features$check_features$fn__47409.invoke(check_features.clj:39)"
    "query_processor.middleware.limit$limit$fn__48153.invoke(limit.clj:37)"
    "query_processor.middleware.cache$run_query_with_cache.invokeStatic(cache.clj:184)"
    "query_processor.middleware.cache$run_query_with_cache.invoke(cache.clj:175)"
    "query_processor.middleware.cache$maybe_return_cached_results$fn__46861.invoke(cache.clj:210)"
    "query_processor.middleware.optimize_temporal_filters$optimize_temporal_filters$fn__48413.invoke(optimize_temporal_filters.clj:204)"
    "query_processor.middleware.validate_temporal_bucketing$validate_temporal_bucketing$fn__50343.invoke(validate_temporal_bucketing.clj:50)"
    "query_processor.middleware.auto_parse_filter_values$auto_parse_filter_values$fn__45980.invoke(auto_parse_filter_values.clj:43)"
    "query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__41576.invoke(wrap_value_literals.clj:161)"
    "query_processor.middleware.annotate$add_column_info$fn__41473.invoke(annotate.clj:605)"
    "query_processor.middleware.permissions$check_query_permissions$fn__47281.invoke(permissions.clj:81)"
    "query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__49271.invoke(pre_alias_aggregations.clj:40)"
    "query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__47482.invoke(cumulative_aggregations.clj:60)"
    "metabase_enterprise.sandbox.query_processor.middleware.row_level_restrictions$apply_row_level_permissions$fn__50861.invoke(row_level_restrictions.clj:335)"
    "query_processor.middleware.resolve_joined_fields$resolve_joined_fields$fn__49570.invoke(resolve_joined_fields.clj:102)"
    "query_processor.middleware.resolve_joins$resolve_joins$fn__49883.invoke(resolve_joins.clj:171)"
    "query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__45556.invoke(add_implicit_joins.clj:190)"
    "query_processor.middleware.large_int_id$convert_id_to_string$fn__48117.invoke(large_int_id.clj:59)"
    "query_processor.middleware.format_rows$format_rows$fn__48098.invoke(format_rows.clj:74)"
    "query_processor.middleware.add_default_temporal_unit$add_default_temporal_unit$fn__44850.invoke(add_default_temporal_unit.clj:23)"
    "query_processor.middleware.desugar$desugar$fn__47548.invoke(desugar.clj:21)"
    "query_processor.middleware.binning$update_binning_strategy$fn__46367.invoke(binning.clj:227)"
    "query_processor.middleware.resolve_fields$resolve_fields$fn__47084.invoke(resolve_fields.clj:34)"
    "query_processor.middleware.add_dimension_projections$add_remapping$fn__45205.invoke(add_dimension_projections.clj:312)"
    "query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__45434.invoke(add_implicit_clauses.clj:147)"
    "metabase_enterprise.sandbox.query_processor.middleware.row_level_restrictions$apply_row_level_permissions$fn__50861.invoke(row_level_restrictions.clj:335)"
    "query_processor.middleware.upgrade_field_literals$upgrade_field_literals$fn__50292.invoke(upgrade_field_literals.clj:40)"
    "query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__45719.invoke(add_source_metadata.clj:123)"
    "metabase_enterprise.sandbox.query_processor.middleware.column_level_perms_check$maybe_apply_column_level_perms_check$fn__50373.invoke(column_level_perms_check.clj:30)"
    "query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__49445.invoke(reconcile_breakout_and_order_by_bucketing.clj:100)"
    "query_processor.middleware.auto_bucket_datetimes$auto_bucket_datetimes$fn__45927.invoke(auto_bucket_datetimes.clj:147)"
    "query_processor.middleware.resolve_source_table$resolve_source_tables$fn__47131.invoke(resolve_source_table.clj:45)"
    "query_processor.middleware.parameters$substitute_parameters$fn__49253.invoke(parameters.clj:111)"
    "query_processor.middleware.resolve_referenced$resolve_referenced_card_resources$fn__47183.invoke(resolve_referenced.clj:79)"
    "query_processor.middleware.expand_macros$expand_macros$fn__47804.invoke(expand_macros.clj:155)"
    "query_processor.middleware.add_timezone_info$add_timezone_info$fn__45728.invoke(add_timezone_info.clj:15)"
    "query_processor.middleware.splice_params_in_response$splice_params_in_response$fn__50245.invoke(splice_params_in_response.clj:32)"
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__49456$fn__49460.invoke(resolve_database_and_driver.clj:31)"
    "driver$do_with_driver.invokeStatic(driver.clj:60)"
    "driver$do_with_driver.invoke(driver.clj:56)"
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__49456.invoke(resolve_database_and_driver.clj:25)"
    "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__48044.invoke(fetch_source_query.clj:274)"
    "query_processor.middleware.store$initialize_store$fn__50254$fn__50255.invoke(store.clj:11)"
    "query_processor.store$do_with_store.invokeStatic(store.clj:44)"
    "query_processor.store$do_with_store.invoke(store.clj:38)"
    "query_processor.middleware.store$initialize_store$fn__50254.invoke(store.clj:10)"
    "query_processor.middleware.validate$validate_query$fn__50299.invoke(validate.clj:10)"
    "query_processor.middleware.normalize_query$normalize$fn__48180.invoke(normalize_query.clj:22)"
    "query_processor.middleware.add_rows_truncated$add_rows_truncated$fn__45574.invoke(add_rows_truncated.clj:35)"
    "metabase_enterprise.audit.query_processor.middleware.handle_audit_queries$handle_internal_queries$fn__31880.invoke(handle_audit_queries.clj:162)"
    "query_processor.middleware.results_metadata$record_and_return_metadata_BANG_$fn__50230.invoke(results_metadata.clj:147)"
    "query_processor.middleware.constraints$add_default_userland_constraints$fn__47425.invoke(constraints.clj:42)"
    "query_processor.middleware.process_userland_query$process_userland_query$fn__49342.invoke(process_userland_query.clj:135)"
    "query_processor.middleware.catch_exceptions$catch_exceptions$fn__47365.invoke(catch_exceptions.clj:173)"
    "query_processor.reducible$async_qp$qp_STAR___33705$thunk__33706.invoke(reducible.clj:103)"
    "query_processor.reducible$async_qp$qp_STAR___33705$fn__33708.invoke(reducible.clj:108)"]}
  {:status :failed,
   :class clojure.lang.ExceptionInfo,
   :error "Error reducing result rows",
   :stacktrace
   ["--> query_processor.context.default$default_reducef$fn__33619.invoke(default.clj:60)"
    "query_processor.context.default$default_reducef.invokeStatic(default.clj:57)"
    "query_processor.context.default$default_reducef.invoke(default.clj:48)"
    "query_processor.context$reducef.invokeStatic(context.clj:69)"
    "query_processor.context$reducef.invoke(context.clj:62)"
    "query_processor.context.default$default_runf$respond_STAR___33623.invoke(default.clj:69)"
    "driver.bigquery$post_process_native$fn__1478.invoke(bigquery.clj:251)"
    "driver.bigquery$do_with_finished_response.invokeStatic(bigquery.clj:171)"
    "driver.bigquery$do_with_finished_response.invoke(bigquery.clj:162)"
    "driver.bigquery$post_process_native.invokeStatic(bigquery.clj:234)"
    "driver.bigquery$post_process_native.invoke(bigquery.clj:228)"
    "driver.bigquery$process_native_STAR_$thunk__1576.invoke(bigquery.clj:277)"
    "driver.bigquery$process_native_STAR_.invokeStatic(bigquery.clj:279)"
    "driver.bigquery$process_native_STAR_.invoke(bigquery.clj:272)"
    "driver.bigquery$eval1580$fn__1582.invoke(bigquery.clj:299)"
    "query_processor.context$executef.invokeStatic(context.clj:59)"
    "query_processor.context$executef.invoke(context.clj:48)"
    "query_processor.context.default$default_runf.invokeStatic(default.clj:68)"
    "query_processor.context.default$default_runf.invoke(default.clj:66)"
    "query_processor.context$runf.invokeStatic(context.clj:45)"
    "query_processor.context$runf.invoke(context.clj:39)"
    "query_processor.reducible$pivot.invokeStatic(reducible.clj:34)"
    "query_processor.reducible$pivot.invoke(reducible.clj:31)"
    "query_processor.middleware.mbql_to_native$mbql__GT_native$fn__48167.invoke(mbql_to_native.clj:25)"
    "query_processor.middleware.check_features$check_features$fn__47409.invoke(check_features.clj:39)"
    "query_processor.middleware.limit$limit$fn__48153.invoke(limit.clj:37)"
    "query_processor.middleware.cache$run_query_with_cache.invokeStatic(cache.clj:184)"
    "query_processor.middleware.cache$run_query_with_cache.invoke(cache.clj:175)"
    "query_processor.middleware.cache$maybe_return_cached_results$fn__46861.invoke(cache.clj:210)"
    "query_processor.middleware.optimize_temporal_filters$optimize_temporal_filters$fn__48413.invoke(optimize_temporal_filters.clj:204)"
    "query_processor.middleware.validate_temporal_bucketing$validate_temporal_bucketing$fn__50343.invoke(validate_temporal_bucketing.clj:50)"
    "query_processor.middleware.auto_parse_filter_values$auto_parse_filter_values$fn__45980.invoke(auto_parse_filter_values.clj:43)"
    "query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__41576.invoke(wrap_value_literals.clj:161)"
    "query_processor.middleware.annotate$add_column_info$fn__41473.invoke(annotate.clj:605)"
    "query_processor.middleware.permissions$check_query_permissions$fn__47281.invoke(permissions.clj:81)"
    "query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__49271.invoke(pre_alias_aggregations.clj:40)"
    "query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__47482.invoke(cumulative_aggregations.clj:60)"
    "metabase_enterprise.sandbox.query_processor.middleware.row_level_restrictions$apply_row_level_permissions$fn__50861.invoke(row_level_restrictions.clj:335)"
    "query_processor.middleware.resolve_joined_fields$resolve_joined_fields$fn__49570.invoke(resolve_joined_fields.clj:102)"
    "query_processor.middleware.resolve_joins$resolve_joins$fn__49883.invoke(resolve_joins.clj:171)"
    "query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__45556.invoke(add_implicit_joins.clj:190)"
    "query_processor.middleware.large_int_id$convert_id_to_string$fn__48117.invoke(large_int_id.clj:59)"
    "query_processor.middleware.format_rows$format_rows$fn__48098.invoke(format_rows.clj:74)"
    "query_processor.middleware.add_default_temporal_unit$add_default_temporal_unit$fn__44850.invoke(add_default_temporal_unit.clj:23)"
    "query_processor.middleware.desugar$desugar$fn__47548.invoke(desugar.clj:21)"
    "query_processor.middleware.binning$update_binning_strategy$fn__46367.invoke(binning.clj:227)"
    "query_processor.middleware.resolve_fields$resolve_fields$fn__47084.invoke(resolve_fields.clj:34)"
    "query_processor.middleware.add_dimension_projections$add_remapping$fn__45205.invoke(add_dimension_projections.clj:312)"
    "query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__45434.invoke(add_implicit_clauses.clj:147)"
    "metabase_enterprise.sandbox.query_processor.middleware.row_level_restrictions$apply_row_level_permissions$fn__50861.invoke(row_level_restrictions.clj:335)"
    "query_processor.middleware.upgrade_field_literals$upgrade_field_literals$fn__50292.invoke(upgrade_field_literals.clj:40)"
    "query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__45719.invoke(add_source_metadata.clj:123)"
    "metabase_enterprise.sandbox.query_processor.middleware.column_level_perms_check$maybe_apply_column_level_perms_check$fn__50373.invoke(column_level_perms_check.clj:30)"
    "query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__49445.invoke(reconcile_breakout_and_order_by_bucketing.clj:100)"
    "query_processor.middleware.auto_bucket_datetimes$auto_bucket_datetimes$fn__45927.invoke(auto_bucket_datetimes.clj:147)"
    "query_processor.middleware.resolve_source_table$resolve_source_tables$fn__47131.invoke(resolve_source_table.clj:45)"
    "query_processor.middleware.parameters$substitute_parameters$fn__49253.invoke(parameters.clj:111)"
    "query_processor.middleware.resolve_referenced$resolve_referenced_card_resources$fn__47183.invoke(resolve_referenced.clj:79)"
    "query_processor.middleware.expand_macros$expand_macros$fn__47804.invoke(expand_macros.clj:155)"
    "query_processor.middleware.add_timezone_info$add_timezone_info$fn__45728.invoke(add_timezone_info.clj:15)"
    "query_processor.middleware.splice_params_in_response$splice_params_in_response$fn__50245.invoke(splice_params_in_response.clj:32)"
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__49456$fn__49460.invoke(resolve_database_and_driver.clj:31)"
    "driver$do_with_driver.invokeStatic(driver.clj:60)"
    "driver$do_with_driver.invoke(driver.clj:56)"
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__49456.invoke(resolve_database_and_driver.clj:25)"
    "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__48044.invoke(fetch_source_query.clj:274)"
    "query_processor.middleware.store$initialize_store$fn__50254$fn__50255.invoke(store.clj:11)"
    "query_processor.store$do_with_store.invokeStatic(store.clj:44)"
    "query_processor.store$do_with_store.invoke(store.clj:38)"
    "query_processor.middleware.store$initialize_store$fn__50254.invoke(store.clj:10)"
    "query_processor.middleware.validate$validate_query$fn__50299.invoke(validate.clj:10)"
    "query_processor.middleware.normalize_query$normalize$fn__48180.invoke(normalize_query.clj:22)"
    "query_processor.middleware.add_rows_truncated$add_rows_truncated$fn__45574.invoke(add_rows_truncated.clj:35)"
    "metabase_enterprise.audit.query_processor.middleware.handle_audit_queries$handle_internal_queries$fn__31880.invoke(handle_audit_queries.clj:162)"
    "query_processor.middleware.results_metadata$record_and_return_metadata_BANG_$fn__50230.invoke(results_metadata.clj:147)"
    "query_processor.middleware.constraints$add_default_userland_constraints$fn__47425.invoke(constraints.clj:42)"
    "query_processor.middleware.process_userland_query$process_userland_query$fn__49342.invoke(process_userland_query.clj:135)"
    "query_processor.middleware.catch_exceptions$catch_exceptions$fn__47365.invoke(catch_exceptions.clj:173)"
    "query_processor.reducible$async_qp$qp_STAR___33705$thunk__33706.invoke(reducible.clj:103)"
    "query_processor.reducible$async_qp$qp_STAR___33705$fn__33708.invoke(reducible.clj:108)"],
   :error_type :qp,
   :ex-data {:type :qp}}],
 :error_type :qp,
 :json_query
 {:constraints {:max-results 10000, :max-results-bare-rows 2000},
  :type :native,
  :middleware {:js-int-to-string? true},
  :native
  {:query
   "WITH BT AS (\nSELECT * FROM `integrated_reporting_precomp.integrated_activity`\nLEFT JOIN \n(SELECT * FROM `integrated_reporting_precomp.total_user_groups` WHERE {{user_group}}) `Total User Groups`\nUSING (user_id)\nWHERE {{company_name}}\n),\nDAU_TBL AS (\nSELECT COUNT(DISTINCT user_id) AS dau, `date`, date_trunc(`date`, month) AS month_date,\nFROM BT\nGROUP BY `date`\nORDER BY `date`\n\n),\nMAU_TBL AS (\nSELECT COUNT(DISTINCT user_id) AS mau, date_trunc(`date`, month) AS month_date,\nFROM BT\nGROUP BY `month_date`\nORDER BY `month_date`\n),\nCOMBINED AS \n(\nSELECT *, dau / mau AS `dau_mau_ratio`\nFROM DAU_TBL\nLEFT JOIN MAU_TBL\nUSING (month_date)\n)\n\n\n\nSELECT * FROM COMBINED",
   :template-tags
   {"user_group"
    {:id "1cb59f39-a306-2e37-9db7-6cd8ba6d96d5",
     :name "user_group",
     :display-name "User group",
     :type :dimension,
     :dimension [:field 190 nil],
     :widget-type :category},
    "company_name"
    {:id "01b0f29f-1417-29b3-3ee0-b6365c7c2ab6",
     :name "company_name",
     :display-name "Company name",
     :type :dimension,
     :dimension [:field 201 nil],
     :widget-type :category}}},
  :database 5,
  :parameters
  [{:type "category", :value ["Sunlife"], :target ["dimension" ["template-tag" "company_name"]]}
   {:type "category", :value ["global"], :target ["dimension" ["template-tag" "user_group"]]}],
  :async? true,
  :cache-ttl 908},
 :status :failed,
 :class java.io.IOException,
 :stacktrace
 ["java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)"
  "java.base/sun.nio.ch.SocketDispatcher.writev(Unknown Source)"
  "java.base/sun.nio.ch.IOUtil.write(Unknown Source)"
  "java.base/sun.nio.ch.IOUtil.write(Unknown Source)"
  "java.base/sun.nio.ch.SocketChannelImpl.write(Unknown Source)"
  "java.base/java.nio.channels.SocketChannel.write(Unknown Source)"
  "org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:273)"
  "org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)"
  "org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)"
  "org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)"
  "org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:820)"
  "org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)"
  "org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)"
  "org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:544)"
  "org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:838)"
  "org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:910)"
  "org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:284)"
  "org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:268)"
  "org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:713)"
  "java.base/java.util.zip.DeflaterOutputStream.flush(Unknown Source)"
  "--> async.streaming_response$delay_output_stream$fn__17621.invoke(streaming_response.clj:119)"
  "async.streaming_response.proxy$java.io.OutputStream$ff19274a.flush(Unknown Source)"
  "query_processor.streaming.json$fn$reify__65061.write_row_BANG_(json.clj:64)"
  "query_processor.streaming$streaming_rff$fn__65556$fn__65557.invoke(streaming.clj:34)"
  "query_processor.middleware.process_userland_query$add_and_save_execution_info_xform_BANG_$execution_info_rf_STAR___49333.invoke(process_userland_query.clj:94)"
  "query_processor.reducible$combine_additional_reducing_fns$fn__33728.invoke(reducible.clj:216)"
  "query_processor.middleware.add_rows_truncated$add_rows_truncated_xform$fn__45569.invoke(add_rows_truncated.clj:27)"
  "query_processor.middleware.format_rows$format_rows_xform$fn__48091.invoke(format_rows.clj:68)"
  "query_processor.reducible$combine_additional_reducing_fns$fn__33728.invoke(reducible.clj:216)"
  "query_processor.middleware.cache$save_results_xform$fn__46819.invoke(cache.clj:114)"
  "query_processor.middleware.limit$limit_xform$fn__48150.invoke(limit.clj:24)"
  "query_processor.context.default$default_reducef$fn__33619.invoke(default.clj:58)"
  "query_processor.context.default$default_reducef.invokeStatic(default.clj:57)"
  "query_processor.context.default$default_reducef.invoke(default.clj:48)"
  "query_processor.context$reducef.invokeStatic(context.clj:69)"
  "query_processor.context$reducef.invoke(context.clj:62)"
  "query_processor.context.default$default_runf$respond_STAR___33623.invoke(default.clj:69)"
  "driver.bigquery$post_process_native$fn__1478.invoke(bigquery.clj:251)"
  "driver.bigquery$do_with_finished_response.invokeStatic(bigquery.clj:171)"
  "driver.bigquery$do_with_finished_response.invoke(bigquery.clj:162)"
  "driver.bigquery$post_process_native.invokeStatic(bigquery.clj:234)"
  "driver.bigquery$post_process_native.invoke(bigquery.clj:228)"
  "driver.bigquery$process_native_STAR_$thunk__1576.invoke(bigquery.clj:277)"
  "driver.bigquery$process_native_STAR_.invokeStatic(bigquery.clj:279)"
  "driver.bigquery$process_native_STAR_.invoke(bigquery.clj:272)"
  "driver.bigquery$eval1580$fn__1582.invoke(bigquery.clj:299)"
  "query_processor.context$executef.invokeStatic(context.clj:59)"
  "query_processor.context$executef.invoke(context.clj:48)"
  "query_processor.context.default$default_runf.invokeStatic(default.clj:68)"
  "query_processor.context.default$default_runf.invoke(default.clj:66)"
  "query_processor.context$runf.invokeStatic(context.clj:45)"
  "query_processor.context$runf.invoke(context.clj:39)"
  "query_processor.reducible$pivot.invokeStatic(reducible.clj:34)"
  "query_processor.reducible$pivot.invoke(reducible.clj:31)"
  "query_processor.middleware.mbql_to_native$mbql__GT_native$fn__48167.invoke(mbql_to_native.clj:25)"
  "query_processor.middleware.check_features$check_features$fn__47409.invoke(check_features.clj:39)"
  "query_processor.middleware.limit$limit$fn__48153.invoke(limit.clj:37)"
  "query_processor.middleware.cache$run_query_with_cache.invokeStatic(cache.clj:184)"
  "query_processor.middleware.cache$run_query_with_cache.invoke(cache.clj:175)"
  "query_processor.middleware.cache$maybe_return_cached_results$fn__46861.invoke(cache.clj:210)"
  "query_processor.middleware.optimize_temporal_filters$optimize_temporal_filters$fn__48413.invoke(optimize_temporal_filters.clj:204)"
  "query_processor.middleware.validate_temporal_bucketing$validate_temporal_bucketing$fn__50343.invoke(validate_temporal_bucketing.clj:50)"
  "query_processor.middleware.auto_parse_filter_values$auto_parse_filter_values$fn__45980.invoke(auto_parse_filter_values.clj:43)"
  "query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__41576.invoke(wrap_value_literals.clj:161)"
  "query_processor.middleware.annotate$add_column_info$fn__41473.invoke(annotate.clj:605)"
  "query_processor.middleware.permissions$check_query_permissions$fn__47281.invoke(permissions.clj:81)"
  "query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__49271.invoke(pre_alias_aggregations.clj:40)"
  "query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__47482.invoke(cumulative_aggregations.clj:60)"
  "metabase_enterprise.sandbox.query_processor.middleware.row_level_restrictions$apply_row_level_permissions$fn__50861.invoke(row_level_restrictions.clj:335)"
  "query_processor.middleware.resolve_joined_fields$resolve_joined_fields$fn__49570.invoke(resolve_joined_fields.clj:102)"
  "query_processor.middleware.resolve_joins$resolve_joins$fn__49883.invoke(resolve_joins.clj:171)"
  "query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__45556.invoke(add_implicit_joins.clj:190)"
  "query_processor.middleware.large_int_id$convert_id_to_string$fn__48117.invoke(large_int_id.clj:59)"
  "query_processor.middleware.format_rows$format_rows$fn__48098.invoke(format_rows.clj:74)"
  "query_processor.middleware.add_default_temporal_unit$add_default_temporal_unit$fn__44850.invoke(add_default_temporal_unit.clj:23)"
  "query_processor.middleware.desugar$desugar$fn__47548.invoke(desugar.clj:21)"
  "query_processor.middleware.binning$update_binning_strategy$fn__46367.invoke(binning.clj:227)"
  "query_processor.middleware.resolve_fields$resolve_fields$fn__47084.invoke(resolve_fields.clj:34)"
  "query_processor.middleware.add_dimension_projections$add_remapping$fn__45205.invoke(add_dimension_projections.clj:312)"
  "query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__45434.invoke(add_implicit_clauses.clj:147)"
  "metabase_enterprise.sandbox.query_processor.middleware.row_level_restrictions$apply_row_level_permissions$fn__50861.invoke(row_level_restrictions.clj:335)"
  "query_processor.middleware.upgrade_field_literals$upgrade_field_literals$fn__50292.invoke(upgrade_field_literals.clj:40)"
  "query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__45719.invoke(add_source_metadata.clj:123)"
  "metabase_enterprise.sandbox.query_processor.middleware.column_level_perms_check$maybe_apply_column_level_perms_check$fn__50373.invoke(column_level_perms_check.clj:30)"
  "query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__49445.invoke(reconcile_breakout_and_order_by_bucketing.clj:100)"
  "query_processor.middleware.auto_bucket_datetimes$auto_bucket_datetimes$fn__45927.invoke(auto_bucket_datetimes.clj:147)"
  "query_processor.middleware.resolve_source_table$resolve_source_tables$fn__47131.invoke(resolve_source_table.clj:45)"
  "query_processor.middleware.parameters$substitute_parameters$fn__49253.invoke(parameters.clj:111)"
  "query_processor.middleware.resolve_referenced$resolve_referenced_card_resources$fn__47183.invoke(resolve_referenced.clj:79)"
  "query_processor.middleware.expand_macros$expand_macros$fn__47804.invoke(expand_macros.clj:155)"
  "query_processor.middleware.add_timezone_info$add_timezone_info$fn__45728.invoke(add_timezone_info.clj:15)"
  "query_processor.middleware.splice_params_in_response$splice_params_in_response$fn__50245.invoke(splice_params_in_response.clj:32)"
  "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__49456$fn__49460.invoke(resolve_database_and_driver.clj:31)"
  "driver$do_with_driver.invokeStatic(driver.clj:60)"
  "driver$do_with_driver.invoke(driver.clj:56)"
  "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__49456.invoke(resolve_database_and_driver.clj:25)"
  "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__48044.invoke(fetch_source_query.clj:274)"
  "query_processor.middleware.store$initialize_store$fn__50254$fn__50255.invoke(store.clj:11)"
  "query_processor.store$do_with_store.invokeStatic(store.clj:44)"
  "query_processor.store$do_with_store.invoke(store.clj:38)"
  "query_processor.middleware.store$initialize_store$fn__50254.invoke(store.clj:10)"
  "query_processor.middleware.validate$validate_query$fn__50299.invoke(validate.clj:10)"
  "query_processor.middleware.normalize_query$normalize$fn__48180.invoke(normalize_query.clj:22)"
  "query_processor.middleware.add_rows_truncated$add_rows_truncated$fn__45574.invoke(add_rows_truncated.clj:35)"
  "metabase_enterprise.audit.query_processor.middleware.handle_audit_queries$handle_internal_queries$fn__31880.invoke(handle_audit_queries.clj:162)"
  "query_processor.middleware.results_metadata$record_and_return_metadata_BANG_$fn__50230.invoke(results_metadata.clj:147)"
  "query_processor.middleware.constraints$add_default_userland_constraints$fn__47425.invoke(constraints.clj:42)"
  "query_processor.middleware.process_userland_query$process_userland_query$fn__49342.invoke(process_userland_query.clj:135)"
  "query_processor.middleware.catch_exceptions$catch_exceptions$fn__47365.invoke(catch_exceptions.clj:173)"
  "query_processor.reducible$async_qp$qp_STAR___33705$thunk__33706.invoke(reducible.clj:103)"
  "query_processor.reducible$async_qp$qp_STAR___33705$fn__33708.invoke(reducible.clj:108)"],
 :context :question,
 :error "Broken pipe",
 :row_count 0,
 :running_time 0,
 :data {:rows [], :cols []}}
2

Hi @Kenneth

Please post "Diagnostic Info" from Admin > Troubleshooting, and which database type you are querying.

I'm unsure if the error you are posting is related to the slow questions - it sounds like it's not.

Which type of questions are slow - QB or SQL based?
If SQL, can you post the query? And if you run the query directly on your database, does that use indexes instead of scans?
Have you tried running with EXPLAIN?
https://www.metabase.com/learn/building-analytics/sql-templates/sql-best-practices.html#explain

3

Hi @flamber, these were queries run using the query builder, the resultant SQL was generated on the metabase end I believe. This is also a new issue, the same queries have been run in the past, and didn't take nearly as long. I believe that they might have been failing a few times before suceeding.

in the case, what I observed was queries would seem to time out, not returning data (run button, but no generated result) multiple times before succeeding sometime in the future.

{
  "browser-info": {
    "language": "en-GB",
    "platform": "Win32",
    "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36",
    "vendor": "Google Inc."
  },
  "system-info": {
    "file.encoding": "UTF-8",
    "java.runtime.name": "OpenJDK Runtime Environment",
    "java.runtime.version": "11.0.11+9",
    "java.vendor": "AdoptOpenJDK",
    "java.vendor.url": "https://adoptopenjdk.net/",
    "java.version": "11.0.11",
    "java.vm.name": "OpenJDK 64-Bit Server VM",
    "java.vm.version": "11.0.11+9",
    "os.name": "Linux",
    "os.version": "4.14.219-164.354.amzn2.x86_64",
    "user.language": "en",
    "user.timezone": "GMT"
  },
  "metabase-info": {
    "databases": [
      "h2",
      "bigquery"
    ],
    "hosting-env": "unknown",
    "application-database": "postgres",
    "application-database-details": {
      "database": {
        "name": "PostgreSQL",
        "version": "11.9"
      },
      "jdbc-driver": {
        "name": "PostgreSQL JDBC Driver",
        "version": "42.2.18"
      }
    },
    "run-mode": "prod",
    "version": {
      "date": "2021-05-17",
      "tag": "v1.39.2",
      "branch": "release-x.39.x",
      "hash": "b0d9436"
    },
    "settings": {
      "report-timezone": null
    }
  }
}
4

@Kenneth Okay, I'm getting really confused, so can we only focus on one issue at the time? You're welcome to create other threads for the other issues, but trying to understand a problem, when there's too much unrelated information can be very difficult.
Which issue do you want to focus on? Then supply only logs related to the time of that issue. And if it's a change compared to previously, then define what "previously" means and how it used to function.

5

@flamber Apologies if it's confusing, there is exactly one issue, that seemed to resolve itself. I'm raising it largely in case it stems from something more critical.

Specifically, a simple group by -> Aggregate through the query builder resulted in no generated table. Checking the logs, I identified that that log stemmed from said error via the timestamps.

6

@Kenneth Okay, so the problem has solved itself now? Then I would guess it might have something to do with connections to your BigQuery region. I have not seen any reports similar to this.

7

The problem seems to have resolved itself, and I'll be keeping an eye on it if it surfaces again.