"Error reducing result rows" in version 0.38.1

On BigQuery, my queries run perfectly, but Metabase isn’t able to return the results.

I’m aware this issue has been reported before, but it was supposed to be corrected by version 0.35.
I faced this same problem back in v0.36 and, after upgrading to 0.38.0.1, it was gone for a while, but it’s happening again now. Upgrading to v0.38.1 was not enough this time.

The issue seems to occur specifically with SQL mode questions. It also seems to be related to query complexity vs. result size, as when I try simpler single-table queries or a more strict limit clause the result is returned perfectly.

Any clue on how to solve this? Any help will be very much appreciated!

Logs

{:database_id 130,
 :started_at #t "2021-03-18T15:46:50.244368Z[GMT]",
 :via
 [{:status :failed,
   :class org.eclipse.jetty.io.EofException,
   :error nil,
   :stacktrace
   ["org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)"
    "org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)"
    "org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)"
    "org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)"
    "org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:826)"
    "org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)"
    "org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)"
    "org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:544)"
    "org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:838)"
    "org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:910)"
    "org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:284)"
    "org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:268)"
    "org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:713)"
    "java.base/java.util.zip.DeflaterOutputStream.flush(Unknown Source)"
    "--> async.streaming_response$delay_output_stream$fn__16072.invoke(streaming_response.clj:119)"
    "async.streaming_response.proxy$java.io.OutputStream$ff19274a.flush(Unknown Source)"
    "query_processor.streaming.json$fn$reify__53817.write_row_BANG_(json.clj:62)"
    "query_processor.streaming$streaming_rff$fn__54285$fn__54286.invoke(streaming.clj:34)"
    "query_processor.middleware.process_userland_query$add_and_save_execution_info_xform_BANG_$execution_info_rf_STAR___47164.invoke(process_userland_query.clj:94)"
    "query_processor.reducible$combine_additional_reducing_fns$fn__37451.invoke(reducible.clj:216)"
    "query_processor.middleware.add_rows_truncated$add_rows_truncated_xform$fn__44077.invoke(add_rows_truncated.clj:27)"
    "query_processor.middleware.format_rows$format_rows_xform$fn__46348.invoke(format_rows.clj:68)"
    "query_processor.reducible$combine_additional_reducing_fns$fn__37451.invoke(reducible.clj:216)"
    "query_processor.middleware.limit$limit_xform$fn__46397.invoke(limit.clj:24)"
    "query_processor.context.default$default_reducef$fn__36875.invoke(default.clj:58)"
    "query_processor.context.default$default_reducef.invokeStatic(default.clj:57)"
    "query_processor.context.default$default_reducef.invoke(default.clj:48)"
    "query_processor.context$reducef.invokeStatic(context.clj:69)"
    "query_processor.context$reducef.invoke(context.clj:62)"
    "query_processor.context.default$default_runf$respond_STAR___36879.invoke(default.clj:69)"
    "driver.bigquery$post_process_native$fn__1498.invoke(bigquery.clj:251)"
    "driver.bigquery$do_with_finished_response.invokeStatic(bigquery.clj:171)"
    "driver.bigquery$do_with_finished_response.invoke(bigquery.clj:162)"
    "driver.bigquery$post_process_native.invokeStatic(bigquery.clj:234)"
    "driver.bigquery$post_process_native.invoke(bigquery.clj:228)"
    "driver.bigquery$process_native_STAR_$thunk__1596.invoke(bigquery.clj:277)"
    "driver.bigquery$process_native_STAR_.invokeStatic(bigquery.clj:279)"
    "driver.bigquery$process_native_STAR_.invoke(bigquery.clj:272)"
    "driver.bigquery$eval1600$fn__1602.invoke(bigquery.clj:299)"
    "query_processor.context$executef.invokeStatic(context.clj:59)"
    "query_processor.context$executef.invoke(context.clj:48)"
    "query_processor.context.default$default_runf.invokeStatic(default.clj:68)"
    "query_processor.context.default$default_runf.invoke(default.clj:66)"
    "query_processor.context$runf.invokeStatic(context.clj:45)"
    "query_processor.context$runf.invoke(context.clj:39)"
    "query_processor.reducible$pivot.invokeStatic(reducible.clj:34)"
    "query_processor.reducible$pivot.invoke(reducible.clj:31)"
    "query_processor.middleware.mbql_to_native$mbql__GT_native$fn__46414.invoke(mbql_to_native.clj:25)"
    "query_processor.middleware.check_features$check_features$fn__45666.invoke(check_features.clj:41)"
    "query_processor.middleware.limit$limit$fn__46400.invoke(limit.clj:37)"
    "query_processor.middleware.cache$maybe_return_cached_results$fn__45324.invoke(cache.clj:211)"
    "query_processor.middleware.optimize_datetime_filters$optimize_datetime_filters$fn__46579.invoke(optimize_datetime_filters.clj:133)"
    "query_processor.middleware.auto_parse_filter_values$auto_parse_filter_values$fn__44470.invoke(auto_parse_filter_values.clj:43)"
    "query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__40787.invoke(wrap_value_literals.clj:147)"
    "query_processor.middleware.annotate$add_column_info$fn__40650.invoke(annotate.clj:582)"
    "query_processor.middleware.permissions$check_query_permissions$fn__45541.invoke(permissions.clj:75)"
    "query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__47102.invoke(pre_alias_aggregations.clj:40)"
    "query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__45739.invoke(cumulative_aggregations.clj:60)"
    "query_processor.middleware.resolve_joined_fields$resolve_joined_fields$fn__47415.invoke(resolve_joined_fields.clj:94)"
    "query_processor.middleware.resolve_joins$resolve_joins$fn__47720.invoke(resolve_joins.clj:178)"
    "query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__44064.invoke(add_implicit_joins.clj:181)"
    "query_processor.middleware.large_int_id$convert_id_to_string$fn__46375.invoke(large_int_id.clj:44)"
    "query_processor.middleware.format_rows$format_rows$fn__46355.invoke(format_rows.clj:74)"
    "query_processor.middleware.desugar$desugar$fn__45805.invoke(desugar.clj:21)"
    "query_processor.middleware.binning$update_binning_strategy$fn__44830.invoke(binning.clj:228)"
    "query_processor.middleware.resolve_fields$resolve_fields$fn__45348.invoke(resolve_fields.clj:24)"
    "query_processor.middleware.add_dimension_projections$add_remapping$fn__43694.invoke(add_dimension_projections.clj:314)"
    "query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__43925.invoke(add_implicit_clauses.clj:146)"
    "query_processor.middleware.upgrade_field_literals$upgrade_field_literals$fn__48150.invoke(upgrade_field_literals.clj:45)"
    "query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__44217.invoke(add_source_metadata.clj:122)"
    "query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__47299.invoke(reconcile_breakout_and_order_by_bucketing.clj:97)"
    "query_processor.middleware.auto_bucket_datetimes$auto_bucket_datetimes$fn__44417.invoke(auto_bucket_datetimes.clj:139)"
    "query_processor.middleware.resolve_source_table$resolve_source_tables$fn__45395.invoke(resolve_source_table.clj:45)"
    "query_processor.middleware.parameters$substitute_parameters$fn__47084.invoke(parameters.clj:111)"
    "query_processor.middleware.resolve_referenced$resolve_referenced_card_resources$fn__45447.invoke(resolve_referenced.clj:79)"
    "query_processor.middleware.expand_macros$expand_macros$fn__46061.invoke(expand_macros.clj:155)"
    "query_processor.middleware.add_timezone_info$add_timezone_info$fn__44226.invoke(add_timezone_info.clj:15)"
    "query_processor.middleware.splice_params_in_response$splice_params_in_response$fn__48086.invoke(splice_params_in_response.clj:32)"
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__47310$fn__47314.invoke(resolve_database_and_driver.clj:31)"
    "driver$do_with_driver.invokeStatic(driver.clj:60)"
    "driver$do_with_driver.invoke(driver.clj:56)"
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__47310.invoke(resolve_database_and_driver.clj:25)"
    "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__46301.invoke(fetch_source_query.clj:274)"
    "query_processor.middleware.store$initialize_store$fn__48095$fn__48096.invoke(store.clj:11)"
    "query_processor.store$do_with_store.invokeStatic(store.clj:44)"
    "query_processor.store$do_with_store.invoke(store.clj:38)"
    "query_processor.middleware.store$initialize_store$fn__48095.invoke(store.clj:10)"
    "query_processor.middleware.validate$validate_query$fn__48157.invoke(validate.clj:10)"
    "query_processor.middleware.normalize_query$normalize$fn__46427.invoke(normalize_query.clj:22)"
    "query_processor.middleware.add_rows_truncated$add_rows_truncated$fn__44082.invoke(add_rows_truncated.clj:35)"
    "query_processor.middleware.results_metadata$record_and_return_metadata_BANG_$fn__48071.invoke(results_metadata.clj:147)"
    "query_processor.middleware.constraints$add_default_userland_constraints$fn__45682.invoke(constraints.clj:42)"
    "query_processor.middleware.process_userland_query$process_userland_query$fn__47173.invoke(process_userland_query.clj:135)"
    "query_processor.middleware.catch_exceptions$catch_exceptions$fn__45625.invoke(catch_exceptions.clj:173)"
    "query_processor.reducible$async_qp$qp_STAR___37428$thunk__37429.invoke(reducible.clj:103)"
    "query_processor.reducible$async_qp$qp_STAR___37428.invoke(reducible.clj:109)"
    "query_processor.reducible$sync_qp$qp_STAR___37437$fn__37440.invoke(reducible.clj:135)"
    "query_processor.reducible$sync_qp$qp_STAR___37437.invoke(reducible.clj:134)"
    "query_processor$process_userland_query.invokeStatic(query_processor.clj:237)"
    "query_processor$process_userland_query.doInvoke(query_processor.clj:233)"
    "query_processor$fn__48203$process_query_and_save_execution_BANG___48212$fn__48215.invoke(query_processor.clj:249)"
    "query_processor$fn__48203$process_query_and_save_execution_BANG___48212.invoke(query_processor.clj:241)"
    "query_processor$fn__48247$process_query_and_save_with_max_results_constraints_BANG___48256$fn__48259.invoke(query_processor.clj:261)"
    "query_processor$fn__48247$process_query_and_save_with_max_results_constraints_BANG___48256.invoke(query_processor.clj:254)"
    "api.dataset$run_query_async$fn__54361.invoke(dataset.clj:56)"
    "query_processor.streaming$streaming_response_STAR_$fn__54340$fn__54341.invoke(streaming.clj:72)"
    "query_processor.streaming$streaming_response_STAR_$fn__54340.invoke(streaming.clj:71)"
    "async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:65)"
    "async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:63)"
    "async.streaming_response$do_f_async$fn__16055.invoke(streaming_response.clj:84)"]}
  {:status :failed,
   :class clojure.lang.ExceptionInfo,
   :error "Error reducing result rows",
   :stacktrace
   ["--> query_processor.context.default$default_reducef$fn__36875.invoke(default.clj:60)"
    "query_processor.context.default$default_reducef.invokeStatic(default.clj:57)"
    "query_processor.context.default$default_reducef.invoke(default.clj:48)"
    "query_processor.context$reducef.invokeStatic(context.clj:69)"
    "query_processor.context$reducef.invoke(context.clj:62)"
    "query_processor.context.default$default_runf$respond_STAR___36879.invoke(default.clj:69)"
    "driver.bigquery$post_process_native$fn__1498.invoke(bigquery.clj:251)"
    "driver.bigquery$do_with_finished_response.invokeStatic(bigquery.clj:171)"
    "driver.bigquery$do_with_finished_response.invoke(bigquery.clj:162)"
    "driver.bigquery$post_process_native.invokeStatic(bigquery.clj:234)"
    "driver.bigquery$post_process_native.invoke(bigquery.clj:228)"
    "driver.bigquery$process_native_STAR_$thunk__1596.invoke(bigquery.clj:277)"
    "driver.bigquery$process_native_STAR_.invokeStatic(bigquery.clj:279)"
    "driver.bigquery$process_native_STAR_.invoke(bigquery.clj:272)"
    "driver.bigquery$eval1600$fn__1602.invoke(bigquery.clj:299)"
    "query_processor.context$executef.invokeStatic(context.clj:59)"
    "query_processor.context$executef.invoke(context.clj:48)"
    "query_processor.context.default$default_runf.invokeStatic(default.clj:68)"
    "query_processor.context.default$default_runf.invoke(default.clj:66)"
    "query_processor.context$runf.invokeStatic(context.clj:45)"
    "query_processor.context$runf.invoke(context.clj:39)"
    "query_processor.reducible$pivot.invokeStatic(reducible.clj:34)"
    "query_processor.reducible$pivot.invoke(reducible.clj:31)"
    "query_processor.middleware.mbql_to_native$mbql__GT_native$fn__46414.invoke(mbql_to_native.clj:25)"
    "query_processor.middleware.check_features$check_features$fn__45666.invoke(check_features.clj:41)"
    "query_processor.middleware.limit$limit$fn__46400.invoke(limit.clj:37)"
    "query_processor.middleware.cache$maybe_return_cached_results$fn__45324.invoke(cache.clj:211)"
    "query_processor.middleware.optimize_datetime_filters$optimize_datetime_filters$fn__46579.invoke(optimize_datetime_filters.clj:133)"
    "query_processor.middleware.auto_parse_filter_values$auto_parse_filter_values$fn__44470.invoke(auto_parse_filter_values.clj:43)"
    "query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__40787.invoke(wrap_value_literals.clj:147)"
    "query_processor.middleware.annotate$add_column_info$fn__40650.invoke(annotate.clj:582)"
    "query_processor.middleware.permissions$check_query_permissions$fn__45541.invoke(permissions.clj:75)"
    "query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__47102.invoke(pre_alias_aggregations.clj:40)"
    "query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__45739.invoke(cumulative_aggregations.clj:60)"
    "query_processor.middleware.resolve_joined_fields$resolve_joined_fields$fn__47415.invoke(resolve_joined_fields.clj:94)"
    "query_processor.middleware.resolve_joins$resolve_joins$fn__47720.invoke(resolve_joins.clj:178)"
    "query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__44064.invoke(add_implicit_joins.clj:181)"
    "query_processor.middleware.large_int_id$convert_id_to_string$fn__46375.invoke(large_int_id.clj:44)"
    "query_processor.middleware.format_rows$format_rows$fn__46355.invoke(format_rows.clj:74)"
    "query_processor.middleware.desugar$desugar$fn__45805.invoke(desugar.clj:21)"
    "query_processor.middleware.binning$update_binning_strategy$fn__44830.invoke(binning.clj:228)"
    "query_processor.middleware.resolve_fields$resolve_fields$fn__45348.invoke(resolve_fields.clj:24)"
    "query_processor.middleware.add_dimension_projections$add_remapping$fn__43694.invoke(add_dimension_projections.clj:314)"
    "query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__43925.invoke(add_implicit_clauses.clj:146)"
    "query_processor.middleware.upgrade_field_literals$upgrade_field_literals$fn__48150.invoke(upgrade_field_literals.clj:45)"
    "query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__44217.invoke(add_source_metadata.clj:122)"
    "query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__47299.invoke(reconcile_breakout_and_order_by_bucketing.clj:97)"
    "query_processor.middleware.auto_bucket_datetimes$auto_bucket_datetimes$fn__44417.invoke(auto_bucket_datetimes.clj:139)"
    "query_processor.middleware.resolve_source_table$resolve_source_tables$fn__45395.invoke(resolve_source_table.clj:45)"
    "query_processor.middleware.parameters$substitute_parameters$fn__47084.invoke(parameters.clj:111)"
    "query_processor.middleware.resolve_referenced$resolve_referenced_card_resources$fn__45447.invoke(resolve_referenced.clj:79)"
    "query_processor.middleware.expand_macros$expand_macros$fn__46061.invoke(expand_macros.clj:155)"
    "query_processor.middleware.add_timezone_info$add_timezone_info$fn__44226.invoke(add_timezone_info.clj:15)"
    "query_processor.middleware.splice_params_in_response$splice_params_in_response$fn__48086.invoke(splice_params_in_response.clj:32)"
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__47310$fn__47314.invoke(resolve_database_and_driver.clj:31)"
    "driver$do_with_driver.invokeStatic(driver.clj:60)"
    "driver$do_with_driver.invoke(driver.clj:56)"
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__47310.invoke(resolve_database_and_driver.clj:25)"
    "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__46301.invoke(fetch_source_query.clj:274)"
    "query_processor.middleware.store$initialize_store$fn__48095$fn__48096.invoke(store.clj:11)"
    "query_processor.store$do_with_store.invokeStatic(store.clj:44)"
    "query_processor.store$do_with_store.invoke(store.clj:38)"
    "query_processor.middleware.store$initialize_store$fn__48095.invoke(store.clj:10)"
    "query_processor.middleware.validate$validate_query$fn__48157.invoke(validate.clj:10)"
    "query_processor.middleware.normalize_query$normalize$fn__46427.invoke(normalize_query.clj:22)"
    "query_processor.middleware.add_rows_truncated$add_rows_truncated$fn__44082.invoke(add_rows_truncated.clj:35)"
    "query_processor.middleware.results_metadata$record_and_return_metadata_BANG_$fn__48071.invoke(results_metadata.clj:147)"
    "query_processor.middleware.constraints$add_default_userland_constraints$fn__45682.invoke(constraints.clj:42)"
    "query_processor.middleware.process_userland_query$process_userland_query$fn__47173.invoke(process_userland_query.clj:135)"
    "query_processor.middleware.catch_exceptions$catch_exceptions$fn__45625.invoke(catch_exceptions.clj:173)"
    "query_processor.reducible$async_qp$qp_STAR___37428$thunk__37429.invoke(reducible.clj:103)"
    "query_processor.reducible$async_qp$qp_STAR___37428.invoke(reducible.clj:109)"
    "query_processor.reducible$sync_qp$qp_STAR___37437$fn__37440.invoke(reducible.clj:135)"
    "query_processor.reducible$sync_qp$qp_STAR___37437.invoke(reducible.clj:134)"
    "query_processor$process_userland_query.invokeStatic(query_processor.clj:237)"
    "query_processor$process_userland_query.doInvoke(query_processor.clj:233)"
    "query_processor$fn__48203$process_query_and_save_execution_BANG___48212$fn__48215.invoke(query_processor.clj:249)"
    "query_processor$fn__48203$process_query_and_save_execution_BANG___48212.invoke(query_processor.clj:241)"
    "query_processor$fn__48247$process_query_and_save_with_max_results_constraints_BANG___48256$fn__48259.invoke(query_processor.clj:261)"
    "query_processor$fn__48247$process_query_and_save_with_max_results_constraints_BANG___48256.invoke(query_processor.clj:254)"
    "api.dataset$run_query_async$fn__54361.invoke(dataset.clj:56)"
    "query_processor.streaming$streaming_response_STAR_$fn__54340$fn__54341.invoke(streaming.clj:72)"
    "query_processor.streaming$streaming_response_STAR_$fn__54340.invoke(streaming.clj:71)"
    "async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:65)"
    "async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:63)"
    "async.streaming_response$do_f_async$fn__16055.invoke(streaming_response.clj:84)"],
   :error_type :qp,
   :ex-data {:type :qp}}],
 :error_type :qp,
 :json_query
 {:type "native",
  :native
  {:query
   "    SELECT \n        o.id,\n        o.simplified_partner_name,\n        o.created_at,\n        o.invoice_total,\n        min(b.beep_date) as tip_hub_beep_date\n    FROM analytics_core.orders o\n    LEFT JOIN analytics_core.beeps b\n        ON b.order_id = o.id\n        AND b.hub_name = o.hub_name\n    WHERE datetime_trunc(o.created_at, month) BETWEEN datetime_trunc(datetime_add(current_datetime(), INTERVAL -12 month), month)\n       AND datetime_trunc(current_datetime(), month)\n    GROUP BY id, created_at, invoice_total, simplified_partner_name\n    limit 100000",
   :template-tags {}},
  :database 130,
  :parameters [],
  :middleware {:js-int-to-string? true, :add-default-userland-constraints? true}},
 :status :failed,
 :class java.io.IOException,
 :stacktrace
 ["java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)"
  "java.base/sun.nio.ch.SocketDispatcher.writev(Unknown Source)"
  "java.base/sun.nio.ch.IOUtil.write(Unknown Source)"
  "java.base/sun.nio.ch.IOUtil.write(Unknown Source)"
  "java.base/sun.nio.ch.SocketChannelImpl.write(Unknown Source)"
  "java.base/java.nio.channels.SocketChannel.write(Unknown Source)"
  "org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:273)"
  "org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)"
  "org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)"
  "org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)"
  "org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:826)"
  "org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)"
  "org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)"
  "org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:544)"
  "org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:838)"
  "org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:910)"
  "org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:284)"
  "org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:268)"
  "org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:713)"
  "java.base/java.util.zip.DeflaterOutputStream.flush(Unknown Source)"
  "--> async.streaming_response$delay_output_stream$fn__16072.invoke(streaming_response.clj:119)"
  "async.streaming_response.proxy$java.io.OutputStream$ff19274a.flush(Unknown Source)"
  "query_processor.streaming.json$fn$reify__53817.write_row_BANG_(json.clj:62)"
  "query_processor.streaming$streaming_rff$fn__54285$fn__54286.invoke(streaming.clj:34)"
  "query_processor.middleware.process_userland_query$add_and_save_execution_info_xform_BANG_$execution_info_rf_STAR___47164.invoke(process_userland_query.clj:94)"
  "query_processor.reducible$combine_additional_reducing_fns$fn__37451.invoke(reducible.clj:216)"
  "query_processor.middleware.add_rows_truncated$add_rows_truncated_xform$fn__44077.invoke(add_rows_truncated.clj:27)"
  "query_processor.middleware.format_rows$format_rows_xform$fn__46348.invoke(format_rows.clj:68)"
  "query_processor.reducible$combine_additional_reducing_fns$fn__37451.invoke(reducible.clj:216)"
  "query_processor.middleware.limit$limit_xform$fn__46397.invoke(limit.clj:24)"
  "query_processor.context.default$default_reducef$fn__36875.invoke(default.clj:58)"
  "query_processor.context.default$default_reducef.invokeStatic(default.clj:57)"
  "query_processor.context.default$default_reducef.invoke(default.clj:48)"
  "query_processor.context$reducef.invokeStatic(context.clj:69)"
  "query_processor.context$reducef.invoke(context.clj:62)"
  "query_processor.context.default$default_runf$respond_STAR___36879.invoke(default.clj:69)"
  "driver.bigquery$post_process_native$fn__1498.invoke(bigquery.clj:251)"
  "driver.bigquery$do_with_finished_response.invokeStatic(bigquery.clj:171)"
  "driver.bigquery$do_with_finished_response.invoke(bigquery.clj:162)"
  "driver.bigquery$post_process_native.invokeStatic(bigquery.clj:234)"
  "driver.bigquery$post_process_native.invoke(bigquery.clj:228)"
  "driver.bigquery$process_native_STAR_$thunk__1596.invoke(bigquery.clj:277)"
  "driver.bigquery$process_native_STAR_.invokeStatic(bigquery.clj:279)"
  "driver.bigquery$process_native_STAR_.invoke(bigquery.clj:272)"
  "driver.bigquery$eval1600$fn__1602.invoke(bigquery.clj:299)"
  "query_processor.context$executef.invokeStatic(context.clj:59)"
  "query_processor.context$executef.invoke(context.clj:48)"
  "query_processor.context.default$default_runf.invokeStatic(default.clj:68)"
  "query_processor.context.default$default_runf.invoke(default.clj:66)"
  "query_processor.context$runf.invokeStatic(context.clj:45)"
  "query_processor.context$runf.invoke(context.clj:39)"
  "query_processor.reducible$pivot.invokeStatic(reducible.clj:34)"
  "query_processor.reducible$pivot.invoke(reducible.clj:31)"
  "query_processor.middleware.mbql_to_native$mbql__GT_native$fn__46414.invoke(mbql_to_native.clj:25)"
  "query_processor.middleware.check_features$check_features$fn__45666.invoke(check_features.clj:41)"
  "query_processor.middleware.limit$limit$fn__46400.invoke(limit.clj:37)"
  "query_processor.middleware.cache$maybe_return_cached_results$fn__45324.invoke(cache.clj:211)"
  "query_processor.middleware.optimize_datetime_filters$optimize_datetime_filters$fn__46579.invoke(optimize_datetime_filters.clj:133)"
  "query_processor.middleware.auto_parse_filter_values$auto_parse_filter_values$fn__44470.invoke(auto_parse_filter_values.clj:43)"
  "query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__40787.invoke(wrap_value_literals.clj:147)"
  "query_processor.middleware.annotate$add_column_info$fn__40650.invoke(annotate.clj:582)"
  "query_processor.middleware.permissions$check_query_permissions$fn__45541.invoke(permissions.clj:75)"
  "query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__47102.invoke(pre_alias_aggregations.clj:40)"
  "query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__45739.invoke(cumulative_aggregations.clj:60)"
  "query_processor.middleware.resolve_joined_fields$resolve_joined_fields$fn__47415.invoke(resolve_joined_fields.clj:94)"
  "query_processor.middleware.resolve_joins$resolve_joins$fn__47720.invoke(resolve_joins.clj:178)"
  "query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__44064.invoke(add_implicit_joins.clj:181)"
  "query_processor.middleware.large_int_id$convert_id_to_string$fn__46375.invoke(large_int_id.clj:44)"
  "query_processor.middleware.format_rows$format_rows$fn__46355.invoke(format_rows.clj:74)"
  "query_processor.middleware.desugar$desugar$fn__45805.invoke(desugar.clj:21)"
  "query_processor.middleware.binning$update_binning_strategy$fn__44830.invoke(binning.clj:228)"
  "query_processor.middleware.resolve_fields$resolve_fields$fn__45348.invoke(resolve_fields.clj:24)"
  "query_processor.middleware.add_dimension_projections$add_remapping$fn__43694.invoke(add_dimension_projections.clj:314)"
  "query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__43925.invoke(add_implicit_clauses.clj:146)"
  "query_processor.middleware.upgrade_field_literals$upgrade_field_literals$fn__48150.invoke(upgrade_field_literals.clj:45)"
  "query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__44217.invoke(add_source_metadata.clj:122)"
  "query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__47299.invoke(reconcile_breakout_and_order_by_bucketing.clj:97)"
  "query_processor.middleware.auto_bucket_datetimes$auto_bucket_datetimes$fn__44417.invoke(auto_bucket_datetimes.clj:139)"
  "query_processor.middleware.resolve_source_table$resolve_source_tables$fn__45395.invoke(resolve_source_table.clj:45)"
  "query_processor.middleware.parameters$substitute_parameters$fn__47084.invoke(parameters.clj:111)"
  "query_processor.middleware.resolve_referenced$resolve_referenced_card_resources$fn__45447.invoke(resolve_referenced.clj:79)"
  "query_processor.middleware.expand_macros$expand_macros$fn__46061.invoke(expand_macros.clj:155)"
  "query_processor.middleware.add_timezone_info$add_timezone_info$fn__44226.invoke(add_timezone_info.clj:15)"
  "query_processor.middleware.splice_params_in_response$splice_params_in_response$fn__48086.invoke(splice_params_in_response.clj:32)"
  "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__47310$fn__47314.invoke(resolve_database_and_driver.clj:31)"
  "driver$do_with_driver.invokeStatic(driver.clj:60)"
  "driver$do_with_driver.invoke(driver.clj:56)"
  "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__47310.invoke(resolve_database_and_driver.clj:25)"
  "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__46301.invoke(fetch_source_query.clj:274)"
  "query_processor.middleware.store$initialize_store$fn__48095$fn__48096.invoke(store.clj:11)"
  "query_processor.store$do_with_store.invokeStatic(store.clj:44)"
  "query_processor.store$do_with_store.invoke(store.clj:38)"
  "query_processor.middleware.store$initialize_store$fn__48095.invoke(store.clj:10)"
  "query_processor.middleware.validate$validate_query$fn__48157.invoke(validate.clj:10)"
  "query_processor.middleware.normalize_query$normalize$fn__46427.invoke(normalize_query.clj:22)"
  "query_processor.middleware.add_rows_truncated$add_rows_truncated$fn__44082.invoke(add_rows_truncated.clj:35)"
  "query_processor.middleware.results_metadata$record_and_return_metadata_BANG_$fn__48071.invoke(results_metadata.clj:147)"
  "query_processor.middleware.constraints$add_default_userland_constraints$fn__45682.invoke(constraints.clj:42)"
  "query_processor.middleware.process_userland_query$process_userland_query$fn__47173.invoke(process_userland_query.clj:135)"
  "query_processor.middleware.catch_exceptions$catch_exceptions$fn__45625.invoke(catch_exceptions.clj:173)"
  "query_processor.reducible$async_qp$qp_STAR___37428$thunk__37429.invoke(reducible.clj:103)"
  "query_processor.reducible$async_qp$qp_STAR___37428.invoke(reducible.clj:109)"
  "query_processor.reducible$sync_qp$qp_STAR___37437$fn__37440.invoke(reducible.clj:135)"
  "query_processor.reducible$sync_qp$qp_STAR___37437.invoke(reducible.clj:134)"
  "query_processor$process_userland_query.invokeStatic(query_processor.clj:237)"
  "query_processor$process_userland_query.doInvoke(query_processor.clj:233)"
  "query_processor$fn__48203$process_query_and_save_execution_BANG___48212$fn__48215.invoke(query_processor.clj:249)"
  "query_processor$fn__48203$process_query_and_save_execution_BANG___48212.invoke(query_processor.clj:241)"
  "query_processor$fn__48247$process_query_and_save_with_max_results_constraints_BANG___48256$fn__48259.invoke(query_processor.clj:261)"
  "query_processor$fn__48247$process_query_and_save_with_max_results_constraints_BANG___48256.invoke(query_processor.clj:254)"
  "api.dataset$run_query_async$fn__54361.invoke(dataset.clj:56)"
  "query_processor.streaming$streaming_response_STAR_$fn__54340$fn__54341.invoke(streaming.clj:72)"
  "query_processor.streaming$streaming_response_STAR_$fn__54340.invoke(streaming.clj:71)"
  "async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:65)"
  "async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:63)"
  "async.streaming_response$do_f_async$fn__16055.invoke(streaming_response.clj:84)"],
 :context :ad-hoc,
 :error "Broken pipe",
 :row_count 0,
 :running_time 0,
 :data {:rows [], :cols []}}

Diagnostic Info

{
  "browser-info": {
    "language": "en-US",
    "platform": "Linux x86_64",
    "userAgent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:86.0) Gecko/20100101 Firefox/86.0",
    "vendor": ""
  },
  "system-info": {
    "file.encoding": "UTF-8",
    "java.runtime.name": "OpenJDK Runtime Environment",
    "java.runtime.version": "11.0.10+9",
    "java.vendor": "AdoptOpenJDK",
    "java.vendor.url": "https://adoptopenjdk.net/",
    "java.version": "11.0.10",
    "java.vm.name": "OpenJDK 64-Bit Server VM",
    "java.vm.version": "11.0.10+9",
    "os.name": "Linux",
    "os.version": "4.19.0-10-cloud-amd64",
    "user.language": "en",
    "user.timezone": "GMT"
  },
  "metabase-info": {
    "databases": [
      "mysql",
      "bigquery",
      "h2"
    ],
    "hosting-env": "unknown",
    "application-database": "h2",
    "application-database-details": {
      "database": {
        "name": "H2",
        "version": "1.4.197 (2018-03-18)"
      },
      "jdbc-driver": {
        "name": "H2 JDBC Driver",
        "version": "1.4.197 (2018-03-18)"
      }
    },
    "run-mode": "prod",
    "version": {
      "date": "2021-03-03",
      "tag": "v0.38.1",
      "branch": "release-x.38.x",
      "hash": "79ef63a"
    },
    "settings": {
      "report-timezone": "America/Sao_Paulo"
    }
  }
}

Hi @lucasmarques
So initially I would have thought that your problem might have been fixed in 0.36.8 with this https://github.com/metabase/metabase/pull/13530, but since you say that it only happens under specific conditions, then I still think it has something to do with hitting the 10MB API limit on BigQuery: https://cloud.google.com/bigquery/quotas#jobsquery_requests

You should open an issue on this with as much information as possible to reproduce:
https://github.com/metabase/metabase/issues/new/choose

Also, you should migrate away from H2 if you are using Metabase in production:
https://www.metabase.com/docs/latest/operations-guide/migrating-from-h2.html

Hi @flamber!

Thanks for your reply!
Since then I've been dealing with unrelated workload and could not yet validate if BigQuery's API limit is my problem.
But today I shifted back to this issue and decided to migrate from H2 to another DB first, as you recommended. However, I'm repeatedly facing another error now.

Original docker-composer file for my metabase service:

version: '2'

services:

  nginx:
    container_name: nginx
    restart: always
    image: nginx:1.19
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./conf.d:/etc/nginx/conf.d

  metabase:
    container_name: metabase
    image: metabase/metabase:v0.38.1
    restart: always
    volumes:
     - /home/metabase-data:/metabase-data
    environment:
      MB_DB_FILE: metabase-data/metabase.db
      JAVA_OPTS: "-Xmx3g"

After taking this service down and backing up the H2 database, I edited the file to be such as this:

version: '2'

services:

  postgres:
    container_name: postgres
    image: postgres
    environment:
        - POSTGRES_USER=metabase
        - POSTGRES_DB=metabase
        - POSTGRES_PASSWORD=*
        - PGDATA=/var/lib/postgresql/data
    volumes:
        - /home/postgres-data:/var/lib/postgresql/data
    ports:
        - 5432:5432

  metabase:
    container_name: metabase
    image: metabase/metabase:v0.38.1
    restart: always
    volumes:
        - /home/metabase-data:/metabase-data
    environment:
        - MB_DB_FILE=/metabase-data/metabase.db
        - MB_DB_TYPE=postgres
        - MB_DB_DBNAME=metabase
        - MB_DB_PORT=5432
        - MB_DB_USER=metabase
        - MB_DB_PASS=*
        - MB_DB_HOST=postgres
    command: ["load-from-h2"]
    restart: always

Then used
docker-composer up -d postgres,
which started the postgres container without any problem.

Yet, when I use
docker-composer run metabase
I keep getting this same error:

Warning: environ value jdk-11.0.10+9 for key :java-version has been overwritten with 11.0.10
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2021-03-24 15:52:37,736 INFO metabase.util :: Maximum memory available to JVM: 988.0 MB
2021-03-24 15:52:52,019 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. 🔓 
 For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
2021-03-24 15:53:02,673 INFO metabase.core :: 
Metabase v0.38.1 (79ef63a release-x.38.x) 

Copyright © 2021 Metabase, Inc. 

Metabase Enterprise Edition extensions are NOT PRESENT.
2021-03-24 15:53:02,691 WARN metabase.core :: WARNING: You have enabled namespace tracing, which could log sensitive information like db passwords.
2021-03-24 15:53:02,786 INFO cmd.copy :: Set up h2 source database and run migrations... 
2021-03-24 15:53:02,797 INFO db.setup :: Verifying h2 Database Connection ...
2021-03-24 15:53:02,872 ERROR cmd.copy :: [FAIL]

clojure.lang.ExceptionInfo: ERROR Set up h2 source database and run migrations... {}
	at metabase.cmd.copy$do_step$fn__74686.invoke(copy.clj:33)
	at metabase.cmd.copy$do_step.invokeStatic(copy.clj:29)
	at metabase.cmd.copy$do_step.invoke(copy.clj:27)
	at metabase.cmd.copy$fn__74847$copy_BANG___74852$fn__74853.invoke(copy.clj:262)
	at metabase.cmd.copy$fn__74847$copy_BANG___74852.invoke(copy.clj:255)
	at metabase.cmd.load_from_h2$load_from_h2_BANG_.invokeStatic(load_from_h2.clj:35)
	at metabase.cmd.load_from_h2$load_from_h2_BANG_.invoke(load_from_h2.clj:25)
	at clojure.lang.Var.invoke(Var.java:384)
	at metabase.cmd$load_from_h2.invokeStatic(cmd.clj:45)
	at metabase.cmd$load_from_h2.invoke(cmd.clj:39)
	at metabase.cmd$load_from_h2.invokeStatic(cmd.clj:42)
	at metabase.cmd$load_from_h2.invoke(cmd.clj:39)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.core$apply.invokeStatic(core.clj:665)
	at clojure.core$apply.invoke(core.clj:660)
	at metabase.cmd$run_cmd$fn__74405.invoke(cmd.clj:191)
	at metabase.cmd$run_cmd.invokeStatic(cmd.clj:191)
	at metabase.cmd$run_cmd.invoke(cmd.clj:187)
	at clojure.lang.Var.invoke(Var.java:388)
	at metabase.core$run_cmd.invokeStatic(core.clj:148)
	at metabase.core$run_cmd.invoke(core.clj:146)
	at metabase.core$_main.invokeStatic(core.clj:170)
	at metabase.core$_main.doInvoke(core.clj:165)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at metabase.core.main(Unknown Source)
Caused by: clojure.lang.ExceptionInfo: Unable to connect to Metabase h2 DB. {}
	at metabase.db.setup$fn__34535$verify_db_connection__34540$fn__34541$fn__34542.invoke(setup.clj:102)
	at metabase.db.setup$fn__34535$verify_db_connection__34540$fn__34541.invoke(setup.clj:100)
	at metabase.db.setup$fn__34535$verify_db_connection__34540.invoke(setup.clj:94)
	at metabase.db.setup$setup_db_BANG_$fn__34570$fn__34571.invoke(setup.clj:142)
	at metabase.util$do_with_us_locale.invokeStatic(util.clj:670)
	at metabase.util$do_with_us_locale.invoke(util.clj:656)
	at metabase.db.setup$setup_db_BANG_$fn__34570.invoke(setup.clj:141)
	at metabase.db.setup$setup_db_BANG_.invokeStatic(setup.clj:140)
	at metabase.db.setup$setup_db_BANG_.invoke(setup.clj:136)
	at metabase.cmd.copy$fn__74847$copy_BANG___74852$fn__74853$fn__74854.invoke(copy.clj:263)
	at metabase.cmd.copy$do_step$fn__74686.invoke(copy.clj:30)
	... 25 more
Caused by: org.h2.jdbc.JdbcSQLException: Invalid database name: "/" [90138-197]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
	at org.h2.message.DbException.get(DbException.java:179)
	at org.h2.message.DbException.get(DbException.java:155)
	at org.h2.engine.ConnectionInfo.getName(ConnectionInfo.java:404)
	at org.h2.engine.Engine.openSession(Engine.java:50)
	at org.h2.engine.Engine.openSession(Engine.java:179)
	at org.h2.engine.Engine.createSessionAndValidate(Engine.java:157)
	at org.h2.engine.Engine.createSession(Engine.java:140)
	at org.h2.engine.Engine.createSession(Engine.java:28)
	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:351)
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:124)
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:103)
	at org.h2.Driver.connect(Driver.java:69)
	at java.sql/java.sql.DriverManager.getConnection(Unknown Source)
	at java.sql/java.sql.DriverManager.getConnection(Unknown Source)
	at clojure.java.jdbc$get_driver_connection.invokeStatic(jdbc.clj:271)
	at clojure.java.jdbc$get_driver_connection.invoke(jdbc.clj:250)
	at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:411)
	at clojure.java.jdbc$get_connection.invoke(jdbc.clj:274)
	at clojure.java.jdbc$db_query_with_resultset_STAR_.invokeStatic(jdbc.clj:1111)
	at clojure.java.jdbc$db_query_with_resultset_STAR_.invoke(jdbc.clj:1093)
	at clojure.java.jdbc$query.invokeStatic(jdbc.clj:1182)
	at clojure.java.jdbc$query.invoke(jdbc.clj:1144)
	at clojure.java.jdbc$query.invokeStatic(jdbc.clj:1160)
	at clojure.java.jdbc$query.invoke(jdbc.clj:1144)
	at metabase.driver.sql_jdbc.connection$can_connect_with_spec_QMARK_.invokeStatic(connection.clj:175)
	at metabase.driver.sql_jdbc.connection$can_connect_with_spec_QMARK_.invoke(connection.clj:172)
	at metabase.db.setup$fn__34535$verify_db_connection__34540$fn__34541$fn__34542.invoke(setup.clj:100)
	... 35 more
Command failed with exception: ERROR Set up h2 source database and run migrations...

I've also tried removing the command and MB_DB_FILE env var from metabase service definition and using docker-compose run metabase load-from-h2 /metabase-data/metabase.db instead, with no luck either...

Any clue on what is the issue here?

@lucasmarques How would these two environment variables know that you are referring to another container in one of them, but a string value in the other?

        - MB_DB_TYPE=postgres
...
        - MB_DB_HOST=postgres

Think you need to wrap this in quotes - "MB_DB_TYPE=postgres"

@flamber wrapping MB_DB_TYPE=postgres in quotes didn't work either.
I also tried changing postgres' service and container name... Still the same error.

Considering this line
Caused by: org.h2.jdbc.JdbcSQLException: Invalid database name: "/" [90138-197]
maybe metabase is having trouble to format the H2 file location specified in MB_DB_FILE?

@lucasmarques What is the actual real full path to the H2 file including all the extensions?
Normally I would just recommend running migrations directly without any containers, since it's just a one-off process.

Thanks @flamber!

After execution of the load-from-h2 command directly with the metabase.jar file, as suggested here, the migration succeeded.

However the migration alone didn't affect the issue I originally reported.
As soon as possible I'll bring results of further validation on the BQ's API limit hypothesis.

@lucasmarques Changing the application database will not fix anything regarding the querying issue. It will just make sure that you won't lose everything if the H2 file became corrupted.