Metabase: Error reducing result rows

Hello,

I am facing issue while sync+scan from Athena. The table is showing old columns and records which is not available in Athena. I am seeing below errors in logs:

[32aa516e-ea2c-44b5-81d5-73ce0db6b752] 2021-07-01T17:02:14+05:30 ERROR metabase.query-processor.middleware.catch-exceptions Error processing query: null
{:database_id 2,
 :started_at #t "2021-07-01T11:32:10.159981Z[GMT]",
 :via
 [{:status :failed,
   :class org.eclipse.jetty.io.EofException,
   :error nil,
   :stacktrace
   ["org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)"
"org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)"
"org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)"
"org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)"
"org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:820)"
"org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)"
"org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)"
"org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:544)"
"org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:838)"
"org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:910)"
"org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:284)"
"org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:268)"
"org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:713)"
"java.base/java.util.zip.DeflaterOutputStream.flush(Unknown Source)"
"--> async.streaming_response$delay_output_stream$fn__16072.invoke(streaming_response.clj:119)"
"async.streaming_response.proxy$java.io.OutputStream$ff19274a.flush(Unknown Source)"
"query_processor.streaming.json$fn$reify__53817.write_row_BANG_(json.clj:62)"
"query_processor.streaming$streaming_rff$fn__54285$fn__54286.invoke(streaming.clj:34)"
"query_processor.middleware.process_userland_query$add_and_save_execution_info_xform_BANG_$execution_info_rf_STAR___47164.invoke(process_userland_query.clj:94)"
"query_processor.reducible$combine_additional_reducing_fns$fn__37451.invoke(reducible.clj:216)"
"query_processor.middleware.add_rows_truncated$add_rows_truncated_xform$fn__44077.invoke(add_rows_truncated.clj:27)"
"query_processor.middleware.format_rows$format_rows_xform$fn__46348.invoke(format_rows.clj:68)"
"query_processor.middleware.limit$limit_xform$fn__46397.invoke(limit.clj:24)"
"query_processor.reducible$reducible_rows$reify__37444.reduce(reducible.clj:161)"
"query_processor.context.default$default_reducef$fn__36875.invoke(default.clj:58)"
"query_processor.context.default$default_reducef.invokeStatic(default.clj:57)"
"query_processor.context.default$default_reducef.invoke(default.clj:48)"
"query_processor.context$reducef.invokeStatic(context.clj:69)"
"query_processor.context$reducef.invoke(context.clj:62)"
"query_processor.context.default$default_runf$respond_STAR___36879.invoke(default.clj:69)"
"driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:398)"
"driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:375)"
"driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:384)"
"driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:375)"
"driver.sql_jdbc$fn__79102.invokeStatic(sql_jdbc.clj:54)"
"driver.sql_jdbc$fn__79102.invoke(sql_jdbc.clj:52)"
"driver.athena$fn__592.invokeStatic(athena.clj:278)"
"driver.athena$fn__592.invoke(athena.clj:276)"
"query_processor.context$executef.invokeStatic(context.clj:59)"
"query_processor.context$executef.invoke(context.clj:48)"
"query_processor.context.default$default_runf.invokeStatic(default.clj:68)"
"query_processor.context.default$default_runf.invoke(default.clj:66)"
"query_processor.context$runf.invokeStatic(context.clj:45)"
"query_processor.context$runf.invoke(context.clj:39)"
"query_processor.reducible$pivot.invokeStatic(reducible.clj:34)"
"query_processor.reducible$pivot.invoke(reducible.clj:31)"
"query_processor.middleware.mbql_to_native$mbql__GT_native$fn__46414.invoke(mbql_to_native.clj:25)"
"query_processor.middleware.check_features$check_features$fn__45666.invoke(check_features.clj:41)"
"query_processor.middleware.limit$limit$fn__46400.invoke(limit.clj:37)"
"query_processor.middleware.cache$maybe_return_cached_results$fn__45324.invoke(cache.clj:211)"
"query_processor.middleware.optimize_datetime_filters$optimize_datetime_filters$fn__46579.invoke(optimize_datetime_filters.clj:133)"
"query_processor.middleware.auto_parse_filter_values$auto_parse_filter_values$fn__44470.invoke(auto_parse_filter_values.clj:43)"
"query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__40787.invoke(wrap_value_literals.clj:147)"
"query_processor.middleware.annotate$add_column_info$fn__40650.invoke(annotate.clj:582)"
"query_processor.middleware.permissions$check_query_permissions$fn__45541.invoke(permissions.clj:75)"
"query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__47102.invoke(pre_alias_aggregations.clj:40)"
"query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__45739.invoke(cumulative_aggregations.clj:60)"
"query_processor.middleware.resolve_joined_fields$resolve_joined_fields$fn__47415.invoke(resolve_joined_fields.clj:94)"
"query_processor.middleware.resolve_joins$resolve_joins$fn__47720.invoke(resolve_joins.clj:178)"
"query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__44064.invoke(add_implicit_joins.clj:181)"
"query_processor.middleware.large_int_id$convert_id_to_string$fn__46375.invoke(large_int_id.clj:44)"
"query_processor.middleware.format_rows$format_rows$fn__46355.invoke(format_rows.clj:74)"
"query_processor.middleware.desugar$desugar$fn__45805.invoke(desugar.clj:21)"
"query_processor.middleware.binning$update_binning_strategy$fn__44830.invoke(binning.clj:228)"
"query_processor.middleware.resolve_fields$resolve_fields$fn__45348.invoke(resolve_fields.clj:24)"
"query_processor.middleware.add_dimension_projections$add_remapping$fn__43694.invoke(add_dimension_projections.clj:316)"
"query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__43925.invoke(add_implicit_clauses.clj:146)"
"query_processor.middleware.upgrade_field_literals$upgrade_field_literals$fn__48150.invoke(upgrade_field_literals.clj:45)"
"query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__44217.invoke(add_source_metadata.clj:122)"
"query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__47299.invoke(reconcile_breakout_and_order_by_bucketing.clj:97)"
"query_processor.middleware.auto_bucket_datetimes$auto_bucket_datetimes$fn__44417.invoke(auto_bucket_datetimes.clj:139)"
"query_processor.middleware.resolve_source_table$resolve_source_tables$fn__45395.invoke(resolve_source_table.clj:45)"
"query_processor.middleware.parameters$substitute_parameters$fn__47084.invoke(parameters.clj:111)"
"query_processor.middleware.resolve_referenced$resolve_referenced_card_resources$fn__45447.invoke(resolve_referenced.clj:79)"
"query_processor.middleware.expand_macros$expand_macros$fn__46061.invoke(expand_macros.clj:155)"
"query_processor.middleware.add_timezone_info$add_timezone_info$fn__44226.invoke(add_timezone_info.clj:15)"
"query_processor.middleware.splice_params_in_response$splice_params_in_response$fn__48086.invoke(splice_params_in_response.clj:32)"
"query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__47310$fn__47314.invoke(resolve_database_and_driver.clj:31)"
"driver$do_with_driver.invokeStatic(driver.clj:60)"
"driver$do_with_driver.invoke(driver.clj:56)"
"query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__47310.invoke(resolve_database_and_driver.clj:25)"
"query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__46301.invoke(fetch_source_query.clj:274)"
"query_processor.middleware.store$initialize_store$fn__48095$fn__48096.invoke(store.clj:11)"
"query_processor.store$do_with_store.invokeStatic(store.clj:44)"
"query_processor.store$do_with_store.invoke(store.clj:38)"
"query_processor.middleware.store$initialize_store$fn__48095.invoke(store.clj:10)"
"query_processor.middleware.validate$validate_query$fn__48157.invoke(validate.clj:10)"
"query_processor.middleware.normalize_query$normalize$fn__46427.invoke(normalize_query.clj:22)"
"query_processor.middleware.add_rows_truncated$add_rows_truncated$fn__44082.invoke(add_rows_truncated.clj:35)"
"query_processor.middleware.results_metadata$record_and_return_metadata_BANG_$fn__48071.invoke(results_metadata.clj:147)"
"query_processor.middleware.constraints$add_default_userland_constraints$fn__45682.invoke(constraints.clj:42)"
"query_processor.middleware.process_userland_query$process_userland_query$fn__47173.invoke(process_userland_query.clj:135)"
"query_processor.middleware.catch_exceptions$catch_exceptions$fn__45625.invoke(catch_exceptions.clj:173)"
"query_processor.reducible$async_qp$qp_STAR___37428$thunk__37429.invoke(reducible.clj:103)"
"query_processor.reducible$async_qp$qp_STAR___37428.invoke(reducible.clj:109)"
"query_processor.reducible$sync_qp$qp_STAR___37437$fn__37440.invoke(reducible.clj:135)"
"query_processor.reducible$sync_qp$qp_STAR___37437.invoke(reducible.clj:134)"
"query_processor$process_userland_query.invokeStatic(query_processor.clj:237)"
"query_processor$process_userland_query.doInvoke(query_processor.clj:233)"
"query_processor$fn__48203$process_query_and_save_execution_BANG___48212$fn__48215.invoke(query_processor.clj:249)"
"query_processor$fn__48203$process_query_and_save_execution_BANG___48212.invoke(query_processor.clj:241)"
"query_processor$fn__48247$process_query_and_save_with_max_results_constraints_BANG___48256$fn__48259.invoke(query_processor.clj:261)"
"query_processor$fn__48247$process_query_and_save_with_max_results_constraints_BANG___48256.invoke(query_processor.clj:254)"
"api.dataset$run_query_async$fn__54361.invoke(dataset.clj:56)"
"query_processor.streaming$streaming_response_STAR_$fn__54340$fn__54341.invoke(streaming.clj:72)"
"query_processor.streaming$streaming_response_STAR_$fn__54340.invoke(streaming.clj:71)"
"async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:65)"
"async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:63)"
"async.streaming_response$do_f_async$fn__16055.invoke(streaming_response.clj:84)"]}
  {:status :failed,
   :class clojure.lang.ExceptionInfo,
   :error "Error reducing result rows",
   :stacktrace
   ["--> query_processor.context.default$default_reducef$fn__36875.invoke(default.clj:60)"
"query_processor.context.default$default_reducef.invokeStatic(default.clj:57)"
"query_processor.context.default$default_reducef.invoke(default.clj:48)"
"query_processor.context$reducef.invokeStatic(context.clj:69)"
"query_processor.context$reducef.invoke(context.clj:62)"
"query_processor.context.default$default_runf$respond_STAR___36879.invoke(default.clj:69)"
"driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:398)"
"driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:375)"
"driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:384)"
"driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:375)"
"driver.sql_jdbc$fn__79102.invokeStatic(sql_jdbc.clj:54)"
"driver.sql_jdbc$fn__79102.invoke(sql_jdbc.clj:52)"
"driver.athena$fn__592.invokeStatic(athena.clj:278)"
"driver.athena$fn__592.invoke(athena.clj:276)"
"query_processor.context$executef.invokeStatic(context.clj:59)"
"query_processor.context$executef.invoke(context.clj:48)"
"query_processor.context.default$default_runf.invokeStatic(default.clj:68)"
"query_processor.context.default$default_runf.invoke(default.clj:66)"
"query_processor.context$runf.invokeStatic(context.clj:45)"
"query_processor.context$runf.invoke(context.clj:39)"
"query_processor.reducible$pivot.invokeStatic(reducible.clj:34)"
"query_processor.reducible$pivot.invoke(reducible.clj:31)"
"query_processor.middleware.mbql_to_native$mbql__GT_native$fn__46414.invoke(mbql_to_native.clj:25)"
"query_processor.middleware.check_features$check_features$fn__45666.invoke(check_features.clj:41)"
"query_processor.middleware.limit$limit$fn__46400.invoke(limit.clj:37)"
"query_processor.middleware.cache$maybe_return_cached_results$fn__45324.invoke(cache.clj:211)"
"query_processor.middleware.optimize_datetime_filters$optimize_datetime_filters$fn__46579.invoke(optimize_datetime_filters.clj:133)"
"query_processor.middleware.auto_parse_filter_values$auto_parse_filter_values$fn__44470.invoke(auto_parse_filter_values.clj:43)"
"query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__40787.invoke(wrap_value_literals.clj:147)"
"query_processor.middleware.annotate$add_column_info$fn__40650.invoke(annotate.clj:582)"
"query_processor.middleware.permissions$check_query_permissions$fn__45541.invoke(permissions.clj:75)"
"query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__47102.invoke(pre_alias_aggregations.clj:40)"
"query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__45739.invoke(cumulative_aggregations.clj:60)"
"query_processor.middleware.resolve_joined_fields$resolve_joined_fields$fn__47415.invoke(resolve_joined_fields.clj:94)"
"query_processor.middleware.resolve_joins$resolve_joins$fn__47720.invoke(resolve_joins.clj:178)"
"query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__44064.invoke(add_implicit_joins.clj:181)"
"query_processor.middleware.large_int_id$convert_id_to_string$fn__46375.invoke(large_int_id.clj:44)"
"query_processor.middleware.format_rows$format_rows$fn__46355.invoke(format_rows.clj:74)"
"query_processor.middleware.desugar$desugar$fn__45805.invoke(desugar.clj:21)"
"query_processor.middleware.binning$update_binning_strategy$fn__44830.invoke(binning.clj:228)"
"query_processor.middleware.resolve_fields$resolve_fields$fn__45348.invoke(resolve_fields.clj:24)"
"query_processor.middleware.add_dimension_projections$add_remapping$fn__43694.invoke(add_dimension_projections.clj:316)"
"query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__43925.invoke(add_implicit_clauses.clj:146)"
"query_processor.middleware.upgrade_field_literals$upgrade_field_literals$fn__48150.invoke(upgrade_field_literals.clj:45)"
"query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__44217.invoke(add_source_metadata.clj:122)"
"query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__47299.invoke(reconcile_breakout_and_order_by_bucketing.clj:97)"
"query_processor.middleware.auto_bucket_datetimes$auto_bucket_datetimes$fn__44417.invoke(auto_bucket_datetimes.clj:139)"
"query_processor.middleware.resolve_source_table$resolve_source_tables$fn__45395.invoke(resolve_source_table.clj:45)"
"query_processor.middleware.parameters$substitute_parameters$fn__47084.invoke(parameters.clj:111)"
"query_processor.middleware.resolve_referenced$resolve_referenced_card_resources$fn__45447.invoke(resolve_referenced.clj:79)"
"query_processor.middleware.expand_macros$expand_macros$fn__46061.invoke(expand_macros.clj:155)"
"query_processor.middleware.add_timezone_info$add_timezone_info$fn__44226.invoke(add_timezone_info.clj:15)"
"query_processor.middleware.splice_params_in_response$splice_params_in_response$fn__48086.invoke(splice_params_in_response.clj:32)"
"query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__47310$fn__47314.invoke(resolve_database_and_driver.clj:31)"
"driver$do_with_driver.invokeStatic(driver.clj:60)"
"driver$do_with_driver.invoke(driver.clj:56)"
"query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__47310.invoke(resolve_database_and_driver.clj:25)"
"query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__46301.invoke(fetch_source_query.clj:274)"
"query_processor.middleware.store$initialize_store$fn__48095$fn__48096.invoke(store.clj:11)"
"query_processor.store$do_with_store.invokeStatic(store.clj:44)"
"query_processor.store$do_with_store.invoke(store.clj:38)"
"query_processor.middleware.store$initialize_store$fn__48095.invoke(store.clj:10)"
"query_processor.middleware.validate$validate_query$fn__48157.invoke(validate.clj:10)"
"query_processor.middleware.normalize_query$normalize$fn__46427.invoke(normalize_query.clj:22)"
"query_processor.middleware.add_rows_truncated$add_rows_truncated$fn__44082.invoke(add_rows_truncated.clj:35)"
"query_processor.middleware.results_metadata$record_and_return_metadata_BANG_$fn__48071.invoke(results_metadata.clj:147)"
"query_processor.middleware.constraints$add_default_userland_constraints$fn__45682.invoke(constraints.clj:42)"
"query_processor.middleware.process_userland_query$process_userland_query$fn__47173.invoke(process_userland_query.clj:135)"
"query_processor.middleware.catch_exceptions$catch_exceptions$fn__45625.invoke(catch_exceptions.clj:173)"
"query_processor.reducible$async_qp$qp_STAR___37428$thunk__37429.invoke(reducible.clj:103)"
"query_processor.reducible$async_qp$qp_STAR___37428.invoke(reducible.clj:109)"
"query_processor.reducible$sync_qp$qp_STAR___37437$fn__37440.invoke(reducible.clj:135)"
"query_processor.reducible$sync_qp$qp_STAR___37437.invoke(reducible.clj:134)"
"query_processor$process_userland_query.invokeStatic(query_processor.clj:237)"
"query_processor$process_userland_query.doInvoke(query_processor.clj:233)"
"query_processor$fn__48203$process_query_and_save_execution_BANG___48212$fn__48215.invoke(query_processor.clj:249)"
"query_processor$fn__48203$process_query_and_save_execution_BANG___48212.invoke(query_processor.clj:241)"
"query_processor$fn__48247$process_query_and_save_with_max_results_constraints_BANG___48256$fn__48259.invoke(query_processor.clj:261)"
"query_processor$fn__48247$process_query_and_save_with_max_results_constraints_BANG___48256.invoke(query_processor.clj:254)"
"api.dataset$run_query_async$fn__54361.invoke(dataset.clj:56)"
"query_processor.streaming$streaming_response_STAR_$fn__54340$fn__54341.invoke(streaming.clj:72)"
"query_processor.streaming$streaming_response_STAR_$fn__54340.invoke(streaming.clj:71)"
"async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:65)"
"async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:63)"
"async.streaming_response$do_f_async$fn__16055.invoke(streaming_response.clj:84)"],
   :error_type :qp,
   :ex-data {:type :qp}}],
 :error_type :qp,
 :json_query
 {:database 2,
  :type "query",
  :query {:source-table 139},
  :parameters [],
  :middleware {:js-int-to-string? true, :add-default-userland-constraints? true}},
 :native
 {:query
  "SELECT \"report_glue_db_prod\".\"employers\".\"allowonlineapplication\" AS \"allowonlineapplication\", \"report_glue_db_prod\".\"employers\".\"alternateinboxemail\" AS \"alternateinboxemail\", \"report_glue_db_prod\".\"employers\".\"autoreplyapplications\" AS \"autoreplyapplications\", \"report_glue_db_prod\".\"employers\".\"callsnaphireapi\" AS \"callsnaphireapi\", \"report_glue_db_prod\".\"employers\".\"caneditalternateinbox\" AS \"caneditalternateinbox\", \"report_glue_db_prod\".\"employers\".\"canmodifyextjobids\" AS \"canmodifyextjobids\", \"report_glue_db_prod\".\"employers\".\"canpostinternal\" AS \"canpostinternal\", \"report_glue_db_prod\".\"employers\".\"cjbid\" AS \"cjbid\", \"report_glue_db_prod\".\"employers\".\"contactmecustom\" AS \"contactmecustom\", \"report_glue_db_prod\".\"employers\".\"contactmeprofile\" AS \"contactmeprofile\", \"report_glue_db_prod\".\"employers\".\"customjobbankid\" AS \"customjobbankid\", \"report_glue_db_prod\".\"employers\".\"dateadded\" AS \"dateadded\", \"report_glue_db_prod\".\"employers\".\"datemodified\" AS \"datemodified\", \"report_glue_db_prod\".\"employers\".\"defaultbrandcode\" AS \"defaultbrandcode\", \"report_glue_db_prod\".\"employers\".\"defaultshowemailonjob\" AS \"defaultshowemailonjob\", \"report_glue_db_prod\".\"employers\".\"defaultshoworgname\" AS \"defaultshoworgname\", \"report_glue_db_prod\".\"employers\".\"defaultshowpropertyname\" AS \"defaultshowpropertyname\", \"report_glue_db_prod\".\"employers\".\"emailapplications\" AS \"emailapplications\", \"report_glue_db_prod\".\"employers\".\"emailfoundresumes\" AS \"emailfoundresumes\", \"report_glue_db_prod\".\"employers\".\"employerid\" AS \"employerid\", \"report_glue_db_prod\".\"employers\".\"erolautoreplyletter\" AS \"erolautoreplyletter\", \"report_glue_db_prod\".\"employers\".\"externaljobbankid\" AS \"externaljobbankid\", \"report_glue_db_prod\".\"employers\".\"externaljobbanktype\" AS \"externaljobbanktype\", \"report_glue_db_prod\".\"employers\".\"guid\" AS \"guid\", \"report_glue_db_prod\".\"employers\".\"hasbranding\" AS \"hasbranding\", \"report_glue_db_prod\".\"employers\".\"hascustomjobbank\" AS \"hascustomjobbank\", \"report_glue_db_prod\".\"employers\".\"hassearchbranding\" AS \"hassearchbranding\", \"report_glue_db_prod\".\"employers\".\"industrycode\" AS \"industrycode\", \"report_glue_db_prod\".\"employers\".\"ipaddressadded\" AS \"ipaddressadded\", \"report_glue_db_prod\".\"employers\".\"ipaddressmodified\" AS \"ipaddressmodified\", \"report_glue_db_prod\".\"employers\".\"isahlamember\" AS \"isahlamember\", \"report_glue_db_prod\".\"employers\".\"isrecruiter\" AS \"isrecruiter\", \"report_glue_db_prod\".\"employers\".\"jobadexpirydays\" AS \"jobadexpirydays\", \"report_glue_db_prod\".\"employers\".\"managementcompany\" AS \"managementcompany\", \"report_glue_db_prod\".\"employers\".\"maxresumedetectives\" AS \"maxresumedetectives\", \"report_glue_db_prod\".\"employers\".\"numberemployees\" AS \"numberemployees\", \"report_glue_db_prod\".\"employers\".\"numberproperties\" AS \"numberproperties\", \"report_glue_db_prod\".\"employers\".\"orgname\" AS \"orgname\", \"report_glue_db_prod\".\"employers\".\"orgurl\" AS \"orgurl\", \"report_glue_db_prod\".\"employers\".\"popupbilling\" AS \"popupbilling\", \"report_glue_db_prod\".\"employers\".\"postingverticals\" AS \"postingverticals\", \"report_glue_db_prod\".\"employers\".\"pricinggroupid\" AS \"pricinggroupid\", \"report_glue_db_prod\".\"employers\".\"profileid\" AS \"profileid\", \"report_glue_db_prod\".\"employers\".\"resumedetectiveexpirydays\" AS \"resumedetectiveexpirydays\", \"report_glue_db_prod\".\"employers\".\"searchverticals\" AS \"searchverticals\", \"report_glue_db_prod\".\"employers\".\"showapplicationsource\" AS \"showapplicationsource\", \"report_glue_db_prod\".\"employers\".\"templateid\" AS \"templateid\", \"report_glue_db_prod\".\"employers\".\"templateversion\" AS \"templateversion\", \"report_glue_db_prod\".\"employers\".\"title\" AS \"title\", \"report_glue_db_prod\".\"employers\".\"useemployerurl\" AS \"useemployerurl\", \"report_glue_db_prod\".\"employers\".\"userid\" AS \"userid\" FROM \"report_glue_db_prod\".\"employers\" LIMIT 2000",
  :params nil},
 :status :failed,
 :class java.io.IOException,
 :stacktrace
 ["java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)"
  "java.base/sun.nio.ch.SocketDispatcher.writev(Unknown Source)"
  "java.base/sun.nio.ch.IOUtil.write(Unknown Source)"
  "java.base/sun.nio.ch.IOUtil.write(Unknown Source)"
  "java.base/sun.nio.ch.SocketChannelImpl.write(Unknown Source)"
  "java.base/java.nio.channels.SocketChannel.write(Unknown Source)"
  "org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:273)"
  "org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)"
  "org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)"
  "org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)"
  "org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:820)"
  "org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)"
  "org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)"
  "org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:544)"
  "org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:838)"
  "org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:910)"
  "org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:284)"
  "org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:268)"
  "org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:713)"
  "java.base/java.util.zip.DeflaterOutputStream.flush(Unknown Source)"
  "--> async.streaming_response$delay_output_stream$fn__16072.invoke(streaming_response.clj:119)"
  "async.streaming_response.proxy$java.io.OutputStream$ff19274a.flush(Unknown Source)"
  "query_processor.streaming.json$fn$reify__53817.write_row_BANG_(json.clj:62)"
  "query_processor.streaming$streaming_rff$fn__54285$fn__54286.invoke(streaming.clj:34)"
  "query_processor.middleware.process_userland_query$add_and_save_execution_info_xform_BANG_$execution_info_rf_STAR___47164.invoke(process_userland_query.clj:94)"
  "query_processor.reducible$combine_additional_reducing_fns$fn__37451.invoke(reducible.clj:216)"
  "query_processor.middleware.add_rows_truncated$add_rows_truncated_xform$fn__44077.invoke(add_rows_truncated.clj:27)"
  "query_processor.middleware.format_rows$format_rows_xform$fn__46348.invoke(format_rows.clj:68)"
  "query_processor.middleware.limit$limit_xform$fn__46397.invoke(limit.clj:24)"
  "query_processor.reducible$reducible_rows$reify__37444.reduce(reducible.clj:161)"
  "query_processor.context.default$default_reducef$fn__36875.invoke(default.clj:58)"
  "query_processor.context.default$default_reducef.invokeStatic(default.clj:57)"
  "query_processor.context.default$default_reducef.invoke(default.clj:48)"
  "query_processor.context$reducef.invokeStatic(context.clj:69)"
  "query_processor.context$reducef.invoke(context.clj:62)"
  "query_processor.context.default$default_runf$respond_STAR___36879.invoke(default.clj:69)"
  "driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:398)"
  "driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:375)"
  "driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:384)"
  "driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:375)"
  "driver.sql_jdbc$fn__79102.invokeStatic(sql_jdbc.clj:54)"
  "driver.sql_jdbc$fn__79102.invoke(sql_jdbc.clj:52)"
  "driver.athena$fn__592.invokeStatic(athena.clj:278)"
  "driver.athena$fn__592.invoke(athena.clj:276)"
  "query_processor.context$executef.invokeStatic(context.clj:59)"
  "query_processor.context$executef.invoke(context.clj:48)"
  "query_processor.context.default$default_runf.invokeStatic(default.clj:68)"
  "query_processor.context.default$default_runf.invoke(default.clj:66)"
  "query_processor.context$runf.invokeStatic(context.clj:45)"
  "query_processor.context$runf.invoke(context.clj:39)"
  "query_processor.reducible$pivot.invokeStatic(reducible.clj:34)"
  "query_processor.reducible$pivot.invoke(reducible.clj:31)"
  "query_processor.middleware.mbql_to_native$mbql__GT_native$fn__46414.invoke(mbql_to_native.clj:25)"
  "query_processor.middleware.check_features$check_features$fn__45666.invoke(check_features.clj:41)"
  "query_processor.middleware.limit$limit$fn__46400.invoke(limit.clj:37)"
  "query_processor.middleware.cache$maybe_return_cached_results$fn__45324.invoke(cache.clj:211)"
  "query_processor.middleware.optimize_datetime_filters$optimize_datetime_filters$fn__46579.invoke(optimize_datetime_filters.clj:133)"
  "query_processor.middleware.auto_parse_filter_values$auto_parse_filter_values$fn__44470.invoke(auto_parse_filter_values.clj:43)"
  "query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__40787.invoke(wrap_value_literals.clj:147)"
  "query_processor.middleware.annotate$add_column_info$fn__40650.invoke(annotate.clj:582)"
  "query_processor.middleware.permissions$check_query_permissions$fn__45541.invoke(permissions.clj:75)"
  "query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__47102.invoke(pre_alias_aggregations.clj:40)"
  "query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__45739.invoke(cumulative_aggregations.clj:60)"
  "query_processor.middleware.resolve_joined_fields$resolve_joined_fields$fn__47415.invoke(resolve_joined_fields.clj:94)"
  "query_processor.middleware.resolve_joins$resolve_joins$fn__47720.invoke(resolve_joins.clj:178)"
  "query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__44064.invoke(add_implicit_joins.clj:181)"
  "query_processor.middleware.large_int_id$convert_id_to_string$fn__46375.invoke(large_int_id.clj:44)"
  "query_processor.middleware.format_rows$format_rows$fn__46355.invoke(format_rows.clj:74)"
  "query_processor.middleware.desugar$desugar$fn__45805.invoke(desugar.clj:21)"
  "query_processor.middleware.binning$update_binning_strategy$fn__44830.invoke(binning.clj:228)"
  "query_processor.middleware.resolve_fields$resolve_fields$fn__45348.invoke(resolve_fields.clj:24)"
  "query_processor.middleware.add_dimension_projections$add_remapping$fn__43694.invoke(add_dimension_projections.clj:316)"
  "query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__43925.invoke(add_implicit_clauses.clj:146)"
  "query_processor.middleware.upgrade_field_literals$upgrade_field_literals$fn__48150.invoke(upgrade_field_literals.clj:45)"
  "query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__44217.invoke(add_source_metadata.clj:122)"
  "query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__47299.invoke(reconcile_breakout_and_order_by_bucketing.clj:97)"
  "query_processor.middleware.auto_bucket_datetimes$auto_bucket_datetimes$fn__44417.invoke(auto_bucket_datetimes.clj:139)"
  "query_processor.middleware.resolve_source_table$resolve_source_tables$fn__45395.invoke(resolve_source_table.clj:45)"
  "query_processor.middleware.parameters$substitute_parameters$fn__47084.invoke(parameters.clj:111)"
  "query_processor.middleware.resolve_referenced$resolve_referenced_card_resources$fn__45447.invoke(resolve_referenced.clj:79)"
  "query_processor.middleware.expand_macros$expand_macros$fn__46061.invoke(expand_macros.clj:155)"
  "query_processor.middleware.add_timezone_info$add_timezone_info$fn__44226.invoke(add_timezone_info.clj:15)"
  "query_processor.middleware.splice_params_in_response$splice_params_in_response$fn__48086.invoke(splice_params_in_response.clj:32)"
  "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__47310$fn__47314.invoke(resolve_database_and_driver.clj:31)"
  "driver$do_with_driver.invokeStatic(driver.clj:60)"
  "driver$do_with_driver.invoke(driver.clj:56)"
  "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__47310.invoke(resolve_database_and_driver.clj:25)"
  "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__46301.invoke(fetch_source_query.clj:274)"
  "query_processor.middleware.store$initialize_store$fn__48095$fn__48096.invoke(store.clj:11)"
  "query_processor.store$do_with_store.invokeStatic(store.clj:44)"
  "query_processor.store$do_with_store.invoke(store.clj:38)"
  "query_processor.middleware.store$initialize_store$fn__48095.invoke(store.clj:10)"
  "query_processor.middleware.validate$validate_query$fn__48157.invoke(validate.clj:10)"
  "query_processor.middleware.normalize_query$normalize$fn__46427.invoke(normalize_query.clj:22)"
  "query_processor.middleware.add_rows_truncated$add_rows_truncated$fn__44082.invoke(add_rows_truncated.clj:35)"
  "query_processor.middleware.results_metadata$record_and_return_metadata_BANG_$fn__48071.invoke(results_metadata.clj:147)"
  "query_processor.middleware.constraints$add_default_userland_constraints$fn__45682.invoke(constraints.clj:42)"
  "query_processor.middleware.process_userland_query$process_userland_query$fn__47173.invoke(process_userland_query.clj:135)"
  "query_processor.middleware.catch_exceptions$catch_exceptions$fn__45625.invoke(catch_exceptions.clj:173)"
  "query_processor.reducible$async_qp$qp_STAR___37428$thunk__37429.invoke(reducible.clj:103)"
  "query_processor.reducible$async_qp$qp_STAR___37428.invoke(reducible.clj:109)"
  "query_processor.reducible$sync_qp$qp_STAR___37437$fn__37440.invoke(reducible.clj:135)"
  "query_processor.reducible$sync_qp$qp_STAR___37437.invoke(reducible.clj:134)"
  "query_processor$process_userland_query.invokeStatic(query_processor.clj:237)"
  "query_processor$process_userland_query.doInvoke(query_processor.clj:233)"
  "query_processor$fn__48203$process_query_and_save_execution_BANG___48212$fn__48215.invoke(query_processor.clj:249)"
  "query_processor$fn__48203$process_query_and_save_execution_BANG___48212.invoke(query_processor.clj:241)"
  "query_processor$fn__48247$process_query_and_save_with_max_results_constraints_BANG___48256$fn__48259.invoke(query_processor.clj:261)"
  "query_processor$fn__48247$process_query_and_save_with_max_results_constraints_BANG___48256.invoke(query_processor.clj:254)"
  "api.dataset$run_query_async$fn__54361.invoke(dataset.clj:56)"
  "query_processor.streaming$streaming_response_STAR_$fn__54340$fn__54341.invoke(streaming.clj:72)"
  "query_processor.streaming$streaming_response_STAR_$fn__54340.invoke(streaming.clj:71)"
  "async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:65)"
  "async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:63)"
  "async.streaming_response$do_f_async$fn__16055.invoke(streaming_response.clj:84)"],
 :context :ad-hoc,
 :error "Broken pipe",
 :row_count 0,
 :running_time 0,
 :preprocessed
 {:database 2,
  :type :query,
  :query
  {:source-table 139,
   :fields
   [[:field-id 1137]
[:field-id 1142]
[:field-id 1118]
[:field-id 1107]
[:field-id 1125]
[:field-id 1148]
[:field-id 1106]
[:field-id 1128]
[:field-id 1130]
[:field-id 1104]
[:field-id 1109]
[:datetime-field [:field-id 1139] :default]
[:datetime-field [:field-id 1105] :default]
[:field-id 1134]
[:field-id 1144]
[:field-id 1113]
[:field-id 1101]
[:field-id 1121]
[:field-id 1132]
[:field-id 1102]
[:field-id 1129]
[:field-id 1108]
[:field-id 1098]
[:field-id 1126]
[:field-id 1115]
[:field-id 1112]
[:field-id 1123]
[:field-id 1114]
[:field-id 1103]
[:field-id 1119]
[:field-id 1133]
[:field-id 1120]
[:field-id 1110]
[:field-id 1136]
[:field-id 1135]
[:field-id 1117]
[:field-id 1122]
[:field-id 1124]
[:field-id 1131]
[:field-id 1140]
[:field-id 1145]
[:field-id 1147]
[:field-id 1138]
[:field-id 1141]
[:field-id 1127]
[:field-id 1143]
[:field-id 1146]
[:field-id 1100]
[:field-id 1111]
[:field-id 1116]
[:field-id 1099]],
   :limit 2000},
  :middleware {:js-int-to-string? true, :add-default-userland-constraints? true},
  :info
  {:executed-by 48,
   :context :ad-hoc,
   :nested? false,
   :query-hash
   [-15, 124, -17, -57, -99, -122, -105, 104, -3, -111, -96, -21, 67, -53, -51, -17, 96, 109, -9, 112, 73, -54, -26,
-19, -62, 94, 85, -122, -126, -44, -81, 114]},
  :constraints {:max-results 10000, :max-results-bare-rows 2000}},
 :data {:rows [], :cols []}}

Here is my diagnostic info:

    {
      "browser-info": {
        "language": "en-GB",
        "platform": "MacIntel",
        "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
        "vendor": "Google Inc."
      },
      "system-info": {
        "file.encoding": "UTF-8",
        "java.runtime.name": "OpenJDK Runtime Environment",
        "java.runtime.version": "11.0.10+9",
        "java.vendor": "AdoptOpenJDK",
        "java.vendor.url": "https://adoptopenjdk.net/",
        "java.version": "11.0.10",
        "java.vm.name": "OpenJDK 64-Bit Server VM",
        "java.vm.version": "11.0.10+9",
        "os.name": "Linux",
        "os.version": "4.14.225-168.357.amzn2.x86_64",
        "user.language": "en",
        "user.timezone": "GMT"
      },
      "metabase-info": {
        "databases": [
          "athena",
          "h2"
        ],
        "hosting-env": "unknown",
        "application-database": "postgres",
        "application-database-details": {
          "database": {
            "name": "PostgreSQL",
            "version": "11.10"
          },
          "jdbc-driver": {
            "name": "PostgreSQL JDBC Driver",
            "version": "42.2.18"
          }
        },
        "run-mode": "prod",
        "version": {
          "date": "2021-03-03",
          "tag": "v0.38.1",
          "branch": "release-x.38.x",
          "hash": "79ef63a"
        },
        "settings": {
          "report-timezone": null
        }
      }
    }

Hi @aalap
Try a later release. There were several fixes to sync. Latest release is 0.39.4: https://github.com/metabase/metabase/releases/latest
The error you are showing is not the interesting part - errors during sync would be interesting.

Hello @flamber,
I have tried updating metabase version but getting below error:

Warning: environ value jdk-11.0.11+9 for key :java-version has been overwritten with 11.0.11
2021-07-07 06:47:37,929 INFO metabase.util :: Maximum memory available to JVM: 494.9 MB
2021-07-07 06:47:57,549 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. ๐Ÿ”“
For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
2021-07-07 06:48:10,418 INFO metabase.core ::
Metabase v0.39.4 (f538050 release-x.39.x)
Copyright ยฉ 2021 Metabase, Inc.
Metabase Enterprise Edition extensions are NOT PRESENT.
2021-07-07 06:48:10,427 WARN metabase.core :: WARNING: You have enabled namespace tracing, which could log sensitive information like db passwords.
2021-07-07 06:48:10,427 INFO metabase.core :: Starting Metabase in STANDALONE mode
2021-07-07 06:48:10,759 INFO metabase.server :: Launching Embedded Jetty Webserver with config:
{:port 80}
2021-07-07 06:48:10,923 INFO metabase.core :: Starting Metabase version v0.39.4 (f538050 release-x.39.x) ...
2021-07-07 06:48:10,959 INFO metabase.core :: System info:
{"file.encoding" "UTF-8",
"java.runtime.name" "OpenJDK Runtime Environment",
"java.runtime.version" "11.0.11+9",
"java.vendor" "AdoptOpenJDK",
"java.vendor.url" "https://adoptopenjdk.net/",
"java.version" "11.0.11",
"java.vm.name" "OpenJDK 64-Bit Server VM",
"java.vm.version" "11.0.11+9",
"os.name" "Linux",
"os.version" "4.14.231-180.360.amzn2.x86_64",
"user.language" "en",
"user.timezone" "GMT"}
2021-07-07 06:48:11,015 INFO metabase.plugins :: Loading plugins in /app/plugins...
2021-07-07 06:48:11,423 INFO util.files :: Extract file /modules/sparksql.metabase-driver.jar -> /app/plugins/sparksql.metabase-driver.jar
2021-07-07 06:48:12,443 INFO util.files :: Extract file /modules/bigquery.metabase-driver.jar -> /app/plugins/bigquery.metabase-driver.jar
2021-07-07 06:48:12,448 INFO util.files :: Extract file /modules/sqlserver.metabase-driver.jar -> /app/plugins/sqlserver.metabase-driver.jar
2021-07-07 06:48:12,457 INFO util.files :: Extract file /modules/sqlite.metabase-driver.jar -> /app/plugins/sqlite.metabase-driver.jar
2021-07-07 06:48:12,532 INFO util.files :: Extract file /modules/vertica.metabase-driver.jar -> /app/plugins/vertica.metabase-driver.jar
2021-07-07 06:48:12,533 INFO util.files :: Extract file /modules/redshift.metabase-driver.jar -> /app/plugins/redshift.metabase-driver.jar
2021-07-07 06:48:12,540 INFO util.files :: Extract file /modules/presto.metabase-driver.jar -> /app/plugins/presto.metabase-driver.jar
2021-07-07 06:48:12,542 INFO util.files :: Extract file /modules/druid.metabase-driver.jar -> /app/plugins/druid.metabase-driver.jar
2021-07-07 06:48:12,546 INFO util.files :: Extract file /modules/mongo.metabase-driver.jar -> /app/plugins/mongo.metabase-driver.jar
2021-07-07 06:48:12,624 INFO util.files :: Extract file /modules/google.metabase-driver.jar -> /app/plugins/google.metabase-driver.jar
2021-07-07 06:48:12,632 INFO util.files :: Extract file /modules/oracle.metabase-driver.jar -> /app/plugins/oracle.metabase-driver.jar
2021-07-07 06:48:12,633 INFO util.files :: Extract file /modules/snowflake.metabase-driver.jar -> /app/plugins/snowflake.metabase-driver.jar
2021-07-07 06:48:13,030 INFO util.files :: Extract file /modules/googleanalytics.metabase-driver.jar -> /app/plugins/googleanalytics.metabase-driver.jar
2021-07-07 06:48:13,433 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :athena...e[0m
2021-07-07 06:48:13,818 INFO driver.impl :: e[34mRegistered abstract driver :sqle[0m ๐Ÿšš
e[32m โ†™ Load driver :sql took 278.8 mse[0m
2021-07-07 06:48:13,833 INFO driver.impl :: e[34mRegistered abstract driver :sql-jdbce[0m (parents: [:sql]) ๐Ÿšš
e[32mLoad driver :sql-jdbc took 401.2 mse[0m
2021-07-07 06:48:13,836 INFO driver.impl :: e[34mRegistered driver :athenae[0m (parents: [:sql-jdbc]) ๐Ÿšš
2021-07-07 06:48:13,842 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sqlite...e[0m
2021-07-07 06:48:13,844 INFO driver.impl :: e[34mRegistered driver :sqlitee[0m (parents: [:sql-jdbc]) ๐Ÿšš
2021-07-07 06:48:13,916 INFO plugins.dependencies :: Plugin 'Metabase BigQuery Driver' depends on plugin 'Metabase Google Drivers Shared Dependencies'
2021-07-07 06:48:13,917 INFO plugins.dependencies :: Metabase BigQuery Driver dependency {:plugin Metabase Google Drivers Shared Dependencies} satisfied? false
2021-07-07 06:48:13,918 INFO plugins.dependencies :: e[33mPlugins with unsatisfied deps: ["Metabase BigQuery Driver"]e[0m
2021-07-07 06:48:13,939 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :hive-like...e[0m
2021-07-07 06:48:13,940 INFO driver.impl :: e[34mRegistered abstract driver :hive-likee[0m (parents: [:sql-jdbc]) ๐Ÿšš
2021-07-07 06:48:13,941 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sparksql...e[0m
2021-07-07 06:48:13,941 INFO driver.impl :: e[34mRegistered driver :sparksqle[0m (parents: [:hive-like]) ๐Ÿšš
2021-07-07 06:48:13,946 INFO plugins.dependencies :: Plugin 'Metabase Google Analytics Driver' depends on plugin 'Metabase Google Drivers Shared Dependencies'
2021-07-07 06:48:13,946 INFO plugins.dependencies :: Metabase Google Analytics Driver dependency {:plugin Metabase Google Drivers Shared Dependencies} satisfied? false
2021-07-07 06:48:14,018 INFO plugins.dependencies :: e[33mPlugins with unsatisfied deps: ["Metabase BigQuery Driver" "Metabase Google Analytics Driver"]e[0m
2021-07-07 06:48:14,023 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sqlserver...e[0m
2021-07-07 06:48:14,023 INFO driver.impl :: e[34mRegistered driver :sqlservere[0m (parents: [:sql-jdbc]) ๐Ÿšš
2021-07-07 06:48:14,026 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :google...e[0m
2021-07-07 06:48:14,026 INFO driver.impl :: e[34mRegistered abstract driver :googlee[0m ๐Ÿšš
2021-07-07 06:48:14,027 INFO plugins.dependencies :: Metabase BigQuery Driver dependency {:plugin Metabase Google Drivers Shared Dependencies} satisfied? true
2021-07-07 06:48:14,027 INFO plugins.dependencies :: Metabase Google Analytics Driver dependency {:plugin Metabase Google Drivers Shared Dependencies} satisfied? true
2021-07-07 06:48:14,028 DEBUG plugins.initialize :: e[33mDependencies satisfied; these plugins will now be loaded: ["Metabase Google Analytics Driver" "Metabase BigQuery Driver"]e[0m
2021-07-07 06:48:14,028 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :googleanalytics...e[0m
2021-07-07 06:48:14,029 INFO driver.impl :: e[34mRegistered driver :googleanalyticse[0m (parents: [:google]) ๐Ÿšš
2021-07-07 06:48:14,029 WARN plugins.lazy-loaded-driver :: e[31mWarning: plugin manifest for :bigquery does not include connection propertiese[0m
2021-07-07 06:48:14,030 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :bigquery...e[0m
2021-07-07 06:48:14,030 INFO driver.impl :: e[34mRegistered driver :bigquerye[0m (parents: [:sql :google]) ๐Ÿšš
2021-07-07 06:48:14,044 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :snowflake...e[0m
2021-07-07 06:48:14,045 INFO driver.impl :: e[34mRegistered driver :snowflakee[0m (parents: [:sql-jdbc]) ๐Ÿšš
2021-07-07 06:48:14,050 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :mongo...e[0m
2021-07-07 06:48:14,050 INFO driver.impl :: e[34mRegistered driver :mongoe[0m ๐Ÿšš
2021-07-07 06:48:14,052 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :presto...e[0m
2021-07-07 06:48:14,053 INFO driver.impl :: e[34mRegistered driver :prestoe[0m (parents: [:sql]) ๐Ÿšš
2021-07-07 06:48:14,119 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :redshift...e[0m
2021-07-07 06:48:14,128 INFO driver.impl :: e[34mRegistered driver :postgrese[0m (parents: [:sql-jdbc]) ๐Ÿšš
e[32mLoad driver :postgres took 16.8 mse[0m
2021-07-07 06:48:14,137 INFO driver.impl :: e[34mRegistered driver :redshifte[0m (parents: [:postgres]) ๐Ÿšš
2021-07-07 06:48:14,141 INFO plugins.dependencies :: e[31mMetabase cannot initialize plugin Metabase Oracle Driver due to required dependencies.e[0m Metabase requires the Oracle JDBC driver in order to connect to Oracle databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/oracle.html for more details.
2021-07-07 06:48:14,141 INFO plugins.dependencies :: Metabase Oracle Driver dependency {:class oracle.jdbc.OracleDriver} satisfied? false
2021-07-07 06:48:14,142 INFO plugins.dependencies :: e[33mPlugins with unsatisfied deps: ["Metabase Oracle Driver"]e[0m
2021-07-07 06:48:14,144 INFO plugins.dependencies :: e[31mMetabase cannot initialize plugin Metabase Vertica Driver due to required dependencies.e[0m Metabase requires the Vertica JDBC driver in order to connect to Vertica databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/vertica.html for more details.
2021-07-07 06:48:14,144 INFO plugins.dependencies :: Metabase Vertica Driver dependency {:class com.vertica.jdbc.Driver} satisfied? false
2021-07-07 06:48:14,145 INFO plugins.dependencies :: e[33mPlugins with unsatisfied deps: ["Metabase Vertica Driver" "Metabase Oracle Driver"]e[0m
2021-07-07 06:48:14,147 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :druid...e[0m
2021-07-07 06:48:14,149 INFO driver.impl :: e[34mRegistered driver :druide[0m ๐Ÿšš
2021-07-07 06:48:14,245 INFO driver.impl :: e[34mRegistered driver :h2e[0m (parents: [:sql-jdbc]) ๐Ÿšš
2021-07-07 06:48:14,266 INFO driver.impl :: e[34mRegistered driver :mysqle[0m (parents: [:sql-jdbc]) ๐Ÿšš
2021-07-07 06:48:14,321 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
2021-07-07 06:48:14,324 INFO db.setup :: e[36mVerifying postgres Database Connection ...e[0m
2021-07-07 06:48:24,369 ERROR metabase.core :: Metabase Initialization FAILED
clojure.lang.ExceptionInfo: Unable to connect to Metabase postgres DB. {}
at metabase.db.setup$fn__35507$verify_db_connection__35512$fn__35513$fn__35514.invoke(setup.clj:102)
at metabase.db.setup$fn__35507$verify_db_connection__35512$fn__35513.invoke(setup.clj:100)
at metabase.db.setup$fn__35507$verify_db_connection__35512.invoke(setup.clj:94)
at metabase.db.setup$setup_db_BANG_$fn__35542$fn__35543.invoke(setup.clj:142)
at metabase.util$do_with_us_locale.invokeStatic(util.clj:687)
at metabase.util$do_with_us_locale.invoke(util.clj:673)
at metabase.db.setup$setup_db_BANG_$fn__35542.invoke(setup.clj:141)
at metabase.db.setup$setup_db_BANG_.invokeStatic(setup.clj:140)
at metabase.db.setup$setup_db_BANG_.invoke(setup.clj:136)
at metabase.db$setup_db_BANG_$fn__35556.invoke(db.clj:61)
at metabase.db$setup_db_BANG_.invokeStatic(db.clj:56)
at metabase.db$setup_db_BANG_.invoke(db.clj:51)
at metabase.core$init_BANG_.invokeStatic(core.clj:95)
at metabase.core$init_BANG_.invoke(core.clj:74)
at metabase.core$start_normally.invokeStatic(core.clj:139)
at metabase.core$start_normally.invoke(core.clj:133)
at metabase.core$_main.invokeStatic(core.clj:172)
at metabase.core$_main.doInvoke(core.clj:166)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at metabase.core.main(Unknown Source)
Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:315)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225)
at org.postgresql.Driver.makeConnection(Driver.java:465)
at org.postgresql.Driver.connect(Driver.java:264)
at java.sql/java.sql.DriverManager.getConnection(Unknown Source)
at java.sql/java.sql.DriverManager.getConnection(Unknown Source)
at clojure.java.jdbc$get_driver_connection.invokeStatic(jdbc.clj:271)
at clojure.java.jdbc$get_driver_connection.invoke(jdbc.clj:250)
at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:411)
at clojure.java.jdbc$get_connection.invoke(jdbc.clj:274)
at clojure.java.jdbc$db_query_with_resultset_STAR_.invokeStatic(jdbc.clj:1111)
at clojure.java.jdbc$db_query_with_resultset_STAR_.invoke(jdbc.clj:1093)
at clojure.java.jdbc$query.invokeStatic(jdbc.clj:1182)
at clojure.java.jdbc$query.invoke(jdbc.clj:1144)
at clojure.java.jdbc$query.invokeStatic(jdbc.clj:1160)
at clojure.java.jdbc$query.invoke(jdbc.clj:1144)
at metabase.driver.sql_jdbc.connection$can_connect_with_spec_QMARK_.invokeStatic(connection.clj:199)
at metabase.driver.sql_jdbc.connection$can_connect_with_spec_QMARK_.invoke(connection.clj:196)
at metabase.db.setup$fn__35507$verify_db_connection__35512$fn__35513$fn__35514.invoke(setup.clj:100)
... 21 more
Caused by: java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at org.postgresql.core.PGStream.createSocket(PGStream.java:231)
at org.postgresql.core.PGStream.<init>(PGStream.java:95)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:98)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)
... 40 more
2021-07-07 06:48:24,379 INFO metabase.core :: Metabase Shutting Down ...
2021-07-07 06:48:24,379 INFO metabase.server :: Shutting Down Embedded Jetty Webserver

@aalap Well, that's a completely different problem. It's errors with Unable to connect to Metabase postgres DB, so that's because there's no connection between Metabase and the application database.

1 Like

Now we are getting below error:
E 46 com.simba.athena.dsi.core.impl.DSIConnection.getProperty(19): +++++ enter +++++

2021-07-09T00:11:16.019+05:30	Jul 08 18:41:16.019 TRACE 56 com.simba.athena.jdbc.common.SConnection.clearWarnings(): +++++ enter +++++

2021-07-09T00:11:16.019+05:30	Jul 08 18:41:16.019 TRACE 56 com.simba.athena.jdbc.common.SConnection.getAutoCommit(): +++++ enter +++++

2021-07-09T00:11:16.019+05:30	Jul 08 18:41:16.019 TRACE 56 com.simba.athena.dsi.core.impl.DSIConnection.getProperty(19): +++++ enter +++++

2021-07-09T00:11:16.054+05:30	2021-07-08 18:41:16,054 INFO sync.util :: e[35mFINISHED: step 'fingerprint-fields' for athena Database 2 'report_db' (2.0 s)e[0m

2021-07-09T00:11:16.055+05:30	2021-07-08 18:41:16,055 INFO sync.util :: e[35mSTARTING: step 'classify-fields' for athena Database 2 'report_db'e[0m

2021-07-09T00:11:16.135+05:30	2021-07-08 18:41:16,135 INFO sync.analyze :: e[34mclassify-fields Analyzed [********************ยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยท] ๐Ÿ˜ 40% Table 45 'report_glue_db_prod.resume_download_statistics'e[0m

2021-07-09T00:11:16.243+05:30	2021-07-08 18:41:16,243 INFO sync.analyze :: e[34mclassify-fields Analyzed [*************************ยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยท] ๐Ÿ˜ฌ 50% Table 235 'report_glue_db_prod.hc1_oldjobads'e[0m

2021-07-09T00:11:16.326+05:30	2021-07-08 18:41:16,325 INFO sync.analyze :: e[34mclassify-fields Analyzed [******************************ยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยทยท] ๐Ÿ˜Œ 60% Table 157 'report_glue_db_prod.hc1_datedpostingactivities'e[0m

2021-07-09T00:11:16.365+05:30	2021-07-08 18:41:16,364 INFO sync.util :: e[35mFINISHED: step 'classify-fields' for athena Database 2 'report_db' (309.6 ms)e[0m

2021-07-09T00:11:16.365+05:30	2021-07-08 18:41:16,365 INFO sync.util :: e[35mSTARTING: step 'classify-tables' for athena Database 2 'report_db'e[0m

2021-07-09T00:11:16.386+05:30	2021-07-08 18:41:16,386 INFO sync.analyze :: e[34mclassify-tables Analyzed [***********************************ยทยทยทยทยทยทยทยทยทยทยทยทยทยทยท] ๐Ÿ˜ 70% Table 13 'report_glue_db_prod.company_change_requests'e[0m

2021-07-09T00:11:16.474+05:30	2021-07-08 18:41:16,474 INFO sync.analyze :: e[34mclassify-tables Analyzed [****************************************ยทยทยทยทยทยทยทยทยทยท] ๐Ÿ˜Š 80% Table 198 'report_glue_db_prod.hc1_employerquestionsanswers'e[0m

2021-07-09T00:11:16.623+05:30	2021-07-08 18:41:16,622 INFO sync.analyze :: e[34mclassify-tables Analyzed [*********************************************ยทยทยทยทยท] ๐Ÿ˜ 90% Table 307 'report_glue_db_prod.hc1_featuredprofiles'e[0m

2021-07-09T00:11:16.748+05:30	2021-07-08 18:41:16,748 INFO sync.util :: e[35mFINISHED: step 'classify-tables' for athena Database 2 'report_db' (382.9 ms)e[0m

2021-07-09T00:11:16.761+05:30	2021-07-08 18:41:16,761 INFO sync.util :: e[35mFINISHED: Analyze data for athena Database 2 'report_db' (2.7 s)e[0m

2021-07-09T00:13:44.193+05:30

Copy
Jul 08 18:43:44.193 TRACE 3 com.simba.athena.jdbc.common.SForwardResultSet.close(): +++++ enter +++++
Jul 08 18:43:44.193 TRACE 3 com.simba.athena.jdbc.common.SForwardResultSet.close(): +++++ enter +++++

2021-07-09T00:13:44.193+05:30	Jul 08 18:43:44.193 TRACE 3 com.simba.athena.jdbc.common.SStatement.close(): +++++ enter +++++

@aalap There's no errors. That's just trace logging. I don't see anything in that log that could help figuring out what is wrong.
Which version of the Athena driver are you using?

@flamber
Oh, sorry, I misunderstood. Please find below Athena version.
Athena driver V1.2.0

And I have updated to 0.39.4, still data is not syncing with athena db, can you please help?

@aalap I don't know what the problem is, but perhaps the driver needs to get fixed:
https://github.com/dacort/metabase-athena-driver/issues

Hey @flamber,
Driver is updated and it works fine in similar environment (dev, stg). But not works only in prod environment. Any idea?
As Metabase read exactly from Athena on Dev and Staging env but it is not updating data on Prod server, where all 3 env are of same.

@aalap If it works in "development" and "staging", but not in "production", then there must be a difference somewhere between the environments.

Hey @flamber,

  1. Can you please share any docs related to AWS Athena and Metabase integration or related.
    As I guess connection string and secret needs to be updated and issue might resolve.

  2. Select Athena as DB source gives 3 schema and I need to show only one so for that purpose, can you guide me if possible.

Thanks

@aalap

  1. Third-party drivers should have documentation on usage, where you download the driver from.
    So in this case: https://github.com/dacort/metabase-athena-driver
    You should compare data in the application database between your environments. Also, some settings might require restart of your instance.

  2. If the user credentials you use only have limited privileges, then Metabase cannot see more than what that user has privileges to.
    I don't use Athena, so you'll have to look in a Athena forum.

Hey @flamber,
I noticed one thing:

  • If I add access key and secret key it shows perfect data which I expect.
  • If I skip access key and just insert secret key it shows old un-synced data.

is this something related metabase database or do you have any idea why it is happening?
Thanks

@aalap So if you don't fill out all the database details correctly, then it doesn't work? Then fill out all the details.
Sounds like you had filled out all the details at some point, which means Metabase synced, and then you removed some details, which means that Metabase cannot sync correctly anymore and will show the information Metabase has (meaning outdated).

Seems like the driver should require all fields filled out, so open an issue on the driver. I cannot help you. I do not use Athena.

Okay @flamber, thanks for all the answers, really appreciate that.

I will raise query in driver.