We're experiencing server issues

Hello,
I am getting this "We're experiencing server issues. Try refreshing the page after waiting a minute or two. If the problem persists we'd recommend you contact an admin." for some of my queries. I googled it out and could not solve it. A simple query that I get this error is "SELECT pg_sleep(10)".

"SELECT pg_sleep(10)" gives error, but "SELECT pg_sleep(9)" works fine. This works for values <=9 and gives this error for values >= 10.

Here are some more details:

  • I am using Metabase 0.45.1 with jar file on Ubuntu 20.04.5 LTS.
  • Java vesion is 11 (openjdk 11.0.18).
  • The Metabase database has already been migrated to Postgresql.
  • Metabase and all Postgresql DBs are on one server (server A).
  • I tested the same situation with another new Metabase Installation (0.45.3 with H2) on another server (Server B) and have the same problem.
  • I have Adminer on same server (Server A) and "SELECT pg_sleep(10)" works fine.
  • I tested "SELECT pg_sleep(10)" from my local Navicat and works fine.
  • I tested "SELECT pg_sleep(10)" from local Metabase (on my laptop) with connection to same DB (Server A) and it works fine.

I really don't now what is happening specially because Metabase is connecting to database on the same server (Server A). There seems to be some configuration on my companies servers that is causing this.
I really appreciate your help.

Here is what I get from the log file:

{:database_id 2,
:started_at #t "2023-03-15T08:44:51.991339+03:30[Asia/Tehran]",
:via
[{:status :failed,
:class clojure.lang.ExceptionInfo,
:error "Error executing query: ERROR: canceling statement due to user request",
:stacktrace
["--> driver.sql_jdbc.execute$execute_reducible_query$fn__56023.invoke(execute.clj:501)"
"driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:498)"
"driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:485)"
"driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:493)"
"driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:485)"
"driver.sql_jdbc$fn__88395.invokeStatic(sql_jdbc.clj:58)"
"driver.sql_jdbc$fn__88395.invoke(sql_jdbc.clj:56)"
"query_processor.context$executef.invokeStatic(context.clj:59)"
"query_processor.context$executef.invoke(context.clj:48)"
"query_processor.context.default$default_runf.invokeStatic(default.clj:67)"
"query_processor.context.default$default_runf.invoke(default.clj:65)"
"query_processor.context$runf.invokeStatic(context.clj:45)"
"query_processor.context$runf.invoke(context.clj:39)"
"query_processor.reducible$identity_qp.invokeStatic(reducible.clj:12)"
"query_processor.reducible$identity_qp.invoke(reducible.clj:9)"
"query_processor.middleware.cache$maybe_return_cached_results$maybe_return_cached_results_STAR___53529.invoke(cache.clj:220)"
"query_processor.middleware.permissions$check_query_permissions$fn__49184.invoke(permissions.clj:109)"
"query_processor.middleware.mbql_to_native$mbql__GT_native$fn__52474.invoke(mbql_to_native.clj:23)"
"query_processor$fn__55005$combined_post_process__55010$combined_post_process_STAR___55011.invoke(query_processor.clj:212)"
"query_processor$fn__55005$combined_pre_process__55006$combined_pre_process_STAR___55007.invoke(query_processor.clj:209)"
"query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__53450$fn__53455.invoke(resolve_database_and_driver.clj:35)"
"driver$do_with_driver.invokeStatic(driver.clj:76)"
"driver$do_with_driver.invoke(driver.clj:72)"
"query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__53450.invoke(resolve_database_and_driver.clj:34)"
"query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__49450.invoke(fetch_source_query.clj:314)"
"query_processor.middleware.store$initialize_store$fn__49640$fn__49641.invoke(store.clj:11)"
"query_processor.store$do_with_store.invokeStatic(store.clj:45)"
"query_processor.store$do_with_store.invoke(store.clj:39)"
"query_processor.middleware.store$initialize_store$fn__49640.invoke(store.clj:10)"
"query_processor.middleware.normalize_query$normalize$fn__53722.invoke(normalize_query.clj:22)"
"query_processor.middleware.constraints$add_default_userland_constraints$fn__50728.invoke(constraints.clj:53)"
"query_processor.middleware.process_userland_query$process_userland_query$fn__53661.invoke(process_userland_query.clj:145)"
"query_processor.middleware.catch_exceptions$catch_exceptions$fn__54033.invoke(catch_exceptions.clj:167)"
"query_processor.reducible$async_qp$qp_STAR___45482$thunk__45484.invoke(reducible.clj:100)"
"query_processor.reducible$async_qp$qp_STAR___45482.invoke(reducible.clj:106)"
"query_processor.reducible$async_qp$qp_STAR___45482.invoke(reducible.clj:91)"
"query_processor.reducible$sync_qp$qp_STAR___45493.doInvoke(reducible.clj:126)"
"query_processor$process_userland_query.invokeStatic(query_processor.clj:331)"
"query_processor$process_userland_query.doInvoke(query_processor.clj:327)"
"query_processor$fn__55052$process_query_and_save_execution_BANG___55061$fn__55064.invoke(query_processor.clj:342)"
"query_processor$fn__55052$process_query_and_save_execution_BANG___55061.invoke(query_processor.clj:335)"
"query_processor$fn__55096$process_query_and_save_with_max_results_constraints_BANG___55105$fn__55108.invoke(query_processor.clj:354)"
"query_processor$fn__55096$process_query_and_save_with_max_results_constraints_BANG___55105.invoke(query_processor.clj:347)"
"api.dataset$run_query_async$fn__70403.invoke(dataset.clj:69)"
"query_processor.streaming$streaming_response_STAR_$fn__40578$fn__40579.invoke(streaming.clj:162)"
"query_processor.streaming$streaming_response_STAR_$fn__40578.invoke(streaming.clj:161)"
"async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:65)"
"async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:63)"
"async.streaming_response$do_f_async$task__20017.invoke(streaming_response.clj:84)"],
:error_type :invalid-query,
:ex-data
{:sql
"-- Metabase:: userID: 1 queryType: native queryHash: c5e83ea841c703a305c8bd3ef34c4eeb5a55583f9dcafe1afbcaa05561e09e5b\nSELECT pg_sleep(11)",
:params nil,
:type :invalid-query}}],
:state "57014",
:error_type :invalid-query,
:json_query
{:type "native",
:native {:query "SELECT pg_sleep(11)", :template-tags {}},
:database 2,
:parameters [],
:middleware {:js-int-to-string? true, :add-default-userland-constraints? true}},
:status :failed,
:class org.postgresql.util.PSQLException,
:stacktrace
["org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676)"
"org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366)"
"org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356)"
"org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496)"
"org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413)"
"org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:333)"
"org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:319)"
"org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:295)"
"org.postgresql.jdbc.PgStatement.execute(PgStatement.java:290)"
"com.mchange.v2.c3p0.impl.NewProxyStatement.execute(NewProxyStatement.java:75)"
"--> driver.sql_jdbc.execute$fn__55943.invokeStatic(execute.clj:366)"
"driver.sql_jdbc.execute$fn__55943.invoke(execute.clj:364)"
"driver.sql_jdbc.execute$execute_statement_or_prepared_statement_BANG_.invokeStatic(execute.clj:374)"
"driver.sql_jdbc.execute$execute_statement_or_prepared_statement_BANG_.invoke(execute.clj:371)"
"driver.sql_jdbc.execute$execute_reducible_query$fn__56023.invoke(execute.clj:499)"
"driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:498)"
"driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:485)"
"driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:493)"
"driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:485)"
"driver.sql_jdbc$fn__88395.invokeStatic(sql_jdbc.clj:58)"
"driver.sql_jdbc$fn__88395.invoke(sql_jdbc.clj:56)"
"query_processor.context$executef.invokeStatic(context.clj:59)"
"query_processor.context$executef.invoke(context.clj:48)"
"query_processor.context.default$default_runf.invokeStatic(default.clj:67)"
"query_processor.context.default$default_runf.invoke(default.clj:65)"
"query_processor.context$runf.invokeStatic(context.clj:45)"
"query_processor.context$runf.invoke(context.clj:39)"
"query_processor.reducible$identity_qp.invokeStatic(reducible.clj:12)"
"query_processor.reducible$identity_qp.invoke(reducible.clj:9)"
"query_processor.middleware.cache$maybe_return_cached_results$maybe_return_cached_results_STAR___53529.invoke(cache.clj:220)"
"query_processor.middleware.permissions$check_query_permissions$fn__49184.invoke(permissions.clj:109)"
"query_processor.middleware.mbql_to_native$mbql__GT_native$fn__52474.invoke(mbql_to_native.clj:23)"
"query_processor$fn__55005$combined_post_process__55010$combined_post_process_STAR___55011.invoke(query_processor.clj:212)"
"query_processor$fn__55005$combined_pre_process__55006$combined_pre_process_STAR___55007.invoke(query_processor.clj:209)"
"query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__53450$fn__53455.invoke(resolve_database_and_driver.clj:35)"
"driver$do_with_driver.invokeStatic(driver.clj:76)"
"driver$do_with_driver.invoke(driver.clj:72)"
"query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__53450.invoke(resolve_database_and_driver.clj:34)"
"query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__49450.invoke(fetch_source_query.clj:314)"
"query_processor.middleware.store$initialize_store$fn__49640$fn__49641.invoke(store.clj:11)"
"query_processor.store$do_with_store.invokeStatic(store.clj:45)"
"query_processor.store$do_with_store.invoke(store.clj:39)"
"query_processor.middleware.store$initialize_store$fn__49640.invoke(store.clj:10)"
"query_processor.middleware.normalize_query$normalize$fn__53722.invoke(normalize_query.clj:22)"
"query_processor.middleware.constraints$add_default_userland_constraints$fn__50728.invoke(constraints.clj:53)"
"query_processor.middleware.process_userland_query$process_userland_query$fn__53661.invoke(process_userland_query.clj:145)"
"query_processor.middleware.catch_exceptions$catch_exceptions$fn__54033.invoke(catch_exceptions.clj:167)"
"query_processor.reducible$async_qp$qp_STAR___45482$thunk__45484.invoke(reducible.clj:100)"
"query_processor.reducible$async_qp$qp_STAR___45482.invoke(reducible.clj:106)"
"query_processor.reducible$async_qp$qp_STAR___45482.invoke(reducible.clj:91)"
"query_processor.reducible$sync_qp$qp_STAR___45493.doInvoke(reducible.clj:126)"
"query_processor$process_userland_query.invokeStatic(query_processor.clj:331)"
"query_processor$process_userland_query.doInvoke(query_processor.clj:327)"
"query_processor$fn__55052$process_query_and_save_execution_BANG___55061$fn__55064.invoke(query_processor.clj:342)"
"query_processor$fn__55052$process_query_and_save_execution_BANG___55061.invoke(query_processor.clj:335)"
"query_processor$fn__55096$process_query_and_save_with_max_results_constraints_BANG___55105$fn__55108.invoke(query_processor.clj:354)"
"query_processor$fn__55096$process_query_and_save_with_max_results_constraints_BANG___55105.invoke(query_processor.clj:347)"
"api.dataset$run_query_async$fn__70403.invoke(dataset.clj:69)"
"query_processor.streaming$streaming_response_STAR_$fn__40578$fn__40579.invoke(streaming.clj:162)"
"query_processor.streaming$streaming_response_STAR_$fn__40578.invoke(streaming.clj:161)"
"async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:65)"
"async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:63)"
"async.streaming_response$do_f_async$task__20017.invoke(streaming_response.clj:84)"],
:card_id nil,
:context :ad-hoc,
:error "ERROR: canceling statement due to user request",
:row_count 0,
:running_time 0,
:data {:rows [], :cols []}}

Here is the diagnostic info:

{
"browser-info": {
"language": "en-US",
"platform": "Win32",
"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/110.0",
"vendor": ""
},
"system-info": {
"file.encoding": "UTF-8",
"java.runtime.name": "OpenJDK Runtime Environment",
"java.runtime.version": "11.0.18+10-post-Ubuntu-0ubuntu120.04.1",
"java.vendor": "Ubuntu",
"java.vendor.url": "https://ubuntu.com/",
"java.version": "11.0.18",
"java.vm.name": "OpenJDK 64-Bit Server VM",
"java.vm.version": "11.0.18+10-post-Ubuntu-0ubuntu120.04.1",
"os.name": "Linux",
"os.version": "5.4.0-137-generic",
"user.language": "en",
"user.timezone": "Asia/Tehran"
},
"metabase-info": {
"databases": [
"sqlserver",
"postgres",
"h2"
],
"hosting-env": "unknown",
"application-database": "postgres",
"application-database-details": {
"database": {
"name": "PostgreSQL",
"version": "12.13 (Ubuntu 12.13-1.pgdg20.04+1)"
},
"jdbc-driver": {
"name": "PostgreSQL JDBC Driver",
"version": "42.5.0"
}
},
"run-mode": "prod",
"version": {
"date": "2022-12-07",
"tag": "v0.45.1",
"branch": "release-x.45.x",
"hash": "019d31c"
},
"settings": {
"report-timezone": null
}
}
}

Seems like your reverse proxy is killing every request of more than 10 seconds, have you checked that config? It’s not a Metabase config, but rather a reverse proxy config

1 Like

That's right. The problem was in Nginx configurations. proxy_read_timeout was set to 10.
Thanks you very much.