We recently upgraded to a new version as we were getting a bug when trying to add multiple filters to a question that was explicitly listed as fixed in the release notes.
Since moving to the new version we now seem to be getting a number of errors with the message.Assert failed: No more than 1024 pending puts are allowed on a single channel. Consider using a windowed buffer. (< (.size puts) impl/MAX-QUEUE-SIZE)
I have noticed there is one other topic with this exact error, but it's from 2020 and just said it was resolved in an old version.
{
"browser-info": {
"language": "en-GB",
"platform": "MacIntel",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/112.0",
"vendor": ""
},
"system-info": {
"file.encoding": "UTF-8",
"java.runtime.name": "OpenJDK Runtime Environment",
"java.runtime.version": "11.0.19+7",
"java.vendor": "Eclipse Adoptium",
"java.vendor.url": "https://adoptium.net/",
"java.version": "11.0.19",
"java.vm.name": "OpenJDK 64-Bit Server VM",
"java.vm.version": "11.0.19+7",
"os.name": "Linux",
"os.version": "5.10.179-166.674.amzn2.x86_64",
"user.language": "en",
"user.timezone": "Europe/London"
},
"metabase-info": {
"databases": [
"mysql"
],
"hosting-env": "unknown",
"application-database": "mysql",
"application-database-details": {
"database": {
"name": "MySQL",
"version": "5.7.12"
},
"jdbc-driver": {
"name": "MariaDB Connector/J",
"version": "2.7.6"
}
},
"run-mode": "prod",
"version": {
"date": "2023-05-24",
"tag": "v0.46.4",
"branch": "release-x.46.x",
"hash": "f858476"
},
"settings": {
"report-timezone": null
}
}
}
1 Like
[7805d989-940d-4e53-9a0f-6a87dd828b6b] 2023-06-13T16:38:00+01:00 DEBUG metabase.server.middleware.log GET /api/public/dashboard/93406bf8-7768-4e1a-acf5-c7debb20da9c/dashcard/124/card/135 202 [ASYNC: completed] 6.0 s (17 DB calls) App DB connections: 1/15 Jetty threads: 3/50 (23 idle, 0 queued) (147 total active threads) Queries in flight: 2 (0 queued); mysql DB 2 connections: 14/15 (0 threads blocked)
[7805d989-940d-4e53-9a0f-6a87dd828b6b] 2023-06-13T16:38:00+01:00 ERROR metabase.query-processor.middleware.catch-exceptions Error processing query: Assert failed: No more than 1024 pending puts are allowed on a single channel. Consider using a windowed buffer.
(< (.size puts) impl/MAX-QUEUE-SIZE)
{:database_id 2,
:started_at #t "2023-06-13T16:37:55.272358+01:00[Europe/London]",
:via
[{:status :failed,
:class clojure.lang.ExceptionInfo,
:error
"Error reducing result rows: Assert failed: No more than 1024 pending puts are allowed on a single channel. Consider using a windowed buffer.\n(< (.size puts) impl/MAX-QUEUE-SIZE)",
:stacktrace
["--> query_processor.context.default$default_reducef$fn__53368.invoke(default.clj:60)"
"query_processor.context.default$default_reducef.invokeStatic(default.clj:57)"
"query_processor.context.default$default_reducef.invoke(default.clj:49)"
"query_processor.middleware.cache$run_query_with_cache$reducef_SINGLEQUOTE___69252$fn__69253.invoke(cache.clj:196)"
"query_processor.middleware.cache.impl$do_with_serialization.invokeStatic(impl.clj:83)"
"query_processor.middleware.cache.impl$do_with_serialization.invoke(impl.clj:54)"
"query_processor.middleware.cache.impl$do_with_serialization.invokeStatic(impl.clj:73)"
"query_processor.middleware.cache.impl$do_with_serialization.invoke(impl.clj:54)"
"query_processor.middleware.cache$run_query_with_cache$reducef_SINGLEQUOTE___69252.invoke(cache.clj:192)"
"query_processor.context$reducef.invokeStatic(context.clj:70)"
"query_processor.context$reducef.invoke(context.clj:63)"
"query_processor.context.default$default_runf$respond_STAR___53372.invoke(default.clj:69)"
"driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:513)"
"driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:489)"
"driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:497)"
"driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:489)"
"driver.sql_jdbc$fn__105125.invokeStatic(sql_jdbc.clj:63)"
"driver.sql_jdbc$fn__105125.invoke(sql_jdbc.clj:61)"
"query_processor.context$executef.invokeStatic(context.clj:60)"
"query_processor.context$executef.invoke(context.clj:49)"
"query_processor.context.default$default_runf.invokeStatic(default.clj:68)"
"query_processor.context.default$default_runf.invoke(default.clj:66)"
"query_processor.context$runf.invokeStatic(context.clj:46)"
"query_processor.context$runf.invoke(context.clj:40)"
"query_processor.reducible$identity_qp.invokeStatic(reducible.clj:15)"
"query_processor.reducible$identity_qp.invoke(reducible.clj:12)"
"query_processor.middleware.cache$run_query_with_cache.invokeStatic(cache.clj:197)"
"query_processor.middleware.cache$run_query_with_cache.invoke(cache.clj:182)"
"query_processor.middleware.cache$maybe_return_cached_results$maybe_return_cached_results_STAR___69265.invoke(cache.clj:223)"
"query_processor.middleware.permissions$check_query_permissions$fn__64737.invoke(permissions.clj:126)"
"query_processor.middleware.mbql_to_native$mbql__GT_native$fn__68156.invoke(mbql_to_native.clj:24)"
"query_processor$fn__70784$combined_post_process__70789$combined_post_process_STAR___70790.invoke(query_processor.clj:243)"
"query_processor$fn__70784$combined_pre_process__70785$combined_pre_process_STAR___70786.invoke(query_processor.clj:240)"
"query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__69176$fn__69181.invoke(resolve_database_and_driver.clj:36)"
"driver$do_with_driver.invokeStatic(driver.clj:90)"
"driver$do_with_driver.invoke(driver.clj:86)"
"query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__69176.invoke(resolve_database_and_driver.clj:35)"
"query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__65046.invoke(fetch_source_query.clj:310)"
"query_processor.middleware.store$initialize_store$fn__65224$fn__65225.invoke(store.clj:12)"
"query_processor.store$do_with_store.invokeStatic(store.clj:47)"
"query_processor.store$do_with_store.invoke(store.clj:41)"
"query_processor.middleware.store$initialize_store$fn__65224.invoke(store.clj:11)"
"query_processor.middleware.normalize_query$normalize$fn__69465.invoke(normalize_query.clj:25)"
"query_processor.middleware.constraints$add_default_userland_constraints$fn__66402.invoke(constraints.clj:54)"
"query_processor.middleware.process_userland_query$process_userland_query$fn__69401.invoke(process_userland_query.clj:151)"
"query_processor.middleware.catch_exceptions$catch_exceptions$fn__69778.invoke(catch_exceptions.clj:171)"
"query_processor.reducible$async_qp$qp_STAR___59546$thunk__59548.invoke(reducible.clj:103)"
"query_processor.reducible$async_qp$qp_STAR___59546$fn__59550.invoke(reducible.clj:108)"],
:error_type :qp,
:ex-data {:type :qp}}],
:error_type :qp,
:json_query
{:constraints {:max-results 10000, :max-results-bare-rows 2000},
:type :native,
:middleware {:js-int-to-string? true, :ignore-cached-results? false},
:native
{:query
"SELECT date(`users`.`created_at`) AS `created_at`, count(*) AS `count`\nFROM `users`\nWHERE (date(`users`.`created_at`) BETWEEN date(date_add(now(), INTERVAL -1 day))\n AND date(now()) AND `users`.`type` = 'admin')\nGROUP BY date(`users`.`created_at`)\nORDER BY date(`users`.`created_at`) ASC",
:template-tags {}},
:database 2,
:parameters [],
:async? true,
:cache-ttl 17},
:status :failed,
:class java.lang.AssertionError,
:stacktrace
["clojure.core.async.impl.channels.ManyToManyChannel.put_BANG_(channels.clj:157)"
"clojure.core.async$put_BANG_.invokeStatic(async.clj:201)"
"clojure.core.async$put_BANG_.invoke(async.clj:189)"
"--> events$publish_event_BANG_.invokeStatic(events.clj:73)"
"events$publish_event_BANG_.invoke(events.clj:66)"
"query_processor.middleware.process_userland_query$add_and_save_execution_info_xform_BANG_$execution_info_rf_STAR___69392.invoke(process_userland_query.clj:94)"
"query_processor.middleware.results_metadata$insights_xform$combine__68138.invoke(results_metadata.clj:67)"
"query_processor.reducible$combine_additional_reducing_fns$combine_additional_reducing_fns_rf_STAR___59569.invoke(reducible.clj:199)"
"query_processor.middleware.add_rows_truncated$add_rows_truncated_xform$fn__66815.invoke(add_rows_truncated.clj:27)"
"query_processor.middleware.format_rows$format_rows_xform$fn__69682.invoke(format_rows.clj:71)"
"query_processor.middleware.annotate$add_column_info_xform$combine__61643.invoke(annotate.clj:689)"
"query_processor.reducible$combine_additional_reducing_fns$combine_additional_reducing_fns_rf_STAR___59569.invoke(reducible.clj:199)"
"query_processor.middleware.cache$save_results_xform$fn__69212.invoke(cache.clj:116)"
"query_processor.context.default$default_reducef$fn__53368.invoke(default.clj:58)"
"query_processor.context.default$default_reducef.invokeStatic(default.clj:57)"
"query_processor.context.default$default_reducef.invoke(default.clj:49)"
"query_processor.middleware.cache$run_query_with_cache$reducef_SINGLEQUOTE___69252$fn__69253.invoke(cache.clj:196)"
"query_processor.middleware.cache.impl$do_with_serialization.invokeStatic(impl.clj:83)"
"query_processor.middleware.cache.impl$do_with_serialization.invoke(impl.clj:54)"
"query_processor.middleware.cache.impl$do_with_serialization.invokeStatic(impl.clj:73)"
"query_processor.middleware.cache.impl$do_with_serialization.invoke(impl.clj:54)"
"query_processor.middleware.cache$run_query_with_cache$reducef_SINGLEQUOTE___69252.invoke(cache.clj:192)"
"query_processor.context$reducef.invokeStatic(context.clj:70)"
"query_processor.context$reducef.invoke(context.clj:63)"
"query_processor.context.default$default_runf$respond_STAR___53372.invoke(default.clj:69)"
"driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:513)"
"driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:489)"
"driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:497)"
"driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:489)"
"driver.sql_jdbc$fn__105125.invokeStatic(sql_jdbc.clj:63)"
"driver.sql_jdbc$fn__105125.invoke(sql_jdbc.clj:61)"
"query_processor.context$executef.invokeStatic(context.clj:60)"
"query_processor.context$executef.invoke(context.clj:49)"
"query_processor.context.default$default_runf.invokeStatic(default.clj:68)"
"query_processor.context.default$default_runf.invoke(default.clj:66)"
"query_processor.context$runf.invokeStatic(context.clj:46)"
"query_processor.context$runf.invoke(context.clj:40)"
"query_processor.reducible$identity_qp.invokeStatic(reducible.clj:15)"
"query_processor.reducible$identity_qp.invoke(reducible.clj:12)"
"query_processor.middleware.cache$run_query_with_cache.invokeStatic(cache.clj:197)"
"query_processor.middleware.cache$run_query_with_cache.invoke(cache.clj:182)"
"query_processor.middleware.cache$maybe_return_cached_results$maybe_return_cached_results_STAR___69265.invoke(cache.clj:223)"
"query_processor.middleware.permissions$check_query_permissions$fn__64737.invoke(permissions.clj:126)"
"query_processor.middleware.mbql_to_native$mbql__GT_native$fn__68156.invoke(mbql_to_native.clj:24)"
"query_processor$fn__70784$combined_post_process__70789$combined_post_process_STAR___70790.invoke(query_processor.clj:243)"
"query_processor$fn__70784$combined_pre_process__70785$combined_pre_process_STAR___70786.invoke(query_processor.clj:240)"
"query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__69176$fn__69181.invoke(resolve_database_and_driver.clj:36)"
"driver$do_with_driver.invokeStatic(driver.clj:90)"
"driver$do_with_driver.invoke(driver.clj:86)"
"query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__69176.invoke(resolve_database_and_driver.clj:35)"
"query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__65046.invoke(fetch_source_query.clj:310)"
"query_processor.middleware.store$initialize_store$fn__65224$fn__65225.invoke(store.clj:12)"
"query_processor.store$do_with_store.invokeStatic(store.clj:47)"
"query_processor.store$do_with_store.invoke(store.clj:41)"
"query_processor.middleware.store$initialize_store$fn__65224.invoke(store.clj:11)"
"query_processor.middleware.normalize_query$normalize$fn__69465.invoke(normalize_query.clj:25)"
"query_processor.middleware.constraints$add_default_userland_constraints$fn__66402.invoke(constraints.clj:54)"
"query_processor.middleware.process_userland_query$process_userland_query$fn__69401.invoke(process_userland_query.clj:151)"
"query_processor.middleware.catch_exceptions$catch_exceptions$fn__69778.invoke(catch_exceptions.clj:171)"
"query_processor.reducible$async_qp$qp_STAR___59546$thunk__59548.invoke(reducible.clj:103)"
"query_processor.reducible$async_qp$qp_STAR___59546$fn__59550.invoke(reducible.clj:108)"],
:card_id 131,
:context :public-dashboard,
:error
"Assert failed: No more than 1024 pending puts are allowed on a single channel. Consider using a windowed buffer.\n(< (.size puts) impl/MAX-QUEUE-SIZE)",
:row_count 0,
:running_time 0,
:data {:rows [], :cols []}}
Luiggi
June 13, 2023, 5:01pm
3
I think that the issue is that your MySQL app db is very old:
The minimum recommended version is MySQL 5.7.7 or MariaDB 10.2.2, and the utf8mb4 character set is required. You can change the application database to use MySQL using environment variables like this:
5.7.12 is newer than 5.7.7 which is the minimum version.
Luiggi
June 13, 2023, 5:28pm
5
What’s the version of MySQL you’re querying? Not the app db, the dw
Luiggi
June 13, 2023, 7:09pm
7
mmmmmmmmmmmm wait, how many users do you have hitting your public dashboards? If I read correctly this is an issue in our Ring middleware so it's a problem in the web server
Not a huge number, it's a "public" dashboard but it's embedded in an internal web app our company use, behind a login. So maybe around 60 users max across the course of a day.
Is there any update to this?
Luiggi
June 19, 2023, 11:55am
10
Unfortunately no since we can’t reproduce. You’re the only user that hit this since the report from many years back
We had this same error when we upgraded Metabase to version v0.46.6.1 to address the security vulnerabilities shared in this blog post: Please upgrade your Metabase immediately
We have opened up a bug ticket with them to get help to address this issue.
opened 08:01PM - 21 Jul 23 UTC
Type:Bug
.Needs Triage
### Describe the bug
Hi Metabase folks,
Today we tried to upgrade our insta… nce to v0.46.6.1 following the recommendations from this post (https://www.metabase.com/blog/security-advisory), and we ran into a lot of errors while users tried to execute their queries. Here's the error we are getting:
`
Assert failed: No more than 1024 pending puts are allowed on a single channel. Consider using a windowed buffer. (< (.size puts) impl/MAX-QUEUE-SIZE)
`
We ended up reverting the upgrade back to v0.46.1, but we still need to address this security vulnerability.
Is this issue known? What recommendation would you make to apply the security patches?
Thank you.
### To Reproduce
When accessing any question page.
### Expected behavior
_No response_
### Logs
Assert failed: No more than 1024 pending puts are allowed on a single channel. Consider using a windowed buffer. (< (.size puts) impl/MAX-QUEUE-SIZE)
### Information about your Metabase installation
```JSON
{
"browser-info": {
"language": "en-US",
"platform": "MacIntel",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36",
"vendor": "Google Inc."
},
"system-info": {
"file.encoding": "UTF-8",
"java.runtime.name": "OpenJDK Runtime Environment",
"java.runtime.version": "11.0.18+10",
"java.vendor": "Eclipse Adoptium",
"java.vendor.url": "https://adoptium.net/",
"java.version": "11.0.18",
"java.vm.name": "OpenJDK 64-Bit Server VM",
"java.vm.version": "11.0.18+10",
"os.name": "Linux",
"os.version": "4.14.314-238.539.amzn2.x86_64",
"user.language": "en",
"user.timezone": "GMT"
},
"metabase-info": {
"databases": [
"postgres",
"snowflake",
"h2"
],
"hosting-env": "unknown",
"application-database": "mysql",
"application-database-details": {
"database": {
"name": "MySQL",
"version": "5.7.38-log"
},
"jdbc-driver": {
"name": "MariaDB Connector/J",
"version": "2.7.6"
}
},
"run-mode": "prod",
"version": {
"date": "2023-04-05",
"tag": "v0.46.1",
"branch": "release-x.46.x",
"hash": "12a6e1d"
},
"settings": {
"report-timezone": "US/Eastern"
}
}
}
```
### Severity
High
### Additional context
_No response_
I will share updates.
Luiggi
July 21, 2023, 8:07pm
12
I just replied: the change has nothing to do on that end