Hello,
I'm experiencing some issues since today on my Metabase instance hosted on an AWS
Elastic Beanstalk Webapp. I'm currently running on Metabase v0.45.2.1 upgraded 20 days ago. During the last 3 days my instance was a bit slow. But today it became unusable.
At first I've upgraded to the lastest docker platform on AWS today as I was running a version deprecated since december 2020. I've also upgraded from a T2.small to a T2.medium instance. I'm only using 20% RAM and have plenty of disk space available.
Both my own DB and Metabase DB are healthy, with a great amount of remaining space, CPU usage is always below 50% for the moment.
Since today, fetching current user data takes a very long time (more than a minute) and most requests are killed by the Gateway. Authentication through Google SSO is impossible. Our pulses are also sent with a delay between 10 and 15 minutes.
For the moment I can't see what's wrong. Does anyone have experienced the same thing in the past ?
Here is the error thrown during auth:
2023-03-02 15:51:00,500 INFO integrations.google :: Successfully authenticated Google Sign-In token for: HIDDEN
2023-03-02 15:52:04,593 DEBUG middleware.log :: e[32mPOST /api/session/google_auth 200 1.1 mins (7 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (35 idle, 0 queued) (69 total active threads) Queries in flight: 0 (0 queued)e[0m
2023-03-02 15:52:04,595 ERROR api.session :: Authentication endpoint error
org.eclipse.jetty.io.EofException
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:280)
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:834)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:555)
at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:1014)
at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:1086)
at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:285)
at org.eclipse.jetty.server.HttpOutput.close(HttpOutput.java:638)
at ring.util.servlet$make_output_stream$fn__834.invoke(servlet.clj:91)
at ring.util.servlet.proxy$java.io.FilterOutputStream$ff19274a.close(Unknown Source)
at ring.core.protocols$fn__789.invokeStatic(protocols.clj:39)
at ring.core.protocols$fn__789.invoke(protocols.clj:19)
at ring.core.protocols$fn__759$G__754__768.invoke(protocols.clj:8)
at ring.util.servlet$update_servlet_response.invokeStatic(servlet.clj:109)
at ring.util.servlet$update_servlet_response.invoke(servlet.clj:94)
at metabase.server.protocols$fn__886.invokeStatic(protocols.clj:27)
at metabase.server.protocols$fn__886.invoke(protocols.clj:20)
at metabase.server.protocols$fn__862$fn__865$G__863__872.invoke(protocols.clj:5)
at metabase.server$async_proxy_handler$fn__83000$fn__83010.invoke(server.clj:76)
at ring.middleware.gzip$wrap_gzip$fn__87330$fn__87331.invoke(gzip.clj:89)
at clojure.core$comp$fn__5876.invoke(core.clj:2586)
at clojure.core$comp$fn__5876.invoke(core.clj:2586)
at ring.middleware.cookies$wrap_cookies$fn__87288$fn__87289.invoke(cookies.clj:217)
at metabase.server.middleware.session$reset_session_timeout$fn__48385$fn__48386.invoke(session.clj:396)
at clojure.core$comp$fn__5876.invoke(core.clj:2586)
at clojure.core$comp$fn__5876.invoke(core.clj:2586)
at clojure.core$comp$fn__5876.invoke(core.clj:2586)
at clojure.core$comp$fn__5876.invoke(core.clj:2586)
at metabase.server.middleware.exceptions$catch_uncaught_exceptions$fn__79072$fn__79073.invoke(exceptions.clj:103)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$wrap_route_matches$fn__29306$fn__29307.invoke(core.clj:151)
at compojure.response$send.invokeStatic(response.clj:28)
at compojure.response$send.invoke(response.clj:22)
at compojure.core$wrap_response$fn__29313.invoke(core.clj:160)
at compojure.core$wrap_route_middleware$fn__29297.invoke(core.clj:132)
at compojure.core$wrap_route_info$fn__29302.invoke(core.clj:139)
at compojure.core$wrap_route_matches$fn__29306.invoke(core.clj:151)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$wrap_route_matches$fn__29306.invoke(core.clj:153)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$wrap_route_matches$fn__29306.invoke(core.clj:152)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$wrap_route_matches$fn__29306.invoke(core.clj:153)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$wrap_route_matches$fn__29306.invoke(core.clj:152)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$wrap_route_matches$fn__29306.invoke(core.clj:153)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$wrap_route_matches$fn__29306.invoke(core.clj:152)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325.invoke(core.clj:200)
at metabase.api.session$_PLUS_log_all_request_failures$fn__81729.invoke(session.clj:311)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325.invoke(core.clj:200)
at compojure.core$make_context$handler__29353.invoke(core.clj:289)
at compojure.core$make_context$fn__29357.invoke(core.clj:299)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$make_context$fn__29357.invoke(core.clj:300)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at metabase.api.routes$fn__82628$fn__82631.invoke(routes.clj:64)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325.invoke(core.clj:200)
at clojure.lang.AFn.applyToHelper(AFn.java:160)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.core$apply.invoke(core.clj:662)
at metabase.server.routes$fn__82792$fn__82793.doInvoke(routes.clj:69)
at clojure.lang.RestFn.invoke(RestFn.java:436)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325.invoke(core.clj:200)
at compojure.core$make_context$handler__29353.invoke(core.clj:289)
at compojure.core$make_context$fn__29357.invoke(core.clj:299)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$wrap_route_matches$fn__29306.invoke(core.clj:153)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$wrap_route_matches$fn__29306.invoke(core.clj:153)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at compojure.core$wrap_route_matches$fn__29306.invoke(core.clj:153)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325$f__29326$respond_SINGLEQUOTE___29327.invoke(core.clj:197)
at metabase.server.routes$fn__82777$fn__82779.invoke(routes.clj:46)
at compojure.core$routes$fn__29325$f__29326.invoke(core.clj:198)
at compojure.core$routes$fn__29325.invoke(core.clj:200)
at metabase.server.middleware.exceptions$catch_uncaught_exceptions$fn__79072.invoke(exceptions.clj:98)
at metabase.server.middleware.exceptions$catch_api_exceptions$fn__79069.invoke(exceptions.clj:86)
at metabase.server.middleware.log$log_api_call$fn__84204$fn__84205$fn__84206.invoke(log.clj:211)
at metabase.driver.sql_jdbc.execute.diagnostic$do_with_diagnostic_info.invokeStatic(diagnostic.clj:15)
at metabase.driver.sql_jdbc.execute.diagnostic$do_with_diagnostic_info.invoke(diagnostic.clj:9)
at metabase.server.middleware.log$log_api_call$fn__84204$fn__84205.invoke(log.clj:203)
at toucan.db$_do_with_call_counting.invokeStatic(db.clj:236)
at toucan.db$_do_with_call_counting.invoke(db.clj:229)
at metabase.server.middleware.log$log_api_call$fn__84204.invoke(log.clj:202)
at metabase.server.middleware.browser_cookie$ensure_browser_id_cookie$fn__87101.invoke(browser_cookie.clj:38)
at metabase.server.middleware.security$add_security_headers$fn__63926.invoke(security.clj:149)
at metabase.server.middleware.json$wrap_json_body$fn__86232.invoke(json.clj:63)
at metabase.server.middleware.json$wrap_streamed_json_response$fn__86250.invoke(json.clj:99)
at metabase.server.middleware.offset_paging$handle_paging$fn__63950.invoke(offset_paging.clj:42)
at ring.middleware.keyword_params$wrap_keyword_params$fn__87368.invoke(keyword_params.clj:55)
at ring.middleware.params$wrap_params$fn__87387.invoke(params.clj:77)
at metabase.server.middleware.misc$maybe_set_site_url$fn__36696.invoke(misc.clj:59)
at metabase.server.middleware.session$reset_session_timeout$fn__48385.invoke(session.clj:394)
at metabase.server.middleware.session$bind_current_user$fn__48368$fn__48369.invoke(session.clj:331)
at metabase.server.middleware.session$do_with_current_user.invokeStatic(session.clj:310)
at metabase.server.middleware.session$do_with_current_user.invoke(session.clj:299)
at metabase.server.middleware.session$bind_current_user$fn__48368.invoke(session.clj:330)
at metabase.server.middleware.session$wrap_current_user_info$fn__48350.invoke(session.clj:280)
at metabase.server.middleware.session$wrap_session_id$fn__48334.invoke(session.clj:213)
at metabase.server.middleware.auth$wrap_api_key$fn__70605.invoke(auth.clj:30)
at ring.middleware.cookies$wrap_cookies$fn__87288.invoke(cookies.clj:216)
at metabase.server.middleware.misc$add_content_type$fn__36679.invoke(misc.clj:27)
at metabase.server.middleware.misc$disable_streaming_buffering$fn__36704.invoke(misc.clj:76)
at ring.middleware.gzip$wrap_gzip$fn__87330.invoke(gzip.clj:86)
at metabase.server.middleware.misc$bind_request$fn__36707.invoke(misc.clj:93)
at metabase.server.middleware.ssl$redirect_to_https_middleware$fn__87117.invoke(ssl.clj:48)
at metabase.server$async_proxy_handler$fn__83000.invoke(server.clj:73)
at metabase.server.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Broken pipe
at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
at java.base/sun.nio.ch.SocketDispatcher.writev(Unknown Source)
at java.base/sun.nio.ch.IOUtil.write(Unknown Source)
at java.base/sun.nio.ch.IOUtil.write(Unknown Source)
at java.base/sun.nio.ch.SocketChannelImpl.write(Unknown Source)
at java.base/java.nio.channels.SocketChannel.write(Unknown Source)
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:274)
... 242 more
2023-03-02 15:52:04,596 INFO middleware.exceptions :: Request canceled before finishing.
