Metabase - Slack

Hi,

Slack messages was working but I’ve problem with Slack since last week. I’ ve bee using 0.38.0-rc1

What does it mean ?

on Pulse page : “SSL peer shut down incorrectly”

logs

PUT /api/slack/settings 400 5.2 s (0 DB calls)
{:url “https://slack.com/api/conversations.list”,
:request
{:query-params {:token “***********”, :limit 1, :exclude_archived true, :types “public_channel,private_channel”},
:as :stream,
:conn-timeout 10000,
:socket-timeout 10000}}

I had changed metabase admin user password last week.

I changed slack admin user password too, it’s ok now, solved.

today, I got this error again. How to fix it ?

Hi @alpertungir
Sounds like you might have a proxy that closes the connection with a timeout, but there should be a full stacktrace in the logs - Admin > Troubleshooting > Logs.

hi @flamber,

we are not using a proxy. I've only this error . Yesterday It was ok but while sending pulse, it occured again. (Admin > Troubleshooting > Logs>> "SSL peer shut down incorrectly")

image

@alpertungir Post “Diagnostic Info” from Admin > Troubleshooting.

@flamber

{
 "browser-info": {
   "language": "tr-TR",
   "platform": "Win32",
   "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.183 Safari/537.36 OPR/72.0.3815.320",
   "vendor": "Google Inc."
 },
 "system-info": {
   "file.encoding": "UTF-8",
   "java.runtime.name": "OpenJDK Runtime Environment",
   "java.runtime.version": "11.0.7+10-post-Ubuntu-2ubuntu219.10",
   "java.vendor": "Ubuntu",
   "java.vendor.url": "https://ubuntu.com/",
   "java.version": "11.0.7",
   "java.vm.name": "OpenJDK 64-Bit Server VM",
   "java.vm.version": "11.0.7+10-post-Ubuntu-2ubuntu219.10",
   "os.name": "Linux",
   "os.version": "5.3.0-64-generic",
   "user.language": "en",
   "user.timezone": "Europe/Istanbul"
 },
 "metabase-info": {
   "databases": [
     "oracle"
   ],
   "hosting-env": "unknown",
   "application-database": "mysql",
   "application-database-details": {
     "database": {
       "name": "MySQL",
       "version": "8.0.20-0ubuntu0.19.10.1"
     },
     "jdbc-driver": {
       "name": "MariaDB Connector/J",
       "version": "2.6.2"
     }
   },
   "run-mode": "prod",
   "version": {
     "date": "2020-12-17",
     "tag": "v0.38.0-rc1",
     "branch": "master",
     "hash": "e8b0bcc"
   },
   "settings": {
     "report-timezone": "Europe/Istanbul"
   }
 }
}

@alpertungir Where are you seeing the error SSL peer shut down incorrectly ?

@flamber

On pulse page. now I can't show you because I reset the slack token and I'cant enable it as shown before.
If I can enable it, pulses which are using slack, shown like this and I can't choose any slack user. I also see on Admin > Troubleshooting > Logs after pulse sending events.

If you tell me , how to see old logs before restart metabase service, I can send it to you.

image

@alpertungir I don’t know how you have setup Metabase and if you are saving the logs to a permanent file, but that would be the only place where logs are kept after restarting.
Otherwise you might need to enable debug/trace logging to see everything, so you have a better idea where it fails:
https://www.metabase.com/docs/latest/operations-guide/log-configuration.html
Try removing the Slack bot and do the setup again. There’s too little information to figure out what is going on.

@flamber

log details while pulse sending

[3db6d28d-6c70-4586-9233-f56544218961] 2021-01-02T18:00:03+03:00 ERROR metabase.pulse Bildirim gönderilirken hata oluştu!
java.io.EOFException: SSL peer shut down incorrectly
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:167)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:108)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1144)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1055)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:395)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
at org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:313)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at clj_http.core$request.invokeStatic(core.clj:643)
at clj_http.core$request.invoke(core.clj:568)
at clj_http.core$request.invokeStatic(core.clj:569)
at clj_http.core$request.invoke(core.clj:568)
at clojure.lang.Var.invoke(Var.java:384)
at clj_http.client$wrap_request_timing$fn__23807.invoke(client.clj:1077)
at clj_http.headers$wrap_header_map$fn__22941.invoke(headers.clj:152)
at clj_http.client$wrap_query_params$fn__23697.invoke(client.clj:811)
at clj_http.client$wrap_basic_auth$fn__23703.invoke(client.clj:834)
at clj_http.client$wrap_oauth$fn__23708.invoke(client.clj:851)
at clj_http.client$wrap_user_info$fn__23717.invoke(client.clj:871)
at clj_http.client$wrap_url$fn__23789.invoke(client.clj:1029)
at clj_http.client$wrap_decompression$fn__23498.invoke(client.clj:418)
at clj_http.client$wrap_input_coercion$fn__23621.invoke(client.clj:631)
at clj_http.client$wrap_additional_header_parsing$fn__23646.invoke(client.clj:686)
at clj_http.client$wrap_output_coercion$fn__23608.invoke(client.clj:575)
at clj_http.client$wrap_exceptions$fn__23450.invoke(client.clj:252)
at clj_http.client$wrap_accept$fn__23661.invoke(client.clj:729)
at clj_http.client$wrap_accept_encoding$fn__23668.invoke(client.clj:751)
at clj_http.client$wrap_content_type$fn__23655.invoke(client.clj:712)
at clj_http.client$wrap_form_params$fn__23754.invoke(client.clj:953)
at clj_http.client$wrap_nested_params$fn__23775.invoke(client.clj:987)
at clj_http.client$wrap_flatten_nested_params$fn__23784.invoke(client.clj:1011)
at clj_http.client$wrap_method$fn__23722.invoke(client.clj:887)
at clj_http.cookies$wrap_cookies$fn__22853.invoke(cookies.clj:131)
at clj_http.links$wrap_links$fn__23277.invoke(links.clj:63)
at clj_http.client$wrap_unknown_host$fn__23792.invoke(client.clj:1040)
at clj_http.client$request_STAR_.invokeStatic(client.clj:1168)
at clj_http.client$request_STAR_.invoke(client.clj:1161)
at clj_http.client$post.invokeStatic(client.clj:1186)
at clj_http.client$post.doInvoke(client.clj:1182)
at clojure.lang.RestFn.invoke(RestFn.java:423)
at metabase.integrations.slack$fn__64578$upload_file_BANG___64583$fn__64584.invoke(slack.clj:171)
at metabase.integrations.slack$fn__64578$upload_file_BANG___64583.invoke(slack.clj:167)
at metabase.pulse$create_and_upload_slack_attachments_BANG_$iter__69983__69987$fn__69988.invoke(pulse.clj:102)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:51)
at clojure.lang.RT.seq(RT.java:535)
at clojure.core$seq__5402.invokeStatic(core.clj:137)
at clojure.core$dorun.invokeStatic(core.clj:3133)
at clojure.core$doall.invokeStatic(core.clj:3148)
at clojure.core$doall.invoke(core.clj:3148)
at metabase.pulse$create_and_upload_slack_attachments_BANG_.invokeStatic(pulse.clj:100)
at metabase.pulse$create_and_upload_slack_attachments_BANG_.invoke(pulse.clj:97)
at metabase.pulse$fn__70108.invokeStatic(pulse.clj:265)
at metabase.pulse$fn__70108.invoke(pulse.clj:263)
at clojure.lang.MultiFn.invoke(MultiFn.java:229)
at metabase.pulse$send_notifications_BANG_$fn__70123.invoke(pulse.clj:281)
at metabase.pulse$send_notifications_BANG_.invokeStatic(pulse.clj:280)
at metabase.pulse$send_notifications_BANG_.invoke(pulse.clj:276)
at metabase.pulse$send_pulse_BANG_.invokeStatic(pulse.clj:303)
at metabase.pulse$send_pulse_BANG_.doInvoke(pulse.clj:285)
at clojure.lang.RestFn.invoke(RestFn.java:439)
at metabase.task.send_pulses$fn__79134$send_pulses_BANG___79143$fn__79146$fn__79164$fn__79165.invoke(send_pulses.clj:59)
at metabase.models.task_history$fn__38433$do_with_task_history__38438$fn__38439.invoke(task_history.clj:77)
at metabase.models.task_history$fn__38433$do_with_task_history__38438.invoke(task_history.clj:72)
at metabase.task.send_pulses$fn__79134$send_pulses_BANG___79143$fn__79146$fn__79164.invoke(send_pulses.clj:56)
at metabase.task.send_pulses$fn__79134$send_pulses_BANG___79143$fn__79146.invoke(send_pulses.clj:55)
at metabase.task.send_pulses$fn__79134$send_pulses_BANG___79143.invoke(send_pulses.clj:42)
at metabase.task.send_pulses$fn__79134$send_pulses_BANG___79143$fn__79144.invoke(send_pulses.clj:49)
at metabase.task.send_pulses$fn__79134$send_pulses_BANG___79143.invoke(send_pulses.clj:42)
at metabase.task.send_pulses.SendPulses$fn__79200.invoke(send_pulses.clj:100)
at metabase.models.task_history$fn__38433$do_with_task_history__38438$fn__38439.invoke(task_history.clj:77)
at metabase.models.task_history$fn__38433$do_with_task_history__38438.invoke(task_history.clj:72)
at metabase.task.send_pulses.SendPulses.execute(send_pulses.clj:86)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)

@alpertungir And you have no outgoing proxy, firewall or similar network intercepting software? If no, then I don’t know what’s going on.

we cannot find any block or similar things on firewall/network layer.

But as you said , there is a timeout for slack api. because of this I created a crontab to keep it alive. For now, it seems ok. Thank you for your helps.

*/15 * * * * curl -H “Authorization: Bearer ${token}” https://slack.com/api/conversations.list?pretty=1

@alpertungir I’m quite sure you have a network intercept somewhere - load balancer, proxy, firewall, WAF - which causes this timeout. Sounds similar to this:
https://github.com/metabase/metabase/issues/5380