[0.30.2: DONE] Metabase 0.30 RC: Queries timing out, other issues (AWS ALB HTTP/2 related)

Just tried with 0.29.3 release as well. I can reproduce the same query lock issue. Pasted server log here for the x-ray operation.

https://pastebin.com/8uAkwYnW

I think it might be related to the database env, and the target database is an instance of RDS PostgreSQL on AWS.

@je2ryw1h are you on Mongo as well?

Nope, the source data is in a read-only replica of PostgreSQL on RDS, and metabase database is also a RDS PostgresSQL. Metabase instance is using the official metabase docker images.

Based on the log pasted here (YLiZPt4V), it seems the query did fetch the data successfully from the target table, just for some reason could not complete the REST response.

My DB connection (to mongo) is also read-only. Probably not related to the issue though.

Is there anything in the JS logs? Do you have access to any server metrics (specifically cpu or memory consumption) of the server post x-ray?

What are you running the docker image on, memory/cpu wise?

On the frontend, it just the usual request (/api/dataset) timeout, and with following error from javascript stack:

api.js:137 POST https://metabase.vdap.ops-ven.com/api/dataset 504 ()
(anonymous) @ api.js:137
_makeRequest @ api.js:106
(anonymous) @ api.js:98
getDatasetQueryResult @ Question.js:461
_callee$ @ app-main.bundle.js?a148caa8cbdb77f1454b:10499
tryCatch @ runtime.js:65
invoke @ runtime.js:303
prototype.(anonymous function) @ runtime.js:117
step @ Question.js:49
(anonymous) @ Question.js:49
(anonymous) @ Question.js:49
apiGetResults @ app-main.bundle.js?a148caa8cbdb77f1454b:10510
_callee15$ @ actions.js:1199
tryCatch @ runtime.js:65
invoke @ runtime.js:303
prototype.(anonymous function) @ runtime.js:117
step @ actions.js:75
(anonymous) @ actions.js:75
(anonymous) @ actions.js:75
(anonymous) @ actions.js:1168
(anonymous) @ store.js:17
dispatch @ applyMiddleware.js:35
(anonymous) @ actions.js:361
setTimeout (async)
_callee2$ @ actions.js:356
tryCatch @ runtime.js:65
invoke @ runtime.js:303
prototype.(anonymous function) @ runtime.js:117
step @ actions.js:75
(anonymous) @ actions.js:75
Promise.then (async)
step @ actions.js:75
(anonymous) @ actions.js:75
(anonymous) @ actions.js:75
(anonymous) @ actions.js:187
(anonymous) @ store.js:17
(anonymous) @ bindActionCreators.js:3
componentWillMount @ QueryBuilder.jsx:147
performInitialMount @ ReactCompositeComponent.js:348
mountComponent @ ReactCompositeComponent.js:255
mountComponent @ ReactReconciler.js:43
performInitialMount @ ReactCompositeComponent.js:368
mountComponent @ ReactCompositeComponent.js:255
mountComponent @ ReactReconciler.js:43
performInitialMount @ ReactCompositeComponent.js:368
mountComponent @ ReactCompositeComponent.js:255
mountComponent @ ReactReconciler.js:43
_updateRenderedComponent @ ReactCompositeComponent.js:762
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
updateChildren @ ReactChildReconciler.js:107
_reconcilerUpdateChildren @ ReactMultiChild.js:209
_updateChildren @ ReactMultiChild.js:308
updateChildren @ ReactMultiChild.js:295
_updateDOMChildren @ ReactDOMComponent.js:944
updateComponent @ ReactDOMComponent.js:758
receiveComponent @ ReactDOMComponent.js:720
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
performUpdateIfNecessary @ ReactCompositeComponent.js:558
performUpdateIfNecessary @ ReactReconciler.js:154
runBatchedUpdates @ ReactUpdates.js:148
perform @ Transaction.js:141
perform @ Transaction.js:141
perform @ ReactUpdates.js:87
flushBatchedUpdates @ ReactUpdates.js:170
closeAll @ Transaction.js:207
perform @ Transaction.js:154
batchedUpdates @ ReactDefaultBatchingStrategy.js:60
batchedUpdates @ ReactUpdates.js:95
dispatchEvent @ ReactEventListener.js:145
settings.js:216 Error getting setting table.columns TypeError: Cannot read property ‘cols’ of undefined
at Object.getDefault (Table.jsx:77)
at getSetting (settings.js:207)
at getSettings (settings.js:255)
at app-main.bundle.js?a148caa8cbdb77f1454b:32907
at index.js:76
at index.js:36
at index.js:90
at index.js:36
at Function.mapStateToProps [as mapToProps] (QueryHeader.jsx:43)
at mapToPropsProxy (wrapMapToProps.js:43)
getSetting @ settings.js:216
getSettings @ settings.js:255
(anonymous) @ app-main.bundle.js?a148caa8cbdb77f1454b:32907
(anonymous) @ index.js:76
(anonymous) @ index.js:36
(anonymous) @ index.js:90
(anonymous) @ index.js:36
mapStateToProps @ QueryHeader.jsx:43
mapToPropsProxy @ wrapMapToProps.js:43
handleNewPropsAndNewState @ selectorFactory.js:34
handleSubsequentCalls @ selectorFactory.js:67
pureFinalPropsSelector @ selectorFactory.js:74
runComponentSelector @ connectAdvanced.js:26
componentWillReceiveProps @ connectAdvanced.js:150
updateComponent @ ReactCompositeComponent.js:611
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
updateChildren @ ReactChildReconciler.js:107
_reconcilerUpdateChildren @ ReactMultiChild.js:209
_updateChildren @ ReactMultiChild.js:308
updateChildren @ ReactMultiChild.js:295
_updateDOMChildren @ ReactDOMComponent.js:944
updateComponent @ ReactDOMComponent.js:758
receiveComponent @ ReactDOMComponent.js:720
receiveComponent @ ReactReconciler.js:122
updateChildren @ ReactChildReconciler.js:107
_reconcilerUpdateChildren @ ReactMultiChild.js:209
_updateChildren @ ReactMultiChild.js:308
updateChildren @ ReactMultiChild.js:295
_updateDOMChildren @ ReactDOMComponent.js:944
updateComponent @ ReactDOMComponent.js:758
receiveComponent @ ReactDOMComponent.js:720
receiveComponent @ ReactReconciler.js:122
updateChildren @ ReactChildReconciler.js:107
_reconcilerUpdateChildren @ ReactMultiChild.js:209
_updateChildren @ ReactMultiChild.js:308
updateChildren @ ReactMultiChild.js:295
_updateDOMChildren @ ReactDOMComponent.js:944
updateComponent @ ReactDOMComponent.js:758
receiveComponent @ ReactDOMComponent.js:720
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
updateChildren @ ReactChildReconciler.js:107
_reconcilerUpdateChildren @ ReactMultiChild.js:209
_updateChildren @ ReactMultiChild.js:308
updateChildren @ ReactMultiChild.js:295
_updateDOMChildren @ ReactDOMComponent.js:944
updateComponent @ ReactDOMComponent.js:758
receiveComponent @ ReactDOMComponent.js:720
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
performUpdateIfNecessary @ ReactCompositeComponent.js:558
performUpdateIfNecessary @ ReactReconciler.js:154
runBatchedUpdates @ ReactUpdates.js:148
perform @ Transaction.js:141
perform @ Transaction.js:141
perform @ ReactUpdates.js:87
flushBatchedUpdates @ ReactUpdates.js:170
closeAll @ Transaction.js:207
perform @ Transaction.js:154
batchedUpdates @ ReactDefaultBatchingStrategy.js:60
enqueueUpdate @ ReactUpdates.js:198
enqueueUpdate @ ReactUpdateQueue.js:22
enqueueSetState @ ReactUpdateQueue.js:216
ReactComponent.setState @ ReactBaseClasses.js:62
onStateChange @ connectAdvanced.js:205
notify @ Subscription.js:26
notifyNestedSubs @ Subscription.js:65
onStateChange @ connectAdvanced.js:202
dispatch @ createStore.js:173
dispatch @ VM420:1
(anonymous) @ middleware.js:22
(anonymous) @ index.js:28
(anonymous) @ store.js:70
(anonymous) @ store.js:17
dispatch @ applyMiddleware.js:35
_callee$ @ redux.js:16
tryCatch @ runtime.js:65
invoke @ runtime.js:303
prototype.(anonymous function) @ runtime.js:117
step @ redux.js:13
(anonymous) @ redux.js:13
Promise.then (async)
step @ redux.js:13
(anonymous) @ redux.js:13
(anonymous) @ redux.js:13
(anonymous) @ redux.js:29
(anonymous) @ store.js:17
dispatch @ applyMiddleware.js:35
(anonymous) @ actions.js:1203
Promise.catch (async)
_callee15$ @ actions.js:1203
tryCatch @ runtime.js:65
invoke @ runtime.js:303
prototype.(anonymous function) @ runtime.js:117
step @ actions.js:75
(anonymous) @ actions.js:75
(anonymous) @ actions.js:75
(anonymous) @ actions.js:1168
(anonymous) @ store.js:17
dispatch @ applyMiddleware.js:35
(anonymous) @ actions.js:361
setTimeout (async)
_callee2$ @ actions.js:356
tryCatch @ runtime.js:65
invoke @ runtime.js:303
prototype.(anonymous function) @ runtime.js:117
step @ actions.js:75
(anonymous) @ actions.js:75
Promise.then (async)
step @ actions.js:75
(anonymous) @ actions.js:75
(anonymous) @ actions.js:75
(anonymous) @ actions.js:187
(anonymous) @ store.js:17
(anonymous) @ bindActionCreators.js:3
componentWillMount @ QueryBuilder.jsx:147
performInitialMount @ ReactCompositeComponent.js:348
mountComponent @ ReactCompositeComponent.js:255
mountComponent @ ReactReconciler.js:43
performInitialMount @ ReactCompositeComponent.js:368
mountComponent @ ReactCompositeComponent.js:255
mountComponent @ ReactReconciler.js:43
performInitialMount @ ReactCompositeComponent.js:368
mountComponent @ ReactCompositeComponent.js:255
mountComponent @ ReactReconciler.js:43
_updateRenderedComponent @ ReactCompositeComponent.js:762
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
updateChildren @ ReactChildReconciler.js:107
_reconcilerUpdateChildren @ ReactMultiChild.js:209
_updateChildren @ ReactMultiChild.js:308
updateChildren @ ReactMultiChild.js:295
_updateDOMChildren @ ReactDOMComponent.js:944
updateComponent @ ReactDOMComponent.js:758
receiveComponent @ ReactDOMComponent.js:720
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
receiveComponent @ ReactCompositeComponent.js:544
receiveComponent @ ReactReconciler.js:122
_updateRenderedComponent @ ReactCompositeComponent.js:751
_performComponentUpdate @ ReactCompositeComponent.js:721
updateComponent @ ReactCompositeComponent.js:642
performUpdateIfNecessary @ ReactCompositeComponent.js:558
performUpdateIfNecessary @ ReactReconciler.js:154
runBatchedUpdates @ ReactUpdates.js:148
perform @ Transaction.js:141
perform @ Transaction.js:141
perform @ ReactUpdates.js:87
flushBatchedUpdates @ ReactUpdates.js:170
closeAll @ Transaction.js:207
perform @ Transaction.js:154
batchedUpdates @ ReactDefaultBatchingStrategy.js:60
batchedUpdates @ ReactUpdates.js:95
dispatchEvent @ ReactEventListener.js:145
ShrinkableList.jsx:44 GET https://metabase.vdap.ops-ven.com/app/app/assets/img/stopwatch.svg 404 ()

The metabase container is running on an AWS t2.medium instance with 2 vCPU and 4G memory. Here is the metabase JVM process from host “top” command, and “free” cmd shows there are still 1.5G physical memory available.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9268 2000 20 0 2582m 644m 15m S 0.3 16.3 1:21.72 java

free -m
total used free shared buffers cached
Mem: 3945 2342 1602 0 173 1050
-/+ buffers/cache: 1118 2827
Swap: 0 0 0

@je2ryw you are seeing these freezes when xraying the Sample Database or your own?

Just did several tests, the issue happens on both Sample Database and external RDS PostgreSQL DB. It is kind depend on how many cards generated from the xray. For the example, I can reproduce the freeze with the “Products” table in the Sample Database as well.

Next step, I am planning to build a new docker image with Oracle JVM to see if it is related to the OpenJDK VM or not.

Finally, the query lock down issue with the “POST /api/dataset” has been resolved. It was caused by the “HTTP/2” option enabled on the AWS Application Load Balancer. Once the “HTTP/2” option is disabled, multiple concurrent API requests generated from the X-Ray operation won’t lock the metabase instance anymore.

2 Likes

Good call, @je2ryw. I changed the following setting in CloudFront, and loading X-Rays no longer breaks my instance.

However, several charts are still not loading. I clicked into one of them and got the following:

This matches the warning I show above in my initial post.

The warnings you were getting should be fixed in 0.30 (final).

Still have to get to the bottom of the fingerprinting error.

I’m seeing a similar issue to this. The api/dataset endpoint returns constant 504’s regardless of the query, interestingly enough running a query or previously saved question works fine. Another interesting find is that the output from the Metabase jarfile doesn’t show this endpoint being logged in the debug text like the other endpoints do.

I’ve got Metabase 0.30.0 running on a EC2 t2.medium as well. I’ve got it running behind an Nginx proxy, but I’m also seeing these issues connecting to it directly on port 3000. This Metabase instance is connecting to a MySQL server but the issues are present with the sample dataset as well.

@mmollick I would suggest to do a clean restart with metabase instance on EC2, then only test the X-Ray operations and query operations with port 3000 directly. If it works fine, then make sure your nginx reverse proxy does not have the keep-alive option enabled for the proxy_pass metabase requests. Also set the root logger as DEBUG level in the log4j.properties can help your verify that no jetty connections remain open after requests complete.

1 Like

@je2ryw Thanks for the help. Setting the keepalive_timeout to 0 in the nginx config seems to have cleared it up.

Great to hear, and glad it helps. :facepunch:

we’ve been seeing pretty regular hangs for a few months too. first the whole app would hang, but more recently just new queries would hang, while the app and existing (maybe cached) queries serve fine. similar symptoms as described here.

(we’re on 0.29.* and now 0.30, bare jar on a single m3.medium EC2 instance behind an application ELB. affected all browsers and all databases we have in metabase, both postgres and bigquery. we were seeing hangs as often as once or more per day, under moderate load.)

disabling http/2 seemed like a far-fetched fix, but i tried it, and we’ve now been up for five days with no hangs. fingers crossed, it may have done the trick. thanks @je2ryw @mmollick @a13n et al!

1 Like

So disabling HTTP/2 is your solution?
Well I too experienced same, when I moved to 0.30.0 it felt like often hanged up.
I looked into logs, memories ramped up very much than prev version I think, I never experienced hanged up in 0.29.*

Hi,
for folks running metabase via the jar and serving up traffic via jetty and an A record in cloudflare - any other options you’d suggest?

V

The freezing should be resolved in 0.30.2.

thanks sbelak - will update today and check…