Dashboarding charts crashing

fatica · October 7, 2020, 4:02pm

Hi,

This dashboard loads initially, but then after a few uses, it eventually stops loading with a screen like this:

Notice the error in the chart, "There was a problem displaying this chart".

Other dashboards do not seem to have this problem. I broke this dashboard into two different dashboards, just to see if it was too big and the problem persists.

A few errors in the log like this:

200 185.0 ms (11 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (70 total active threads) Queries in flight: 0 (0 queued)
[76b7cfd2-6728-4256-9762-bc91e2ff3dc3] 2020-10-06T15:01:58-05:00 DEBUG metabase.middleware.log GET /api/field/12731 401 242.5 µs (0 DB calls)
"Unauthenticated"

[76b7cfd2-6728-4256-9762-bc91e2ff3dc3] 2020-10-06T15:01:58-05:00 DEBUG metabase.middleware.log GET /api/embed/dashboard/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyZXNvdXJjZSI6eyJkYXNoYm9hcmQiOjExfSwicGFyYW1zIjp7InBhcmFtcyI6W119fQ.-5uN5xfEeZmqyHwJBg4B3Ya386CXayeVBvT9ZqF6cxU/dashcard/71/card/64 400 38.1 ms (2 DB calls)
"Unknown parameter :params."

[76b7cfd2-6728-4256-9762-bc91e2ff3dc3] 2020-10-06T15:01:58-05:00 DEBUG metabase.middleware.log GET /api/embed/dashboard/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyZXNvdXJjZSI6eyJkYXNoYm9hcmQiOjExfSwicGFyYW1zIjp7InBhcmFtcyI6W119fQ.-5uN5xfEeZmqyHwJBg4B3Ya386CXayeVBvT9ZqF6cxU/dashcard/69/card/63 400 18.9 ms (2 DB calls)
"Unknown parameter :params."

[76b7cfd2-6728-4256-9762-bc91e2ff3dc3] 2020-10-06T15:01:58-05:00 DEBUG metabase.middleware.log GET /api/embed/dashboard/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyZXNvdXJjZSI6eyJkYXNoYm9hcmQiOjExfSwicGFyYW1zIjp7InBhcmFtcyI6W119fQ.-5uN5xfEeZmqyHwJBg4B3Ya386CXayeVBvT9ZqF6cxU/field/12731/values 200 145.8 ms (5 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (70 total active threads) Queries in flight: 0 (0 queued)
[76b7cfd2-6728-4256-9762-bc91e2ff3dc3] 2020-10-06T15:59:03-05:00 WARN metabase.middleware.log POST /api/jsonws/invoke 404 579.4 µs (0 DB calls)
"API endpoint does not exist."

[76b7cfd2-6728-4256-9762-bc91e2ff3dc3] 2020-10-06T16:00:00-05:00 INFO metabase.task.send-pulses Sending scheduled pulses...
[76b7cfd2-6728-4256-9762-bc91e2ff3dc3] 2020-10-06T16:25:45-05:00 DEBUG metabase.middleware.log GET /api/session/properties 200 1.2 ms (0 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (67 total active threads) Queries in flight: 0 (0 queued)
[76b7cfd2-6728-4256-9762-bc91e2ff3dc3] 2020-10-06T16:25:46-05:00 DEBUG metabase.middleware.log GET /api/user/current 401 272.2 µs (0 DB calls)
"Unauthenticated"

[76b7cfd2-6728-4256-9762-bc91e2ff3dc3] 2020-10-06T17:00:00-05:00 INFO metabase.task.send-pulses Sending scheduled pulses...
[76b7cfd2-6728-4256-9762-bc91e2ff3dc3] 2020-10-06T18:00:00-05:00 INFO metabase.task.send-pulses Sending scheduled pulses...
[76b7cfd2-6728-4256-9762-bc91e2ff3dc3] 2020-10-06T18:01:48-05:00 WARN metabase.middleware.log POST /api/jsonws/invoke 404 347.9 µs (0 DB calls)
"API endpoint does not exist."

Diagnostic info as follows:
{
"browser-info": {
"language": "en-US",
"platform": "MacIntel",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36",
"vendor": "Google Inc."
},
"system-info": {
"file.encoding": "UTF-8",
"java.runtime.name": "OpenJDK Runtime Environment",
"java.runtime.version": "11.0.8+10",
"java.vendor": "AdoptOpenJDK",
"java.vendor.url": "https://adoptopenjdk.net/",
"java.version": "11.0.8",
"java.vm.name": "OpenJDK 64-Bit Server VM",
"java.vm.version": "11.0.8+10",
"os.name": "Linux",
"os.version": "4.14.67-66.56.amzn1.x86_64",
"user.language": "en",
"user.timezone": "GMT"
},
"metabase-info": {
"databases": [
"mysql"
],
"hosting-env": "elastic-beanstalk",
"application-database": "postgres",
"application-database-details": {
"database": {
"name": "PostgreSQL",
"version": "10.6"
},
"jdbc-driver": {
"name": "PostgreSQL JDBC Driver",
"version": "42.2.8"
}
},
"run-mode": "prod",
"version": {
"date": "2020-09-15",
"tag": "v0.36.6",
"branch": "release-0.36.x",
"hash": "cb258fb"
},
"settings": {
"report-timezone": "US/Mountain"
}
}
}

flamber · October 7, 2020, 4:09pm

Hi @fatica
Since you’re embedding, how long time is the exp (expiration) of the token set to?

fatica · October 7, 2020, 4:50pm

Hi, thanks for the reply, the chart is embedded, but this is happening directly in MetaBase, not while viewing it in the embedded view. When embedding the charts never load, but I believe that might a different issue. Would the embedded dashboard affect the dashboard in MetaBase?

I did not author the PHP integration, but I don’t see any exp being set. Again, I’d be surprised if these were related issues.

public static function getIframeUrl($params)
{
    $dashboard_id = (int) $params->get('metabase_dashboard_id');
    $metabaseSiteUrl = $params->get('metabase_site_url');
    $metabaseSecretKey = $params->get('metabase_secret_key');
    
    $signer = new \Lcobucci\JWT\Signer\Hmac\Sha256();
    $token = (new \Lcobucci\JWT\Builder())
        ->set('resource', ['dashboard' => $dashboard_id])
        ->set('params', [
            'params' => []
        ])
        ->sign($signer, $metabaseSecretKey)
        ->getToken();
    
    return "$metabaseSiteUrl/embed/dashboard/$token#bordered=true&titled=true";
    
}

fatica · October 7, 2020, 5:27pm

Console warnings include:

app-main.bundle.js?b33c92a4486dd445d864:5 DEPRECATED: metabase/redux/metadata addParamValues
v @ app-main.bundle.js?b33c92a4486dd445d864:5
app-main.bundle.js?b33c92a4486dd445d864:5 DEPRECATED: metabase/redux/metadata addFields
v @ app-main.bundle.js?b33c92a4486dd445d864:5
app-main.bundle.js?b33c92a4486dd445d864:5 DEPRECATED: metabase/redux/metadata fetchField
v @ app-main.bundle.js?b33c92a4486dd445d864:5
app-main.bundle.js?b33c92a4486dd445d864:5 DEPRECATED: metabase/redux/metadata fetchFieldValues
v @ app-main.bundle.js?b33c92a4486dd445d864:5
app-main.bundle.js?b33c92a4486dd445d864:5 Error getting setting graph.x_axis._is_timeseries TypeError: Cannot read property ‘cols’ of undefined
at Object.getDefault (app-main.bundle.js?b33c92a4486dd445d864:5)
at g (app-main.bundle.js?b33c92a4486dd445d864:5)
at t.getComputedSettings (app-main.bundle.js?b33c92a4486dd445d864:5)
at h (app-main.bundle.js?b33c92a4486dd445d864:5)
at t.value (app-main.bundle.js?b33c92a4486dd445d864:5)
at t.value (app-main.bundle.js?b33c92a4486dd445d864:5)
at c.performInitialMount (vendor.bundle.js?b33c92a4486dd445d864:1)
at c.mountComponent (vendor.bundle.js?b33c92a4486dd445d864:1)
at Object.mountComponent (vendor.bundle.js?b33c92a4486dd445d864:1)
at c.performInitialMount (vendor.bundle.js?b33c92a4486dd445d864:1)
g @ app-main.bundle.js?b33c92a4486dd445d864:5
app-main.bundle.js?b33c92a4486dd445d864:5 Error getting setting graph.x_axis._is_numeric TypeError: Cannot read property ‘cols’ of undefined
at Object.getDefault (app-main.bundle.js?b33c92a4486dd445d864:5)
at g (app-main.bundle.js?b33c92a4486dd445d864:5)
at t.getComputedSettings (app-main.bundle.js?b33c92a4486dd445d864:5)
at h (app-main.bundle.js?b33c92a4486dd445d864:5)
at t.value (app-main.bundle.js?b33c92a4486dd445d864:5)
at t.value (app-main.bundle.js?b33c92a4486dd445d864:5)
at c.performInitialMount (vendor.bundle.js?b33c92a4486dd445d864:1)
at c.mountComponent (vendor.bundle.js?b33c92a4486dd445d864:1)
at Object.mountComponent (vendor.bundle.js?b33c92a4486dd445d864:1)
at c.performInitialMount (vendor.bundle.js?b33c92a4486dd445d864:1)
g @ app-main.bundle.js?b33c92a4486dd445d864:5
app-main.bundle.js?b33c92a4486dd445d864:5 Error getting setting graph.x_axis._is_histogram TypeError: Cannot read property ‘cols’ of undefined
at Object.getDefault (app-main.bundle.js?b33c92a4486dd445d864:5)
at g (app-main.bundle.js?b33c92a4486dd445d864:5)
at t.getComputedSettings (app-main.bundle.js?b33c92a4486dd445d864:5)
at h (app-main.bundle.js?b33c92a4486dd445d864:5)
at t.value (app-main.bundle.js?b33c92a4486dd445d864:5)
at t.value (app-main.bundle.js?b33c92a4486dd445d864:5)
at c.performInitialMount (vendor.bundle.js?b33c92a4486dd445d864:1)
at c.mountComponent (vendor.bundle.js?b33c92a4486dd445d864:1)
at Object.mountComponent (vendor.bundle.js?b33c92a4486dd445d864:1)

fatica · October 7, 2020, 5:28pm

Console errors show this:

Uncaught Invariant Violation: Minified React error #44; visit http://facebook.github.io/react/docs/error-decoder.html?invariant=44 for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
at e.exports (https://base.metalocator.com/app/dist/vendor.bundle.js?b33c92a4486dd445d864:1:317976)
at Object.e.exports [as findDOMNode] (https://base.metalocator.com/app/dist/vendor.bundle.js?b33c92a4486dd445d864:1:2527194)
at d.value (https://base.metalocator.com/app/dist/app-main.bundle.js?b33c92a4486dd445d864:5:206445)
at n._updateSize (https://base.metalocator.com/app/dist/app-main.bundle.js?b33c92a4486dd445d864:5:205872)

fatica · October 7, 2020, 5:54pm

Looking at an individual question this is what I see. Restarting the application servers fixes it, until it inevitably happens again.

flamber · October 7, 2020, 7:05pm

@fatica I’m guessing you’re connecting over SSH? The problem is that the connection is timing out and never re-established. Workaround is to create the tunnel manually instead of using the built-in:
https://www.metabase.com/docs/latest/administration-guide/01-managing-databases.html#what-if-the-built-in-ssh-tunnels-dont-fit-my-needs
You’re likely seeing this issue:
https://github.com/metabase/metabase/issues/6114 - upvote by clicking on the first post

fatica · October 7, 2020, 8:19pm

Interesting. We are hosted on AWS Elastic BeanStalk. As I understand it, the instances are transient, does this mean every time it scales I will somehow need to update this tunnel?

flamber · October 7, 2020, 8:24pm

@fatica Uhh… yeah. We are currently trying to fix several issues, which requires manual outside workarounds, since that doesn’t work for setups like yours, but I don’t have a timeline for it.
We just fixed a related issue https://github.com/metabase/metabase/issues/8396

fatica · October 7, 2020, 8:35pm

O.k. that puts us in a tough spot, we’re using this in production which is my fault.

Is there a different strategy here that avoids this issue entirely I wonder, like putting the database somewhere else? What’s the recommended approach for connecting to external systems? Should we instead move the data to RDS?

flamber · October 7, 2020, 8:56pm

@fatica That seems like a big change, but I don’t know you structure. Alternatively you can create your own image script, which sets up a tunnel on startup in the container, and then you don’t use the SSH feature inside of Metabase, but use the tunnel create in the container instead.

fatica · October 7, 2020, 9:07pm

I am investigating a VPC Peer to get the database server addressable on the local network, avoiding SSH

flamber · October 7, 2020, 9:08pm

@fatica Yeah, that would probably be the best/easiest non-hacky solution.