Metabase timeout configuration

#1

Hi, I’m receiving error Your question took too long when executing a long query over a JDBC Mysql connection in Metabase(I tried increasing the timeout parameters on the JDBC connection but the timeout termination signal is sent by Metabase not by the DB connection). I’d like to configure jetty timeout to a bigger value to see if this error can be avoided.

Does anyone know if env variables
MB_JETTY_ASYNC_RESPONSE_TIMEOUT and MB_JETTY_MAXIDLETIME will work for those purposes?

I see a similar solution here but I don’t know if there is a better way to increase the default timeout value.

Also in this thread a similar solution is tried but over Nginx(I’m using Metabase through docker so I don’t think this solution will work for me).

Does anyone has had any luck increasing Metabase default timeout?
Thanks!

Metabase 504 Timeout Error
Your question took too long on AWS ELB V 0.36.4
#2

Hi @fahernandez
The problem is not between Metabase and MySQL, but between your browser and Metabase.
You have a proxy that is killing the connection.
Read this: https://github.com/metabase/metabase/issues/12423

#3

@flamber what is the recommended method for setting the nginx timeouts when using AWS Elastic Beanstalk (the recommended platform)? I’ve increased the load balancer timeout to 4000 and verified the timeout is coming from nginx. I’ve SSH’d into the app instance and added the increased timeout config myself to get it working, but this will be overwritten as soon as the system updates. The methods for permanently setting the nginx config all seem to require changes to the code repo.

#4

@tjm

Do you have queries that run for 66 minutes? (4000 seconds / 60 = 66.66 minutes)

Currently there’s a 600 seconds (10 minutes) timeout in the EBS image:
https://github.com/metabase/metabase/blob/master/bin/aws-eb-docker/.ebextensions/metabase_config/metabase-setup.sh

The only solution currently would be to manually change it on every update, or create your own image.

Also, where do you see EBS being “the recommended platform”?

Your question took too long on AWS ELB V 0.36.4
#5

Sorry, the 4000 seconds was a distraction - the point was that I’ve set it to the maximum, so it’s definitely not the load balancer. :slight_smile:

Line 37 of that file shows an env var switch: if [[ "x$NGINX_FORCE_SSL" == "x1" ]]. I’ve tried with & without the NGINX_FORCE_SSL var, but I can’t seem to get that branch of the config to install at all. Either way, it’s in a VPC so I don’t want to force SSL. My nginx.conf file contains no references to timeouts, so is using the default 60s. Guessing the config I’m getting is from Line 98 onwards.

And here is the recommendation to use EB (sorry, Discourse won’t let me have 3 links in a post.): metabase. com/docs/v0.35.4/operations-guide/running-metabase-on-elastic-beanstalk.html#running-metabase-on-aws-elastic-beanstalk

The Metabase team runs a number of production installations on AWS using Elastic Beanstalk and currently recommend it as the preferred choice for production deployments.

We’ve just moved to EB from Heroku because it’s not possible to export large CSVs within Heroku’s 30 second timeout.

Thanks!

#6

@tjm Hmm, that’s a terrible config that desperately needs an update. We will need to update that, but it’s something that would require a lot more testing, so there’s no problems for new instances or upgraded ones.

Anyways, the workaround is still valid - change it manually upon every update or create your own image.

#7

Yeah thanks, I’ve made a PR to get the timeouts in the other path for now. Hopefully that’s small enough to get merged for the next patch release.

The change needs to be made much more frequently than every update of Metabase though - it seems like EB is rebuilding the VMs daily. Or maybe the config is lost when they restart.

At some point I’ll get sick of doing it and investigate how to create my own image.

#8

@flamber I’ve figured out what the issue is. The default “Platform branch” for Docker is “Docker running on 64bit Amazon Linux 2”, but it has an incompatible nginx configuration: there is no /etc/nginx/sites-available/ directory anymore so the server_https task fails and the error is ignored.

It seems the solution is to choose “Docker running on 64bit Amazon Linux” when creating the application.

1 Like
Your question took too long on AWS ELB V 0.36.4
#9

@tjm I have manually edited the metabase-setup.sh and zipped . Then created a new elasticbean application using the zip over Docker running on 64bit Amazon Linux But my application still misses the timeout options. Any suggestion ?

#10

Try this one; it works (v0.36.4). Only change is the timeouts, but they’re set to 600.

You may not be putting the files back inside the zip file correctly? That caught me a couple of times. This command creates the metabase-aws-eb.zip file containing only the extensions dir & json file at the root:

zip -r metabase-aws-eb.zip .ebextensions Dockerrun.aws.json

One other thing that might be helpful: by default your load balancer timeout will be reset whenever the environment is rebuilt. Use something like the below command (for classic ELB) to persist the load balancer timeout in the environment config:

aws elasticbeanstalk update-environment --environment-name Metabase-prod --option-settings Namespace=aws:elb:policies,OptionName=ConnectionSettingIdleTimeout,Value=3600

1 Like
#11

Hey @tjm, I am using this image with proxy timeout set to 600s in metabase.sh, but still getting ‘Your question took too long’ error after 60s, with 504 nginx timeout in response headers. Not using load balancers so that is ruled out as well, and the platform is ‘Docker running on 64 bit Amazon Linux/2.xx’. Any suggestions on what could be going wrong, and if it is definitely nginx that is timing out ?
Thanks!

#12

Hm, I didn’t think it was possible to run Elastic Beanstalk without a load balancer…? If that’s the case, then I’m really not sure where it would be coming from.

You could also +1 the pull request so that one potential source of confusion is removed. :slight_smile:

Ugh, Discourse has barred from me posting links to Github (I must be a spammer trying to promote github). The PR is at /metabase/metabase/pull/13568

Thanks!

#13

Thanks so much for getting back.
EBS allows not using a load-balancer for single zone availability configuration only I think, I have Metabase running on a single instance anyway.
Thanks for the not-a-link, have +1d and will give it a go.

#15

+1 for making this more configurable above 10 min

#16

Hey, @flamber, I stumbled upon this discourse and I was wondering if there's now a way to setup the timeout (long query) for the latest Metabase version (0.43.4)? The problem with the workaround (to set timeouts in metabase-setup.sh) is that it doesn't work if the Docker platform is 64bit Amazon Linux 2 as tjm mentioned.

Any info would be very appreciated. Thank you!

#17

@fatin, the timeout should be always setup at the reverse proxy level (in the case of Elastic Beanstalk, that's set up for you already for 10 min)

#18

Hi Luiggi, noted. Unfortunately, for large queries, it's still timing out after 1 min. Even though I've already set the timeout attribute to 360 seconds for the Load Balancer. (EB timeout is 10 min as you've stated)

#19

@fatin Without seeing the stacktrace (Admin > Troubleshooting > Logs), then it's difficult to tell where the timeout is, but perhaps your database has a max execution timeout.