Metabase Recently Failing on Elastic Beanstalk

Get Help
1

Hey There,

Recently, our Elastic Beanstalk has been a bit unstable. By unstable, I mean it frequently has been going into a warning / severe status. Up until today, I was ignoring the alarms as at least one instance was still working and reachable.

Today, however, metabase has been unreachable and I can’t seem to get it healthy after multiple restarts and upgrading docker. If I look in the log files, this is what I see:

2019/02/26 16:25:54 [warn] 4000#0: duplicate MIME type "text/html" in /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf:11
2019/02/26 16:25:54 [emerg] 4000#0: host not found in upstream "docker" in /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf:23
2019/02/26 16:26:03 [warn] 4323#0: duplicate MIME type "text/html" in /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf:11
2019/02/26 16:26:03 [warn] 4329#0: duplicate MIME type "text/html" in /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf:11
2019/02/26 16:26:05 [warn] 4487#0: duplicate MIME type "text/html" in /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf:11
2019/02/26 16:26:06 [warn] 4502#0: duplicate MIME type "text/html" in /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf:11

Any help would be appreciated!

2

Hi @pjatx
It looks like you’re missing the upstream docker { server 127.0.0.1:3000 } section, which line 23 is referring to. It should be located in one of the Nginx configuration files in /etc/nginx/
Which version of Metabase? Did you manually change the Nginx configuration?

3

Thanks @flamber

Ah got it, I'll dig into that. We're running version v0.31.2.metabase-aws-eb.

I did add an environment property in EB > Configuration > Software to force SSL. Could that be the culprit?

image
image1035×317 30.3 KB

4

@pjatx

I can see the configuration uses different sections depending on if NGINX_FORCE_SSL is enabled or not - but both sections are almost identically.
Reference: https://github.com/metabase/metabase/blob/release-0.31.2/bin/aws-eb-docker/.ebextensions/metabase_config/metabase-setup.sh

But did you find the upstream section in your configuration? Or did you rename your docker or domain after creating it?
I don’t use AWS, so don’t know how EB works.

5

@flamber

So I finally got a chance to look at the config file and don’t see that section. Here’s what mine looks like:

	# Elastic Beanstalk Nginx Configuration File

	user  nginx;
	worker_processes  auto;

	error_log  /var/log/nginx/error.log;

	pid        /var/run/nginx.pid;

	events {
	    worker_connections  1024;
	}

	http {
	    include	  /etc/nginx/mime.types;
	    default_type  application/octet-stream;

	    access_log    /var/log/nginx/access.log;

	    log_format  healthd '$msec"$uri"$status"$request_time"$upstream_response_time"$http_x_forwarded_for';

	    include	  /etc/nginx/conf.d/*.conf;
	    include	  /etc/nginx/sites-enabled/*;
	}
6

@pjatx
There’s multiple files and sub-directories inside /etc/nginx/ - try running this to find it in all files:

grep -r "upstream" /etc/nginx/
7

@flamber

OK, so I think I figured out what was going on. The real issue, was the healthchecks were expecting 200’s but getting 301’s because / redirects to /auth by default. After rebuilding the environment and working with Amazon support, we fixed that problem.

The error message I think was a red herring or a premature check that only failed once then resolved itself. The upstream docker section IS within one of the nginx conf files (nested).

So now, I’ve rebuilt the environment which worked. I then had a clean instance of Metabase, but need to reimport all the settings, questions, etc. I then restored the database from a snapshot.

Now, I think the app is having trouble talking to the database since it’s probably a different host. Do you know what setting I need to change in the database or elsewhere to make Metabase talk to the new DB?

1 Like
8

Alright, anyone who finds this and is in a similar situation.

If you want to swap or restore a metabase app database in elastic beanstalk over a clean install (there may be a better way).

  1. Restore the active EB RDS database with the following command
    aws elasticbeanstalk update-environment --environment-id {your_environment-id} --option-settings Namespace=aws:rds:dbinstance,OptionName=DBSnapshotIdentifier,Value={your_rds-db-name}
  2. The environment will rebuild. At this point Metabase won’t be reachable (you’ll get 504 responses)
  3. SSH into the metabase EC2 instance
  4. Switch to root sudo -s
  5. See which docker container is running docker ps -a
  6. Grab your LONG container ID by typing docker inspect {CONTAINER ID} Paste this somewhere you can reference. It should be the first key: value when you invoke the inspect command
  7. Stop the docker service service docker stop
  8. Find the docker container config file cd /var/lib/docker/containers/{container-name}/
  9. Open up config.v2.json with nano, VIM, or your editor of choice nano config.v2.json
  10. Find the RDS_HOSTNAME key:value in the setting file
  11. Replace the current value with the NEW RDS endpoint
  12. Save the file
  13. Restart docker service docker start
  14. Done. Now the app should be connected to the restored snapshot and behaving normally.

I’m sure there’s a better way to do this, but this worked.

1 Like