Errors when trying to decouple and update to 0.40

kd_thl · July 21, 2021, 2:53am

Hello. I am trying to decouple my RDS database and create a new EB environment to run 0.40, but I keep failing. I would really appreciate some help! Here is what I have done:

https://www.metabase.com/docs/latest/operations-guide/creating-RDS-database-on-AWS.html#decouple-your-rds-database-from-the-elastic-beanstalk-deployment Decoupled the database using this guide, but I did not terminate the old environment. I created a snapshot of my database, and then I "restored" the snapshot to create a new db in RDS. (The snapshot is from Metabase running 0.39.4.)
- I could not do this: Enter metabase as the Initial database name because the "initial database name" field was not there.
https://www.metabase.com/docs/latest/operations-guide/running-metabase-on-elastic-beanstalk.html Created a new EB environment with platform branch "Docker running on 64bit Amazon Linux 2" version 3.4.2 with the settings noted in the guide: health check path, the same VPC as the new RDS db with two zones for load balancer subnets and one for instance subnets, and load balancer from 4 to 1.
I edited the security group settings of the RDS database. When I edit "CIDR/IP - Inbound" security rules, I saw PostgreSQL but when I tried to remove the IP address and add the group ID of my new EB environment I got an error: You may not specify a referenced group id for an existing IPv4 CIDR rule. So instead I added a new rule with the group ID and left the other one with the IP address alone.
After the environment finished building, Metabase was accessible but runs me through the setup. So I tried to connect it to the RDS database I created from the snapshot. I added the following environment variables:
- MB_DB_DBNAME I tried "metabase", the DB identifier name in RDS, and leaving it out
- MB_DB_HOST with the endpoint shown on the RDS db
- MB_DB_PASS with the password
- MB_DB_USER with the username
- MB_DB_TYPE with value "postgres"
- MB_DB_PORT with value 5432

But I got the following errors when I tried to apply the configuration:

Time	Type	Details
2021-07-20 19:45:41 UTC-0700	ERROR	Failed to deploy configuration.
2021-07-20 19:45:41 UTC-0700	ERROR	Unsuccessful command execution on instance id(s) 'i-0d3202972532132e4'. Aborting the operation.
2021-07-20 19:45:41 UTC-0700	INFO	Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
2021-07-20 19:45:40 UTC-0700	ERROR	[Instance: i-0d3202972532132e4] Command failed on instance. Return code: 1 Output: Engine execution has encountered an error..
2021-07-20 19:45:40 UTC-0700	ERROR	Instance deployment failed to build the Docker image. The deployment failed.

2021/07/21 02:45:35.901913 [INFO] successfully execute command: docker pull metabase/metabase:v0.40.1
2021/07/21 02:45:35.901936 [INFO] successfully pulled docker image metabase/metabase:v0.40.1
2021/07/21 02:45:35.901941 [INFO] start to build docker image
2021/07/21 02:45:35.918658 [INFO] Running command /bin/sh -c docker build -t aws_beanstalk/staging-app /var/app/staging/
2021/07/21 02:45:37.958202 [INFO] Sending build context to Docker daemon 24.58kB
2021/07/21 02:45:37.960456 [WARN] failed to execute command: docker build -t aws_beanstalk/staging-app /var/app/staging/, retrying...
2021/07/21 02:45:37.965955 [INFO] Running command /bin/sh -c docker build -t aws_beanstalk/staging-app /var/app/staging/
2021/07/21 02:45:40.196729 [INFO] Sending build context to Docker daemon 24.58kB
2021/07/21 02:45:40.200741 [ERROR] An error occurred during execution of command [config-deploy] - [Docker Specific Build Application]. Stop running the command. Error: failed to build docker image: Command /bin/sh -c docker build -t aws_beanstalk/staging-app /var/app/staging/ failed with error exit status 1. Stderr:Error response from daemon: Error processing tar file(exit status 2): fatal error: runtime: out of memory
runtime stack:
runtime.throw(0x55a3a88c82cd, 0x16)
/usr/lib/golang/src/runtime/panic.go:1116 +0x74 fp=0x7ffed71f4140 sp=0x7ffed71f4110 pc=0x55a3a6ccfa74
runtime.sysMap(0xc000000000, 0x4000000, 0x55a3aa4bcdf8)
/usr/lib/golang/src/runtime/mem_linux.go:169 +0xc7 fp=0x7ffed71f4180 sp=0x7ffed71f4140 pc=0x55a3a6cb1347
runtime.(*mheap).sysAlloc(0x55a3aa4a0280, 0x400000, 0x0, 0x4)
/usr/lib/golang/src/runtime/malloc.go:727 +0x1d4 fp=0x7ffed71f4228 sp=0x7ffed71f4180 pc=0x55a3a6ca4af4
runtime.(*mheap).grow(0x55a3aa4a0280, 0x1, 0x0)
/usr/lib/golang/src/runtime/mheap.go:1344 +0x85 fp=0x7ffed71f4290 sp=0x7ffed71f4228 pc=0x55a3a6cc09a5
runtime.(*mheap).allocSpan(0x55a3aa4a0280, 0x1, 0x37312d7069002a00, 0x55a3aa4bce08, 0x2e3263652e393631)
/usr/lib/golang/src/runtime/mheap.go:1160 +0x6b6 fp=0x7ffed71f4310 sp=0x7ffed71f4290 pc=0x55a3a6cc0756
runtime.(*mheap).alloc.func1()
/usr/lib/golang/src/runtime/mheap.go:907 +0x66 fp=0x7ffed71f4368 sp=0x7ffed71f4310 pc=0x55a3a6cfece6
runtime.(*mheap).alloc(0x55a3aa4a0280, 0x1, 0x4012a, 0x2200000003)
/usr/lib/golang/src/runtime/mheap.go:901 +0x85 fp=0x7ffed71f43b8 sp=0x7ffed71f4368 pc=0x55a3a6cbfc25
runtime.(*mcentral).grow(0x55a3aa4b3138, 0x0)
/usr/lib/golang/src/runtime/mcentral.go:506 +0x7c fp=0x7ffed71f4400 sp=0x7ffed71f43b8 pc=0x55a3a6cb0d1c
runtime.(*mcentral).cacheSpan(0x55a3aa4b3138, 0x55a3a6cfcdfa)
/usr/lib/golang/src/runtime/mcentral.go:177 +0x3e5 fp=0x7ffed71f4478 sp=0x7ffed71f4400 pc=0x55a3a6cb0aa5
runtime.(*mcache).refill(0x7fbb625e2108, 0x2a)
/usr/lib/golang/src/runtime/mcache.go:142 +0xa5 fp=0x7ffed71f4498 sp=0x7ffed71f4478 pc=0x55a3a6cb0445
runtime.(*mcache).nextFree(0x7fbb625e2108, 0x55a3aa48622a, 0x7fbb625e2108, 0xfffffffffffffff8, 0x7ffed71f4528)
/usr/lib/golang/src/runtime/malloc.go:880 +0x8d fp=0x7ffed71f44d0 sp=0x7ffed71f4498 pc=0x55a3a6ca538d
runtime.mallocgc(0x180, 0x55a3a93c9060, 0x7ffed71f4501, 0x7ffed71f45d0)
/usr/lib/golang/src/runtime/malloc.go:1061 +0x854 fp=0x7ffed71f4570 sp=0x7ffed71f44d0 pc=0x55a3a6ca5d94
runtime.newobject(0x55a3a93c9060, 0x55a3a6cfd9e0)
/usr/lib/golang/src/runtime/malloc.go:1195 +0x3a fp=0x7ffed71f45a0 sp=0x7ffed71f4570 pc=0x55a3a6ca623a
runtime.malg(0x8000, 0x0)
/usr/lib/golang/src/runtime/proc.go:3520 +0x33 fp=0x7ffed71f45e0 sp=0x7ffed71f45a0 pc=0x55a3a6cda773
runtime.mpreinit(0x55a3aa4862c0)
/usr/lib/golang/src/runtime/os_linux.go:340 +0x2f fp=0x7ffed71f4600 sp=0x7ffed71f45e0 pc=0x55a3a6ccc5ef
runtime.mcommoninit(0x55a3aa4862c0, 0xffffffffffffffff)
/usr/lib/golang/src/runtime/proc.go:663 +0xfa fp=0x7ffed71f4648 sp=0x7ffed71f4600 pc=0x55a3a6cd38da
runtime.schedinit()
/usr/lib/golang/src/runtime/proc.go:565 +0xa5 fp=0x7ffed71f46a0 sp=0x7ffed71f4648 pc=0x55a3a6cd3465
runtime.rt0_go(0x7ffed71f47a8, 0x3, 0x7ffed71f47a8, 0x0, 0x7fbb617e30ba, 0x0, 0x7ffed71f47a8, 0x300000000, 0x55a3a6d07340, 0x0, ...)
/usr/lib/golang/src/runtime/asm_amd64.s:214 +0x129 fp=0x7ffed71f46a8 sp=0x7ffed71f46a0 pc=0x55a3a6d07489
2021/07/21 02:45:40.200761 [INFO] Executing cleanup logic
2021/07/21 02:45:40.210767 [INFO] CommandService Response: {"status":"FAILURE","api_version":"1.0","results":[{"status":"FAILURE","msg":"Engine execution has encountered an error.","returncode":1,"events":[{"msg":"Instance deployment failed to build the Docker image. The deployment failed.","timestamp":1626835540,"severity":"ERROR"},{"msg":"Instance deployment failed. For details, see 'eb-engine.log'.","timestamp":1626835540,"severity":"ERROR"}]}]}

Luiggi · July 22, 2021, 1:10am

2 things here:

did you use the instance type t3a-small? I'm asking since it's returning an out of memory which should imply lack of RAM
the MB_DB_DBNAME is a parameter you should have set up when you deployed Metabase for the first time on your previous Elastic Beanstalk deployment. You should be able to get that info from your current configuration by going to the instance and then to configuration...Database

kd_thl · July 22, 2021, 1:19am

Thanks, @Luiggi! I just tried again with the MB_DB_DBNAME I got from the config of the old db.

Originally I used t2.micro for EB with for RDS db.t3.small (our 0.39.4 db is db.t2.micro). I just switched EB to t3a-small and changed "Docker running on 64bit Amazon Linux 2/3.4.2" to "Docker running on 64bit Amazon Linux 2/3.4.3."

I'm not sure how it went because the health is severe and I'm getting 502 errors when I try to access the environment…

kd_thl · July 22, 2021, 2:50am

Thanks again, @Luiggi. I think I've got it running! Here were my issues:

I was using the wrong MB_DB_DBNAME.
Not enough memory. I switched to t2.small in my EB configuration.
Wrong RDS password. The password that worked with my original database didn't work with the new one that I created when restoring my snapshot, so I reset the password. (The logs showed this was causing the 502 errors.)

Now my issue is that when I try to access the URL I get a "timed out" error similar to this issue Metabase on AWS Beanstalk. I increased my EB environment size to t2.medium but that didn't help. The logs don't have any errors, so I'm puzzled.

I have a subdomain (CNAME) configured to point to Metabase. It seems like when I try to access the EB URL, it forwards me to the subdomain, which points to the EB URL, forming a loop. I tried setting MB_SITE_URL to the EB URL and got the Metabase login screen using the EB URL, but I couldn't log in because we use Google Auth and it's tied to the subdomain. I'm not sure what I need to do from here.

Luiggi · July 22, 2021, 12:39pm

Interesting, so you can't log in as email based login is disabled? you'll have to get inside the database and disable that feature from the settings table so you can log in again to tweak that parameter

kd_thl · July 22, 2021, 1:44pm

Well, now I can't even get to the Metabase login screen because when I go to the EB URL it forwards me to the subdomain, which points at the EB URL. So I'm in a loop. I'm wondering if I turn off the https redirect via an environment variable if it will work? I'l going to give that a try today.

Luiggi · July 22, 2021, 1:49pm

that's why I suggest you get into the RDS database and tweak the settings there, if possible. Let us know if you managed to solve that

kd_thl · July 22, 2021, 1:57pm

Which parameters do you suggest I try tweaking? I tried setting MB_REDIRECT_ALL_REQUESTS_TO_HTTPS to false but I'm still getting forwarded to the subdomain. I don't want to enable email-based login; I want it to work the same as the old version.

jins-agridigital · July 22, 2021, 2:18pm

Literally In the exact same position as you!

Doesn't seem to be able to detach properly and then get setup again.

jins-agridigital · July 22, 2021, 2:27pm

Interesting....when I go to the Metabase EB URL it loads to show the following...

Is this something to do with database migration or something?

While at the same time, the health is "severe"

Config:

Ive upgraded to t3.medium EC2 instance type
Used the environment variables to add the decoupled RDS instance

kd_thl · July 22, 2021, 2:32pm

Yes, that seems to happen after every update. Mine did that at the EB URL, and from then on the EB URL began forwarding me to the subdomain.

jins-agridigital · July 22, 2021, 2:51pm

Ive tried several times to rebuild from scratch and it just simply isn't working. I'm stumped.

I keep getting this error:

100.0 % of the requests are failing with HTTP 5xx.
Process default has been unhealthy for 2 minutes (Target.ResponseCodeMismatch).
Following services are not running: eb-docker-log.
100 % of CPU is in use.

Instance size: t2.medium
RDS: db.t2.small
Confirm that it restored from snapshot properly

kd_thl · July 22, 2021, 3:29pm

Check your logs to see if the RDS password is correct. The password that worked with my original database didn't work with the new one that I created when restoring my snapshot, so I reset the password and then it worked. (My logs showed incorrect password was causing the 502 errors.)

jins-agridigital · July 22, 2021, 3:35pm

Ok thanks! I'll give that a go now

jins-agridigital · July 22, 2021, 5:35pm

Im stuck where you are. I have the service "running" but when I go to the EB URL it redirects to the CNAME I setup for my previous one.

Is there a way we can upgrade to 0.40 without having to decouple the DB?

Luiggi · July 22, 2021, 9:30pm

Hi @kd_thl and @jins-agridigital, a quick idea: can you connect your CURRENT elastic beanstalk deployment (considering it's updated to the latest platform) to the coupled RDS with environment variables? that should work without decoupling

kd_thl · July 22, 2021, 9:47pm

Thanks, I'll give that a try. Though that means I'd need to keep the old environment running, right? Because terminating the environment would destroy the db.

Luiggi · July 22, 2021, 9:58pm

Exactly, and that's what we want to avoid, but if there are issues like the domain stuff (we had customers report that the change would imply major changes to scripts), connecting the current deployment to the current RDS via env vars should work, but as soon as someone deletes the deployment, everything is gone (if there was no snapshot taken). Just a thought

kd_thl · July 22, 2021, 9:58pm

Are there any settings in EB or Metabase that could account for this redirect issue? I've tried searching documentation and general searches, but I can't find any examples of this issue. I can easily point my CNAME at the old environment's EB url and I'm good to go.

Luiggi · July 22, 2021, 10:02pm

what do you have as the site url? https://www.metabase.com/docs/latest/operations-guide/environment-variables.html#mb_site_url

is it the CNAME or the URL of Elastic Beanstalk?