Errors when trying to decouple and update to 0.40

I've tried with both and not setting it in the environment variables.

When it's the subdomain/CNAME, the EB url redirects to the subdomain.
The first time (and only the first time) I set it to the EB url I saw Metabase, it showed me the typical post-update initialization screen, and from then on it has been redirecting me to the subdomain. That's how I know Metabase is running: I saw that initialization screen, and thereafter it knew where to redirect me (the subdomain).

do you have any load balancers in front of Metabase or you did any tweak to Nginx conf?

I followed the guide explicitly. So I set max load balancers to 1. Didn't tweak anything with Nginx.

Likewise. on everything you're experiencing.
I actually tried your suggestion Luiggi, however having ENV B (new) point to ENV A(old) RDS using env variables still did not work.

The same issue about the CNAME is what Im experiencing.

It is as if the CNAME / EB URL is not resetting - does this get stored in the DB? If yes, does that override the ENV VARs? because I've set the ENV VARs URL to the EB URL and it doesn't work.

1 Like

my suggestion was to use in the old environment (the one you have running with 39.1) environment variables to connect to the actual RDS that you have set up (the one that is coupled with the Elastic Beanstalk deployment), and on that deployment upgrade to 40.1, so you can keep the current URL of EB

Would the environment variables override the RDS instance inside?

In 40.1 yes, as this should be the correct behaviour (env vars should always prevail no matter other configurations). In < 40 no, but I haven't tested this as it was reported by a customer

I tried connecting my old EB environment (which uses a deprecated version of Docker) with my new db (because I can't risk breaking my old one) via environment variables (the same ones that "worked" with the new EB environment). It failed, and I got these errors:

2021-07-23T00:01:52.388Z] ERROR [13650] : Command execution failed: Activity failed. (ElasticBeanstalk::ActivityFatalError)
caused by: Loaded plugins: priorities, update-motd, upgrade-helper
No package amazon-cloudwatch-agent available.
Error: Nothing to do
(ElasticBeanstalk::ExternalInvocationError)

[2021-07-23T00:01:52.388Z] ERROR [13650] : Command CMD-AppDeploy failed!
[2021-07-23T00:01:52.388Z] INFO [13650] : Command processor returning results:
{"status":"FAILURE","api_version":"1.0","results":[{"status":"FAILURE","msg":"Loaded plugins: priorities, update-motd, upgrade-helper\nNo package amazon-cloudwatch-agent available.\nError: Nothing to do. \ncontainer_command 07-install_cloudwatch_agent in .ebextensions/01_metabase.config failed. For more detail, check /var/log/eb-activity.log using console or EB CLI","returncode":1,"events":[{"msg":"Successfully pulled metabase/metabase:v0.40.1","severity":"TRACE","timestamp":1626998496},{"msg":"Successfully built aws_beanstalk/staging-app","severity":"TRACE","timestamp":1626998500}]}]}

i am getting the same issue..
docker ps -a shows nothing running..
I was able to get one running in us-east-1 three days ago.. i am trying o get one working in ca-central-1 and i have this error.

edit: i stand corrected. not the same, not enough coffee yesterday...

try connecting your old Elastic Beanstalk deployment to RDS via environment variables but use this deployment file when upgrading https://github.com/metabase/metabase/files/6811226/metabase-aws-eb-deprecated-ElasticBeanstalk.zip istead of the one from the website (this does not try to install Cloudwatch from yum repo)

do you access the Elastic Beanstalk machine directly? please post the logs :slight_smile:

2021/07/22 20:30:06.845105 [INFO] Starting...

2021/07/22 20:30:06.845158 [INFO] Starting EBPlatform-PlatformEngine

2021/07/22 20:30:06.845224 [INFO] reading event message file

2021/07/22 20:30:06.845343 [INFO] no eb envtier info file found, skip loading env tier info.

2021/07/22 20:30:06.845777 [INFO] Engine received EB command userdata-exec

2021/07/22 20:30:06.970223 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:ca-central-1:855445117064:stack/awseb-e-2zduwrbmwm-stack/7847bae0-eb2b-11eb-9464-06bc95a06192 -r AWSEBAutoScalingGroup --region ca-central-1

2021/07/22 20:30:07.239043 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:ca-central-1:855445117064:stack/awseb-e-2zduwrbmwm-stack/7847bae0-eb2b-11eb-9464-06bc95a06192 -r AWSEBBeanstalkMetadata --region ca-central-1

2021/07/22 20:30:07.554228 [INFO] This is a workflow controlled instance.

2021/07/22 20:30:07.554334 [INFO] Engine command: (env-launch)

2021/07/22 20:30:07.554540 [INFO] Executing instruction: SyncClock

2021/07/22 20:30:07.554544 [INFO] Starting SyncClock

2021/07/22 20:30:07.554555 [INFO] Running command /bin/sh -c /usr/bin/chronyc tracking

2021/07/22 20:30:07.558693 [INFO] Reference ID : A9FEA97B (169.254.169.123)

Stratum : 4

Ref time (UTC) : Thu Jul 22 20:30:05 2021

System time : 0.000053712 seconds fast of NTP time

Last offset : -0.066565529 seconds

RMS offset : 0.066565529 seconds

Frequency : 5.729 ppm slow

Residual freq : -401.983 ppm

Skew : 0.664 ppm

Root delay : 0.017418470 seconds

Root dispersion : 0.002243736 seconds

Update interval : 0.0 seconds

Leap status : Normal

2021/07/22 20:30:07.558715 [INFO] Running command /bin/sh -c /usr/bin/chronyc -a makestep

2021/07/22 20:30:07.561175 [INFO] 200 OK

2021/07/22 20:30:07.561602 [INFO] Skipping Configure OS

2021/07/22 20:30:07.561609 [INFO] Skipping LockGUID

2021/07/22 20:30:07.561614 [INFO] Skipping GenerateEBBanner

2021/07/22 20:30:07.561619 [INFO] Skipping Install Process Manager

2021/07/22 20:30:07.561622 [INFO] Skipping install proxy

2021/07/22 20:30:07.561627 [INFO] Skipping installhealthd

2021/07/22 20:30:07.561631 [INFO] Skipping Install Log Streaming Manager

2021/07/22 20:30:07.561635 [INFO] Skipping install X-Ray

2021/07/22 20:30:07.561639 [INFO] Skipping install Third Party License

2021/07/22 20:30:07.561645 [INFO] Instance has NOT been bootstrapped

2021/07/22 20:30:07.561647 [INFO] Executing instruction: installSqsd

2021/07/22 20:30:07.561650 [INFO] This is a web server environment instance, skip install sqsd daemon ...

2021/07/22 20:30:07.561654 [INFO] Instance has NOT been bootstrapped

2021/07/22 20:30:07.561656 [INFO] Executing instruction: bootstraphealthd

2021/07/22 20:30:07.561660 [INFO] this is an enhanced health env ...

2021/07/22 20:30:07.561669 [INFO] Running command /bin/sh -c id -u apache

2021/07/22 20:30:07.563618 [INFO] bootstrap healthd....

2021/07/22 20:30:07.563633 [INFO] Running command /bin/sh -c /usr/bin/id -u healthd || /usr/sbin/useradd --user-group healthd -s /sbin/nologin --create-home

2021/07/22 20:30:07.596992 [INFO] Skipping Install yum packages

2021/07/22 20:30:07.597003 [INFO] Skipping install docker-compose

2021/07/22 20:30:07.597007 [INFO] Skipping MarkBaked

2021/07/22 20:30:07.597013 [INFO] Instance has NOT been bootstrapped

2021/07/22 20:30:07.597015 [INFO] Executing instruction: TuneSystemSettings

2021/07/22 20:30:07.597018 [INFO] Starting TuneSystemSettings

2021/07/22 20:30:07.597021 [INFO] Instance has NOT been bootstrapped

2021/07/22 20:30:07.597677 [INFO] Executing instruction: GetSetupLogRotate

2021/07/22 20:30:07.597683 [INFO] Initialize LogRotate files and directories

2021/07/22 20:30:07.609843 [INFO] Instance has NOT been bootstrapped

2021/07/22 20:30:07.609852 [INFO] Executing instruction: BootstrapCFNHup

2021/07/22 20:30:07.609855 [INFO] Bootstrap cfn-hup

2021/07/22 20:30:07.613758 [INFO] Copying file /opt/elasticbeanstalk/config/private/aws-eb-command-handler.conf to /etc/cfn/hooks.d/aws-eb-command-handler.conf

2021/07/22 20:30:07.615255 [INFO] Executing instruction: StartCFNHup

2021/07/22 20:30:07.615265 [INFO] Start cfn-hup

2021/07/22 20:30:07.615279 [INFO] Running command /bin/sh -c systemctl is-active cfn-hup.service

2021/07/22 20:30:07.618808 [INFO] Running command /bin/sh -c systemctl is-active cfn-hup.service

2021/07/22 20:30:07.622021 [INFO] Running command /bin/sh -c systemctl show -p PartOf cfn-hup.service

2021/07/22 20:30:07.629442 [INFO] Running command /bin/sh -c systemctl daemon-reload

2021/07/22 20:30:07.680324 [INFO] Running command /bin/sh -c systemctl reset-failed

2021/07/22 20:30:07.684548 [INFO] Running command /bin/sh -c systemctl is-enabled aws-eb.target

2021/07/22 20:30:07.688957 [INFO] Running command /bin/sh -c systemctl enable aws-eb.target

2021/07/22 20:30:07.745663 [INFO] Running command /bin/sh -c systemctl start aws-eb.target

2021/07/22 20:30:07.751368 [INFO] Running command /bin/sh -c systemctl enable cfn-hup.service

2021/07/22 20:30:07.917705 [INFO] Running command /bin/sh -c systemctl show -p PartOf cfn-hup.service

2021/07/22 20:30:07.922290 [INFO] Running command /bin/sh -c systemctl is-active cfn-hup.service

2021/07/22 20:30:07.925587 [INFO] Running command /bin/sh -c systemctl start cfn-hup.service

2021/07/22 20:30:07.963041 [INFO] Instance has NOT been bootstrapped

2021/07/22 20:30:07.963057 [INFO] Executing instruction: SetupPublishLogCronjob

2021/07/22 20:30:07.963061 [INFO] Setup publish logs cron job...

2021/07/22 20:30:07.963066 [INFO] Copying file /opt/elasticbeanstalk/config/private/logtasks/cron/publishlogs to /etc/cron.d/publishlogs

2021/07/22 20:30:07.964829 [INFO] Executing instruction: Start Docker

2021/07/22 20:30:07.964847 [INFO] Starting Docker...

2021/07/22 20:30:07.964901 [INFO] Copying file /opt/elasticbeanstalk/config/private/docker-log.conf to /etc/systemd/system/docker.service.d/docker-log.conf

2021/07/22 20:30:07.969095 [INFO] Copying file /opt/elasticbeanstalk/config/private/docker-rsyslog.conf to /etc/rsyslog.d/docker-log.conf

2021/07/22 20:30:07.970697 [INFO] Running command /bin/sh -c systemctl restart rsyslog.service

2021/07/22 20:30:07.994884 [INFO] Running command /bin/sh -c systemctl show -p PartOf docker.service

2021/07/22 20:30:08.011468 [INFO] Running command /bin/sh -c systemctl daemon-reload

2021/07/22 20:30:08.121221 [INFO] Running command /bin/sh -c systemctl reset-failed

2021/07/22 20:30:08.127637 [INFO] Running command /bin/sh -c systemctl show -p PartOf docker.service

2021/07/22 20:30:08.135798 [INFO] Running command /bin/sh -c systemctl is-active docker.service

2021/07/22 20:30:08.140740 [INFO] Running command /bin/sh -c systemctl start docker.service

2021/07/22 20:30:09.476690 [INFO] Instance has NOT been bootstrapped

2021/07/22 20:30:09.476706 [INFO] Executing instruction: Setup Docker Options

2021/07/22 20:30:09.476710 [INFO] Setting up configurations for docker platform

2021/07/22 20:30:09.486615 [INFO] Running command /bin/sh -c systemctl show -p PartOf eb-docker-events.service

2021/07/22 20:30:09.494108 [INFO] Running command /bin/sh -c systemctl daemon-reload

2021/07/22 20:30:09.558169 [INFO] Running command /bin/sh -c systemctl reset-failed

2021/07/22 20:30:09.562407 [INFO] Running command /bin/sh -c systemctl enable eb-docker-events.service

2021/07/22 20:30:09.647785 [INFO] Running command /bin/sh -c systemctl show -p PartOf eb-docker-events.service

2021/07/22 20:30:09.656738 [INFO] Running command /bin/sh -c systemctl is-active eb-docker-events.service

2021/07/22 20:30:09.660539 [INFO] Running command /bin/sh -c systemctl start eb-docker-events.service

2021/07/22 20:30:09.790402 [INFO] Instance has NOT been bootstrapped

2021/07/22 20:30:09.790420 [INFO] Executing instruction: MarkBootstrapped

2021/07/22 20:30:09.790423 [INFO] Starting MarkBootstrapped

2021/07/22 20:30:09.790428 [INFO] Instance has NOT been bootstrapped

2021/07/22 20:30:09.791272 [INFO] Marked instance as Bootstrapped

2021/07/22 20:30:09.791281 [INFO] Executing instruction: Save CFN Stack Info

2021/07/22 20:30:09.791326 [INFO] Starting SwitchCFNStack

2021/07/22 20:30:09.791329 [INFO] Executing cleanup logic

2021/07/22 20:30:09.791337 [INFO] Platform Engine finished execution on command: env-launch

2021/07/22 20:30:30.883498 [INFO] Starting...

2021/07/22 20:30:30.883545 [INFO] Starting EBPlatform-PlatformEngine

2021/07/22 20:30:30.883562 [INFO] reading event message file

2021/07/22 20:30:30.883656 [INFO] no eb envtier info file found, skip loading env tier info.

2021/07/22 20:30:30.883728 [INFO] Engine received EB command cfn-hup-exec

2021/07/22 20:30:30.964914 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:ca-central-1:855445117064:stack/awseb-e-2zduwrbmwm-stack/7847bae0-eb2b-11eb-9464-06bc95a06192 -r AWSEBAutoScalingGroup --region ca-central-1

2021/07/22 20:30:31.334359 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:ca-central-1:855445117064:stack/awseb-e-2zduwrbmwm-stack/7847bae0-eb2b-11eb-9464-06bc95a06192 -r AWSEBBeanstalkMetadata --region ca-central-1

2021/07/22 20:30:31.643091 [INFO] checking whether command app-deploy is applicable to this instance...

2021/07/22 20:30:31.643102 [INFO] this command is applicable to the instance, thus instance should execute command

2021/07/22 20:30:31.643104 [INFO] Engine command: (app-deploy)

2021/07/22 20:30:31.643109 [INFO] Downloading EB Application...

2021/07/22 20:30:31.643112 [INFO] Region: ca-central-1

2021/07/22 20:30:31.643115 [INFO] envID: e-2zduwrbmwm

2021/07/22 20:30:31.643118 [INFO] envBucket: elasticbeanstalk-ca-central-1-855445117064

2021/07/22 20:30:31.643122 [INFO] Using manifest file name from command request

2021/07/22 20:30:31.643128 [INFO] Unable to get version manifest file.

2021/07/22 20:30:31.643131 [INFO] Downloading latest manifest available.

2021/07/22 20:30:31.643134 [INFO] Download latest app version manifest

2021/07/22 20:30:31.643206 [INFO] resources/environments/e-2zduwrbmwm/_runtime/versions/manifest

2021/07/22 20:30:31.681799 [INFO] latestManifest key *: resources/environments/e-2zduwrbmwm/_runtime/versions/manifest_1626985749352

2021/07/22 20:30:31.681957 [INFO] Downloading: bucket: elasticbeanstalk-ca-central-1-855445117064, object: /resources/environments/e-2zduwrbmwm/_runtime/versions/manifest_1626985749352

2021/07/22 20:30:31.702185 [INFO] Download successful120bytes downloaded

2021/07/22 20:30:31.702267 [INFO] Trying to read and parse version manifest...

2021/07/22 20:30:31.702326 [INFO] Downloading: bucket: elasticbeanstalk-ca-central-1-855445117064, object: /resources/environments/e-2zduwrbmwm/_runtime/_versions/docupet-mb-ca-stage/v0.40.1

2021/07/22 20:30:31.717128 [INFO] Download successful7072bytes downloaded

2021/07/22 20:30:31.717368 [INFO] Executing instruction: ElectLeader

2021/07/22 20:30:31.717374 [INFO] Running leader election for instance i-0bebb779c582476a0...

2021/07/22 20:30:31.717377 [INFO] Calling the cfn-elect-cmd-leader to elect the command leader.

2021/07/22 20:30:31.717391 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-elect-cmd-leader --stack arn:aws:cloudformation:ca-central-1:855445117064:stack/awseb-e-2zduwrbmwm-stack/7847bae0-eb2b-11eb-9464-06bc95a06192 --command-name ElasticBeanstalkCommand-AWSEBAutoScalingGroup --invocation-id d4edfd5e-0634-456b-a9b5-3d2ef0926c67 --listener-id i-0bebb779c582476a0 --region ca-central-1

2021/07/22 20:30:32.042563 [INFO] Instance is Leader.

2021/07/22 20:30:32.042607 [INFO] Executing instruction: stopSqsd

2021/07/22 20:30:32.042613 [INFO] This is a web server environment instance, skip stop sqsd daemon ...

2021/07/22 20:30:32.042617 [INFO] Executing instruction: PreBuildEbExtension

2021/07/22 20:30:32.042621 [INFO] Starting executing the config set Infra-EmbeddedPreBuild.

2021/07/22 20:30:32.042634 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-init -s arn:aws:cloudformation:ca-central-1:855445117064:stack/awseb-e-2zduwrbmwm-stack/7847bae0-eb2b-11eb-9464-06bc95a06192 -r AWSEBAutoScalingGroup --region ca-central-1 --configsets Infra-EmbeddedPreBuild

2021/07/22 20:30:32.389685 [INFO] Finished executing the config set Infra-EmbeddedPreBuild.

2021/07/22 20:30:32.389705 [INFO] Executing instruction: StagedockerApplication

2021/07/22 20:30:32.389727 [INFO] Running command /bin/sh -c docker ps -a -q -f status=running

2021/07/22 20:30:32.425016 [INFO] Running command /bin/sh -c /bin/file -m /opt/elasticbeanstalk/config/private/beanstalk-magic -b --mime-type /opt/elasticbeanstalk/deployment/app_source_bundle

2021/07/22 20:30:32.428666 [INFO] application/zip

2021/07/22 20:30:32.428681 [INFO] app source bundle is zip file ...

2021/07/22 20:30:32.428685 [INFO] extracting /opt/elasticbeanstalk/deployment/app_source_bundle to /var/app/staging/

2021/07/22 20:30:32.428696 [INFO] Running command /bin/sh -c /usr/bin/unzip -q -o /opt/elasticbeanstalk/deployment/app_source_bundle -d /var/app/staging/

2021/07/22 20:30:32.431335 [INFO] finished extracting /opt/elasticbeanstalk/deployment/app_source_bundle to /var/app/staging/ successfully

2021/07/22 20:30:32.431348 [INFO] Executing instruction: RunAppDeployPreBuildHooks

2021/07/22 20:30:32.431365 [INFO] The dir .platform/hooks/prebuild/ does not exist in the application. Skipping this step...

2021/07/22 20:30:32.431367 [INFO] Executing instruction: Generate environment file

2021/07/22 20:30:32.431423 [INFO] Executing instruction: Docker Specific Build Application

2021/07/22 20:30:32.431426 [INFO] start build docker app

2021/07/22 20:30:32.431556 [INFO] generate dockerfile with the image and ports from Dockerrun.aws.json

2021/07/22 20:30:32.431559 [INFO] fetching image and ports

2021/07/22 20:30:32.433235 [INFO] fetch image name

2021/07/22 20:30:32.433262 [INFO] pull docker image if update is not false in Dockerrun.aws.json

2021/07/22 20:30:32.433275 [INFO] Running command /bin/sh -c docker pull metabase/metabase:v0.40.1

2021/07/22 20:30:40.513485 [INFO] v0.40.1: Pulling from metabase/metabase

5843afab3874: Pulling fs layer

2005a1987a07: Pulling fs layer

9acc5d8c95e7: Pulling fs layer

5b46e08f6e91: Pulling fs layer

2ab3e75bc5bc: Pulling fs layer

5b46e08f6e91: Waiting

2ab3e75bc5bc: Waiting

2005a1987a07: Verifying Checksum

2005a1987a07: Download complete

5843afab3874: Verifying Checksum

5843afab3874: Download complete

9acc5d8c95e7: Verifying Checksum

9acc5d8c95e7: Download complete

5b46e08f6e91: Verifying Checksum

5b46e08f6e91: Download complete

5843afab3874: Pull complete

2005a1987a07: Pull complete

2ab3e75bc5bc: Verifying Checksum

2ab3e75bc5bc: Download complete

9acc5d8c95e7: Pull complete

5b46e08f6e91: Pull complete

2ab3e75bc5bc: Pull complete

Digest: sha256:bbe0c9b2aa8fbec2cd3140158ea6e83e47a37937d35aaafa57e1587790e46669

Status: Downloaded newer image for metabase/metabase:v0.40.1

docker.io/metabase/metabase:v0.40.1

2021/07/22 20:30:40.513506 [INFO] successfully execute command: docker pull metabase/metabase:v0.40.1

2021/07/22 20:30:40.513510 [INFO] successfully pulled docker image metabase/metabase:v0.40.1

2021/07/22 20:30:40.513513 [INFO] start to build docker image

2021/07/22 20:30:40.513529 [INFO] Running command /bin/sh -c docker build -t aws_beanstalk/staging-app /var/app/staging/

2021/07/22 20:30:41.061916 [INFO] Sending build context to Docker daemon 24.58kB

Step 1/2 : FROM metabase/metabase:v0.40.1

---> 18d6b793cf0c

Step 2/2 : EXPOSE 3000

---> Running in 7fc9ffbf7c8f

Removing intermediate container 7fc9ffbf7c8f

---> d8efe4b89ea8

Successfully built d8efe4b89ea8

Successfully tagged aws_beanstalk/staging-app:latest

2021/07/22 20:30:41.061936 [INFO] successfully execute command: docker build -t aws_beanstalk/staging-app /var/app/staging/

2021/07/22 20:30:41.061941 [INFO] successfully build docker image metabase/metabase:v0.40.1

2021/07/22 20:30:41.061945 [INFO] save staging docker image id

2021/07/22 20:30:41.061957 [INFO] Running command /bin/sh -c docker images aws_beanstalk/staging-app -q

2021/07/22 20:30:41.100187 [INFO] d8efe4b89ea8

2021/07/22 20:30:41.100272 [INFO] Executing instruction: configure X-Ray

2021/07/22 20:30:41.100276 [INFO] X-Ray is not enabled.

2021/07/22 20:30:41.100280 [INFO] Executing instruction: configure proxy Nginx

2021/07/22 20:30:41.105119 [INFO] Executing instruction: configure healthd specific proxy conf

2021/07/22 20:30:41.106550 [INFO] Running command /bin/sh -c systemctl show -p PartOf healthd.service

2021/07/22 20:30:41.112346 [INFO] Running command /bin/sh -c systemctl daemon-reload

2021/07/22 20:30:41.180979 [INFO] Running command /bin/sh -c systemctl reset-failed

2021/07/22 20:30:41.184960 [INFO] Running command /bin/sh -c systemctl is-enabled aws-eb.target

2021/07/22 20:30:41.188522 [INFO] Running command /bin/sh -c systemctl enable aws-eb.target

2021/07/22 20:30:41.262552 [INFO] Running command /bin/sh -c systemctl start aws-eb.target

2021/07/22 20:30:41.266452 [INFO] Running command /bin/sh -c systemctl enable healthd.service

2021/07/22 20:30:41.344887 [INFO] Running command /bin/sh -c systemctl show -p PartOf healthd.service

2021/07/22 20:30:41.349415 [INFO] Running command /bin/sh -c systemctl is-active healthd.service

2021/07/22 20:30:41.352707 [INFO] Running command /bin/sh -c systemctl start healthd.service

2021/07/22 20:30:41.389147 [INFO] Copying file /opt/elasticbeanstalk/config/private/healthd/healthd_logformat.conf to /var/proxy/staging/nginx/conf.d/healthd_logformat.conf

2021/07/22 20:30:41.390792 [INFO] Copying file /opt/elasticbeanstalk/config/private/healthd/healthd_nginx.conf to /var/proxy/staging/nginx/conf.d/elasticbeanstalk/healthd.conf

2021/07/22 20:30:41.392284 [INFO] Executing instruction: configure log streaming

2021/07/22 20:30:41.392293 [INFO] log streaming is not enabled

2021/07/22 20:30:41.392296 [INFO] disable log stream

2021/07/22 20:30:41.392315 [INFO] Running command /bin/sh -c systemctl show -p PartOf amazon-cloudwatch-agent.service

2021/07/22 20:30:41.399774 [INFO] Running command /bin/sh -c systemctl stop amazon-cloudwatch-agent.service

2021/07/22 20:30:41.404678 [INFO] Executing instruction: GetToggleForceRotate

2021/07/22 20:30:41.404699 [INFO] Checking if logs need forced rotation

2021/07/22 20:30:41.404723 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:ca-central-1:855445117064:stack/awseb-e-2zduwrbmwm-stack/7847bae0-eb2b-11eb-9464-06bc95a06192 -r AWSEBAutoScalingGroup --region ca-central-1

2021/07/22 20:30:41.753580 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:ca-central-1:855445117064:stack/awseb-e-2zduwrbmwm-stack/7847bae0-eb2b-11eb-9464-06bc95a06192 -r AWSEBBeanstalkMetadata --region ca-central-1

2021/07/22 20:30:42.108813 [INFO] Procfile not found. Generating default rsyslog config

2021/07/22 20:30:42.110412 [INFO] Running command /bin/sh -c systemctl restart rsyslog.service

2021/07/22 20:30:42.141609 [INFO] Executing instruction: Configure Docker Logging

2021/07/22 20:30:42.157330 [INFO] Executing instruction: PostBuildEbExtension

2021/07/22 20:30:42.157354 [INFO] Starting executing the config set Infra-EmbeddedPostBuild.

2021/07/22 20:30:42.157370 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-init -s arn:aws:cloudformation:ca-central-1:855445117064:stack/awseb-e-2zduwrbmwm-stack/7847bae0-eb2b-11eb-9464-06bc95a06192 -r AWSEBAutoScalingGroup --region ca-central-1 --configsets Infra-EmbeddedPostBuild

2021/07/22 20:30:42.675371 [ERROR] An error occurred during execution of command [app-deploy] - [PostBuildEbExtension]. Stop running the command. Error: Container commands build failed. Please refer to /var/log/cfn-init.log for more details.

2021/07/22 20:30:42.675391 [INFO] Executing cleanup logic

2021/07/22 20:30:42.675477 [INFO] CommandService Response: {"status":"FAILURE","api_version":"1.0","results":[{"status":"FAILURE","msg":"Engine execution has encountered an error.","returncode":1,"events":[{"msg":"Instance deployment failed. For details, see 'eb-engine.log'.","timestamp":1626985842,"severity":"ERROR"}]}]}

2021/07/22 20:30:42.675651 [INFO] Platform Engine finished execution on command: app-deploy

[cfn-init.log]
2021-07-22 20:30:04,674 [INFO] -----------------------Starting build-----------------------
2021-07-22 20:30:04,680 [INFO] Running configSets: _OnInstanceBoot
2021-07-22 20:30:04,683 [INFO] Running configSet _OnInstanceBoot
2021-07-22 20:30:04,685 [INFO] Running config AWSEBBaseConfig
2021-07-22 20:30:04,820 [INFO] Command clearbackupfiles succeeded
2021-07-22 20:30:04,828 [INFO] Running config AWSEBCfnHupEndpointOverride
2021-07-22 20:30:04,841 [INFO] Command clearbackupfiles succeeded
2021-07-22 20:30:04,842 [INFO] ConfigSets completed
2021-07-22 20:30:04,843 [INFO] -----------------------Build complete-----------------------
2021-07-22 20:30:32,326 [INFO] -----------------------Starting build-----------------------
2021-07-22 20:30:32,331 [INFO] Running configSets: Infra-EmbeddedPreBuild
2021-07-22 20:30:32,334 [INFO] Running configSet Infra-EmbeddedPreBuild
2021-07-22 20:30:32,336 [INFO] Running config prebuild_0_docupet_mb_ca_stage
2021-07-22 20:30:32,366 [INFO] ConfigSets completed
2021-07-22 20:30:32,366 [INFO] -----------------------Build complete-----------------------
2021-07-22 20:30:42,422 [INFO] -----------------------Starting build-----------------------
2021-07-22 20:30:42,428 [INFO] Running configSets: Infra-EmbeddedPostBuild
2021-07-22 20:30:42,430 [INFO] Running configSet Infra-EmbeddedPostBuild
2021-07-22 20:30:42,432 [INFO] Running config postbuild_0_docupet_mb_ca_stage
2021-07-22 20:30:42,450 [INFO] Command 01_server_https succeeded
2021-07-22 20:30:42,464 [INFO] Command 02_log_x_real_ip succeeded
2021-07-22 20:30:42,474 [INFO] Test failed with code 1
2021-07-22 20:30:42,484 [INFO] Test failed with code 1
2021-07-22 20:30:42,579 [INFO] Command 05_try_nginx succeeded
2021-07-22 20:30:42,646 [ERROR] Command 06-install_inspector (bash -x install -u false) failed
2021-07-22 20:30:42,646 [ERROR] Error encountered during build of postbuild_0_docupet_mb_ca_stage: Command 06-install_inspector failed
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 573, in run_config
CloudFormationCarpenter(config, self._auth_config).build(worklog)
File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 273, in build
self._config.commands)
File "/usr/lib/python3.7/site-packages/cfnbootstrap/command_tool.py", line 127, in apply
raise ToolError(u"Command %s failed" % name)
cfnbootstrap.construction_errors.ToolError: Command 06-install_inspector failed
2021-07-22 20:30:42,648 [ERROR] -----------------------BUILD FAILED!------------------------
2021-07-22 20:30:42,648 [ERROR] Unhandled exception during build: Command 06-install_inspector failed
Traceback (most recent call last):
File "/opt/aws/bin/cfn-init", line 176, in
worklog.build(metadata, configSets)
File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 135, in build
Contractor(metadata).build(configSets, self)
File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 561, in build
self.run_config(config, worklog)
File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 573, in run_config
CloudFormationCarpenter(config, self._auth_config).build(worklog)
File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 273, in build
self._config.commands)
File "/usr/lib/python3.7/site-packages/cfnbootstrap/command_tool.py", line 127, in apply
raise ToolError(u"Command %s failed" % name)
cfnbootstrap.construction_errors.ToolError: Command 06-install_inspector failed

ok your error is another one: AWS Inspector is not available in our region, try configuring the environment variables and then upgrading with https://github.com/metabase/metabase/files/6869156/metabase-aws-eb-without-inspector-ElasticBeanstalk.zip

Thanks, @Luiggi, but my old EB environment is using the deprecated version of Docker, so I need to update, and I don't want to incur the costs of leaving the old environment running alongside the new one.

With all these issues people are encountering, are there any plans for an updated version of Metabase to make updating easier (and maybe patch this redirect issue, which seems common with people who are decoupling), or more comprehensive troubleshooting documentation, or something that will help us resolve these issues? I'm thinking I may hold off on updating to 0.40.x until 0.41 is out…

Ok thanks, This worked for me.. I was able to get my metabase up and running in CA-Central, i didnt realize there was no AWS Inspector.. thanks!

We plan to fix the issues that aroused in this upgrade as we identified 3 right now:

  • lack of AWS inspector in some regions (will be fixed by disabling this feature completely)
  • AWS Cloudwatch not being able to be installed from yum repos as deprecated Elastic Beanstalk deployments don't use these (will be fixed by pulling from an s3 bucket that AWS publishes)
  • Metabase does not take anymore the RDS parameters unless you pass them through environment variables (this one is something that we are not sure to fix, as we can't be overriding the environment variables depending on the platform; we should leave this to be configurable by users as it is right now)

We will try hard to provide a "simple" experience for Elastic Beanstalk users, but we can't be solving issues for deprecated platforms, you have to upgrade ASAP or AWS will do it forcefully at some point in time.

None of my issues have been because of the deprecated platform except for when you suggested I try using it as an experiment. I did try it, and I was able to update to 0.40.0 on the deprecated platform branch thanks to the modified file you provided that does not try to install Cloudwatch from yum repo. I then created a new snapshot of my RDS db, restored it, and connected it to the EB environment running on the new platform branch. Same issue: the EB url redirects to the subdomain, which points to the EB url.

Can anything be done about this redirect issue?

I'm reading again the whole thread to understand the issue better, so:

  • If you point the DNS (CNAME) subdomain to the old Elastic Beanstalk URL it works without issue (your old Metabase instance shows the login page)
  • If you point the DNS subdomain to the new Elastic Beanstalk URL it creates a redirect loop

Right now you have the CNAME pointing to the old instance, what happens if you go to the URL of the the new instance in an incognito session?

By the way, what do you have configured in the Authorized javascript origins and authorized redirect uri's in the Google SSO config? https://www.metabase.com/docs/latest/administration-guide/10-single-sign-on.html#enabling-google-sign-in

The new instance forwards me to the subdomain/old instance.

"Authorized JavaScript origins" is the subdomain. "Authorized redirect URIs" is empty.

If on the Site URL (Settings -> Admin -> General-> Site URL) you have the CNAME, then you need to change the DNS to the new Elastic Beanstalk URL and wait till the DNS cache expires (or wait till it propagates). Have you tried this?

If on the Site URL you have the old elastic beanstalk URL, you need to change it to the new one (do a backup first so you can revert this change!)