Somehow I got my ElasticBeanStalk environment wedged after it was working for a year or so.
At this point I always get the Unable to connect to Metabase postgres DB error when the app is trying to start, so the EB always fails on any attempt to change a config and the Web UI won’t come up (503 error). I think the issue might have been that I asked EB to reboot the app instance, but not sure what caused it to get in this state though. I had done an upgrade from v0.27 to the latest v0.29.3 and had changed the instance type. But not sure what actually got it into this state.
Unfortunately the AWS SSH key is not set on the app instance, so I can’t ssh in to inspect things. I did confirm from another instance in the same VPC/Security Group can connect to the DB using the same credentials as is set in the EB Environment configuration. And now EB will never complete a change in config (adding the AWS SSH Key Pair) since the App never starts because of the DB error.
The system was using an RDS Postgres DB for its App DB.
Is there anyway to recover? Its not clear which DB its having problems using. I presume the internal DB for the app itself and not the DB its trying to access for analysis.
07-04 23:47:29 e[1mINFO metabase.utile[0m :: Loading Metabase...
07-04 23:47:33 e[1mINFO util.encryptione[0m :: DB details encryption is DISABLED for this Metabase instance. 🔓
See http://www.metabase.com/docs/latest/operations-guide/start.html#encrypting-your-database-connection-details-at-rest for more information.
07-04 23:47:38 e[1mINFO metabase.coree[0m :: Starting Metabase in STANDALONE mode
07-04 23:47:38 e[1mINFO metabase.coree[0m :: Launching Embedded Jetty Webserver with config:
{:port 3000, :host "0.0.0.0"}
07-04 23:47:38 e[1mINFO metabase.coree[0m :: Starting Metabase version v0.27.2 (35acf6e release-0.27.0) ...
07-04 23:47:38 e[1mINFO metabase.coree[0m :: System timezone is 'GMT' ...
07-04 23:47:40 e[1mWARN metabase.drivere[0m :: No -init-driver function found for 'metabase.driver.google'
07-04 23:47:40 e[1mINFO metabase.coree[0m :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
07-04 23:47:40 e[1mINFO metabase.dbe[0m :: e[36mVerifying postgres Database Connection ...e[0m
07-04 23:47:45 e[1mERROR metabase.drivere[0m :: Failed to connect to database: Timed out after 5000 milliseconds.
java.lang.AssertionError: Assert failed: Unable to connect to Metabase postgres DB.
(binding [*allow-potentailly-unsafe-connections* true] (require (quote metabase.driver)) ((resolve (quote metabase.driver/can-connect-with-details?)) engine details))
at metabase.db$verify_db_connection.invokeStatic(db.clj:334)
at metabase.db$verify_db_connection.invoke(db.clj:327)
at metabase.db$verify_db_connection.invokeStatic(db.clj:330)
at metabase.db$verify_db_connection.invoke(db.clj:327)
at metabase.db$setup_db_BANG_.invokeStatic(db.clj:382)
at metabase.db$setup_db_BANG_.doInvoke(db.clj:376)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at metabase.core$init_BANG_.invokeStatic(core.clj:108)
at metabase.core$init_BANG_.invoke(core.clj:87)
at metabase.core$start_normally.invokeStatic(core.clj:194)
at metabase.core$start_normally.invoke(core.clj:188)
at metabase.core$_main.invokeStatic(core.clj:292)
at metabase.core$_main.doInvoke(core.clj:287)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at metabase.core.main(Unknown Source)
07-04 23:47:45 e[1mERROR metabase.coree[0m :: Metabase Initialization FAILED: Assert failed: Unable to connect to Metabase postgres DB.
(binding [*allow-potentailly-unsafe-connections* true] (require (quote metabase.driver)) ((resolve (quote metabase.driver/can-connect-with-details?)) engine details))
07-04 23:47:45 e[1mINFO metabase.coree[0m :: Metabase Shutting Down ...
07-04 23:47:45 e[1mINFO metabase.coree[0m :: Metabase Shutdown COMPLETE