User is getting 401 "did not match stored password" after upgrading to version 46

When you failed the upgrade did you simply downgrade or you had a clean backup before the upgrade?

You seem to be hitting this

Hi Tony,

I couldn't find the answer helpful in that last issue. Besides it looks like trying to solve the problem with downgrading and not upgrading. I am still stuck with this issue.

Is there any other solution that I can try, I am happy to pay for the solution if any premium support is there.

When you failed the upgrade did you simply downgrade or you had a clean backup before the upgrade which you restored?

I restored clean backup before the upgrade. Without it, I didn't even get login page.

Strange cause the examples all seem to point to when downgrading from a version to any previous versions. Would cause that.

So did you followed the example in the ticket or not?

Hi Tony,

We managed to downgrade without any issues, that validationexception we faced only when we tried to UPGRADE from 43.4 to 43.7. And I have checked there is no hash mentioned in the solution present in table and since the solution is for DOWNGRADE issue I haven't checked on it further.

The issue I am facing is with upgrading again to 0.46 from 0.43, that is our original issue which we were trying to solve. So during the upgrade we faced the issue where all users locked out and log isn't showing any error either

Can you share the whole logs again from beginning to start.

So you are saying you managed to upgrade and metabase showed the login screen but they simply couldn't login? do you have errros for these logins

Hi Tony,

Here is the log. Is that ClassNotFound error cause this issue? We haven't seen it before. we have done the fresh rollback and deploy 46 version again yesterday and found this error in log but I guess it isn't related to login unless there is some link.

Warning: environ value jdk-11.0.18+10 for key :java-version has been overwritten with 11.0.18
2023-05-24 05:27:18,549 INFO metabase.util :: Maximum memory available to JVM: 492.0 MB
2023-05-24 05:27:22,035 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. 🔓 
 For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
..instrumented #'metabase.util.malli/with-api-error-message
..instrumented #'metabase.util.honey-sql-2/identifier
..instrumented #'metabase.util.honey-sql-2/normalize-type-info
..instrumented #'metabase.util.honey-sql-2/with-database-type-info
..instrumented #'metabase.util.honey-sql-2/cast
..instrumented #'metabase.util.honey-sql-2/quoted-cast
..instrumented #'metabase.util.honey-sql-2/maybe-cast
..instrumented #'metabase.models.permissions/classify-path
..instrumented #'metabase.models.permissions/classify-data-path
..instrumented #'metabase.models.permissions/generate-graph
..instrumented #'metabase.models.permissions/->v2-path
..instrumented #'metabase.models.permissions/update-db-data-access-permissions!
..instrumented #'metabase.models.permissions/update-group-permissions!
..instrumented #'metabase.models.permissions/update-data-perms-graph!
..instrumented #'metabase.models.parameter-card/upsert-or-delete-from-parameters!
WARNING: abs already refers to: #'clojure.core/abs in namespace: kixi.stats.math, being replaced by: #'kixi.stats.math/abs
WARNING: abs already refers to: #'clojure.core/abs in namespace: kixi.stats.test, being replaced by: #'kixi.stats.math/abs
WARNING: abs already refers to: #'clojure.core/abs in namespace: kixi.stats.distribution, being replaced by: #'kixi.stats.math/abs
..instrumented #'metabase.query-processor.middleware.permissions/check-query-permissions*
..instrumented #'metabase.driver.sql-jdbc.sync.describe-database/simple-select-probe-query
2023-05-24 05:27:31,725 INFO driver.impl :: e[34mRegistered abstract driver :sqle[0m  🚚
2023-05-24 05:27:31,733 INFO metabase.util :: e[36m ⮦ Load driver :sql took 171.7 mse[0m
2023-05-24 05:27:31,739 INFO driver.impl :: e[34mRegistered abstract driver :sql-jdbce[0m (parents: [:sql]) 🚚
2023-05-24 05:27:31,745 INFO metabase.util :: e[32mLoad driver :sql-jdbc took 190.6 mse[0m
2023-05-24 05:27:31,746 INFO driver.impl :: e[34mRegistered driver :h2e[0m (parents: [:sql-jdbc]) 🚚
..instrumented #'metabase.driver.h2/classify-query
2023-05-24 05:27:31,857 INFO driver.impl :: e[34mRegistered driver :mysqle[0m (parents: [:sql-jdbc]) 🚚
2023-05-24 05:27:31,908 INFO driver.impl :: e[34mRegistered driver :postgrese[0m (parents: [:sql-jdbc]) 🚚
..instrumented #'metabase.models.params.custom-values/values-from-card
..instrumented #'metabase.api.card/param-values
..instrumented #'metabase.api.dashboard/chain-filter
2023-05-24 05:27:34,726 INFO metabase.core :: 
Metabase v0.46.0 (0ca7df3 release-x.46.x) 

Copyright © 2023 Metabase, Inc. 

Metabase Enterprise Edition extensions are NOT PRESENT.
2023-05-24 05:27:34,741 INFO metabase.core :: Starting Metabase in STANDALONE mode
2023-05-24 05:27:34,822 INFO metabase.server :: Launching Embedded Jetty Webserver with config: 
 {:port 3000, :host "0.0.0.0"}

2023-05-24 05:27:34,956 INFO metabase.core :: Starting Metabase version v0.46.0 (0ca7df3 release-x.46.x) ...
2023-05-24 05:27:35,042 INFO metabase.core :: System info:
 {"file.encoding" "UTF-8",
 "java.runtime.name" "OpenJDK Runtime Environment",
 "java.runtime.version" "11.0.18+10",
 "java.vendor" "Eclipse Adoptium",
 "java.vendor.url" "https://adoptium.net/",
 "java.version" "11.0.18",
 "java.vm.name" "OpenJDK 64-Bit Server VM",
 "java.vm.version" "11.0.18+10",
 "os.name" "Linux",
 "os.version" "4.14.281-212.502.amzn2.x86_64",
 "user.language" "en",
 "user.timezone" "GMT"}

2023-05-24 05:27:35,045 INFO metabase.plugins :: Loading plugins in /plugins...
2023-05-24 05:27:35,610 INFO util.files :: Extract file /modules/sparksql.metabase-driver.jar -> /plugins/sparksql.metabase-driver.jar
2023-05-24 05:27:35,713 INFO util.files :: Extract file /modules/athena.metabase-driver.jar -> /plugins/athena.metabase-driver.jar
2023-05-24 05:27:35,848 INFO util.files :: Extract file /modules/sqlserver.metabase-driver.jar -> /plugins/sqlserver.metabase-driver.jar
2023-05-24 05:27:35,866 INFO util.files :: Extract file /modules/presto-jdbc.metabase-driver.jar -> /plugins/presto-jdbc.metabase-driver.jar
2023-05-24 05:27:35,957 INFO util.files :: Extract file /modules/sqlite.metabase-driver.jar -> /plugins/sqlite.metabase-driver.jar
2023-05-24 05:27:36,000 INFO util.files :: Extract file /modules/vertica.metabase-driver.jar -> /plugins/vertica.metabase-driver.jar
2023-05-24 05:27:36,004 INFO util.files :: Extract file /modules/redshift.metabase-driver.jar -> /plugins/redshift.metabase-driver.jar
2023-05-24 05:27:36,016 INFO util.files :: Extract file /modules/druid.metabase-driver.jar -> /plugins/druid.metabase-driver.jar
2023-05-24 05:27:36,023 INFO util.files :: Extract file /modules/mongo.metabase-driver.jar -> /plugins/mongo.metabase-driver.jar
2023-05-24 05:27:36,053 INFO util.files :: Extract file /modules/oracle.metabase-driver.jar -> /plugins/oracle.metabase-driver.jar
2023-05-24 05:27:36,057 INFO util.files :: Extract file /modules/snowflake.metabase-driver.jar -> /plugins/snowflake.metabase-driver.jar
2023-05-24 05:27:36,326 INFO util.files :: Extract file /modules/googleanalytics.metabase-driver.jar -> /plugins/googleanalytics.metabase-driver.jar
2023-05-24 05:27:36,345 INFO util.files :: Extract file /modules/bigquery-cloud-sdk.metabase-driver.jar -> /plugins/bigquery-cloud-sdk.metabase-driver.jar
2023-05-24 05:27:36,686 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :hive-like...e[0m
2023-05-24 05:27:36,690 INFO driver.impl :: e[34mRegistered abstract driver :hive-likee[0m (parents: [:sql-jdbc]) 🚚
2023-05-24 05:27:36,694 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sparksql...e[0m
2023-05-24 05:27:36,696 INFO driver.impl :: e[34mRegistered driver :sparksqle[0m (parents: [:hive-like]) 🚚
2023-05-24 05:27:36,746 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :athena...e[0m
2023-05-24 05:27:36,755 INFO driver.impl :: e[34mRegistered driver :athenae[0m (parents: [:sql-jdbc]) 🚚
2023-05-24 05:27:36,766 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sqlserver...e[0m
2023-05-24 05:27:36,770 INFO driver.impl :: e[34mRegistered driver :sqlservere[0m (parents: [:sql-jdbc]) 🚚
2023-05-24 05:27:36,867 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :presto-jdbc...e[0m
2023-05-24 05:27:36,869 INFO driver.impl :: e[34mRegistered driver :presto-jdbce[0m (parents: [:sql-jdbc]) 🚚
2023-05-24 05:27:36,914 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sqlite...e[0m
2023-05-24 05:27:36,920 INFO driver.impl :: e[34mRegistered driver :sqlitee[0m (parents: [:sql-jdbc]) 🚚
2023-05-24 05:27:36,928 INFO plugins.dependencies :: e[31mMetabase cannot initialize plugin Metabase Vertica Driver due to required dependencies.e[0m Metabase requires the Vertica JDBC driver in order to connect to Vertica databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/vertica.html for more details.

2023-05-24 05:27:36,939 INFO plugins.dependencies :: Metabase Vertica Driver dependency {:class com.vertica.jdbc.Driver} satisfied? false
2023-05-24 05:27:36,946 INFO plugins.dependencies :: e[33mPlugins with unsatisfied deps: ["Metabase Vertica Driver"]e[0m
2023-05-24 05:27:36,969 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :redshift...e[0m
2023-05-24 05:27:36,986 INFO driver.impl :: e[34mRegistered driver :redshifte[0m (parents: [:postgres]) 🚚
2023-05-24 05:27:36,994 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :druid...e[0m
2023-05-24 05:27:36,996 INFO driver.impl :: e[34mRegistered driver :druide[0m  🚚
2023-05-24 05:27:37,023 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :mongo...e[0m
2023-05-24 05:27:37,025 INFO driver.impl :: e[34mRegistered driver :mongoe[0m  🚚
2023-05-24 05:27:37,038 INFO plugins.dependencies :: e[31mMetabase cannot initialize plugin Metabase Oracle Driver due to required dependencies.e[0m Metabase requires the Oracle JDBC driver in order to connect to Oracle databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/oracle.html for more details.

2023-05-24 05:27:37,039 INFO plugins.dependencies :: Metabase Oracle Driver dependency {:class oracle.jdbc.OracleDriver} satisfied? false
2023-05-24 05:27:37,040 INFO plugins.dependencies :: e[33mPlugins with unsatisfied deps: ["Metabase Vertica Driver" "Metabase Oracle Driver"]e[0m
2023-05-24 05:27:37,076 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :snowflake...e[0m
2023-05-24 05:27:37,078 INFO driver.impl :: e[34mRegistered driver :snowflakee[0m (parents: [:sql-jdbc]) 🚚
2023-05-24 05:27:37,090 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :googleanalytics...e[0m
2023-05-24 05:27:37,091 INFO driver.impl :: e[34mRegistered driver :googleanalyticse[0m  🚚
2023-05-24 05:27:37,106 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :bigquery-cloud-sdk...e[0m
2023-05-24 05:27:37,107 INFO driver.impl :: e[34mRegistered driver :bigquery-cloud-sdke[0m (parents: [:sql]) 🚚
2023-05-24 05:27:37,109 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
2023-05-24 05:27:37,114 INFO db.setup :: e[36mVerifying mysql Database Connection ...e[0m
2023-05-24 05:27:37,294 INFO db.setup :: Successfully verified MySQL 8.0.28 application database connection. ✅
2023-05-24 05:27:37,296 INFO db.setup :: Running Database Migrations...
2023-05-24 05:27:37,299 INFO db.setup :: Setting up Liquibase...
2023-05-24 05:27:37,590 INFO db.setup :: Liquibase is ready.
2023-05-24 05:27:37,592 INFO db.liquibase :: Checking if Database has unrun migrations...
2023-05-24 05:27:40,250 INFO db.setup :: Database Migrations Current ...  ✅
2023-05-24 05:27:40,276 INFO db.data-migrations :: Running all necessary data migrations, this may take a minute.
2023-05-24 05:27:41,248 INFO db.data-migrations :: Finished running data migrations.
2023-05-24 05:27:41,250 INFO metabase.util :: e[32mDatabase setup took 4.1 se[0m
2023-05-24 05:27:41,409 INFO metabase.events :: Starting events listener: e[34m:metabase.events.activity-feed/ActivityFeede[0m 👂
2023-05-24 05:27:41,425 INFO metabase.events :: Starting events listener: e[34m:metabase.events.last-login/LastLogine[0m 👂
2023-05-24 05:27:41,428 INFO metabase.events :: Starting events listener: e[34m:metabase.events.persisted-info/PersistedInfoe[0m 👂
2023-05-24 05:27:41,428 INFO metabase.events :: Starting events listener: e[34m:metabase.events.sync-database/Synce[0m 👂
2023-05-24 05:27:41,429 INFO metabase.events :: Starting events listener: e[34m:metabase.events.driver-notifications/DriverNotificationse[0m 👂
2023-05-24 05:27:41,430 INFO metabase.events :: Starting events listener: e[34m:metabase.events.revision/Revisionse[0m 👂
2023-05-24 05:27:41,433 INFO metabase.events :: Starting events listener: e[34m:metabase.events.view-log/ViewLoge[0m 👂
2023-05-24 05:27:41,554 INFO driver.impl :: e[33mInitializing driver :sql...e[0m
2023-05-24 05:27:41,555 INFO driver.impl :: e[33mInitializing driver :sql-jdbc...e[0m
2023-05-24 05:27:41,556 INFO driver.impl :: e[33mInitializing driver :h2...e[0m
2023-05-24 05:27:41,836 INFO util.files :: Extract file /sample-database.db.mv.db -> /plugins/sample-database.db.mv.db
2023-05-24 05:27:42,276 INFO impl.StdSchedulerFactory :: Using default implementation for ThreadExecutor
2023-05-24 05:27:42,321 INFO core.SchedulerSignalerImpl :: Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
2023-05-24 05:27:42,321 INFO core.QuartzScheduler :: Quartz Scheduler v.2.3.2 created.
2023-05-24 05:27:42,323 INFO jdbcjobstore.JobStoreTX :: Using db table-based data access locking (synchronization).
2023-05-24 05:27:42,325 INFO jdbcjobstore.JobStoreTX :: JobStoreTX initialized.
2023-05-24 05:27:42,328 INFO core.QuartzScheduler :: Scheduler meta-data: Quartz Scheduler (v2.3.2) 'MetabaseScheduler' with instanceId '1e0b4b7470ff1684906062278'
  Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
  NOT STARTED.
  Currently in standby mode.
  Number of jobs executed: 0
  Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
  Using job-store 'org.quartz.impl.jdbcjobstore.JobStoreTX' - which supports persistence. and is clustered.

2023-05-24 05:27:42,329 INFO impl.StdSchedulerFactory :: Quartz scheduler 'MetabaseScheduler' initialized from default resource file in Quartz package: 'quartz.properties'
2023-05-24 05:27:42,330 INFO impl.StdSchedulerFactory :: Quartz scheduler version: 2.3.2
2023-05-24 05:27:42,542 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_1e0b4b7470ff1684906062278 paused.
2023-05-24 05:27:42,544 INFO metabase.task :: Task scheduler initialized into standby mode.
2023-05-24 05:27:42,547 INFO metabase.task :: Initializing task e[32mSyncDatabasese[0m 📆
2023-05-24 05:27:42,650 INFO task.sync-databases :: Updated default schedules for 0 databases
2023-05-24 05:27:42,652 INFO metabase.task :: Initializing task e[32mPersistRefreshe[0m 📆
2023-05-24 05:27:42,688 INFO driver.impl :: e[33mInitializing driver :mysql...e[0m
2023-05-24 05:27:42,735 INFO metabase.task :: Initializing task e[32mCheckForNewVersionse[0m 📆
2023-05-24 05:27:42,801 INFO metabase.task :: Initializing task e[32mPersistPrunee[0m 📆
2023-05-24 05:27:42,812 INFO metabase.task :: Initializing task e[32mSendAnonymousUsageStatse[0m 📆
2023-05-24 05:27:42,852 INFO metabase.task :: Initializing task e[32mRefreshSlackChannelsAndUserse[0m 📆
2023-05-24 05:27:42,899 INFO metabase.task :: Initializing task e[32mSendPulsese[0m 📆
2023-05-24 05:27:42,931 INFO metabase.task :: Initializing task e[32mSendFollowUpEmailse[0m 📆
2023-05-24 05:27:42,965 INFO metabase.task :: Initializing task e[32mTaskHistoryCleanupe[0m 📆
2023-05-24 05:27:42,988 INFO metabase.task :: Initializing task e[32mTruncateAuditLoge[0m 📆
2023-05-24 05:27:43,035 INFO jdbcjobstore.JobStoreTX :: ClusterManager: detected 1 failed or restarted instances.
2023-05-24 05:27:43,035 INFO jdbcjobstore.JobStoreTX :: ClusterManager: Scanning for instance "abfde8a47d061683022047253"'s failed in-progress jobs.
2023-05-24 05:27:43,044 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_1e0b4b7470ff1684906062278 started.
2023-05-24 05:27:43,045 INFO metabase.task :: Task scheduler started
2023-05-24 05:27:43,046 INFO metabase.core :: Metabase Initialization COMPLETE
2023-05-24 05:27:43,076 INFO jdbcjobstore.JobStoreTX :: Handling 5 trigger(s) that missed their scheduled fire-time.
2023-05-24 05:27:43,112 ERROR jdbcjobstore.JobStoreTX :: MisfireHandler: Error handling misfires: Couldn't store trigger 'DEFAULT.metabase.task.abandonment-emails.trigger' for 'DEFAULT.metabase.task.abandonment-emails.job' job:Couldn't retrieve job because a required class was not found: metabase.task.follow_up_emails.AbandonmentEmail
org.quartz.JobPersistenceException: Couldn't store trigger 'DEFAULT.metabase.task.abandonment-emails.trigger' for 'DEFAULT.metabase.task.abandonment-emails.job' job:Couldn't retrieve job because a required class was not found: metabase.task.follow_up_emails.AbandonmentEmail [See nested exception: org.quartz.JobPersistenceException: Couldn't retrieve job because a required class was not found: metabase.task.follow_up_emails.AbandonmentEmail [See nested exception: java.lang.ClassNotFoundException: metabase.task.follow_up_emails.AbandonmentEmail]]
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1228)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.doUpdateOfMisfiredTrigger(JobStoreSupport.java:1042)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:991)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3264)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:4012)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:4033)
Caused by: org.quartz.JobPersistenceException: Couldn't retrieve job because a required class was not found: metabase.task.follow_up_emails.AbandonmentEmail [See nested exception: java.lang.ClassNotFoundException: metabase.task.follow_up_emails.AbandonmentEmail]
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1393)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1210)
	... 5 more
Caused by: java.lang.ClassNotFoundException: metabase.task.follow_up_emails.AbandonmentEmail
	at java.base/java.net.URLClassLoader.findClass(Unknown Source)
	at clojure.lang.DynamicClassLoader.findClass(DynamicClassLoader.java:69)
	at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
	at clojure.lang.DynamicClassLoader.loadClass(DynamicClassLoader.java:77)
	at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Unknown Source)
	at metabase.task$load_class.invokeStatic(task.clj:118)
	at metabase.task$load_class.invoke(task.clj:117)
	at metabase.task.ClassLoadHelper.loadClass(task.clj:128)
	at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1390)
	... 6 more
2023-05-24 05:27:43,130 INFO task.refresh-slack-channel-user-cache :: Slack is not configured, not refreshing slack user/channel cache.
2023-05-24 05:28:17,505 INFO i18n.impl :: Reading available locales from locales.clj...
2023-05-24 05:28:17,747 INFO util.fonts :: Reading available fonts from /frontend_client/app/fonts
2023-05-24 05:28:22,119 DEBUG middleware.log :: e[31mGET /api/user/current 401 2.8 ms (0 DB calls) 
"Unauthenticated"
e[0m
2023-05-24 05:28:22,220 DEBUG middleware.log :: e[32mGET /api/session/properties 200 79.9 ms (1 DB calls) App DB connections: 0/4 Jetty threads: 3/50 (3 idle, 0 queued) (41 total active threads) Queries in flight: 0 (0 queued)e[0m
2023-05-24 05:28:28,269 ERROR api.session :: Authentication endpoint error
clojure.lang.ExceptionInfo: Password did not match stored password. {:status-code 401, :errors {:password "did not match stored password"}}
	at metabase.api.session$do_http_401_on_error.invokeStatic(session.clj:156)
	at metabase.api.session$do_http_401_on_error.invoke(session.clj:152)
	at metabase.api.session$fn__97456.invokeStatic(session.clj:178)
	at metabase.api.session$fn__97456.invoke(session.clj:165)
	at compojure.core$wrap_response$fn__42375.invoke(core.clj:160)
	at compojure.core$wrap_route_middleware$fn__42359.invoke(core.clj:132)
	at compojure.core$wrap_route_info$fn__42364.invoke(core.clj:139)
	at compojure.core$wrap_route_matches$fn__42368.invoke(core.clj:151)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387.invoke(core.clj:200)
	at metabase.api.session$_PLUS_log_all_request_failures$fn__97517.invoke(session.clj:313)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387.invoke(core.clj:200)
	at compojure.core$make_context$handler__42415.invoke(core.clj:290)
	at compojure.core$make_context$fn__42419.invoke(core.clj:300)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$make_context$fn__42419.invoke(core.clj:301)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at metabase.api.routes$fn__98449$fn__98452.invoke(routes.clj:64)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387.invoke(core.clj:200)
	at clojure.lang.AFn.applyToHelper(AFn.java:160)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.core$apply.invokeStatic(core.clj:667)
	at clojure.core$apply.invoke(core.clj:662)
	at metabase.server.routes$fn__98610$fn__98611.doInvoke(routes.clj:70)
	at clojure.lang.RestFn.invoke(RestFn.java:436)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387.invoke(core.clj:200)
	at compojure.core$make_context$handler__42415.invoke(core.clj:290)
	at compojure.core$make_context$fn__42419.invoke(core.clj:300)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$wrap_route_matches$fn__42368.invoke(core.clj:153)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$wrap_route_matches$fn__42368.invoke(core.clj:153)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at compojure.core$wrap_route_matches$fn__42368.invoke(core.clj:153)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387$f__42388$respond_SINGLEQUOTE___42389.invoke(core.clj:197)
	at metabase.server.routes$fn__98595$fn__98597.invoke(routes.clj:47)
	at compojure.core$routes$fn__42387$f__42388.invoke(core.clj:198)
	at compojure.core$routes$fn__42387.invoke(core.clj:200)
	at metabase.server.middleware.exceptions$catch_uncaught_exceptions$fn__95133.invoke(exceptions.clj:103)
	at metabase.server.middleware.exceptions$catch_api_exceptions$fn__95130.invoke(exceptions.clj:91)
	at metabase.server.middleware.log$log_api_call$fn__100334$fn__100335$fn__100336.invoke(log.clj:216)
	at metabase.driver.sql_jdbc.execute.diagnostic$do_with_diagnostic_info.invokeStatic(diagnostic.clj:18)
	at metabase.driver.sql_jdbc.execute.diagnostic$do_with_diagnostic_info.invoke(diagnostic.clj:12)
	at metabase.server.middleware.log$log_api_call$fn__100334$fn__100335.invoke(log.clj:208)
	at toucan2.execute$do_with_call_counts.invokeStatic(execute.clj:112)
	at toucan2.execute$do_with_call_counts.invoke(execute.clj:103)
	at metabase.server.middleware.log$log_api_call$fn__100334.invoke(log.clj:207)
	at metabase.server.middleware.browser_cookie$ensure_browser_id_cookie$fn__103755.invoke(browser_cookie.clj:42)
	at metabase.server.middleware.security$add_security_headers$fn__79453.invoke(security.clj:153)
	at metabase.server.middleware.json$wrap_json_body$fn__102499.invoke(json.clj:67)
	at metabase.server.middleware.offset_paging$handle_paging$fn__79477.invoke(offset_paging.clj:45)
	at metabase.server.middleware.json$wrap_streamed_json_response$fn__102517.invoke(json.clj:103)
	at ring.middleware.keyword_params$wrap_keyword_params$fn__104022.invoke(keyword_params.clj:55)
	at ring.middleware.params$wrap_params$fn__104041.invoke(params.clj:77)
	at metabase.server.middleware.misc$maybe_set_site_url$fn__50479.invoke(misc.clj:61)
	at metabase.server.middleware.session$reset_session_timeout$fn__63769.invoke(session.clj:430)
	at metabase.server.middleware.session$bind_current_user$fn__63739$fn__63740.invoke(session.clj:336)
	at metabase.server.middleware.session$do_with_current_user.invokeStatic(session.clj:315)
	at metabase.server.middleware.session$do_with_current_user.invoke(session.clj:304)
	at metabase.server.middleware.session$bind_current_user$fn__63739.invoke(session.clj:335)
	at metabase.server.middleware.session$wrap_current_user_info$fn__63721.invoke(session.clj:285)
	at metabase.server.middleware.session$wrap_session_id$fn__63704.invoke(session.clj:217)
	at metabase.server.middleware.auth$wrap_api_key$fn__86632.invoke(auth.clj:30)
	at ring.middleware.cookies$wrap_cookies$fn__103942.invoke(cookies.clj:216)
	at metabase.server.middleware.misc$add_content_type$fn__50461.invoke(misc.clj:29)
	at metabase.server.middleware.misc$disable_streaming_buffering$fn__50487.invoke(misc.clj:78)
	at ring.middleware.gzip$wrap_gzip$fn__103984.invoke(gzip.clj:86)
	at metabase.server.middleware.misc$bind_request$fn__50490.invoke(misc.clj:95)
	at metabase.server.middleware.ssl$redirect_to_https_middleware$fn__103771.invoke(ssl.clj:41)
	at metabase.server$async_proxy_handler$fn__99100.invoke(server.clj:77)
	at metabase.server.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.Server.handle(Server.java:563)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:936)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1080)
	at java.base/java.lang.Thread.run(Unknown Source)
2023-05-24 05:28:28,293 DEBUG middleware.log :: e[31mPOST /api/session 401 277.8 ms (1 DB calls) 
{:errors {:password "did not match stored password"}}
e[0m

please go to 46.4 if possible.

I'm completely puzzled by the issue you're seeing. I'm guessing you're running the JAR right?

We are using docker image of 46.0 not the JAR. I will try 46.4 if that works.

I have tried to upgrade to 46.4 with no luck. It's showing same error on login.

Here is the log.

Warning: environ value jdk-11.0.19+7 for key :java-version has been overwritten with 11.0.19
2023-05-31 01:08:02,828 INFO metabase.util :: Maximum memory available to JVM: 492.0 MB
2023-05-31 01:08:06,467 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. 🔓 
 For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
..instrumented #'metabase.util.malli/with-api-error-message
..instrumented #'metabase.util.honey-sql-2/identifier
..instrumented #'metabase.util.honey-sql-2/normalize-type-info
..instrumented #'metabase.util.honey-sql-2/with-database-type-info
..instrumented #'metabase.util.honey-sql-2/cast
..instrumented #'metabase.util.honey-sql-2/quoted-cast
..instrumented #'metabase.util.honey-sql-2/maybe-cast
..instrumented #'metabase.models.permissions/classify-path
..instrumented #'metabase.models.permissions/classify-data-path
..instrumented #'metabase.models.permissions/generate-graph
..instrumented #'metabase.models.permissions/->v2-path
..instrumented #'metabase.models.permissions/update-db-data-access-permissions!
..instrumented #'metabase.models.permissions/update-group-permissions!
..instrumented #'metabase.models.permissions/update-data-perms-graph!
..instrumented #'metabase.models.parameter-card/upsert-or-delete-from-parameters!
WARNING: abs already refers to: #'clojure.core/abs in namespace: kixi.stats.math, being replaced by: #'kixi.stats.math/abs
WARNING: abs already refers to: #'clojure.core/abs in namespace: kixi.stats.test, being replaced by: #'kixi.stats.math/abs
WARNING: abs already refers to: #'clojure.core/abs in namespace: kixi.stats.distribution, being replaced by: #'kixi.stats.math/abs
..instrumented #'metabase.query-processor.middleware.permissions/check-query-permissions*
..instrumented #'metabase.driver.sql-jdbc.sync.describe-database/simple-select-probe-query
2023-05-31 01:08:17,749 INFO driver.impl :: e[34mRegistered abstract driver :sqle[0m  🚚
2023-05-31 01:08:17,755 INFO metabase.util :: e[36m ⮦ Load driver :sql took 151.7 mse[0m
2023-05-31 01:08:17,760 INFO driver.impl :: e[34mRegistered abstract driver :sql-jdbce[0m (parents: [:sql]) 🚚
2023-05-31 01:08:17,766 INFO metabase.util :: e[32mLoad driver :sql-jdbc took 168.9 mse[0m
2023-05-31 01:08:17,767 INFO driver.impl :: e[34mRegistered driver :h2e[0m (parents: [:sql-jdbc]) 🚚
..instrumented #'metabase.driver.h2/classify-query
2023-05-31 01:08:17,844 INFO driver.impl :: e[34mRegistered driver :mysqle[0m (parents: [:sql-jdbc]) 🚚
2023-05-31 01:08:17,884 INFO driver.impl :: e[34mRegistered driver :postgrese[0m (parents: [:sql-jdbc]) 🚚
..instrumented #'metabase.models.params.custom-values/values-from-card
..instrumented #'metabase.api.card/param-values
..instrumented #'metabase.api.dashboard/chain-filter
..instrumented #'metabase.api.setup/state-for-checklist
..instrumented #'metabase.api.setup/checklist-items
2023-05-31 01:08:21,411 INFO metabase.core :: 
Metabase v0.46.4 (f858476 release-x.46.x) 

Copyright © 2023 Metabase, Inc. 

Metabase Enterprise Edition extensions are NOT PRESENT.
2023-05-31 01:08:21,431 INFO metabase.core :: Starting Metabase in STANDALONE mode
2023-05-31 01:08:21,544 INFO metabase.server :: Launching Embedded Jetty Webserver with config: 
 {:port 3000, :host "0.0.0.0"}

2023-05-31 01:08:21,670 INFO metabase.core :: Starting Metabase version v0.46.4 (f858476 release-x.46.x) ...
2023-05-31 01:08:21,723 INFO metabase.core :: System info:
 {"file.encoding" "UTF-8",
 "java.runtime.name" "OpenJDK Runtime Environment",
 "java.runtime.version" "11.0.19+7",
 "java.vendor" "Eclipse Adoptium",
 "java.vendor.url" "https://adoptium.net/",
 "java.version" "11.0.19",
 "java.vm.name" "OpenJDK 64-Bit Server VM",
 "java.vm.version" "11.0.19+7",
 "os.name" "Linux",
 "os.version" "4.14.281-212.502.amzn2.x86_64",
 "user.language" "en",
 "user.timezone" "GMT"}

2023-05-31 01:08:21,727 INFO metabase.plugins :: Loading plugins in /plugins...
2023-05-31 01:08:22,295 INFO util.files :: Extract file /modules/sparksql.metabase-driver.jar -> /plugins/sparksql.metabase-driver.jar
2023-05-31 01:08:22,458 INFO util.files :: Extract file /modules/athena.metabase-driver.jar -> /plugins/athena.metabase-driver.jar
2023-05-31 01:08:22,585 INFO util.files :: Extract file /modules/sqlserver.metabase-driver.jar -> /plugins/sqlserver.metabase-driver.jar
2023-05-31 01:08:22,599 INFO util.files :: Extract file /modules/presto-jdbc.metabase-driver.jar -> /plugins/presto-jdbc.metabase-driver.jar
2023-05-31 01:08:22,677 INFO util.files :: Extract file /modules/sqlite.metabase-driver.jar -> /plugins/sqlite.metabase-driver.jar
2023-05-31 01:08:22,715 INFO util.files :: Extract file /modules/vertica.metabase-driver.jar -> /plugins/vertica.metabase-driver.jar
2023-05-31 01:08:22,717 INFO util.files :: Extract file /modules/redshift.metabase-driver.jar -> /plugins/redshift.metabase-driver.jar
2023-05-31 01:08:22,727 INFO util.files :: Extract file /modules/druid.metabase-driver.jar -> /plugins/druid.metabase-driver.jar
2023-05-31 01:08:22,733 INFO util.files :: Extract file /modules/mongo.metabase-driver.jar -> /plugins/mongo.metabase-driver.jar
2023-05-31 01:08:22,760 INFO util.files :: Extract file /modules/oracle.metabase-driver.jar -> /plugins/oracle.metabase-driver.jar
2023-05-31 01:08:22,762 INFO util.files :: Extract file /modules/snowflake.metabase-driver.jar -> /plugins/snowflake.metabase-driver.jar
2023-05-31 01:08:22,993 INFO util.files :: Extract file /modules/googleanalytics.metabase-driver.jar -> /plugins/googleanalytics.metabase-driver.jar
2023-05-31 01:08:23,009 INFO util.files :: Extract file /modules/bigquery-cloud-sdk.metabase-driver.jar -> /plugins/bigquery-cloud-sdk.metabase-driver.jar
2023-05-31 01:08:23,683 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :hive-like...e[0m
2023-05-31 01:08:23,685 INFO driver.impl :: e[34mRegistered abstract driver :hive-likee[0m (parents: [:sql-jdbc]) 🚚
2023-05-31 01:08:23,687 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sparksql...e[0m
2023-05-31 01:08:23,689 INFO driver.impl :: e[34mRegistered driver :sparksqle[0m (parents: [:hive-like]) 🚚
2023-05-31 01:08:23,738 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :athena...e[0m
2023-05-31 01:08:23,740 INFO driver.impl :: e[34mRegistered driver :athenae[0m (parents: [:sql-jdbc]) 🚚
2023-05-31 01:08:23,762 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sqlserver...e[0m
2023-05-31 01:08:23,765 INFO driver.impl :: e[34mRegistered driver :sqlservere[0m (parents: [:sql-jdbc]) 🚚
2023-05-31 01:08:23,835 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :presto-jdbc...e[0m
2023-05-31 01:08:23,845 INFO driver.impl :: e[34mRegistered driver :presto-jdbce[0m (parents: [:sql-jdbc]) 🚚
2023-05-31 01:08:23,854 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sqlite...e[0m
2023-05-31 01:08:23,857 INFO driver.impl :: e[34mRegistered driver :sqlitee[0m (parents: [:sql-jdbc]) 🚚
2023-05-31 01:08:23,874 INFO plugins.dependencies :: e[31mMetabase cannot initialize plugin Metabase Vertica Driver due to required dependencies.e[0m Metabase requires the Vertica JDBC driver in order to connect to Vertica databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/vertica.html for more details.

2023-05-31 01:08:23,876 INFO plugins.dependencies :: Metabase Vertica Driver dependency {:class com.vertica.jdbc.Driver} satisfied? false
2023-05-31 01:08:23,877 INFO plugins.dependencies :: e[33mPlugins with unsatisfied deps: ["Metabase Vertica Driver"]e[0m
2023-05-31 01:08:23,883 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :redshift...e[0m
2023-05-31 01:08:23,884 INFO driver.impl :: e[34mRegistered driver :redshifte[0m (parents: [:postgres]) 🚚
2023-05-31 01:08:23,889 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :druid...e[0m
2023-05-31 01:08:23,890 INFO driver.impl :: e[34mRegistered driver :druide[0m  🚚
2023-05-31 01:08:23,902 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :mongo...e[0m
2023-05-31 01:08:23,903 INFO driver.impl :: e[34mRegistered driver :mongoe[0m  🚚
2023-05-31 01:08:23,910 INFO plugins.dependencies :: e[31mMetabase cannot initialize plugin Metabase Oracle Driver due to required dependencies.e[0m Metabase requires the Oracle JDBC driver in order to connect to Oracle databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/oracle.html for more details.

2023-05-31 01:08:23,911 INFO plugins.dependencies :: Metabase Oracle Driver dependency {:class oracle.jdbc.OracleDriver} satisfied? false
2023-05-31 01:08:23,912 INFO plugins.dependencies :: e[33mPlugins with unsatisfied deps: ["Metabase Oracle Driver" "Metabase Vertica Driver"]e[0m
2023-05-31 01:08:24,012 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :snowflake...e[0m
2023-05-31 01:08:24,013 INFO driver.impl :: e[34mRegistered driver :snowflakee[0m (parents: [:sql-jdbc]) 🚚
2023-05-31 01:08:24,020 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :googleanalytics...e[0m
2023-05-31 01:08:24,027 INFO driver.impl :: e[34mRegistered driver :googleanalyticse[0m  🚚
2023-05-31 01:08:24,048 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :bigquery-cloud-sdk...e[0m
2023-05-31 01:08:24,056 INFO driver.impl :: e[34mRegistered driver :bigquery-cloud-sdke[0m (parents: [:sql]) 🚚
2023-05-31 01:08:24,057 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
2023-05-31 01:08:24,063 INFO db.setup :: e[36mVerifying mysql Database Connection ...e[0m
2023-05-31 01:08:24,433 INFO db.setup :: Successfully verified MySQL 8.0.28 application database connection. ✅
2023-05-31 01:08:24,436 INFO db.setup :: Running Database Migrations...
2023-05-31 01:08:24,443 INFO db.setup :: Setting up Liquibase...
2023-05-31 01:08:24,818 INFO db.setup :: Liquibase is ready.
2023-05-31 01:08:24,819 INFO db.liquibase :: Checking if Database has unrun migrations...
2023-05-31 01:08:28,066 INFO db.setup :: Database Migrations Current ...  ✅
2023-05-31 01:08:28,104 INFO db.data-migrations :: Running all necessary data migrations, this may take a minute.
2023-05-31 01:08:29,132 INFO db.data-migrations :: Finished running data migrations.
2023-05-31 01:08:29,133 INFO metabase.util :: e[32mDatabase setup took 5.1 se[0m
2023-05-31 01:08:29,278 INFO metabase.events :: Starting events listener: e[34m:metabase.events.activity-feed/ActivityFeede[0m 👂
2023-05-31 01:08:29,297 INFO metabase.events :: Starting events listener: e[34m:metabase.events.last-login/LastLogine[0m 👂
2023-05-31 01:08:29,300 INFO metabase.events :: Starting events listener: e[34m:metabase.events.persisted-info/PersistedInfoe[0m 👂
2023-05-31 01:08:29,301 INFO metabase.events :: Starting events listener: e[34m:metabase.events.sync-database/Synce[0m 👂
2023-05-31 01:08:29,302 INFO metabase.events :: Starting events listener: e[34m:metabase.events.driver-notifications/DriverNotificationse[0m 👂
2023-05-31 01:08:29,302 INFO metabase.events :: Starting events listener: e[34m:metabase.events.revision/Revisionse[0m 👂
2023-05-31 01:08:29,305 INFO metabase.events :: Starting events listener: e[34m:metabase.events.view-log/ViewLoge[0m 👂
2023-05-31 01:08:29,400 INFO driver.impl :: e[33mInitializing driver :sql...e[0m
2023-05-31 01:08:29,401 INFO driver.impl :: e[33mInitializing driver :sql-jdbc...e[0m
2023-05-31 01:08:29,402 INFO driver.impl :: e[33mInitializing driver :h2...e[0m
2023-05-31 01:08:29,530 INFO util.files :: Extract file /sample-database.db.mv.db -> /plugins/sample-database.db.mv.db
2023-05-31 01:08:29,615 INFO impl.StdSchedulerFactory :: Using default implementation for ThreadExecutor
2023-05-31 01:08:29,645 INFO core.SchedulerSignalerImpl :: Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
2023-05-31 01:08:29,646 INFO core.QuartzScheduler :: Quartz Scheduler v.2.3.2 created.
2023-05-31 01:08:29,649 INFO jdbcjobstore.JobStoreTX :: Using db table-based data access locking (synchronization).
2023-05-31 01:08:29,652 INFO jdbcjobstore.JobStoreTX :: JobStoreTX initialized.
2023-05-31 01:08:29,654 INFO core.QuartzScheduler :: Scheduler meta-data: Quartz Scheduler (v2.3.2) 'MetabaseScheduler' with instanceId 'f58d8c08a7c61685495309618'
  Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
  NOT STARTED.
  Currently in standby mode.
  Number of jobs executed: 0
  Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
  Using job-store 'org.quartz.impl.jdbcjobstore.JobStoreTX' - which supports persistence. and is clustered.

2023-05-31 01:08:29,655 INFO impl.StdSchedulerFactory :: Quartz scheduler 'MetabaseScheduler' initialized from default resource file in Quartz package: 'quartz.properties'
2023-05-31 01:08:29,656 INFO impl.StdSchedulerFactory :: Quartz scheduler version: 2.3.2
2023-05-31 01:08:29,863 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_f58d8c08a7c61685495309618 paused.
2023-05-31 01:08:29,864 INFO metabase.task :: Task scheduler initialized into standby mode.
2023-05-31 01:08:29,866 INFO metabase.task :: Initializing task e[32mSyncDatabasese[0m 📆
2023-05-31 01:08:29,945 INFO task.sync-databases :: Updated default schedules for 0 databases
2023-05-31 01:08:29,946 INFO metabase.task :: Initializing task e[32mPersistRefreshe[0m 📆
2023-05-31 01:08:29,978 INFO driver.impl :: e[33mInitializing driver :mysql...e[0m
2023-05-31 01:08:30,064 INFO metabase.task :: Initializing task e[32mCheckForNewVersionse[0m 📆
2023-05-31 01:08:30,104 INFO metabase.task :: Initializing task e[32mPersistPrunee[0m 📆
2023-05-31 01:08:30,114 INFO metabase.task :: Initializing task e[32mSendAnonymousUsageStatse[0m 📆
2023-05-31 01:08:30,140 INFO metabase.task :: Initializing task e[32mRefreshSlackChannelsAndUserse[0m 📆
2023-05-31 01:08:30,193 INFO metabase.task :: Initializing task e[32mSendPulsese[0m 📆
2023-05-31 01:08:30,228 INFO metabase.task :: Initializing task e[32mSendFollowUpEmailse[0m 📆
2023-05-31 01:08:30,260 INFO metabase.task :: Initializing task e[32mTaskHistoryCleanupe[0m 📆
2023-05-31 01:08:30,287 INFO metabase.task :: Initializing task e[32mTruncateAuditLoge[0m 📆
2023-05-31 01:08:30,318 INFO jdbcjobstore.JobStoreTX :: ClusterManager: detected 1 failed or restarted instances.
2023-05-31 01:08:30,318 INFO jdbcjobstore.JobStoreTX :: ClusterManager: Scanning for instance "ccae72e657051685391208182"'s failed in-progress jobs.
2023-05-31 01:08:30,329 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_f58d8c08a7c61685495309618 started.
2023-05-31 01:08:30,338 INFO metabase.task :: Task scheduler started
2023-05-31 01:08:30,341 INFO metabase.core :: Metabase Initialization COMPLETE
2023-05-31 01:08:30,335 INFO jdbcjobstore.JobStoreTX :: Handling 5 trigger(s) that missed their scheduled fire-time.
2023-05-31 01:08:30,446 INFO task.refresh-slack-channel-user-cache :: Slack is not configured, not refreshing slack user/channel cache.
2023-05-31 01:08:30,429 ERROR jdbcjobstore.JobStoreTX :: MisfireHandler: Error handling misfires: Couldn't store trigger 'DEFAULT.metabase.task.abandonment-emails.trigger' for 'DEFAULT.metabase.task.abandonment-emails.job' job:Couldn't retrieve job because a required class was not found: metabase.task.follow_up_emails.AbandonmentEmail
org.quartz.JobPersistenceException: Couldn't store trigger 'DEFAULT.metabase.task.abandonment-emails.trigger' for 'DEFAULT.metabase.task.abandonment-emails.job' job:Couldn't retrieve job because a required class was not found: metabase.task.follow_up_emails.AbandonmentEmail [See nested exception: org.quartz.JobPersistenceException: Couldn't retrieve job because a required class was not found: metabase.task.follow_up_emails.AbandonmentEmail [See nested exception: java.lang.ClassNotFoundException: metabase.task.follow_up_emails.AbandonmentEmail]]
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1228)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.doUpdateOfMisfiredTrigger(JobStoreSupport.java:1042)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:991)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3264)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:4012)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:4033)
Caused by: org.quartz.JobPersistenceException: Couldn't retrieve job because a required class was not found: metabase.task.follow_up_emails.AbandonmentEmail [See nested exception: java.lang.ClassNotFoundException: metabase.task.follow_up_emails.AbandonmentEmail]
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1393)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1210)
	... 5 more
Caused by: java.lang.ClassNotFoundException: metabase.task.follow_up_emails.AbandonmentEmail
	at java.base/java.net.URLClassLoader.findClass(Unknown Source)
	at clojure.lang.DynamicClassLoader.findClass(DynamicClassLoader.java:69)
	at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
	at clojure.lang.DynamicClassLoader.loadClass(DynamicClassLoader.java:77)
	at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Unknown Source)
	at metabase.task$load_class.invokeStatic(task.clj:118)
	at metabase.task$load_class.invoke(task.clj:117)
	at metabase.task.ClassLoadHelper.loadClass(task.clj:128)
	at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1390)
	... 6 more
2023-05-31 01:09:55,785 INFO i18n.impl :: Reading available locales from locales.clj...
2023-05-31 01:09:56,124 INFO util.fonts :: Reading available fonts from /frontend_client/app/fonts
2023-05-31 01:10:00,134 DEBUG middleware.log :: e[31mGET /api/user/current 401 8.0 ms (0 DB calls) 
"Unauthenticated"
e[0m
2023-05-31 01:10:00,218 DEBUG middleware.log :: e[32mGET /api/session/properties 200 85.4 ms (1 DB calls) App DB connections: 0/4 Jetty threads: 3/50 (3 idle, 0 queued) (40 total active threads) Queries in flight: 0 (0 queued)e[0m
2023-05-31 01:10:16,523 ERROR api.session :: Authentication endpoint error
clojure.lang.ExceptionInfo: Password did not match stored password. {:status-code 401, :errors {:password "did not match stored password"}}
	at metabase.api.session$do_http_401_on_error.invokeStatic(session.clj:156)
	at metabase.api.session$do_http_401_on_error.invoke(session.clj:152)
	at metabase.api.session$fn__97518.invokeStatic(session.clj:178)
	at metabase.api.session$fn__97518.invoke(session.clj:165)
	at compojure.core$wrap_response$fn__42386.invoke(core.clj:160)
	at compojure.core$wrap_route_middleware$fn__42370.invoke(core.clj:132)
	at compojure.core$wrap_route_info$fn__42375.invoke(core.clj:139)
	at compojure.core$wrap_route_matches$fn__42379.invoke(core.clj:151)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398.invoke(core.clj:200)
	at metabase.api.session$_PLUS_log_all_request_failures$fn__97579.invoke(session.clj:313)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398.invoke(core.clj:200)
	at compojure.core$make_context$handler__42426.invoke(core.clj:290)
	at compojure.core$make_context$fn__42430.invoke(core.clj:300)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$make_context$fn__42430.invoke(core.clj:301)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at metabase.api.routes$fn__98511$fn__98514.invoke(routes.clj:64)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398.invoke(core.clj:200)
	at clojure.lang.AFn.applyToHelper(AFn.java:160)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.core$apply.invokeStatic(core.clj:667)
	at clojure.core$apply.invoke(core.clj:662)
	at metabase.server.routes$fn__98672$fn__98673.doInvoke(routes.clj:70)
	at clojure.lang.RestFn.invoke(RestFn.java:436)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398.invoke(core.clj:200)
	at compojure.core$make_context$handler__42426.invoke(core.clj:290)
	at compojure.core$make_context$fn__42430.invoke(core.clj:300)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$wrap_route_matches$fn__42379.invoke(core.clj:153)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$wrap_route_matches$fn__42379.invoke(core.clj:153)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at compojure.core$wrap_route_matches$fn__42379.invoke(core.clj:153)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398$f__42399$respond_SINGLEQUOTE___42400.invoke(core.clj:197)
	at metabase.server.routes$fn__98657$fn__98659.invoke(routes.clj:47)
	at compojure.core$routes$fn__42398$f__42399.invoke(core.clj:198)
	at compojure.core$routes$fn__42398.invoke(core.clj:200)
	at metabase.server.middleware.exceptions$catch_uncaught_exceptions$fn__95304.invoke(exceptions.clj:103)
	at metabase.server.middleware.exceptions$catch_api_exceptions$fn__95301.invoke(exceptions.clj:91)
	at metabase.server.middleware.log$log_api_call$fn__100396$fn__100397$fn__100398.invoke(log.clj:216)
	at metabase.driver.sql_jdbc.execute.diagnostic$do_with_diagnostic_info.invokeStatic(diagnostic.clj:18)
	at metabase.driver.sql_jdbc.execute.diagnostic$do_with_diagnostic_info.invoke(diagnostic.clj:12)
	at metabase.server.middleware.log$log_api_call$fn__100396$fn__100397.invoke(log.clj:208)
	at toucan2.execute$do_with_call_counts.invokeStatic(execute.clj:112)
	at toucan2.execute$do_with_call_counts.invoke(execute.clj:103)
	at metabase.server.middleware.log$log_api_call$fn__100396.invoke(log.clj:207)
	at metabase.server.middleware.browser_cookie$ensure_browser_id_cookie$fn__103823.invoke(browser_cookie.clj:42)
	at metabase.server.middleware.security$add_security_headers$fn__79562.invoke(security.clj:153)
	at metabase.server.middleware.json$wrap_json_body$fn__102563.invoke(json.clj:67)
	at metabase.server.middleware.offset_paging$handle_paging$fn__79586.invoke(offset_paging.clj:45)
	at metabase.server.middleware.json$wrap_streamed_json_response$fn__102581.invoke(json.clj:103)
	at ring.middleware.keyword_params$wrap_keyword_params$fn__104090.invoke(keyword_params.clj:55)
	at ring.middleware.params$wrap_params$fn__104109.invoke(params.clj:77)
	at metabase.server.middleware.misc$maybe_set_site_url$fn__50518.invoke(misc.clj:61)
	at metabase.server.middleware.session$reset_session_timeout$fn__63863.invoke(session.clj:441)
	at metabase.server.middleware.session$bind_current_user$fn__63832$fn__63833.invoke(session.clj:336)
	at metabase.server.middleware.session$do_with_current_user.invokeStatic(session.clj:315)
	at metabase.server.middleware.session$do_with_current_user.invoke(session.clj:304)
	at metabase.server.middleware.session$bind_current_user$fn__63832.invoke(session.clj:335)
	at metabase.server.middleware.session$wrap_current_user_info$fn__63813.invoke(session.clj:285)
	at metabase.server.middleware.session$wrap_session_id$fn__63796.invoke(session.clj:217)
	at metabase.server.middleware.auth$wrap_api_key$fn__86714.invoke(auth.clj:30)
	at ring.middleware.cookies$wrap_cookies$fn__104010.invoke(cookies.clj:216)
	at metabase.server.middleware.misc$add_content_type$fn__50500.invoke(misc.clj:29)
	at metabase.server.middleware.misc$disable_streaming_buffering$fn__50526.invoke(misc.clj:78)
	at ring.middleware.gzip$wrap_gzip$fn__104052.invoke(gzip.clj:86)
	at metabase.server.middleware.misc$bind_request$fn__50529.invoke(misc.clj:95)
	at metabase.server.middleware.ssl$redirect_to_https_middleware$fn__103839.invoke(ssl.clj:41)
	at metabase.server$async_proxy_handler$fn__99162.invoke(server.clj:77)
	at metabase.server.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.Server.handle(Server.java:563)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:936)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1080)
	at java.base/java.lang.Thread.run(Unknown Source)
2023-05-31 01:10:16,550 DEBUG middleware.log :: e[31mPOST /api/session 401 301.2 ms (1 DB calls) 
{:errors {:password "did not match stored password"}}
e[0m

Ok, I have downloaded the latest JAR file and ran it locally and pointed it to our staging db and voila it worked.

So, this is undoubtedly not a migration mess or something as DB is the same and I can still log in without any issues. Any idea if it could be because we are running it from docker and upgrading the docker image is messing things up somehow?

I will try to create a new beanstalk instance that serves the JAR and see how it goes.