We recently upgraded our Beanstalk instance with a docker image of metabase version 0.46 and it all went well. We have an external DB to which I confirmed the instance is still pointing. However, all the users are now getting 401 "did not match stored password" error on login.
We have checked the log but there is nothing in there except the same error which is showing on UI. There is no way I found using which we can reset a user's password as well. There is only one error during a start-up which is as below. Although, I think it has nothing to do with the password not matching issue here. Can someone please help?
2023-04-18 14:08:02,491 ERROR jdbcjobstore.JobStoreTX :: MisfireHandler: Error handling misfires: Couldn't store trigger 'DEFAULT.metabase.task.abandonment-emails.trigger' for 'DEFAULT.metabase.task.abandonment-emails.job' job:Couldn't retrieve job because a required class was not found: metabase.task.follow_up_emails.AbandonmentEmail org.quartz.JobPersistenceException: Couldn't store trigger 'DEFAULT.metabase.task.abandonment-emails.trigger' for 'DEFAULT.metabase.task.abandonment-emails.job' job:Couldn't retrieve job because a required class was not found: metabase.task.follow_up_emails.AbandonmentEmail [See nested exception: org.quartz.JobPersistenceException: Couldn't retrieve job because a required class was not found: metabase.task.follow_up_emails.AbandonmentEmail [See nested exception: java.lang.ClassNotFoundException: metabase.task.follow_up_emails.AbandonmentEmail]] at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1228) at org.quartz.impl.jdbcjobstore.JobStoreSupport.doUpdateOfMisfiredTrigger(JobStoreSupport.java:1042) at org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:991) at org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3264) at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:4012) at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:4033) Caused by: org.quartz.JobPersistenceException: Couldn't retrieve job because a required class was not found: metabase.task.follow_up_emails.AbandonmentEmail [See nested exception: java.lang.ClassNotFoundException: metabase.task.follow_up_emails.AbandonmentEmail] at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1393) at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1210) ... 5 more Caused by: java.lang.ClassNotFoundException: metabase.task.follow_up_emails.AbandonmentEmail at java.base/java.net.URLClassLoader.findClass(Unknown Source) at clojure.lang.DynamicClassLoader.findClass(DynamicClassLoader.java:69) at java.base/java.lang.ClassLoader.loadClass(Unknown Source) at clojure.lang.DynamicClassLoader.loadClass(DynamicClassLoader.java:77) at java.base/java.lang.ClassLoader.loadClass(Unknown Source) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Unknown Source) at metabase.task$load_class.invokeStatic(task.clj:118) at metabase.task$load_class.invoke(task.clj:117) at metabase.task.ClassLoadHelper.loadClass(task.clj:128) at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852) at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1390) ... 6 more
Yes, all users are getting the same message including Admin. Not sure about the protocol used in Authentication. It's simple username password based authentication, and we don't allow users to change password by themselves, only the admin can reset it but since the admin also can't log in we are completely locked out of Metabase.
We have restored the old version on production and it helped to regain the access but we need to upgrade and have the staging instance still locked out so would be great if someone can give a pointer.
I am using external MySQL DB and not the H2 db. Can you please confirm where exactly I should be checking for Admin -> Troubleshooting -> Diagnostic Info
Yeah, I figured the Diagnostic Info can be seen after login only which I can't. And yes I already check the core_user table and everything is just the same as of production db where we are able to login. The only diff is the production have older version and staging have latest one.
I have downloaded the log from beanstalk and it looks like it is trying to connect to the H2 db instead of the MySQL. I have the environment variable set in staging instance just like production. Do you have any idea if the latest version has something changed around how to point to the external DB instance other than setting the environment variables?
I upgraded directly to 0.46. And ignore my last comment on the app is using H2 Db, I can see it is connecting to MySQL too just like prod.
So should I try to upgrade to 0.44 -> 0.45 -> 0.46? I think we first tried to upgrade to 0.46 but since it was't working we tried to downgraded to 45 -> 44 but since nothing worked we moved back to 43.
Okay so jackpot Metabase doesn't really support downgrades so make sure to load the old backup back in the staging environment ... start upgrading slowly major version after major version and make sure if works after every upgrade
This didn't work. It still shows the same error and the start-up log doesn't have a single error in it. Also, I found this sequential update only requires if we are upgrading from prior to 0.40 version. Is there any other work around you know that we can try? Thanks in advance.
Yes, we tried to upgrade from 43 to 44. And same issue in incognito as well.
I am checking table DATABASECHANGELOG and finding it only has entries till version 0.43. There are no new entries added in there for 0.44. I am not sure but would expect the migration should add some new entries for the new version in there. No?
Not sure how I can check if it's a rights issue. Won't it give at least an error in the log if that is the case?
I can see no table named persisted_info is there so definitely the DB migration is not working properly but wondering why the application is running fine if migration is failing and showing a login page
I just tried to upgrade from 43.4 to 43.7 and got this error. Any suggestion?
2023-05-02 11:04:06,672 ERROR metabase.core :: Metabase Initialization FAILED
liquibase.exception.ValidationFailedException: Validation Failed:
1 change sets check sum
migrations/000_migrations.yaml::268::rlotun was: 8:4ccc4d50f9b233bc6515780d9cae360b but is now: 8:9da2f706a7cd42b5101601e0106fa929
at liquibase.changelog.DatabaseChangeLog.validate(DatabaseChangeLog.java:296)
at liquibase.Liquibase$16.run(Liquibase.java:1988)
at liquibase.Scope.lambda$child$0(Scope.java:180)
at liquibase.Scope.child(Scope.java:189)
at liquibase.Scope.child(Scope.java:179)
at liquibase.Scope.child(Scope.java:158)
at liquibase.Liquibase.runInScope(Liquibase.java:2405)
at liquibase.Liquibase.listUnrunChangeSets(Liquibase.java:1978)
at liquibase.Liquibase.listUnrunChangeSets(Liquibase.java:1969)
at metabase.db.liquibase$has_unrun_migrations_QMARK_.invokeStatic(liquibase.clj:109)
at metabase.db.liquibase$has_unrun_migrations_QMARK_.invoke(liquibase.clj:99)
at metabase.db.liquibase$migrate_up_if_needed_BANG_.invokeStatic(liquibase.clj:134)
at metabase.db.liquibase$migrate_up_if_needed_BANG_.invoke(liquibase.clj:130)
at metabase.db.setup$fn__34835$migrate_BANG___34840$fn__34841$fn__34842.invoke(setup.clj:68)
at metabase.db.liquibase$fn__30492$do_with_liquibase__30497$fn__30498.invoke(liquibase.clj:59)
at metabase.db.liquibase$fn__30492$do_with_liquibase__30497.invoke(liquibase.clj:51)
at metabase.db.setup$fn__34835$migrate_BANG___34840$fn__34841.invoke(setup.clj:63)
at metabase.db.setup$fn__34835$migrate_BANG___34840.invoke(setup.clj:40)
at metabase.db.setup$fn__34894$run_schema_migrations_BANG___34899$fn__34900.invoke(setup.clj:121)
at metabase.db.setup$fn__34894$run_schema_migrations_BANG___34899.invoke(setup.clj:115)
at metabase.db.setup$fn__34946$setup_db_BANG___34951$fn__34952$fn__34955$fn__34956.invoke(setup.clj:147)
at metabase.util$do_with_us_locale.invokeStatic(util.clj:724)
at metabase.util$do_with_us_locale.invoke(util.clj:710)
at metabase.db.setup$fn__34946$setup_db_BANG___34951$fn__34952$fn__34955.invoke(setup.clj:145)
at metabase.db.setup$fn__34946$setup_db_BANG___34951$fn__34952.invoke(setup.clj:144)
at metabase.db.setup$fn__34946$setup_db_BANG___34951.invoke(setup.clj:138)
at metabase.db$setup_db_BANG_$fn__34981.invoke(db.clj:65)
at metabase.db$setup_db_BANG_.invokeStatic(db.clj:60)
at metabase.db$setup_db_BANG_.invoke(db.clj:51)
at metabase.core$init_BANG__STAR_.invokeStatic(core.clj:98)
at metabase.core$init_BANG__STAR_.invoke(core.clj:81)
at metabase.core$init_BANG_.invokeStatic(core.clj:138)
at metabase.core$init_BANG_.invoke(core.clj:133)
at metabase.core$start_normally.invokeStatic(core.clj:150)
at metabase.core$start_normally.invoke(core.clj:144)
at metabase.core$_main.invokeStatic(core.clj:183)
at metabase.core$_main.doInvoke(core.clj:177)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at metabase.core.main(Unknown Source)
Metabase doesn't really support downgrades so make sure to load the old backup back in the staging environment ... start upgrading slowly major version after major version and make sure if works after every upgrade
