Hello, everyone.
I've faced with the problem that some models stopped to refresh their cache (cache stores in mysql db).
Unfortunately, I have not found an appropiate solution in previous discussion.
I've tried to recreate models but it did not help. The new models have a status "Refreshing" for more than several days now. Enabling and disabling the cache via settings panel did not change the situation. The models are pretty small (about several hundreds rows per model).
I see in "jobs" section that several triggers do not start (see problem_triggers.png). The log shows the following message: "Handling 5 trigger(s) that missed their scheduled fire-time"
Also I see the error with job: "metabase.task.truncate-audit-log.job" (trigger _state = ERROR, output of query: select * from QRTZ_TRIGGERS).
Could anyone give me a hint what can cause the described problem?
Maybe there is a way to manually recreate some refreshing triggers named metabase.task.PersistenceRefresh.database.trigger ?
Metabase log
[f9e0e6d4-6c5c-4f24-8a51-0060f2eb7218] 2024-03-12T14:45:16+03:00 INFO metabase.task.persist-refresh Scheduling persistence refreshes for database 12: trigger: metabase.task.PersistenceRefresh.database.trigger.12
[f9e0e6d4-6c5c-4f24-8a51-0060f2eb7218] 2024-03-12T14:45:16+03:00 INFO metabase.task.persist-refresh Scheduling persistence refreshes for database 13: trigger: metabase.task.PersistenceRefresh.database.trigger.13
[f9e0e6d4-6c5c-4f24-8a51-0060f2eb7218] 2024-03-12T14:45:16+03:00 INFO metabase.task.persist-refresh Scheduling persistence refreshes for database 15: trigger: metabase.task.PersistenceRefresh.database.trigger.15
[f9e0e6d4-6c5c-4f24-8a51-0060f2eb7218] 2024-03-12T14:45:16+03:00 INFO metabase.task.persist-refresh Scheduling persistence refreshes for database 17: trigger: metabase.task.PersistenceRefresh.database.trigger.17
[f9e0e6d4-6c5c-4f24-8a51-0060f2eb7218] 2024-03-12T14:45:16+03:00 INFO metabase.task.persist-refresh Scheduling persistence refreshes for database 18: trigger: metabase.task.PersistenceRefresh.database.trigger.18
[f9e0e6d4-6c5c-4f24-8a51-0060f2eb7218] 2024-03-12T14:45:16+03:00 INFO metabase.task.persist-refresh Scheduling persistence refreshes for database 20: trigger: metabase.task.PersistenceRefresh.database.trigger.20
[f9e0e6d4-6c5c-4f24-8a51-0060f2eb7218] 2024-03-12T14:45:16+03:00 INFO metabase.task.persist-refresh Scheduling persistence refreshes for database 22: trigger: metabase.task.PersistenceRefresh.database.trigger.22
[f9e0e6d4-6c5c-4f24-8a51-0060f2eb7218] 2024-03-12T14:45:16+03:00 INFO metabase.task.persist-refresh Scheduling persistence refreshes for database 24: trigger: metabase.task.PersistenceRefresh.database.trigger.24
[f9e0e6d4-6c5c-4f24-8a51-0060f2eb7218] 2024-03-12T14:45:16+03:00 INFO metabase.task.persist-refresh Scheduling persistence refreshes for database 25: trigger: metabase.task.PersistenceRefresh.database.trigger.25
...
[f9e0e6d4-6c5c-4f24-8a51-0060f2eb7218] 2024-03-12T15:03:25+03:00 INFO org.quartz.impl.jdbcjobstore.JobStoreTX Handling 5 trigger(s) that missed their scheduled fire-time.
...
[f9e0e6d4-6c5c-4f24-8a51-0060f2eb7218] 2024-03-12T15:04:00+03:00 WARN metabase.task Ошибка при получении сведений о задании Quartz: metabase.task.truncate-audit-log.job
org.quartz.JobPersistenceException: Couldn't retrieve job because a required class was not found: metabase.task.truncate_audit_log.TruncateAuditLog [See nested exception: java.lang.ClassNotFoundException: metabase.task.truncate_audit_log.TruncateAuditLog]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1393)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$9.execute(JobStoreSupport.java:1382)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3864)
at org.quartz.impl.jdbcjobstore.JobStoreTX.executeInLock(JobStoreTX.java:93)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeWithoutLock(JobStoreSupport.java:3800)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1379)
at org.quartz.core.QuartzScheduler.getJobDetail(QuartzScheduler.java:1493)
at org.quartz.impl.StdScheduler.getJobDetail(StdScheduler.java:498)
at clojurewerkz.quartzite.scheduler$get_job.invokeStatic(scheduler.clj:188)
at clojurewerkz.quartzite.scheduler$get_job.invoke(scheduler.clj:185)
at metabase.task$job_info.invokeStatic(task.clj:290)
at metabase.task$job_info.invoke(task.clj:282)
at clojure.core$map$fn__5935.invoke(core.clj:2772)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:51)
at clojure.lang.RT.seq(RT.java:535)
at clojure.core$seq__5467.invokeStatic(core.clj:139)
at clojure.core$filter$fn__5962.invoke(core.clj:2826)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:51)
at clojure.lang.Cons.next(Cons.java:39)
at clojure.lang.RT.next(RT.java:713)
at clojure.core$next__5451.invokeStatic(core.clj:64)
at clojure.core.protocols$fn__8249.invokeStatic(protocols.clj:169)
at clojure.core.protocols$fn__8249.invoke(protocols.clj:124)
at clojure.core.protocols$fn__8204$G__8199__8213.invoke(protocols.clj:19)
at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:31)
at clojure.core.protocols$fn__8236.invokeStatic(protocols.clj:75)
at clojure.core.protocols$fn__8236.invoke(protocols.clj:75)
at clojure.core.protocols$fn__8178$G__8173__8191.invoke(protocols.clj:13)
at clojure.core$reduce.invokeStatic(core.clj:6886)
at clojure.core$reduce.invoke(core.clj:6868)
at cheshire.generate$generate.invokeStatic(generate.clj:133)
at cheshire.generate$generate.invoke(generate.clj:119)
at cheshire.generate$generate$fn__4046.invoke(generate.clj:125)
at clojure.core.protocols$iter_reduce.invokeStatic(protocols.clj:49)
at clojure.core.protocols$fn__8230.invokeStatic(protocols.clj:75)
at clojure.core.protocols$fn__8230.invoke(protocols.clj:75)
at clojure.core.protocols$fn__8178$G__8173__8191.invoke(protocols.clj:13)
at clojure.core$reduce.invokeStatic(core.clj:6886)
at clojure.core$reduce.invoke(core.clj:6868)
at cheshire.generate$generate.invokeStatic(generate.clj:125)
at cheshire.generate$generate.invoke(generate.clj:119)
at cheshire.core$generate_stream.invokeStatic(core.clj:106)
at cheshire.core$generate_stream.invoke(core.clj:81)
at metabase.server.middleware.json$streamed_json_response$fn__45149.invoke(json.clj:83)
at ring.util.io$piped_input_stream$fn__242.invoke(io.clj:28)
at clojure.core$binding_conveyor_fn$fn__5823.invoke(core.clj:2047)
at clojure.lang.AFn.call(AFn.java:18)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassNotFoundException: metabase.task.truncate_audit_log.TruncateAuditLog
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
at clojure.lang.DynamicClassLoader.findClass(DynamicClassLoader.java:69)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
at clojure.lang.DynamicClassLoader.loadClass(DynamicClassLoader.java:77)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at metabase.task$load_class.invokeStatic(task.clj:118)
at metabase.task$load_class.invoke(task.clj:117)
at metabase.task.ClassLoadHelper.loadClass(task.clj:128)
at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1390)
... 52 more
[f9e0e6d4-6c5c-4f24-8a51-0060f2eb7218] 2024-03-12T15:04:00+03:00 DEBUG metabase.server.middleware.log GET /api/task/info 200 2.5 ms (0 вызовы БД) Соединения с БД приложения: 1 / 10 Потоки Jetty: 6 / 50 (2 бездействует, 0 в очереди) (222 всего активных тем) Запросы в полете: 0 (0 в очереди)
[f9e0e6d4-6c5c-4f24-8a51-0060f2eb7218] 2024-03-12T15:04:00+03:00 WARN metabase.task Ошибка при получении сведений о задании Quartz: metabase.task.truncate-audit-log.job
org.quartz.JobPersistenceException: Couldn't retrieve job because a required class was not found: metabase.task.truncate_audit_log.TruncateAuditLog [See nested exception: java.lang.ClassNotFoundException: metabase.task.truncate_audit_log.TruncateAuditLog]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1393)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$9.execute(JobStoreSupport.java:1382)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3864)
at org.quartz.impl.jdbcjobstore.JobStoreTX.executeInLock(JobStoreTX.java:93)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeWithoutLock(JobStoreSupport.java:3800)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1379)
at org.quartz.core.QuartzScheduler.getJobDetail(QuartzScheduler.java:1493)
at org.quartz.impl.StdScheduler.getJobDetail(StdScheduler.java:498)
at clojurewerkz.quartzite.scheduler$get_job.invokeStatic(scheduler.clj:188)
at clojurewerkz.quartzite.scheduler$get_job.invoke(scheduler.clj:185)
at metabase.task$job_info.invokeStatic(task.clj:290)
at metabase.task$job_info.invoke(task.clj:282)
at clojure.core$map$fn__5935.invoke(core.clj:2772)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:51)
at clojure.lang.RT.seq(RT.java:535)
at clojure.core$seq__5467.invokeStatic(core.clj:139)
at clojure.core$filter$fn__5962.invoke(core.clj:2826)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:51)
at clojure.lang.Cons.next(Cons.java:39)
at clojure.lang.RT.next(RT.java:713)
at clojure.core$next__5451.invokeStatic(core.clj:64)
at clojure.core.protocols$fn__8249.invokeStatic(protocols.clj:169)
at clojure.core.protocols$fn__8249.invoke(protocols.clj:124)
at clojure.core.protocols$fn__8204$G__8199__8213.invoke(protocols.clj:19)
at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:31)
at clojure.core.protocols$fn__8236.invokeStatic(protocols.clj:75)
at clojure.core.protocols$fn__8236.invoke(protocols.clj:75)
at clojure.core.protocols$fn__8178$G__8173__8191.invoke(protocols.clj:13)
at clojure.core$reduce.invokeStatic(core.clj:6886)
at clojure.core$reduce.invoke(core.clj:6868)
at cheshire.generate$generate.invokeStatic(generate.clj:133)
at cheshire.generate$generate.invoke(generate.clj:119)
at cheshire.generate$generate$fn__4046.invoke(generate.clj:125)
at clojure.core.protocols$iter_reduce.invokeStatic(protocols.clj:49)
at clojure.core.protocols$fn__8230.invokeStatic(protocols.clj:75)
at clojure.core.protocols$fn__8230.invoke(protocols.clj:75)
at clojure.core.protocols$fn__8178$G__8173__8191.invoke(protocols.clj:13)
at clojure.core$reduce.invokeStatic(core.clj:6886)
at clojure.core$reduce.invoke(core.clj:6868)
at cheshire.generate$generate.invokeStatic(generate.clj:125)
at cheshire.generate$generate.invoke(generate.clj:119)
at cheshire.core$generate_stream.invokeStatic(core.clj:106)
at cheshire.core$generate_stream.invoke(core.clj:81)
at metabase.server.middleware.json$streamed_json_response$fn__45149.invoke(json.clj:83)
at ring.util.io$piped_input_stream$fn__242.invoke(io.clj:28)
at clojure.core$binding_conveyor_fn$fn__5823.invoke(core.clj:2047)
at clojure.lang.AFn.call(AFn.java:18)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassNotFoundException: metabase.task.truncate_audit_log.TruncateAuditLog
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
at clojure.lang.DynamicClassLoader.findClass(DynamicClassLoader.java:69)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
at clojure.lang.DynamicClassLoader.loadClass(DynamicClassLoader.java:77)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at metabase.task$load_class.invokeStatic(task.clj:118)
at metabase.task$load_class.invoke(task.clj:117)
at metabase.task.ClassLoadHelper.loadClass(task.clj:128)
at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1390)
... 52 more
Diagnostic info
{
"browser-info": {
"language": "en-US",
"platform": "Win32",
"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
"vendor": "Google Inc."
},
"system-info": {
"file.encoding": "UTF-8",
"java.runtime.name": "OpenJDK Runtime Environment",
"java.runtime.version": "11.0.20.1+1-post-Ubuntu-0ubuntu120.04",
"java.vendor": "Ubuntu",
"java.vendor.url": "https://ubuntu.com/",
"java.version": "11.0.20.1",
"java.vm.name": "OpenJDK 64-Bit Server VM",
"java.vm.version": "11.0.20.1+1-post-Ubuntu-0ubuntu120.04",
"os.name": "Linux",
"os.version": "5.4.0-152-generic",
"user.language": "en"
},
"metabase-info": {
"databases": [
"h2",
"mysql"
],
"hosting-env": "unknown",
"application-database": "mysql",
"application-database-details": {
"database": {
"name": "MySQL",
"version": "8.0.35-0ubuntu0.20.04.1"
},
"jdbc-driver": {
"name": "MariaDB Connector/J",
"version": "2.7.10"
}
},
"run-mode": "prod",
"version": {
"date": "2024-03-04",
"tag": "v0.48.8",
"hash": "a900c85"
},
"settings": {
"report-timezone": null
}
}
}