After running the service for a few minutes, an OOM (Out of Memory) exception occurs

The service is running in a Kubernetes (K8S) environment. The Deployment has been allocated 4 CPU cores and 4GB of memory resources.

The following is the overall service run log. Please help me analyze what is causing this issue, or where should I start to troubleshoot it.

Warning: environ value jdk-11.0.22+7 for key :java-version has been overwritten with 11.0.22

2024-05-09 19:10:05,127 INFO metabase.util :: Maximum memory available to JVM: 942.0 MB

2024-05-09 19:10:07,582 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. 🔓

For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html

2024-05-09 19:10:14,530 INFO driver.impl :: [34mRegistered abstract driver :sql[0m 🚚

2024-05-09 19:10:14,543 INFO driver.impl :: [34mRegistered abstract driver :sql-jdbc[0m (parents: [:sql]) 🚚

2024-05-09 19:10:14,548 INFO metabase.util :: [32mLoad driver :sql-jdbc took 95.9 ms[0m

2024-05-09 19:10:14,549 INFO driver.impl :: [34mRegistered driver :h2[0m (parents: [:sql-jdbc]) 🚚

2024-05-09 19:10:14,722 INFO driver.impl :: [34mRegistered driver :mysql[0m (parents: [:sql-jdbc]) 🚚

2024-05-09 19:10:14,757 INFO driver.impl :: [34mRegistered driver :postgres[0m (parents: [:sql-jdbc]) 🚚

2024-05-09 19:10:16,922 INFO metabase.core ::

Metabase v0.48.8-SNAPSHOT (4597504)

Copyright © 2024 Metabase, Inc.

Metabase Enterprise Edition extensions are NOT PRESENT.

2024-05-09 19:10:16,933 INFO metabase.core :: Starting Metabase in STANDALONE mode

2024-05-09 19:10:16,993 INFO metabase.server :: Launching Embedded Jetty Webserver with config:

{:port 3000, :host "0.0.0.0", :max-threads 100}

2024-05-09 19:10:17,061 INFO metabase.core :: Starting Metabase version v0.48.8-SNAPSHOT (4597504) ...

2024-05-09 19:10:17,069 INFO metabase.core :: System info:

{"file.encoding" "UTF-8",

"java.runtime.name" "OpenJDK Runtime Environment",

"java.runtime.version" "11.0.22+7",

"java.vendor" "Eclipse Adoptium",

"java.vendor.url" "https://adoptium.net/",

"java.version" "11.0.22",

"java.vm.name" "OpenJDK 64-Bit Server VM",

"java.vm.version" "11.0.22+7",

"os.name" "Linux",

"os.version" "5.10.134-15.1.2.lifsea8.x86_64",

"user.language" "en",

"user.timezone" "Asia/Shanghai"}

2024-05-09 19:10:17,072 INFO metabase.plugins :: Loading plugins in /plugins...

2024-05-09 19:10:17,328 INFO util.files :: Extract file /modules/sparksql.metabase-driver.jar -> /plugins/sparksql.metabase-driver.jar

2024-05-09 19:10:17,437 INFO util.files :: Extract file /modules/athena.metabase-driver.jar -> /plugins/athena.metabase-driver.jar

2024-05-09 19:10:17,556 INFO util.files :: Extract file /modules/sqlserver.metabase-driver.jar -> /plugins/sqlserver.metabase-driver.jar

2024-05-09 19:10:17,570 INFO util.files :: Extract file /modules/presto-jdbc.metabase-driver.jar -> /plugins/presto-jdbc.metabase-driver.jar

2024-05-09 19:10:17,656 INFO util.files :: Extract file /modules/sqlite.metabase-driver.jar -> /plugins/sqlite.metabase-driver.jar

2024-05-09 19:10:17,700 INFO util.files :: Extract file /modules/vertica.metabase-driver.jar -> /plugins/vertica.metabase-driver.jar

2024-05-09 19:10:17,702 INFO util.files :: Extract file /modules/redshift.metabase-driver.jar -> /plugins/redshift.metabase-driver.jar

2024-05-09 19:10:17,713 INFO util.files :: Extract file /modules/presto.metabase-driver.jar -> /plugins/presto.metabase-driver.jar

2024-05-09 19:10:17,715 INFO util.files :: Extract file /modules/druid.metabase-driver.jar -> /plugins/druid.metabase-driver.jar

2024-05-09 19:10:17,720 INFO util.files :: Extract file /modules/mongo.metabase-driver.jar -> /plugins/mongo.metabase-driver.jar

2024-05-09 19:10:17,746 INFO util.files :: Extract file /modules/oracle.metabase-driver.jar -> /plugins/oracle.metabase-driver.jar

2024-05-09 19:10:17,748 INFO util.files :: Extract file /modules/snowflake.metabase-driver.jar -> /plugins/snowflake.metabase-driver.jar

2024-05-09 19:10:18,173 INFO util.files :: Extract file /modules/googleanalytics.metabase-driver.jar -> /plugins/googleanalytics.metabase-driver.jar

2024-05-09 19:10:18,187 INFO util.files :: Extract file /modules/bigquery-cloud-sdk.metabase-driver.jar -> /plugins/bigquery-cloud-sdk.metabase-driver.jar

2024-05-09 19:10:18,376 INFO util.files :: Extract file /modules/presto-common.metabase-driver.jar -> /plugins/presto-common.metabase-driver.jar

2024-05-09 19:10:18,720 DEBUG plugins.lazy-loaded-driver :: [35mRegistering lazy loading driver :presto-jdbc...[0m

2024-05-09 19:10:18,722 INFO driver.impl :: [34mRegistered driver :presto-jdbc[0m (parents: [:sql-jdbc]) 🚚

2024-05-09 19:10:18,726 DEBUG plugins.lazy-loaded-driver :: [35mRegistering lazy loading driver :presto-common...[0m

2024-05-09 19:10:18,727 INFO driver.impl :: [34mRegistered abstract driver :presto-common[0m 🚚

2024-05-09 19:10:18,733 INFO plugins.dependencies :: [31mMetabase cannot initialize plugin Metabase Vertica Driver due to required dependencies.[0m Metabase requires the Vertica JDBC driver in order to connect to Vertica databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/vertica.html for more details.

2024-05-09 19:10:18,735 INFO plugins.dependencies :: Metabase Vertica Driver dependency {:class com.vertica.jdbc.Driver} satisfied? false

2024-05-09 19:10:18,736 INFO plugins.dependencies :: [33mPlugins with unsatisfied deps: ["Metabase Vertica Driver"][0m

2024-05-09 19:10:18,750 DEBUG plugins.lazy-loaded-driver :: [35mRegistering lazy loading driver :druid...[0m

2024-05-09 19:10:18,750 INFO driver.impl :: [34mRegistered driver :druid[0m 🚚

2024-05-09 19:10:18,756 DEBUG plugins.lazy-loaded-driver :: [35mRegistering lazy loading driver :googleanalytics...[0m

2024-05-09 19:10:18,757 INFO driver.impl :: [34mRegistered driver :googleanalytics[0m 🚚

2024-05-09 19:10:18,771 INFO plugins.dependencies :: Plugin 'Metabase Presto Driver' depends on plugin 'Presto Common Driver'

2024-05-09 19:10:18,772 INFO plugins.dependencies :: Metabase Presto Driver dependency {:plugin Presto Common Driver} satisfied? true

2024-05-09 19:10:18,777 DEBUG plugins.lazy-loaded-driver :: [35mRegistering lazy loading driver :presto...[0m

2024-05-09 19:10:18,779 INFO driver.impl :: [34mRegistered driver :presto[0m (parents: [:presto-common]) 🚚

2024-05-09 19:10:18,796 DEBUG plugins.lazy-loaded-driver :: [35mRegistering lazy loading driver :bigquery-cloud-sdk...[0m

2024-05-09 19:10:18,797 INFO driver.impl :: [34mRegistered driver :bigquery-cloud-sdk[0m (parents: [:sql]) 🚚

2024-05-09 19:10:18,813 DEBUG plugins.lazy-loaded-driver :: [35mRegistering lazy loading driver :sqlserver...[0m

2024-05-09 19:10:18,814 INFO driver.impl :: [34mRegistered driver :sqlserver[0m (parents: [:sql-jdbc]) 🚚

2024-05-09 19:10:18,835 DEBUG plugins.lazy-loaded-driver :: [35mRegistering lazy loading driver :mongo...[0m

2024-05-09 19:10:18,836 INFO driver.impl :: [34mRegistered driver :mongo[0m 🚚

2024-05-09 19:10:18,870 DEBUG plugins.lazy-loaded-driver :: [35mRegistering lazy loading driver :athena...[0m

2024-05-09 19:10:18,871 INFO driver.impl :: [34mRegistered driver :athena[0m (parents: [:sql-jdbc]) 🚚

2024-05-09 19:10:18,882 DEBUG plugins.lazy-loaded-driver :: [35mRegistering lazy loading driver :hive-like...[0m

2024-05-09 19:10:18,883 INFO driver.impl :: [34mRegistered abstract driver :hive-like[0m (parents: [:sql-jdbc]) 🚚

2024-05-09 19:10:18,884 DEBUG plugins.lazy-loaded-driver :: [35mRegistering lazy loading driver :sparksql...[0m

2024-05-09 19:10:18,884 INFO driver.impl :: [34mRegistered driver :sparksql[0m (parents: [:hive-like]) 🚚

2024-05-09 19:10:18,893 DEBUG plugins.lazy-loaded-driver :: [35mRegistering lazy loading driver :sqlite...[0m

2024-05-09 19:10:18,894 INFO driver.impl :: [34mRegistered driver :sqlite[0m (parents: [:sql-jdbc]) 🚚

2024-05-09 19:10:18,899 DEBUG plugins.lazy-loaded-driver :: [35mRegistering lazy loading driver :redshift...[0m

2024-05-09 19:10:18,899 INFO driver.impl :: [34mRegistered driver :redshift[0m (parents: [:postgres]) 🚚

2024-05-09 19:10:18,942 DEBUG plugins.lazy-loaded-driver :: [35mRegistering lazy loading driver :snowflake...[0m

2024-05-09 19:10:18,943 INFO driver.impl :: [34mRegistered driver :snowflake[0m (parents: [:sql-jdbc]) 🚚

2024-05-09 19:10:18,949 INFO plugins.dependencies :: [31mMetabase cannot initialize plugin Metabase Oracle Driver due to required dependencies.[0m Metabase requires the Oracle JDBC driver in order to connect to Oracle databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/oracle.html for more details.

2024-05-09 19:10:18,950 INFO plugins.dependencies :: Metabase Oracle Driver dependency {:class oracle.jdbc.OracleDriver} satisfied? false

2024-05-09 19:10:18,950 INFO plugins.dependencies :: [33mPlugins with unsatisfied deps: ["Metabase Oracle Driver" "Metabase Vertica Driver"][0m

2024-05-09 19:10:18,959 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...

2024-05-09 19:10:18,961 INFO db.setup :: [36mVerifying mysql Database Connection ...[0m

2024-05-09 19:10:19,007 ERROR middleware.log :: [31mGET /api/health 503 4.5 ms (0 DB calls)

{:status "initializing", :progress 0.3}

[0m

2024-05-09 19:10:19,134 INFO db.setup :: Successfully verified MySQL 8.0.13 application database connection. ✅

2024-05-09 19:10:19,134 INFO db.setup :: [36mChecking if a database downgrade is required...[0m

2024-05-09 19:10:19,718 INFO db.setup :: Running Database Migrations...

2024-05-09 19:10:19,719 INFO db.setup :: Setting up Liquibase...

2024-05-09 19:10:19,863 INFO db.setup :: Liquibase is ready.

2024-05-09 19:10:19,864 INFO db.liquibase :: Checking if Database has unrun migrations...

2024-05-09 19:10:20,437 INFO db.liquibase :: No unrun migrations found.

2024-05-09 19:10:20,438 INFO db.setup :: Database Migrations Current ... ✅

2024-05-09 19:10:20,439 INFO metabase.util :: [32mDatabase setup took 1.5 s[0m

2024-05-09 19:10:21,864 ERROR middleware.log :: [31mGET /api/health 503 470.6 µs(0个数据库调用)

{:status "initializing", :progress 0.5}

[0m

2024-05-09 19:10:21,910 ERROR middleware.log :: [31mGET /api/health 503 427.2 µs(0个数据库调用)

{:status "initializing", :progress 0.5}

[0m

2024-05-09 19:10:24,711 ERROR middleware.log :: [31mGET /api/health 503 459.2 µs(0个数据库调用)

{:status "initializing", :progress 0.5}

[0m

2024-05-09 19:10:25,202 INFO impl.StdSchedulerFactory :: Using default implementation for ThreadExecutor

2024-05-09 19:10:25,222 INFO core.SchedulerSignalerImpl :: Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl

2024-05-09 19:10:25,222 INFO core.QuartzScheduler :: Quartz Scheduler v.2.3.2 created.

2024-05-09 19:10:25,224 INFO jdbcjobstore.JobStoreTX :: Using db table-based data access locking (synchronization).

2024-05-09 19:10:25,226 INFO jdbcjobstore.JobStoreTX :: JobStoreTX initialized.

2024-05-09 19:10:25,227 INFO core.QuartzScheduler :: Scheduler meta-data: Quartz Scheduler (v2.3.2) 'MetabaseScheduler' with instanceId 'swan-bias-core-app-5477847cf6-86hxw1715253025204'

Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.

NOT STARTED.

Currently in standby mode.

Number of jobs executed: 0

Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.

Using job-store 'org.quartz.impl.jdbcjobstore.JobStoreTX' - which supports persistence. and is clustered.

2024-05-09 19:10:25,227 INFO impl.StdSchedulerFactory :: Quartz scheduler 'MetabaseScheduler' initialized from default resource file in Quartz package: 'quartz.properties'

2024-05-09 19:10:25,228 INFO impl.StdSchedulerFactory :: Quartz scheduler version: 2.3.2

2024-05-09 19:10:25,335 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_swan-bias-core-app-5477847cf6-86hxw1715253025204 paused.

2024-05-09 19:10:25,336 INFO metabase.task :: 定时任务被初始为备用模式。

2024-05-09 19:10:25,337 INFO metabase.task :: 初始化任务 [32mSyncDatabases[0m 📆

2024-05-09 19:10:25,379 INFO task.sync-databases :: 升级了0个数据库默认定时任务

2024-05-09 19:10:25,380 INFO metabase.task :: 初始化任务 [32mPersistRefresh[0m 📆

2024-05-09 19:10:25,393 INFO driver.impl :: [33m初始化驱动程序:sql...[0m

2024-05-09 19:10:25,393 INFO driver.impl :: [33m初始化驱动程序:sql-jdbc...[0m

2024-05-09 19:10:25,394 INFO driver.impl :: [33m初始化驱动程序:mysql...[0m

2024-05-09 19:10:25,418 INFO driver.impl :: [33m初始化驱动程序:mongo...[0m

2024-05-09 19:10:25,419 INFO plugins.classloader :: Added URL file:/plugins/mongo.metabase-driver.jar to classpath

2024-05-09 19:10:25,421 DEBUG plugins.init-steps :: [34m正在加载插件命名空间 metabase.driver.mongo...[0m

2024-05-09 19:10:25,649 INFO driver.impl :: [34m注册司机 :mongo[0m 🚚

2024-05-09 19:10:25,661 INFO metabase.util :: [32m[35m同步数据库:mongo时出错[0m took 240.3 ms[0m

2024-05-09 19:10:25,978 INFO metabase.task :: 初始化任务 [32mCheckForNewVersions[0m 📆

2024-05-09 19:10:26,001 INFO metabase.task :: 初始化任务 [32mPersistPrune[0m 📆

2024-05-09 19:10:26,005 INFO metabase.task :: 初始化任务 [32mSendAnonymousUsageStats[0m 📆

2024-05-09 19:10:26,021 INFO metabase.task :: 初始化任务 [32mModelIndexValues[0m 📆

2024-05-09 19:10:26,024 INFO metabase.task :: 初始化任务 [32mRefreshSlackChannelsAndUsers[0m 📆

2024-05-09 19:10:26,045 INFO metabase.task :: 初始化任务 [32mTruncateAuditTables[0m 📆

2024-05-09 19:10:26,058 INFO metabase.task :: 初始化任务 [32mSendPulses[0m 📆

2024-05-09 19:10:26,072 INFO metabase.task :: 初始化任务 [32mSendFollowUpEmails[0m 📆

2024-05-09 19:10:26,088 INFO metabase.task :: 初始化任务 [32mTaskHistoryCleanup[0m 📆

2024-05-09 19:10:26,102 INFO metabase.task :: 初始化任务 [32mSendWarnPulseRemovalEmail[0m 📆

2024-05-09 19:10:26,112 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_swan-bias-core-app-5477847cf6-86hxw1715253025204 started.

2024-05-09 19:10:26,113 INFO metabase.task :: 定时任务开始执行

2024-05-09 19:10:26,114 INFO metabase.core :: Metabase 初始化完成,用时 27.3 s

2024-05-09 19:10:26,140 INFO task.refresh-slack-channel-user-cache :: Slack没有配置,没有刷新slack用户/频道缓存。

2024-05-09 19:10:27,711 DEBUG middleware.log :: [32mGET /api/health 200 960.9 µs(0个数据库调用) 应用数据库连接:1 / 4 Jetty线程。4/100(3空闲,0排队) (31总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:31,909 DEBUG middleware.log :: [32mGET /api/health 200 488.8 µs(0个数据库调用) 应用数据库连接:1 / 4 Jetty线程。4/100(3空闲,0排队) (31总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:34,496 DEBUG middleware.log :: [32mDELETE /api/session 204 23.3 ms(2个数据库调用) 应用数据库连接:1 / 4 Jetty线程。4/100(3空闲,0排队) (31总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:35,646 DEBUG middleware.log :: [32mGET /api/user/current 200 133.6 ms(11个数据库调用) 应用数据库连接:1 / 4 Jetty线程。4/100(2空闲,0排队) (31总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:35,901 DEBUG middleware.log :: [32mGET /api/collection/root 200 159.6 ms(2个数据库调用) 应用数据库连接:6 / 10 Jetty线程。9/100(3空闲,0排队) (38总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:35,995 DEBUG middleware.log :: [32mGET /api/user/current 200 113.0 ms(10个数据库调用) 应用数据库连接:4 / 10 Jetty线程。8/100(3空闲,0排队) (38总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:36,004 DEBUG middleware.log :: [32mGET /api/activity/recent_views 200 215.0 ms(3个数据库调用) 应用数据库连接:4 / 10 Jetty线程。7/100(3空闲,0排队) (38总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:36,264 DEBUG middleware.log :: [32mGET /api/util/bug_report_details 200 472.8 ms(1个数据库调用) 应用数据库连接:2 / 10 Jetty线程。8/100(4空闲,0排队) (38总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:36,314 DEBUG middleware.log :: [32mGET /api/search 200 191.7 ms(4个数据库调用) 应用数据库连接:2 / 10 Jetty线程。7/100(5空闲,0排队) (38总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:36,368 DEBUG middleware.log :: [32mGET /api/database 200 596.8 ms(3个数据库调用) 应用数据库连接:2 / 10 Jetty线程。6/100(5空闲,0排队) (38总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:36,379 INFO metabase.events :: 加载事件命名空间: [34mmetabase.events.audit-log[0m 👂

2024-05-09 19:10:36,468 INFO metabase.events :: 加载事件命名空间: [34mmetabase.events.driver-notifications[0m 👂

2024-05-09 19:10:36,484 INFO metabase.events :: 加载事件命名空间: [34mmetabase.events.last-login[0m 👂

2024-05-09 19:10:36,509 INFO metabase.events :: 加载事件命名空间: [34mmetabase.events.persisted-info[0m 👂

2024-05-09 19:10:36,526 INFO metabase.events :: 加载事件命名空间: [34mmetabase.events.recent-views[0m 👂

2024-05-09 19:10:36,549 INFO metabase.events :: 加载事件命名空间: [34mmetabase.events.revision[0m 👂

2024-05-09 19:10:36,601 INFO metabase.events :: 加载事件命名空间: [34mmetabase.events.schema[0m 👂

2024-05-09 19:10:36,602 INFO metabase.events :: 加载事件命名空间: [34mmetabase.events.sync-database[0m 👂

2024-05-09 19:10:36,614 INFO metabase.events :: 加载事件命名空间: [34mmetabase.events.view-log[0m 👂

2024-05-09 19:10:36,837 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef 200 1.1 s(10个数据库调用) 应用数据库连接:3 / 10 Jetty线程。5/100(5空闲,0排队) (38总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:38,765 DEBUG middleware.log :: [32mGET /api/user/current 200 271.5 ms(11个数据库调用) 应用数据库连接:2 / 16 Jetty线程。6/100(8空闲,0排队) (72总活动线程) 飞行中的查询:10。 (0排队)[0m

2024-05-09 19:10:39,206 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/275553/card/305616 202 [适应性测试(ASYNC): completed] 2.3 s(23个数据库调用) 应用数据库连接:4 / 16 Jetty线程。6/100(8空闲,0排队) (87总活动线程) 飞行中的查询:8。 (0排队); mysql DB 382 connections: 0/1 (4 threads blocked)[0m

2024-05-09 19:10:39,210 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277115/card/307103 202 [适应性测试(ASYNC): completed] 2.3 s(23个数据库调用) 应用数据库连接:4 / 16 Jetty线程。6/100(8空闲,0排队) (87总活动线程) 飞行中的查询:9。 (0排队); mysql DB 382 connections: 1/1 (6 threads blocked)[0m

2024-05-09 19:10:39,222 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/275551/card/305617 202 [适应性测试(ASYNC): completed] 2.3 s(23个数据库调用) 应用数据库连接:3 / 16 Jetty线程。6/100(8空闲,0排队) (87总活动线程) 飞行中的查询:8。 (0排队); mysql DB 382 connections: 0/0 (3 threads blocked)[0m

2024-05-09 19:10:39,227 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277114/card/307107 202 [适应性测试(ASYNC): completed] 2.3 s(23个数据库调用) 应用数据库连接:6 / 16 Jetty线程。6/100(8空闲,0排队) (87总活动线程) 飞行中的查询:6。 (0排队); mysql DB 382 connections: 1/1 (4 threads blocked)[0m

2024-05-09 19:10:39,242 DEBUG middleware.log :: [32mGET /api/search 200 346.6 ms(4个数据库调用) 应用数据库连接:7 / 16 Jetty线程。6/100(8空闲,0排队) (87总活动线程) 飞行中的查询:6。 (0排队)[0m

2024-05-09 19:10:39,331 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277068/card/307058 202 [适应性测试(ASYNC): completed] 2.4 s(23个数据库调用) 应用数据库连接:5 / 16 Jetty线程。5/100(8空闲,0排队) (89总活动线程) 飞行中的查询:5。 (0排队); mysql DB 382 connections: 1/1 (4 threads blocked)[0m

2024-05-09 19:10:39,334 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277066/card/307056 202 [适应性测试(ASYNC): completed] 2.4 s(23个数据库调用) 应用数据库连接:5 / 16 Jetty线程。5/100(8空闲,0排队) (89总活动线程) 飞行中的查询:4。 (0排队); mysql DB 382 connections: 1/1 (5 threads blocked)[0m

2024-05-09 19:10:39,346 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277081/card/307071 202 [适应性测试(ASYNC): completed] 2.4 s(23个数据库调用) 应用数据库连接:6 / 16 Jetty线程。5/100(8空闲,0排队) (89总活动线程) 飞行中的查询:3。 (0排队); mysql DB 382 connections: 0/0 (2 threads blocked)[0m

2024-05-09 19:10:39,359 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277109/card/307099 202 [适应性测试(ASYNC): completed] 2.4 s(23个数据库调用) 应用数据库连接:6 / 16 Jetty线程。5/100(8空闲,0排队) (89总活动线程) 飞行中的查询:2。 (0排队); mysql DB 382 connections: 0/0 (1 threads blocked)[0m

2024-05-09 19:10:39,389 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277013/card/306998 202 [适应性测试(ASYNC): completed] 2.5 s(23个数据库调用) 应用数据库连接:5 / 16 Jetty线程。6/100(8空闲,0排队) (89总活动线程) 飞行中的查询:1。 (0排队); mysql DB 382 connections: 0/0 (0 threads blocked)[0m

2024-05-09 19:10:39,433 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277064/card/307004 202 [适应性测试(ASYNC): completed] 2.5 s(23个数据库调用) 应用数据库连接:4 / 16 Jetty线程。5/100(8空闲,0排队) (89总活动线程) 飞行中的查询:0。 (0排队); mysql DB 382 connections: 1/1 (7 threads blocked)[0m

2024-05-09 19:10:39,655 DEBUG middleware.log :: [32mGET /api/collection/tree 200 3.5 s(6个数据库调用) 应用数据库连接:2 / 16 Jetty线程。5/100(8空闲,0排队) (89总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:39,844 DEBUG middleware.log :: [32mDELETE /api/session 204 30.2 ms(2个数据库调用) 应用数据库连接:2 / 16 Jetty线程。5/100(8空闲,0排队) (89总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:40,596 DEBUG middleware.log :: [32mGET /api/collection/tree 200 1.7 s(6个数据库调用) 应用数据库连接:1 / 16 Jetty线程。4/100(8空闲,0排队) (89总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:41,909 DEBUG middleware.log :: [32mGET /api/health 200 471.8 µs(0个数据库调用) 应用数据库连接:1 / 16 Jetty线程。4/100(9空闲,0排队) (89总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:43,757 DEBUG middleware.log :: [32mGET /api/user/current 200 35.8 ms(10个数据库调用) 应用数据库连接:1 / 16 Jetty线程。4/100(9空闲,0排队) (89总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:43,940 DEBUG middleware.log :: [32mGET /api/search 200 57.8 ms(4个数据库调用) 应用数据库连接:2 / 16 Jetty线程。5/100(9空闲,0排队) (89总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:44,054 DEBUG middleware.log :: [32mGET /api/database/31/schemas 200 68.8 ms(4个数据库调用) 应用数据库连接:2 / 16 Jetty线程。5/100(9空闲,0排队) (89总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:44,368 DEBUG middleware.log :: [32mGET /api/database/31 200 19.3 ms(3个数据库调用) 应用数据库连接:2 / 16 Jetty线程。5/100(9空闲,0排队) (89总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:44,976 DEBUG middleware.log :: [32mGET /api/collection/tree 200 1.1 s(6个数据库调用) 应用数据库连接:1 / 16 Jetty线程。4/100(9空闲,0排队) (89总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:45,290 DEBUG middleware.log :: [32mGET /api/database/26/schemas 200 18.5 ms(4个数据库调用) 应用数据库连接:1 / 16 Jetty线程。4/100(9空闲,0排队) (89总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:45,672 DEBUG middleware.log :: [32mGET /api/database/26 200 12.2 ms(3个数据库调用) 应用数据库连接:1 / 16 Jetty线程。4/100(9空闲,0排队) (89总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:46,626 DEBUG middleware.log :: [32mGET /api/native-query-snippet 200 39.9 ms(5个数据库调用) 应用数据库连接:1 / 16 Jetty线程。4/100(9空闲,0排队) (89总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:50,556 DEBUG middleware.log :: [32mGET /api/database/31/schemas 200 19.8 ms(4个数据库调用) 应用数据库连接:1 / 16 Jetty线程。4/100(9空闲,0排队) (89总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:50,847 DEBUG middleware.log :: [32mGET /api/database/31 200 14.6 ms(3个数据库调用) 应用数据库连接:1 / 16 Jetty线程。4/100(9空闲,0排队) (89总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:10:51,908 DEBUG middleware.log :: [32mGET /api/health 200 317.5 µs(0个数据库调用) 应用数据库连接:1 / 16 Jetty线程。4/100(9空闲,0排队) (89总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:01,073 INFO i18n.impl :: Reading available locales from locales.clj...

2024-05-09 19:11:01,214 INFO util.fonts :: Reading available fonts from /frontend_client/app/fonts

2024-05-09 19:11:01,910 DEBUG middleware.log :: [32mGET /api/health 200 312.9 µs(0个数据库调用) 应用数据库连接:2 / 16 Jetty线程。5/100(9空闲,0排队) (89总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:02,717 INFO public-settings.premium-features :: [33mGETTING ACTIVE USER COUNT![0m

2024-05-09 19:11:02,764 INFO public-settings.premium-features :: [32m=>[0m 25211

2024-05-09 19:11:02,860 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef 200 292.3 ms(10个数据库调用) 应用数据库连接:10 / 16 Jetty线程。16/100(2空闲,0排队) (92总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:02,946 DEBUG middleware.log :: [32mGET /api/session/properties 200 318.8 ms(7个数据库调用) 应用数据库连接:11 / 16 Jetty线程。15/100(2空闲,0排队) (92总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:03,845 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/275550/card/305615 202 [适应性测试(ASYNC): completed] 1.1 s(23个数据库调用) 应用数据库连接:11 / 40 Jetty线程。13/100(7空闲,0排队) (109总活动线程) 飞行中的查询:11。 (0排队); mysql DB 382 connections: 0/10 (0 threads blocked)[0m

2024-05-09 19:11:03,926 DEBUG middleware.log :: [32mGET /api/database/31/autocomplete_suggestions 200 394.7 ms(5个数据库调用) 应用数据库连接:10 / 40 Jetty线程。13/100(7空闲,0排队) (109总活动线程) 飞行中的查询:11。 (0排队)[0m

2024-05-09 19:11:04,195 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277117/card/307109 202 [适应性测试(ASYNC): completed] 1.4 s(23个数据库调用) 应用数据库连接:7 / 40 Jetty线程。5/100(15空闲,0排队) (121总活动线程) 飞行中的查询:17。 (0排队); mysql DB 382 connections: 1/10 (0 threads blocked)[0m

2024-05-09 19:11:04,324 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/275550/card/305615 202 [适应性测试(ASYNC): completed] 1.2 s(23个数据库调用) 应用数据库连接:6 / 40 Jetty线程。4/100(16空闲,0排队) (123总活动线程) 飞行中的查询:17。 (0排队); mysql DB 382 connections: 5/10 (0 threads blocked)[0m

2024-05-09 19:11:04,384 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277069/card/307059 202 [适应性测试(ASYNC): completed] 1.6 s(23个数据库调用) 应用数据库连接:6 / 40 Jetty线程。4/100(16空闲,0排队) (123总活动线程) 飞行中的查询:16。 (0排队); mysql DB 382 connections: 2/10 (0 threads blocked)[0m

2024-05-09 19:11:04,405 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277110/card/307098 202 [适应性测试(ASYNC): completed] 1.6 s(23个数据库调用) 应用数据库连接:7 / 40 Jetty线程。5/100(15空闲,0排队) (123总活动线程) 飞行中的查询:15。 (0排队); mysql DB 382 connections: 3/10 (0 threads blocked)[0m

2024-05-09 19:11:04,430 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277067/card/307057 202 [适应性测试(ASYNC): completed] 1.6 s(23个数据库调用) 应用数据库连接:6 / 40 Jetty线程。4/100(16空闲,0排队) (123总活动线程) 飞行中的查询:14。 (0排队); mysql DB 382 connections: 6/10 (0 threads blocked)[0m

2024-05-09 19:11:04,477 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277083/card/307072 202 [适应性测试(ASYNC): completed] 1.7 s(23个数据库调用) 应用数据库连接:4 / 40 Jetty线程。4/100(16空闲,0排队) (123总活动线程) 飞行中的查询:13。 (0排队); mysql DB 382 connections: 0/10 (0 threads blocked)[0m

2024-05-09 19:11:04,570 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/275552/card/305618 202 [适应性测试(ASYNC): completed] 1.8 s(23个数据库调用) 应用数据库连接:6 / 40 Jetty线程。5/100(15空闲,0排队) (123总活动线程) 飞行中的查询:12。 (0排队); mysql DB 382 connections: 4/10 (0 threads blocked)[0m

2024-05-09 19:11:04,613 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277065/card/307005 202 [适应性测试(ASYNC): completed] 1.8 s(23个数据库调用) 应用数据库连接:6 / 40 Jetty线程。4/100(16空闲,0排队) (123总活动线程) 飞行中的查询:11。 (0排队); mysql DB 382 connections: 6/10 (0 threads blocked)[0m

2024-05-09 19:11:04,632 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277117/card/307109 202 [适应性测试(ASYNC): completed] 1.5 s(23个数据库调用) 应用数据库连接:4 / 40 Jetty线程。4/100(16空闲,0排队) (123总活动线程) 飞行中的查询:10。 (0排队); mysql DB 382 connections: 4/10 (0 threads blocked)[0m

2024-05-09 19:11:04,691 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/275552/card/305618 202 [适应性测试(ASYNC): completed] 1.6 s(23个数据库调用) 应用数据库连接:7 / 40 Jetty线程。4/100(16空闲,0排队) (123总活动线程) 飞行中的查询:9。 (0排队); mysql DB 382 connections: 6/10 (0 threads blocked)[0m

2024-05-09 19:11:04,696 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/275549/card/305619 202 [适应性测试(ASYNC): completed] 1.9 s(24个数据库调用) 应用数据库连接:7 / 40 Jetty线程。4/100(16空闲,0排队) (123总活动线程) 飞行中的查询:9。 (0排队); mysql DB 382 connections: 5/10 (0 threads blocked)[0m

2024-05-09 19:11:04,740 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277110/card/307098 202 [适应性测试(ASYNC): completed] 1.5 s(23个数据库调用) 应用数据库连接:5 / 40 Jetty线程。4/100(16空闲,0排队) (123总活动线程) 飞行中的查询:7。 (0排队); mysql DB 382 connections: 7/10 (0 threads blocked)[0m

2024-05-09 19:11:04,754 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277069/card/307059 202 [适应性测试(ASYNC): completed] 1.6 s(23个数据库调用) 应用数据库连接:4 / 40 Jetty线程。4/100(16空闲,0排队) (123总活动线程) 飞行中的查询:6。 (0排队); mysql DB 382 connections: 6/10 (0 threads blocked)[0m

2024-05-09 19:11:04,774 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277067/card/307057 202 [适应性测试(ASYNC): completed] 1.6 s(23个数据库调用) 应用数据库连接:3 / 40 Jetty线程。4/100(16空闲,0排队) (123总活动线程) 飞行中的查询:5。 (0排队); mysql DB 382 connections: 5/10 (0 threads blocked)[0m

2024-05-09 19:11:04,854 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/275549/card/305619 202 [适应性测试(ASYNC): completed] 1.7 s(24个数据库调用) 应用数据库连接:4 / 40 Jetty线程。4/100(16空闲,0排队) (123总活动线程) 飞行中的查询:4。 (0排队); mysql DB 382 connections: 6/10 (0 threads blocked)[0m

2024-05-09 19:11:04,886 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277083/card/307072 202 [适应性测试(ASYNC): completed] 1.7 s(23个数据库调用) 应用数据库连接:3 / 40 Jetty线程。4/100(16空闲,0排队) (123总活动线程) 飞行中的查询:3。 (0排队); mysql DB 382 connections: 6/10 (0 threads blocked)[0m

2024-05-09 19:11:04,931 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277018/card/307001 202 [适应性测试(ASYNC): completed] 2.1 s(24个数据库调用) 应用数据库连接:3 / 40 Jetty线程。5/100(15空闲,0排队) (123总活动线程) 飞行中的查询:2。 (0排队); mysql DB 382 connections: 7/10 (0 threads blocked)[0m

2024-05-09 19:11:05,015 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277065/card/307005 202 [适应性测试(ASYNC): completed] 1.8 s(23个数据库调用) 应用数据库连接:4 / 40 Jetty线程。4/100(16空闲,0排队) (123总活动线程) 飞行中的查询:1。 (0排队); mysql DB 382 connections: 4/10 (0 threads blocked)[0m

2024-05-09 19:11:05,039 DEBUG middleware.log :: [32mGET /api/public/dashboard/18191ede-8a63-4196-a14d-63c6e6d735ef/dashcard/277018/card/307001 202 [适应性测试(ASYNC): completed] 1.8 s(24个数据库调用) 应用数据库连接:3 / 40 Jetty线程。4/100(16空闲,0排队) (123总活动线程) 飞行中的查询:0。 (0排队); mysql DB 382 connections: 4/10 (0 threads blocked)[0m

2024-05-09 19:11:05,193 DEBUG middleware.log :: [32mDELETE /api/session 204 6.1 ms(2个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(16空闲,0排队) (123总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:08,644 DEBUG middleware.log :: [32mGET /api/database/31/autocomplete_suggestions 200 21.6 ms(5个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(16空闲,0排队) (123总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:08,798 DEBUG middleware.log :: [32mGET /api/user/current 200 86.1 ms(10个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(16空闲,0排队) (123总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:08,983 DEBUG middleware.log :: [32mGET /api/search 200 85.3 ms(4个数据库调用) 应用数据库连接:3 / 40 Jetty线程。6/100(16空闲,0排队) (123总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:10,093 DEBUG middleware.log :: [32mGET /api/timeline 200 28.5 ms(3个数据库调用) 应用数据库连接:3 / 40 Jetty线程。6/100(16空闲,0排队) (123总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:10,313 DEBUG middleware.log :: [32mGET /api/collection/tree 200 1.4 s(6个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(16空闲,0排队) (123总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:11,505 DEBUG middleware.log :: [32mGET /api/database/31/metadata 200 214.0 ms(19个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(16空闲,0排队) (123总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:11,558 DEBUG middleware.log :: [32mGET /api/database/26/schemas 200 16.4 ms(4个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(16空闲,0排队) (123总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:11,784 DEBUG middleware.log :: [32mGET /api/database/26 200 12.2 ms(3个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(16空闲,0排队) (123总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:11,908 DEBUG middleware.log :: [32mGET /api/health 200 373.0 µs(0个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(16空闲,0排队) (123总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:16,933 DEBUG middleware.log :: [32mGET /api/database/31/autocomplete_suggestions 200 27.2 ms(5个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(16空闲,0排队) (123总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:17,160 DEBUG middleware.log :: [32mGET /api/table/180/query_metadata 200 67.8 ms(9个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(16空闲,0排队) (123总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:21,910 DEBUG middleware.log :: [32mGET /api/health 200 353.2 µs(0个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(16空闲,0排队) (123总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:31,911 DEBUG middleware.log :: [32mGET /api/health 200 559.3 µs(0个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(16空闲,0排队) (123总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:34,385 DEBUG middleware.log :: [32mGET /api/session/properties 200 28.1 ms(4个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(16空闲,0排队) (123总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:38,848 DEBUG middleware.log :: [32mGET /api/session/properties 200 24.3 ms(4个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(16空闲,0排队) (118总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:39,572 DEBUG middleware.log :: [32mGET /api/bookmark 200 9.0 ms(1个数据库调用) 应用数据库连接:4 / 40 Jetty线程。8/100(15空闲,0排队) (115总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:39,602 DEBUG middleware.log :: [32mGET /api/database/31/autocomplete_suggestions 200 32.8 ms(5个数据库调用) 应用数据库连接:4 / 40 Jetty线程。7/100(15空闲,0排队) (115总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:39,617 WARN middleware.log :: [31mGET /api/collection/root 403 51.7 ms(2个数据库调用)

"抱歉,您暂无此功能的操作权限!"

[0m

2024-05-09 19:11:41,617 DEBUG middleware.log :: [32mGET /api/database 200 2.1 s(3个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(15空闲,0排队) (115总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:42,326 DEBUG middleware.log :: [32mGET /api/health 200 273.7 µs(0个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(16空闲,0排队) (115总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:43,727 DEBUG middleware.log :: [32mGET /api/database/26/metadata 200 687.4 ms(15个数据库调用) 应用数据库连接:3 / 40 Jetty线程。5/100(16空闲,0排队) (115总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:45,124 DEBUG middleware.log :: [32mGET /api/timeline 200 13.6 ms(3个数据库调用) 应用数据库连接:2 / 40 Jetty线程。6/100(16空闲,0排队) (115总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:45,155 DEBUG middleware.log :: [32mGET /api/table/158/query_metadata 200 36.9 ms(9个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(16空闲,0排队) (115总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:11:51,971 DEBUG middleware.log :: [32mGET /api/health 200 387.9 µs(0个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(16空闲,0排队) (115总活动线程) 飞行中的查询:0。 (0排队)[0m

2024-05-09 19:12:02,469 DEBUG middleware.log :: [32mGET /api/health 200 375.9 µs(0个数据库调用) 应用数据库连接:2 / 40 Jetty线程。5/100(16空闲,0排队) (115总活动线程) 飞行中的查询:0。 (0排队)[0m

Aborting due to java.lang.OutOfMemoryError: Java heap space

#

# A fatal error has been detected by the Java Runtime Environment:

#

# Internal Error (debug.cpp:339), pid=1, tid=85

# fatal error: OutOfMemory encountered: Java heap space

#

# JRE version: OpenJDK Runtime Environment Temurin-11.0.22+7 (11.0.22+7) (build 11.0.22+7)

# Java VM: OpenJDK 64-Bit Server VM Temurin-11.0.22+7 (11.0.22+7, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)

# Core dump will be written. Default location: Core dumps may be processed with " /dev/null" (or dumping to //core.1)

#

# An error report file with more information is saved as:

# /tmp/hs_err_pid1.log

Use Java opts and increase the heap size (xmx parameter)

This is the log where I have set the xmx parameter to 3072m.

From the logs, it can be seen that after startup, within 2-3 minutes, and less than 100 API calls, the service encountered an OOM issue. Do these APIs really need to consume 3GB of memory? Even 1GB seems excessively high to me.

What DB are you connected to? Since the sync and scan will require different memory needs based on what DB you are using and how is your data/schema structured

I connected to a MySQL database, version 8.0+.

When connecting to the database, I have set the sync and scan tasks to run at 2 AM, so the time period when the OOM occurs should not have any sync or scan tasks running.

Now when starting the project, I have set the parameter -XX:+HeapDumpOnOutOfMemoryError. The next time an OOM occurs, I will provide the dump file.

Do you connect the Database after the setup or during the setup? If you remove that database do you get the same OOMs? so metabase without database connected to it

After the setup.
The error does not occur every time. Our Metabase is deployed on Kubernetes, and it performs elastic scaling to meet the demands. This error occurs after the scaling, and it is not occur every time.

Can you share the memory metrics of your pod then to see if we can find any patterns in the RAM allocation


19:10 ——19:12,This purple line

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: '1'
  creationTimestamp: '2024-05-09T16:35:30Z'
  generation: 9
  managedFields:
    - apiVersion: apps/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          'f:replicas': {}
      manager: kube-controller-manager
      operation: Update
      subresource: scale
    - apiVersion: apps/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          'f:progressDeadlineSeconds': {}
          'f:revisionHistoryLimit': {}
          'f:selector': {}
          'f:strategy':
            'f:rollingUpdate':
              .: {}
              'f:maxSurge': {}
              'f:maxUnavailable': {}
            'f:type': {}
          'f:template':
            'f:metadata':
              'f:labels':
                .: {}
                'f:app': {}
            'f:spec':
              'f:containers':
                'k:{"name":"swan-metabase-core-app"}':
                  .: {}
                  'f:env':
                    .: {}
                    'k:{"name":"JAVA_OPTS"}':
                      .: {}
                      'f:name': {}
                      'f:value': {}
                    'k:{"name":"MB_APPLICATION_DB_MAX_CONNECTION_POOL_SIZE"}':
                      .: {}
                      'f:name': {}
                      'f:value': {}
                    'k:{"name":"MB_DB_DBNAME"}':
                      .: {}
                      'f:name': {}
                      'f:value': {}
                    'k:{"name":"MB_DB_HOST"}':
                      .: {}
                      'f:name': {}
                      'f:value': {}
                    'k:{"name":"MB_DB_PASS"}':
                      .: {}
                      'f:name': {}
                      'f:value': {}
                    'k:{"name":"MB_DB_PORT"}':
                      .: {}
                      'f:name': {}
                      'f:value': {}
                    'k:{"name":"MB_DB_TYPE"}':
                      .: {}
                      'f:name': {}
                      'f:value': {}
                    'k:{"name":"MB_DB_USER"}':
                      .: {}
                      'f:name': {}
                      'f:value': {}
                    'k:{"name":"MB_JDBC_DATA_WAREHOUSE_MAX_CONNECTION_POOL_SIZE"}':
                      .: {}
                      'f:name': {}
                      'f:value': {}
                    'k:{"name":"MB_JDBC_DATA_WAREHOUSE_UNRETURNED_CONNECTION_TIMEOUT_SECONDS"}':
                      .: {}
                      'f:name': {}
                      'f:value': {}
                    'k:{"name":"MB_JETTY_MAXTHREADS"}':
                      .: {}
                      'f:name': {}
                      'f:value': {}
                    'k:{"name":"TZ"}':
                      .: {}
                      'f:name': {}
                      'f:value': {}
                  'f:image': {}
                  'f:imagePullPolicy': {}
                  'f:name': {}
                  'f:ports':
                    .: {}
                    'k:{"containerPort":3000,"protocol":"TCP"}':
                      .: {}
                      'f:containerPort': {}
                      'f:protocol': {}
                  'f:readinessProbe':
                    .: {}
                    'f:failureThreshold': {}
                    'f:httpGet':
                      .: {}
                      'f:path': {}
                      'f:port': {}
                      'f:scheme': {}
                    'f:initialDelaySeconds': {}
                    'f:periodSeconds': {}
                    'f:successThreshold': {}
                    'f:timeoutSeconds': {}
                  'f:resources':
                    .: {}
                    'f:limits':
                      .: {}
                      'f:cpu': {}
                      'f:memory': {}
                  'f:terminationMessagePath': {}
                  'f:terminationMessagePolicy': {}
                  'f:volumeMounts':
                    .: {}
                    'k:{"mountPath":"/tmp/logs"}':
                      .: {}
                      'f:mountPath': {}
                      'f:name': {}
              'f:dnsPolicy': {}
              'f:restartPolicy': {}
              'f:schedulerName': {}
              'f:securityContext': {}
              'f:terminationGracePeriodSeconds': {}
              'f:volumes':
                .: {}
                'k:{"name":"metabaseapp-pvc"}':
                  .: {}
                  'f:name': {}
                  'f:persistentVolumeClaim':
                    .: {}
                    'f:claimName': {}
      manager: okhttp
      operation: Update
      time: '2024-05-09T16:35:30Z'
    - apiVersion: apps/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:deployment.kubernetes.io/revision': {}
        'f:status':
          'f:availableReplicas': {}
          'f:conditions':
            .: {}
            'k:{"type":"Available"}':
              .: {}
              'f:lastTransitionTime': {}
              'f:lastUpdateTime': {}
              'f:message': {}
              'f:reason': {}
              'f:status': {}
              'f:type': {}
            'k:{"type":"Progressing"}':
              .: {}
              'f:lastTransitionTime': {}
              'f:lastUpdateTime': {}
              'f:message': {}
              'f:reason': {}
              'f:status': {}
              'f:type': {}
          'f:observedGeneration': {}
          'f:readyReplicas': {}
          'f:replicas': {}
          'f:updatedReplicas': {}
      manager: kube-controller-manager
      operation: Update
      subresource: status
      time: '2024-05-13T11:13:49Z'
  name: swan-metabase-core-app
  namespace: default
  resourceVersion: '67276367'
  uid: 34b8e0eb-87f9-4346-910f-4f5ce6ac1591
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: swan-metabase-core-app
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: swan-metabase-core-app
    spec:
      containers:
        - env:
            - name: MB_DB_DBNAME
              value: metabase_app_prod
            - name: MB_DB_HOST
              value: 127.0.0.1
            - name: MB_DB_PASS
              value: ***********
            - name: MB_DB_PORT
              value: '3306'
            - name: MB_DB_TYPE
              value: mysql
            - name: MB_DB_USER
              value: metabase_app_prod
            - name: MB_JETTY_MAXTHREADS
              value: '100'
            - name: MB_APPLICATION_DB_MAX_CONNECTION_POOL_SIZE
              value: '100'
            - name: MB_JDBC_DATA_WAREHOUSE_MAX_CONNECTION_POOL_SIZE
              value: '30'
            - name: TZ
              value: Asia/Shanghai
            - name: MB_JDBC_DATA_WAREHOUSE_UNRETURNED_CONNECTION_TIMEOUT_SECONDS
              value: '30'
            - name: JAVA_OPTS
              value: '-Dlog4j.configurationFile=file:///tmp/customer-log4j2.xml'
          image: 'metabase/metabase:0.48.8'
          imagePullPolicy: IfNotPresent
          name: swan-metabase-core-app
          ports:
            - containerPort: 3000
              protocol: TCP
          readinessProbe:
            failureThreshold: 1
            httpGet:
              path: /api/health
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 20
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 3
          resources:
            limits:
              cpu: '4'
              memory: 4Gi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /tmp/logs
              name: metabaseapp-pvc
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
        - name: metabaseapp-pvc
          persistentVolumeClaim:
            claimName: metabaseapp-pvc
status:
  availableReplicas: 1
  conditions:
    - lastTransitionTime: '2024-05-09T16:35:30Z'
      lastUpdateTime: '2024-05-09T16:36:25Z'
      message: ReplicaSet "swan-metabase-core-app-d4b7575f6" has successfully progressed.
      reason: NewReplicaSetAvailable
      status: 'True'
      type: Progressing
    - lastTransitionTime: '2024-05-13T11:08:26Z'
      lastUpdateTime: '2024-05-13T11:08:26Z'
      message: Deployment has minimum availability.
      reason: MinimumReplicasAvailable
      status: 'True'
      type: Available
  observedGeneration: 9
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

But if you are giving 4 G how are you getting OOM? In your case the memory isn't even reaching the limit.

Are you sure the kubernetes node pool has enough resources?

The Kubernetes resources are sufficient, but the issue is that when the OOM occurred, the monitoring service crashed, so it only recorded up to 1.8G of memory usage. In reality, the service was using the full 4G of allocated memory.

When starting the project, I added the parameter -XX:+HeapDumpOnOutOfMemoryError. The next time an OOM occurs, it will record the memory usage details.

The JVM sees only 1GB, please use the XMX flag.

Is the default heap size set to 1/4 of the total memory?

This is one of the data points in the log. The service has been idle for a long time, so why are there so many active threads? They are consuming a significant amount of memory.

2024-05-20 18:40:39,627 DEBUG middleware.log :: e[32mGET /api/health 200 150.6 µs(0 DB calls) App DB connections: 1 / 46 Jetty threads。4/100(3 idle,0 queued) (1,071 total active threads) Queries in flight: 0。 (0queued) 

@Luiggi I can provide you with the OOM hprof file. Would that be helpful? If so, how should I go about providing it? The file size is approximately 8GB.

Additionally, what could be the reasons leading to excessive thread usage? Why are the threads not being properly collected/released? Which service endpoint/request is causing these threads to be opened?