Unknown activities on certain period

We saw the following anomaly activities in the CloudWatch AWS.

This spike introduce an issue in Metabase instance. It becomes 503.

However, we didn't set any specific job between 15:30 - 15:40.
Does anybody know to find the root cause of this issue?

We are running version 0.47.6

Thank you

Please post logs

GET /api/collection/264 200 15.1 ms (3 DB calls) App DB connections: 0/4 Jetty threads: 3/50 (4 idle, 0 queued) (63 total active threads) Queries in flight: 0 (0 queued)
[76c268ee-42ea-4538-982d-5c3a04246cee] 2024-02-08T15:23:54+01:00 DEBUG metabase.server.middleware.log GET /api/activity/recent_views 200 22.3 ms (5 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (105 total active threads) Queries in flight: 0 (0 queued)
[2919db54-e9a1-44b6-8639-ceb61e078de5] 2024-02-08T15:23:54+01:00 ERROR metabase.query-processor.middleware.catch-exceptions Error processing query: You'll need to pick a value for 'Academy ID' before this query can run.
{:database_id 3,
 :started_at #t "2024-02-08T14:23:53.964718Z[UTC]",
 :via
 [{:status :failed,
   :class clojure.lang.ExceptionInfo,
   :error
   "Error determining value for parameter \"academy_id\": You'll need to pick a value for 'Academy ID' before this query can run.",
   :stacktrace
   ["--> driver.common.parameters.values$fn__87699$value_for_tag__87704$fn__87705.invoke(values.clj:332)"
    "driver.common.parameters.values$fn__87699$value_for_tag__87704.invoke(values.clj:325)"
    "driver.common.parameters.values$fn__87724$query__GT_params_map__87729$fn__87733$iter__87735__87739$fn__87740.invoke(values.clj:349)"
    "driver.common.parameters.values$fn__87724$query__GT_params_map__87729$fn__87733.invoke(values.clj:348)"
    "driver.common.parameters.values$fn__87724$query__GT_params_map__87729.invoke(values.clj:339)"
    "driver.sql$fn__87811$substitute_native_parameters__87803__87816$fn__87820.invoke(sql.clj:48)"
    "driver.sql$fn__87811$substitute_native_parameters__87803__87816.invoke(sql.clj:44)"
    "query_processor.middleware.parameters.native$expand_inner.invokeStatic(native.clj:41)"
    "query_processor.middleware.parameters.native$expand_inner.invoke(native.clj:32)"
    "query_processor.middleware.parameters$expand_one.invokeStatic(parameters.clj:50)"
    "query_processor.middleware.parameters$expand_one.invoke(parameters.clj:41)"
    "query_processor.middleware.parameters$expand_all$replace_75046__75047.invoke(parameters.clj:59)"
    "mbql.util.match.impl$replace_in_collection$iter__30881__30885$fn__30886.invoke(impl.cljc:44)"
    "mbql.util.match.impl$replace_in_collection.invokeStatic(impl.cljc:43)"
    "mbql.util.match.impl$replace_in_collection.invoke(impl.cljc:38)"
    "query_processor.middleware.parameters$expand_all$replace_75046__75047.invoke(parameters.clj:59)"
    "query_processor.middleware.parameters$expand_all.invokeStatic(parameters.clj:59)"
    "query_processor.middleware.parameters$expand_all.invoke(parameters.clj:53)"
    "query_processor.middleware.parameters$expand_all.invokeStatic(parameters.clj:56)"
    "query_processor.middleware.parameters$expand_all.invoke(parameters.clj:53)"
    "query_processor.middleware.parameters$expand_parameters.invokeStatic(parameters.clj:77)"
    "query_processor.middleware.parameters$expand_parameters.invoke(parameters.clj:73)"
    "query_processor.middleware.parameters$fn__75063$substitute_parameters_STAR___75068$fn__75069.invoke(parameters.clj:82)"
    "query_processor.middleware.parameters$fn__75063$substitute_parameters_STAR___75068.invoke(parameters.clj:79)"
    "query_processor.middleware.parameters$substitute_parameters.invokeStatic(parameters.clj:109)"
    "query_processor.middleware.parameters$substitute_parameters.invoke(parameters.clj:101)"
    "query_processor$preprocess_STAR_$fn__78297.invoke(query_processor.clj:167)"
    "query_processor$preprocess_STAR_.invokeStatic(query_processor.clj:165)"
    "query_processor$preprocess_STAR_.invoke(query_processor.clj:160)"
    "query_processor$fn__78305$combined_pre_process__78306$combined_pre_process_STAR___78307.invoke(query_processor.clj:257)"
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__76983$fn__76988.invoke(resolve_database_and_driver.clj:36)"
    "driver$do_with_driver.invokeStatic(driver.clj:93)"
    "driver$do_with_driver.invoke(driver.clj:88)"
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__76983.invoke(resolve_database_and_driver.clj:35)"
    "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__72965.invoke(fetch_source_query.clj:316)"
    "query_processor.middleware.store$initialize_store$fn__73146$fn__73147.invoke(store.clj:12)"
    "query_processor.store$do_with_store.invokeStatic(store.clj:56)"
    "query_processor.store$do_with_store.invoke(store.clj:50)"
    "query_processor.middleware.store$initialize_store$fn__73146.invoke(store.clj:11)"
    "query_processor.middleware.normalize_query$normalize$fn__77279.invoke(normalize_query.clj:36)"
    "query_processor.middleware.constraints$add_default_userland_constraints$fn__74673.invoke(constraints.clj:54)"
    "query_processor.middleware.process_userland_query$process_userland_query$fn__77208.invoke(process_userland_query.clj:151)"
    "query_processor.middleware.catch_exceptions$catch_exceptions$fn__77605.invoke(catch_exceptions.clj:171)"
    "query_processor.reducible$async_qp$qp_STAR___66814$thunk__66816.invoke(reducible.clj:103)"
    "query_processor.reducible$async_qp$qp_STAR___66814$fn__66818.invoke(reducible.clj:108)"],
   :error_type :missing-required-parameter,
   :ex-data
   {:tag
    {:id "94f84e50-eacc-fdea-a42f-1521166d60d4",
     :name "academy_id",
     :display-name "Academy ID",
     :type :number,
     :required true,
     :default nil},
    :type :missing-required-parameter}}
  {:status :failed,
   :class clojure.lang.ExceptionInfo,
   :error
   "Error building query parameter map: Error determining value for parameter \"academy_id\": You'll need to pick a value for 'Academy ID' before this query can run.",
   :stacktrace
   ["--> driver.common.parameters.values$fn__87724$query__GT_params_map__87729$fn__87733.invoke(values.clj:357)"
    "driver.common.parameters.values$fn__87724$query__GT_params_map__87729.invoke(values.clj:339)"
    "driver.sql$fn__87811$substitute_native_parameters__87803__87816$fn__87820.invoke(sql.clj:48)"
    "driver.sql$fn__87811$substitute_native_parameters__87803__87816.invoke(sql.clj:44)"
    "query_processor.middleware.parameters.native$expand_inner.invokeStatic(native.clj:41)"
    "query_processor.middleware.parameters.native$expand_inner.invoke(native.clj:32)"
    "query_processor.middleware.parameters$expand_one.invokeStatic(parameters.clj:50)"
    "query_processor.middleware.parameters$expand_one.invoke(parameters.clj:41)"
    "query_processor.middleware.parameters$expand_all$replace_75046__75047.invoke(parameters.clj:59)"
    "mbql.util.match.impl$replace_in_collection$iter__30881__30885$fn__30886.invoke(impl.cljc:44)"
    "mbql.util.match.impl$replace_in_collection.invokeStatic(impl.cljc:43)"
    "mbql.util.match.impl$replace_in_collection.invoke(impl.cljc:38)"
    "query_processor.middleware.parameters$expand_all$replace_75046__75047.invoke(parameters.clj:59)"
    "query_processor.middleware.parameters$expand_all.invokeStatic(parameters.clj:59)"
    "query_processor.middleware.parameters$expand_all.invoke(parameters.clj:53)"
    "query_processor.middleware.parameters$expand_all.invokeStatic(parameters.clj:56)"
    "query_processor.middleware.parameters$expand_all.invoke(parameters.clj:53)"
    "query_processor.middleware.parameters$expand_parameters.invokeStatic(parameters.clj:77)"
    "query_processor.middleware.parameters$expand_parameters.invoke(parameters.clj:73)"
    "query_processor.middleware.parameters$fn__75063$substitute_parameters_STAR___75068$fn__75069.invoke(parameters.clj:82)"
    "query_processor.middleware.parameters$fn__75063$substitute_parameters_STAR___75068.invoke(parameters.clj:79)"
    "query_processor.middleware.parameters$substitute_parameters.invokeStatic(parameters.clj:109)"
    "query_processor.middleware.parameters$substitute_parameters.invoke(parameters.clj:101)"
    "query_processor$preprocess_STAR_$fn__78297.invoke(query_processor.clj:167)"
    "query_processor$preprocess_STAR_.invokeStatic(query_processor.clj:165)"
    "query_processor$preprocess_STAR_.invoke(query_processor.clj:160)"
    "query_processor$fn__78305$combined_pre_process__78306$combined_pre_process_STAR___78307.invoke(query_processor.clj:257)"
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__76983$fn__76988.invoke(resolve_database_and_driver.clj:36)"
    "driver$do_with_driver.invokeStatic(driver.clj:93)"
    "driver$do_with_driver.invoke(driver.clj:88)"
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__76983.invoke(resolve_database_and_driver.clj:35)"
    "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__72965.invoke(fetch_source_query.clj:316)"
    "query_processor.middleware.store$initialize_store$fn__73146$fn__73147.invoke(store.clj:12)"
    "query_processor.store$do_with_store.invokeStatic(store.clj:56)"
    "query_processor.store$do_with_store.invoke(store.clj:50)"
    "query_processor.middleware.store$initialize_store$fn__73146.invoke(store.clj:11)"
    "query_processor.middleware.normalize_query$normalize$fn__77279.invoke(normalize_query.clj:36)"
    "query_processor.middleware.constraints$add_default_userland_constraints$fn__74673.invoke(constraints.clj:54)"
    "query_processor.middleware.process_userland_query$process_userland_query$fn__77208.invoke(process_userland_query.clj:151)"
    "query_processor.middleware.catch_exceptions$catch_exceptions$fn__77605.invoke(catch_exceptions.clj:171)"
    "query_processor.reducible$async_qp$qp_STAR___66814$thunk__66816.invoke(reducible.clj:103)"
    "query_processor.reducible$async_qp$qp_STAR___66814$fn__66818.invoke(reducible.clj:108)"],
   :error_type :missing-required-parameter,
   :ex-data
   {:type :missing-required-parameter,
    :tags
    {"academy_id"
     {:id "94f84e50-eacc-fdea-a42f-1521166d60d4",
      :name "academy_id",
      :display-name "Academy ID",
      :type :number,
      :required true,
      :default nil}},
    :params nil}}],
 :error_type :missing-required-parameter,
 :json_query
 {:constraints {:max-results 10000, :max-results-bare-rows 2000},
  :type :native,
  :middleware {:js-int-to-string? true, :ignore-cached-results? false, :process-viz-settings? false},
  :native
  {:template-tags
   {"academy_id"
    {:id "94f84e50-eacc-fdea-a42f-1521166d60d4",
     :name "academy_id",
     :display-name "Academy ID",
     :type :number,
     :required true,
     :default nil}},
   :query
   "select  a.id 'Academy ID', \n        a.subdomain 'Academy subdomain', \n        u.id 'User ID', \n        concat(u.first_name,' ',u.last_name) 'User name', \n        u.email 'User email', \n        u.source_type 'Source type', \n        u.source_id 'Source ID' \nfrom users u \ninner join academy_users au on au.user_id = u.id \ninner join academies a on a.id = au.academy_id \nwhere a.id = {{academy_id}}\nand u.deactivated_at is null"},
  :database 3,
  :parameters [],
  :async? true,
  :cache-ttl 1},
 :status :failed,
 :class clojure.lang.ExceptionInfo,
 :stacktrace
 ["--> driver.common.parameters.values$missing_required_param_exception.invokeStatic(values.clj:96)"
  "driver.common.parameters.values$missing_required_param_exception.invoke(values.clj:95)"
  "driver.common.parameters.values$fn__87528$param_value_for_raw_value_tag__87533$fn__87534.invoke(values.clj:215)"
  "driver.common.parameters.values$fn__87528$param_value_for_raw_value_tag__87533.invoke(values.clj:198)"
  "driver.common.parameters.values$fn__87550.invokeStatic(values.clj:220)"
  "driver.common.parameters.values$fn__87550.invoke(values.clj:218)"
  "driver.common.parameters.values$fn__87699$value_for_tag__87704$fn__87705.invoke(values.clj:330)"
  "driver.common.parameters.values$fn__87699$value_for_tag__87704.invoke(values.clj:325)"
  "driver.common.parameters.values$fn__87724$query__GT_params_map__87729$fn__87733$iter__87735__87739$fn__87740.invoke(values.clj:349)"
  "driver.common.parameters.values$fn__87724$query__GT_params_map__87729$fn__87733.invoke(values.clj:348)"
  "driver.common.parameters.values$fn__87724$query__GT_params_map__87729.invoke(values.clj:339)"
  "driver.sql$fn__87811$substitute_native_parameters__87803__87816$fn__87820.invoke(sql.clj:48)"
  "driver.sql$fn__87811$substitute_native_parameters__87803__87816.invoke(sql.clj:44)"
  "query_processor.middleware.parameters.native$expand_inner.invokeStatic(native.clj:41)"
  "query_processor.middleware.parameters.native$expand_inner.invoke(native.clj:32)"
  "query_processor.middleware.parameters$expand_one.invokeStatic(parameters.clj:50)"
  "query_processor.middleware.parameters$expand_one.invoke(parameters.clj:41)"
  "query_processor.middleware.parameters$expand_all$replace_75046__75047.invoke(parameters.clj:59)"
  "mbql.util.match.impl$replace_in_collection$iter__30881__30885$fn__30886.invoke(impl.cljc:44)"
  "mbql.util.match.impl$replace_in_collection.invokeStatic(impl.cljc:43)"
  "mbql.util.match.impl$replace_in_collection.invoke(impl.cljc:38)"
  "query_processor.middleware.parameters$expand_all$replace_75046__75047.invoke(parameters.clj:59)"
  "query_processor.middleware.parameters$expand_all.invokeStatic(parameters.clj:59)"
  "query_processor.middleware.parameters$expand_all.invoke(parameters.clj:53)"
  "query_processor.middleware.parameters$expand_all.invokeStatic(parameters.clj:56)"
  "query_processor.middleware.parameters$expand_all.invoke(parameters.clj:53)"
  "query_processor.middleware.parameters$expand_parameters.invokeStatic(parameters.clj:77)"
  "query_processor.middleware.parameters$expand_parameters.invoke(parameters.clj:73)"
  "query_processor.middleware.parameters$fn__75063$substitute_parameters_STAR___75068$fn__75069.invoke(parameters.clj:82)"
  "query_processor.middleware.parameters$fn__75063$substitute_parameters_STAR___75068.invoke(parameters.clj:79)"
  "query_processor.middleware.parameters$substitute_parameters.invokeStatic(parameters.clj:109)"
  "query_processor.middleware.parameters$substitute_parameters.invoke(parameters.clj:101)"
  "query_processor$preprocess_STAR_$fn__78297.invoke(query_processor.clj:167)"
  "query_processor$preprocess_STAR_.invokeStatic(query_processor.clj:165)"
  "query_processor$preprocess_STAR_.invoke(query_processor.clj:160)"
  "query_processor$fn__78305$combined_pre_process__78306$combined_pre_process_STAR___78307.invoke(query_processor.clj:257)"
  "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__76983$fn__76988.invoke(resolve_database_and_driver.clj:36)"
  "driver$do_with_driver.invokeStatic(driver.clj:93)"
  "driver$do_with_driver.invoke(driver.clj:88)"
  "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__76983.invoke(resolve_database_and_driver.clj:35)"
  "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__72965.invoke(fetch_source_query.clj:316)"
  "query_processor.middleware.store$initialize_store$fn__73146$fn__73147.invoke(store.clj:12)"
  "query_processor.store$do_with_store.invokeStatic(store.clj:56)"
  "query_processor.store$do_with_store.invoke(store.clj:50)"
  "query_processor.middleware.store$initialize_store$fn__73146.invoke(store.clj:11)"
  "query_processor.middleware.normalize_query$normalize$fn__77279.invoke(normalize_query.clj:36)"
  "query_processor.middleware.constraints$add_default_userland_constraints$fn__74673.invoke(constraints.clj:54)"
  "query_processor.middleware.process_userland_query$process_userland_query$fn__77208.invoke(process_userland_query.clj:151)"
  "query_processor.middleware.catch_exceptions$catch_exceptions$fn__77605.invoke(catch_exceptions.clj:171)"
  "query_processor.reducible$async_qp$qp_STAR___66814$thunk__66816.invoke(reducible.clj:103)"
  "query_processor.reducible$async_qp$qp_STAR___66814$fn__66818.invoke(reducible.clj:108)"],
 :card_id 472,
 :context :question,
 :error "You'll need to pick a value for 'Academy ID' before this query can run.",
 :row_count 0,
 :running_time 0,
 :ex-data {:type :missing-required-parameter},
 :data {:rows [], :cols []}}

[2919db54-e9a1-44b6-8639-ceb61e078de5] 2024-02-08T15:23:54+01:00 DEBUG metabase.server.middleware.log POST /api/card/472/query 202 [ASYNC: completed] 551.8 ms (9 DB calls) App DB connections: 0/15 Jetty threads: 2/50 (5 idle, 0 queued) (106 total active threads) Queries in flight: 0 (0 queued)
[2919db54-e9a1-44b6-8639-ceb61e078de5] 2024-02-08T15:23:54+01:00 DEBUG metabase.server.middleware.log GET /api/database/3 200 12.0 ms (3 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (106 total active threads) Queries in flight: 0 (0 queued)
[12b7beed-5a80-47a3-a4e1-6b9301928f82] 2024-02-08T15:23:55+01:00 DEBUG metabase.server.middleware.log GET /api/search 200 781.3 ms (3 DB calls) App DB connections: 0/4 Jetty threads: 3/50 (4 idle, 0 queued) (63 total active threads) Queries in flight: 0 (0 queued)
[76c268ee-42ea-4538-982d-5c3a04246cee] 2024-02-08T15:23:55+01:00 DEBUG metabase.server.middleware.log GET /api/search 200 223.0 ms (3 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (105 total active threads) Queries in flight: 0 (0 queued)