Pin map pins not loading, 500 response from /api/tiles

wghm · April 9, 2020, 12:25am

Hi there, I have a public dashboard with a pin-map. The map pins do not load, the rest of the dashboard is fine. Logged in (ie not public), viewing the pin-map shows little squares (rather than pins) and gets 200 responses from the /api/tile API. Looking at the public URL I get 500 responses from /api/tile and no map pins show.

It worked until recently. I'm not sure if it was the 0.34 -> 0.35.1 upgrade that cause this. Downgrading doesn't fix anything, but that could be because the DB has changed and I didn't also roll that back

There are not more than 350 pins. So I don't think it's some issue to do with pin quantity.

Here's what it looks like when logged in:

23%20am

And logged out (on the public dash) I just see a map with no pins.

The 500 responses I get look like this:

{"message":"Don't know how to create ISeq from: clojure.lang.ExceptionInfo","type":"class java.lang.IllegalArgumentException","stacktrace":["clojure.lang.RT.seqFrom(RT.java:557)","clojure.lang.RT.seq(RT.java:537)","clojure.lang.APersistentMap.cons(APersistentMap.java:40)","clojure.lang.RT.conj(RT.java:677)","clojure.core$conj__5390.invokeStatic(core.clj:85)","clojure.core$merge$fn__5958.invoke(core.clj:3049)","clojure.core$reduce1.invokeStatic(core.clj:944)","clojure.core$reduce1.invokeStatic(core.clj:934)","clojure.core$merge.invokeStatic(core.clj:3048)","clojure.core$merge.doInvoke(core.clj:3041)","clojure.lang.RestFn.invoke(RestFn.java:421)","--> query_processor.middleware.catch_exceptions$format_exception_STAR_.invokeStatic(catch_exceptions.clj:137)","query_processor.middleware.catch_exceptions$format_exception_STAR_.invoke(catch_exceptions.clj:134)","query_processor.middleware.catch_exceptions$catch_exceptions$fn__42190$raisef_STAR___42196.invoke(catch_exceptions.clj:164)","query_processor.middleware.process_userland_query$process_userland_query$fn__43652$raisef_STAR___43656.invoke(process_userland_query.clj:131)","query_processor.middleware.process_userland_query$process_userland_query$fn__43652.invoke(process_userland_query.clj:138)","query_processor.middleware.catch_exceptions$catch_exceptions$fn__42190.invoke(catch_exceptions.clj:166)","query_processor.reducible$async_qp$qp_STAR___37866$thunk__37867.invoke(reducible.clj:100)","query_processor.reducible$async_qp$qp_STAR___37866.invoke(reducible.clj:106)","query_processor.reducible$async_qp$qp_STAR___37866.invoke(reducible.clj:93)","query_processor.reducible$sync_qp$qp_STAR___37875$fn__37876.invoke(reducible.clj:128)","query_processor.reducible$sync_qp$qp_STAR___37875.invoke(reducible.clj:127)","query_processor$process_userland_query.invokeStatic(query_processor.clj:218)","query_processor$process_userland_query.doInvoke(query_processor.clj:214)","query_processor$fn__47048$process_query_and_save_execution_BANG___47057$fn__47058.invoke(query_processor.clj:230)","query_processor$fn__47048$process_query_and_save_execution_BANG___47057.invoke(query_processor.clj:225)","api.tiles$fn__65632.invokeStatic(tiles.clj:152)","api.tiles$fn__65632.invoke(tiles.clj:123)","middleware.auth$enforce_authentication$fn__66723.invoke(auth.clj:14)","routes$fn__68247$fn__68248.doInvoke(routes.clj:56)","middleware.exceptions$catch_uncaught_exceptions$fn__66822.invoke(exceptions.clj:97)","middleware.exceptions$catch_api_exceptions$fn__66819.invoke(exceptions.clj:85)","middleware.log$log_api_call$fn__68625$fn__68626.invoke(log.clj:197)","middleware.log$log_api_call$fn__68625.invoke(log.clj:191)","middleware.security$add_security_headers$fn__66785.invoke(security.clj:121)","middleware.json$wrap_json_body$fn__68328.invoke(json.clj:66)","middleware.json$wrap_streamed_json_response$fn__68346.invoke(json.clj:100)","middleware.misc$maybe_set_site_url$fn__68655.invoke(misc.clj:59)","middleware.session$bind_current_user$fn__63312$fn__63313.invoke(session.clj:209)","middleware.session$do_with_current_user.invokeStatic(session.clj:184)","middleware.session$do_with_current_user.invoke(session.clj:177)","middleware.session$bind_current_user$fn__63312.invoke(session.clj:208)","middleware.session$wrap_current_user_id$fn__63297.invoke(session.clj:161)","middleware.session$wrap_session_id$fn__63241.invoke(session.clj:125)","middleware.auth$wrap_api_key$fn__66731.invoke(auth.clj:27)","middleware.misc$bind_user_locale$fn__68660.invoke(misc.clj:78)","middleware.misc$add_content_type$fn__68641.invoke(misc.clj:29)","middleware.misc$disable_streaming_buffering$fn__68669.invoke(misc.clj:94)","server$async_proxy_handler$fn__68403.invoke(server.clj:70)","server.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)"]}

I can see that same stack trace in the logs:

[93f6a372-ab9c-44d8-bb60-6db1c8cc7878] 2020-04-08T18:14:38+10:00 ERROR metabase.middleware.log GET /api/tiles/2/1/0/2273/2244/0/2/ 500 94.3 ms (0 DB calls) 
{:message "Don't know how to create ISeq from: clojure.lang.ExceptionInfo",
 :type java.lang.IllegalArgumentException,
 :stacktrace
 ["clojure.lang.RT.seqFrom(RT.java:557)"
  "clojure.lang.RT.seq(RT.java:537)"
  "clojure.lang.APersistentMap.cons(APersistentMap.java:40)"
  "clojure.lang.RT.conj(RT.java:677)"
  "clojure.core$conj__5390.invokeStatic(core.clj:85)"
  "clojure.core$merge$fn__5958.invoke(core.clj:3049)"
  "clojure.core$reduce1.invokeStatic(core.clj:944)"
  "clojure.core$reduce1.invokeStatic(core.clj:934)"
  "clojure.core$merge.invokeStatic(core.clj:3048)"
  "clojure.core$merge.doInvoke(core.clj:3041)"
  "clojure.lang.RestFn.invoke(RestFn.java:421)"
  "--> query_processor.middleware.catch_exceptions$format_exception_STAR_.invokeStatic(catch_exceptions.clj:137)"
  "query_processor.middleware.catch_exceptions$format_exception_STAR_.invoke(catch_exceptions.clj:134)"
  "query_processor.middleware.catch_exceptions$catch_exceptions$fn__42190$raisef_STAR___42196.invoke(catch_exceptions.clj:164)"
  "query_processor.middleware.process_userland_query$process_userland_query$fn__43652$raisef_STAR___43656.invoke(process_userland_query.clj:131)"
  "query_processor.middleware.process_userland_query$process_userland_query$fn__43652.invoke(process_userland_query.clj:138)"
  "query_processor.middleware.catch_exceptions$catch_exceptions$fn__42190.invoke(catch_exceptions.clj:166)"
  "query_processor.reducible$async_qp$qp_STAR___37866$thunk__37867.invoke(reducible.clj:100)"
  "query_processor.reducible$async_qp$qp_STAR___37866.invoke(reducible.clj:106)"
  "query_processor.reducible$async_qp$qp_STAR___37866.invoke(reducible.clj:93)"
  "query_processor.reducible$sync_qp$qp_STAR___37875$fn__37876.invoke(reducible.clj:128)"
  "query_processor.reducible$sync_qp$qp_STAR___37875.invoke(reducible.clj:127)"
  "query_processor$process_userland_query.invokeStatic(query_processor.clj:218)"
  "query_processor$process_userland_query.doInvoke(query_processor.clj:214)"
  "query_processor$fn__47048$process_query_and_save_execution_BANG___47057$fn__47058.invoke(query_processor.clj:230)"
  "query_processor$fn__47048$process_query_and_save_execution_BANG___47057.invoke(query_processor.clj:225)"
  "api.tiles$fn__65632.invokeStatic(tiles.clj:152)"
  "api.tiles$fn__65632.invoke(tiles.clj:123)"
  "middleware.auth$enforce_authentication$fn__66723.invoke(auth.clj:14)"
  "routes$fn__68247$fn__68248.doInvoke(routes.clj:56)"
  "middleware.exceptions$catch_uncaught_exceptions$fn__66822.invoke(exceptions.clj:97)"
  "middleware.exceptions$catch_api_exceptions$fn__66819.invoke(exceptions.clj:85)"
  "middleware.log$log_api_call$fn__68625$fn__68626.invoke(log.clj:197)"
  "middleware.log$log_api_call$fn__68625.invoke(log.clj:191)"
  "middleware.security$add_security_headers$fn__66785.invoke(security.clj:121)"
  "middleware.json$wrap_json_body$fn__68328.invoke(json.clj:66)"
  "middleware.json$wrap_streamed_json_response$fn__68346.invoke(json.clj:100)"
  "middleware.misc$maybe_set_site_url$fn__68655.invoke(misc.clj:59)"
  "middleware.session$bind_current_user$fn__63312$fn__63313.invoke(session.clj:209)"
  "middleware.session$do_with_current_user.invokeStatic(session.clj:184)"
  "middleware.session$do_with_current_user.invoke(session.clj:177)"
  "middleware.session$bind_current_user$fn__63312.invoke(session.clj:208)"
  "middleware.session$wrap_current_user_id$fn__63297.invoke(session.clj:161)"
  "middleware.session$wrap_session_id$fn__63241.invoke(session.clj:125)"
  "middleware.auth$wrap_api_key$fn__66731.invoke(auth.clj:27)"
  "middleware.misc$bind_user_locale$fn__68660.invoke(misc.clj:78)"
  "middleware.misc$add_content_type$fn__68641.invoke(misc.clj:29)"
  "middleware.misc$disable_streaming_buffering$fn__68669.invoke(misc.clj:94)"
  "server$async_proxy_handler$fn__68403.invoke(server.clj:70)"
  "server.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)"]}

Diagnostic info:

{
  "browser-info": {
    "language": "en-AU",
    "platform": "MacIntel",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:75.0) Gecko/20100101 Firefox/75.0",
    "vendor": ""
  },
  "system-info": {
    "file.encoding": "UTF-8",
    "java.runtime.name": "OpenJDK Runtime Environment",
    "java.runtime.version": "11.0.5+10",
    "java.vendor": "AdoptOpenJDK",
    "java.vendor.url": "https://adoptopenjdk.net/",
    "java.version": "11.0.5",
    "java.vm.name": "OpenJDK 64-Bit Server VM",
    "java.vm.version": "11.0.5+10",
    "os.name": "Linux",
    "os.version": "4.14.158-101.185.amzn1.x86_64",
    "user.language": "en",
    "user.timezone": "GMT"
  },
  "metabase-info": {
    "databases": [
      "redshift"
    ],
    "hosting-env": "elastic-beanstalk",
    "application-database": "postgres",
    "application-database-details": {
      "database": {
        "name": "PostgreSQL",
        "version": "11.5"
      },
      "jdbc-driver": {
        "name": "PostgreSQL JDBC Driver",
        "version": "42.2.8"
      }
    },
    "run-mode": "prod",
    "version": {
      "date": "2020-04-02",
      "tag": "v0.35.1",
      "branch": "release-0.35.x",
      "hash": "e67f169"
    },
    "settings": {
      "report-timezone": "Australia/Sydney"
    }
  }
}

I also get a HUGE number of errors and warnings in JS console. However they seem to be mostly unrelated, eg Error getting setting pie._colors TypeError: "e is undefined" is probably not important (the pie chart looks fine). However I do get this:

Invariant Violation: Minified React error #44; visit http://facebook.github.io/react/docs/error-decoder.html?invariant=44 for the full message or use the non-minified dev environment for full errors and additional helpful warnings. 19 vendor.bundle.js:1:291123

Any help would be greatly appreciated!

wghm · April 9, 2020, 2:17am

Nope I was wrong, I did get to over 1000 pins. Which means I think it’s the known issue: https://github.com/metabase/metabase/issues/4394

Although that’s marked as a dupe of https://github.com/metabase/metabase/issues/7560 which looks closer to what I see. But my tiles api endpoint fails differently than both of those bugs suggest!