This morning when I logged in I got this error message:
We're experiencing server issues
Try refreshing the page after waiting a minute or two. If the problem persists we'd recommend you contact an admin.
I hadn't changed anything in my setup. Still using posgres on heroku (no recent changes on the database). Using metabase 0.20.3.
Can you check your logs for any more info?
Also, if you’re using a heroku database as the source of data to generate reports from, you should be aware that the db host they use occasionally changes. You’ll unfortunately need to update it manually.
So checked the credentials and they all seem to be fine. here is what I saw on the logs:
» 28 Dec 2016 19:35:41.483 301 <158>1 2016-12-29T00:35:41.366240+00:00 heroku router - - at=info method=PUT path="/api/database/2" host=metabasebillsync.herokuapp.com request_id=c9cc1758-ad9d-4bde-9147-077430bd0caf fwd="108.176.41.146" dyno=web.1 connect=0ms service=71ms status=200 bytes=746
» 28 Dec 2016 19:35:41.704 199 <190>1 2016-12-29T00:35:41.347393+00:00 app web.1 - - 12-29 00:35:41 e[1mDEBUG metabase.middlewaree[0m :: e[32mPUT /api/database/2 200 (60 ms) (3 DB calls)e[0m
» 28 Dec 2016 19:35:56.016 288 <158>1 2016-12-29T00:35:55.793662+00:00 heroku router - - at=info method=GET path="/" host=metabasebillsync.herokuapp.com request_id=ddc2e04f-dee0-4ce5-9f91-d505badd8c24 fwd="108.176.41.146" dyno=web.1 connect=0ms service=33ms status=200 bytes=4570
» 28 Dec 2016 19:35:57.125 310 <158>1 2016-12-29T00:35:56.954728+00:00 heroku router - - at=info method=GET path="/api/session/properties" host=metabasebillsync.herokuapp.com request_id=9100abff-9628-46ac-bb7c-5eaac4ccd478 fwd="108.176.41.146" dyno=web.1 connect=0ms service=12ms status=200 bytes=2749
» 28 Dec 2016 19:35:57.241 303 <158>1 2016-12-29T00:35:56.945363+00:00 heroku router - - at=info method=GET path="/api/user/current" host=metabasebillsync.herokuapp.com request_id=bffd7ff7-bdce-4fb8-ad70-9280319ff98f fwd="108.176.41.146" dyno=web.1 connect=0ms service=13ms status=200 bytes=590
» 28 Dec 2016 19:35:57.318 200 <190>1 2016-12-29T00:35:56.942842+00:00 app web.1 - - 12-29 00:35:56 e[1mDEBUG metabase.middlewaree[0m :: e[32mGET /api/user/current 200 (5 ms) (1 DB calls)e[0m
» 28 Dec 2016 19:35:57.318 206 <190>1 2016-12-29T00:35:56.951700+00:00 app web.1 - - 12-29 00:35:56 e[1mDEBUG metabase.middlewaree[0m :: e[32mGET /api/session/properties 200 (4 ms) (1 DB calls)e[0m
» 28 Dec 2016 19:35:57.318 211 <190>1 2016-12-29T00:35:57.076872+00:00 app web.1 - - 12-29 00:35:57 e[1mDEBUG metabase.middlewaree[0m :: e[32mGET /api/setup/admin_checklist 200 (29 ms) (10 DB calls)e[0m
» 28 Dec 2016 19:35:57.318 210 <190>1 2016-12-29T00:35:57.103512+00:00 app web.1 - - 12-29 00:35:57 e[1mDEBUG metabase.middlewaree[0m :: e[32mGET /api/activity/recent_views 200 (11 ms) (1 DB calls)e[0m
» 28 Dec 2016 19:35:57.318 287 <190>1 2016-12-29T00:35:57.149519+00:00 app web.1 - - 12-29 00:35:57 e[1mWARN query-processor.expande[0m :: e[33mSpecifying :rows as the aggregation type is deprecated in MBQL '98. This is the default behavior, so you don't need to specify it.e[0m
» 28 Dec 2016 19:35:57.318 262 <190>1 2016-12-29T00:35:57.153427+00:00 app web.1 - - 12-29 00:35:57 e[1mWARN query-processor.expande[0m :: e[33mReferring to fields by their bare ID (481) is deprecated in MBQL '98. Please use [:field-id 481] instead.e[0m
» 28 Dec 2016 19:35:57.377 313 <158>1 2016-12-29T00:35:57.078162+00:00 heroku router - - at=info method=GET path="/api/setup/admin_checklist" host=metabasebillsync.herokuapp.com request_id=d5857027-41cc-42f9-ae88-4d99e2bb18ea fwd="108.176.41.146" dyno=web.1 connect=0ms service=36ms status=200 bytes=1031
» 28 Dec 2016 19:35:57.398 262 <190>1 2016-12-29T00:35:57.169205+00:00 app web.1 - - 12-29 00:35:57 e[1mWARN query-processor.expande[0m :: e[33mReferring to fields by their bare ID (481) is deprecated in MBQL '98. Please use [:field-id 481] instead.e[0m
» 28 Dec 2016 19:35:57.398 218 <190>1 2016-12-29T00:35:57.188390+00:00 app web.1 - - 12-29 00:35:57 e[1mWARN query-processor.expande[0m :: e[33mYou shouldn't specify an :and filter with only one subclause.e[0m
» 28 Dec 2016 19:35:57.454 307 <158>1 2016-12-29T00:35:57.364886+00:00 heroku router - - at=info method=GET path="/api/dashboard?f=all" host=metabasebillsync.herokuapp.com request_id=bfdb67ed-5c0d-463f-a314-dd1d719b18c6 fwd="108.176.41.146" dyno=web.1 connect=0ms service=318ms status=200 bytes=706
» 28 Dec 2016 19:35:57.539 313 <158>1 2016-12-29T00:35:57.265719+00:00 heroku router - - at=info method=GET path="/api/activity/recent_views" host=metabasebillsync.herokuapp.com request_id=53005460-0a25-447c-b211-9cf9feb06bdb fwd="108.176.41.146" dyno=web.1 connect=0ms service=179ms status=200 bytes=905
» 28 Dec 2016 19:35:57.744 199 <190>1 2016-12-29T00:35:57.356757+00:00 app web.1 - - 12-29 00:35:57 e[1mDEBUG metabase.middlewaree[0m :: e[32mGET /api/dashboard 200 (308 ms) (2 DB calls)e[0m
» 28 Dec 2016 19:35:57.744 198 <190>1 2016-12-29T00:35:57.374553+00:00 app web.1 - - 12-29 00:35:57 e[1mDEBUG metabase.middlewaree[0m :: e[32mGET /api/activity 200 (304 ms) (7 DB calls)e[0m
» 28 Dec 2016 19:35:57.744 287 <190>1 2016-12-29T00:35:57.421379+00:00 app web.1 - - 12-29 00:35:57 e[1mWARN query-processor.expande[0m :: e[33mSpecifying :rows as the aggregation type is deprecated in MBQL '98. This is the default behavior, so you don't need to specify it.e[0m
» 28 Dec 2016 19:35:57.744 262 <190>1 2016-12-29T00:35:57.421899+00:00 app web.1 - - 12-29 00:35:57 e[1mWARN query-processor.expande[0m :: e[33mReferring to fields by their bare ID (439) is deprecated in MBQL '98. Please use [:field-id 439] instead.e[0m
» 28 Dec 2016 19:35:57.744 262 <190>1 2016-12-29T00:35:57.422854+00:00 app web.1 - - 12-29 00:35:57 e[1mWARN query-processor.expande[0m :: e[33mReferring to fields by their bare ID (494) is deprecated in MBQL '98. Please use [:field-id 494] instead.e[0m
» 28 Dec 2016 19:35:57.744 262 <190>1 2016-12-29T00:35:57.423354+00:00 app web.1 - - 12-29 00:35:57 e[1mWARN query-processor.expande[0m :: e[33mReferring to fields by their bare ID (481) is deprecated in MBQL '98. Please use [:field-id 481] instead.e[0m
» 28 Dec 2016 19:35:57.802 301 <158>1 2016-12-29T00:35:57.537642+00:00 heroku router - - at=info method=GET path="/api/activity" host=metabasebillsync.herokuapp.com request_id=6e90b23a-356f-454e-a40c-d96767a63906 fwd="108.176.41.146" dyno=web.1 connect=0ms service=484ms status=200 bytes=2694
» 28 Dec 2016 19:35:57.827 262 <190>1 2016-12-29T00:35:57.423929+00:00 app web.1 - - 12-29 00:35:57 e[1mWARN query-processor.expande[0m :: e[33mReferring to fields by their bare ID (481) is deprecated in MBQL '98. Please use [:field-id 481] instead.e[0m
» 28 Dec 2016 19:35:57.827 262 <190>1 2016-12-29T00:35:57.426462+00:00 app web.1 - - 12-29 00:35:57 e[1mWARN query-processor.expande[0m :: e[33mReferring to fields by their bare ID (424) is deprecated in MBQL '98. Please use [:field-id 424] instead.e[0m
» 28 Dec 2016 19:35:59.504 128 <45>1 2016-12-29T00:35:59.153076+00:00 heroku web.1 - - Process running mem=652M(127.4%)
» 28 Dec 2016 19:35:59.570 129 <45>1 2016-12-29T00:35:59.153152+00:00 heroku web.1 - - Error R14 (Memory quota exceeded) Critical
» 28 Dec 2016 19:35:59.713 197 <190>1 2016-12-29T00:35:59.410531+00:00 app web.1 - - 12-29 00:35:59 e[1mDEBUG metabase.middlewaree[0m :: e[32mGET /api/database 200 (41 ms) (3 DB calls)e[0m
» 28 Dec 2016 19:35:59.770 320 <158>1 2016-12-29T00:35:59.443344+00:00 heroku router - - at=info method=GET path="/api/database?include_tables=true" host=metabasebillsync.herokuapp.com request_id=2b6fb691-fa66-429a-87f0-a8557d2049ec fwd="108.176.41.146" dyno=web.1 connect=0ms service=75ms status=200 bytes=4425
» 28 Dec 2016 19:35:59.796 197 <190>1 2016-12-29T00:35:59.643700+00:00 app web.1 - - 12-29 00:35:59 e[1mDEBUG metabase.middlewaree[0m :: e[32mGET /api/card/21 200 (122 ms) (5 DB calls)e[0m
» 28 Dec 2016 19:35:59.932 299 <158>1 2016-12-29T00:35:59.653643+00:00 heroku router - - at=info method=GET path="/api/card/21" host=metabasebillsync.herokuapp.com request_id=d79c2ad6-f20f-4012-ab1d-eb588feb2d3b fwd="108.176.41.146" dyno=web.1 connect=0ms service=137ms status=200 bytes=926
» 28 Dec 2016 19:35:59.982 309 <158>1 2016-12-29T00:35:59.815169+00:00 heroku router - - at=info method=GET path="/card/snippets/text.js" host=metabasebillsync.herokuapp.com request_id=9871fe01-8b6d-4fc0-a3b5-9bce900e7843 fwd="108.176.41.146" dyno=web.1 connect=0ms service=15ms status=200 bytes=4570
» 28 Dec 2016 19:36:01.121 204 <190>1 2016-12-29T00:36:00.763709+00:00 app web.1 - - 12-29 00:36:00 e[1mERROR metabase.middlewaree[0m :: e[31mPOST /api/card/21/query 500 (665 ms) (7 DB calls)e[0m
» 28 Dec 2016 19:36:01.121 165 <190>1 2016-12-29T00:36:00.763720+00:00 app web.1 - - {:message "ERROR: permission denied for relation query_queryexecution",
» 28 Dec 2016 19:36:01.121 106 <190>1 2016-12-29T00:36:00.763721+00:00 app web.1 - - :stacktrace
» 28 Dec 2016 19:36:01.121 150 <190>1 2016-12-29T00:36:00.763722+00:00 app web.1 - - ["db$simple_insert_many_BANG_.invokeStatic(db.clj:733)"
» 28 Dec 2016 19:36:01.121 144 <190>1 2016-12-29T00:36:00.763723+00:00 app web.1 - - "db$simple_insert_many_BANG_.invoke(db.clj:720)"
» 28 Dec 2016 19:36:01.121 145 <190>1 2016-12-29T00:36:00.763723+00:00 app web.1 - - "db$simple_insert_BANG_.invokeStatic(db.clj:758)"
» 28 Dec 2016 19:36:01.121 139 <190>1 2016-12-29T00:36:00.763724+00:00 app web.1 - - "db$simple_insert_BANG_.invoke(db.clj:748)"
» 28 Dec 2016 19:36:01.121 138 <190>1 2016-12-29T00:36:00.763725+00:00 app web.1 - - "db$insert_BANG_.invokeStatic(db.clj:773)"
» 28 Dec 2016 19:36:01.121 132 <190>1 2016-12-29T00:36:00.763726+00:00 app web.1 - - "db$insert_BANG_.invoke(db.clj:762)"
» 28 Dec 2016 19:36:01.121 178 <190>1 2016-12-29T00:36:00.763727+00:00 app web.1 - - "query_processor$save_query_execution_BANG_.invokeStatic(query_processor.clj:672)"
» 28 Dec 2016 19:36:01.121 172 <190>1 2016-12-29T00:36:00.763727+00:00 app web.1 - - "query_processor$save_query_execution_BANG_.invoke(query_processor.clj:664)"
» 28 Dec 2016 19:36:01.121 162 <190>1 2016-12-29T00:36:00.763728+00:00 app web.1 - - "query_processor$query_fail.invokeStatic(query_processor.clj:640)"
» 28 Dec 2016 19:36:01.121 156 <190>1 2016-12-29T00:36:00.763729+00:00 app web.1 - - "query_processor$query_fail.invoke(query_processor.clj:628)"
» 28 Dec 2016 19:36:01.121 165 <190>1 2016-12-29T00:36:00.763729+00:00 app web.1 - - "query_processor$dataset_query.invokeStatic(query_processor.clj:626)"
» 28 Dec 2016 19:36:01.121 159 <190>1 2016-12-29T00:36:00.763730+00:00 app web.1 - - "query_processor$dataset_query.invoke(query_processor.clj:579)"
» 28 Dec 2016 19:36:01.121 152 <190>1 2016-12-29T00:36:00.763730+00:00 app web.1 - - "api.card$run_query_for_card.invokeStatic(card.clj:272)"
» 28 Dec 2016 19:36:01.121 146 <190>1 2016-12-29T00:36:00.763731+00:00 app web.1 - - "api.card$run_query_for_card.invoke(card.clj:265)"
» 28 Dec 2016 19:36:01.121 147 <190>1 2016-12-29T00:36:00.763731+00:00 app web.1 - - "api.card$fn__22759$fn__22762.invoke(card.clj:277)"
» 28 Dec 2016 19:36:01.121 174 <190>1 2016-12-29T00:36:00.763732+00:00 app web.1 - - "api.common.internal$wrap_catch_api_exceptions.invokeStatic(internal.clj:222)"
» 28 Dec 2016 19:36:01.121 168 <190>1 2016-12-29T00:36:00.763733+00:00 app web.1 - - "api.common.internal$wrap_catch_api_exceptions.invoke(internal.clj:219)"
» 28 Dec 2016 19:36:01.121 143 <190>1 2016-12-29T00:36:00.763733+00:00 app web.1 - - "api.card$fn__22759.invokeStatic(card.clj:274)"
» 28 Dec 2016 19:36:01.121 137 <190>1 2016-12-29T00:36:00.763734+00:00 app web.1 - - "api.card$fn__22759.invoke(card.clj:274)"
» 28 Dec 2016 19:36:01.121 167 <190>1 2016-12-29T00:36:00.763735+00:00 app web.1 - - "middleware$enforce_authentication$fn__30829.invoke(middleware.clj:84)"
» 28 Dec 2016 19:36:01.121 146 <190>1 2016-12-29T00:36:00.763735+00:00 app web.1 - - "api.routes$fn__30929.invokeStatic(routes.clj:40)"
» 28 Dec 2016 19:36:01.121 140 <190>1 2016-12-29T00:36:00.763736+00:00 app web.1 - - "api.routes$fn__30929.invoke(routes.clj:40)"
» 28 Dec 2016 19:36:01.121 142 <190>1 2016-12-29T00:36:00.763736+00:00 app web.1 - - "routes$fn__31914.invokeStatic(routes.clj:24)"
» 28 Dec 2016 19:36:01.121 136 <190>1 2016-12-29T00:36:00.763737+00:00 app web.1 - - "routes$fn__31914.invoke(routes.clj:24)"
» 28 Dec 2016 19:36:01.121 168 <190>1 2016-12-29T00:36:00.763738+00:00 app web.1 - - "middleware$log_api_call$fn__30917$fn__30919.invoke(middleware.clj:275)"
» 28 Dec 2016 19:36:01.121 148 <190>1 2016-12-29T00:36:00.763738+00:00 app web.1 - - "db$_do_with_call_counting.invokeStatic(db.clj:458)"
» 28 Dec 2016 19:36:01.197 142 <190>1 2016-12-29T00:36:00.763739+00:00 app web.1 - - "db$_do_with_call_counting.invoke(db.clj:452)"
» 28 Dec 2016 19:36:01.197 158 <190>1 2016-12-29T00:36:00.763740+00:00 app web.1 - - "middleware$log_api_call$fn__30917.invoke(middleware.clj:274)"
» 28 Dec 2016 19:36:01.197 166 <190>1 2016-12-29T00:36:00.763740+00:00 app web.1 - - "middleware$add_security_headers$fn__30876.invoke(middleware.clj:209)"
» 28 Dec 2016 19:36:01.197 165 <190>1 2016-12-29T00:36:00.763741+00:00 app web.1 - - "middleware$bind_current_user$fn__30833.invoke(middleware.clj:104)"],
» 28 Dec 2016 19:36:01.197 239 <190>1 2016-12-29T00:36:00.763742+00:00 app web.1 - - :sql-exception-chain ["PSQLException:" "Message: ERROR: permission denied for relation query_queryexecution" "SQLState: 42501" "Error Code: 0"]}
» 28 Dec 2016 19:36:01.197 94 <190>1 2016-12-29T00:36:00.763743+00:00 app web.1 - -
» 28 Dec 2016 19:36:01.326 307 <158>1 2016-12-29T00:36:00.765372+00:00 heroku router - - at=info method=POST path="/api/card/21/query" host=metabasebillsync.herokuapp.com request_id=d2eeb36d-ca5f-44e1-a4fb-938ccb79fc1e fwd="108.176.41.146" dyno=web.1 connect=0ms service=864ms status=500 bytes=2332
» 28 Dec 2016 19:36:01.350 156 <190>1 2016-12-29T00:36:00.015164+00:00 app web.1 - - 12-29 00:36:00 e[1mDEBUG metabase.query-processore[0m :: e[34m
» 28 Dec 2016 19:36:01.350 105 <190>1 2016-12-29T00:36:00.015175+00:00 app web.1 - - QUERY:
» 28 Dec 2016 19:36:01.350 107 <190>1 2016-12-29T00:36:00.015176+00:00 app web.1 - - {:database 2,
» 28 Dec 2016 19:36:01.350 110 <190>1 2016-12-29T00:36:00.015177+00:00 app web.1 - - :type "native",
» 28 Dec 2016 19:36:01.350 102 <190>1 2016-12-29T00:36:00.015178+00:00 app web.1 - - :native
» 28 Dec 2016 19:36:01.350 102 <190>1 2016-12-29T00:36:00.015178+00:00 app web.1 - - {:query
» 28 Dec 2016 19:36:01.350 366 <190>1 2016-12-29T00:36:00.015193+00:00 app web.1 - - "select i.user_id, u.business_name, i.id, i.amount_due\nfrom invoices as i\nleft join users u on i.user_id = u.id\nwhere i.payment_send_date < now() - interval '1 day' AND i.status = 5 AND i.created_at > date '2016-08-1' AND u.account_status = 0 and i.amount_due > 0;"},
» 28 Dec 2016 19:36:01.350 110 <190>1 2016-12-29T00:36:00.015194+00:00 app web.1 - - :parameters ,
» 28 Dec 2016 19:36:01.350 158 <190>1 2016-12-29T00:36:00.015195+00:00 app web.1 - - :constraints {:max-results 10000, :max-results-bare-rows 2000},
» 28 Dec 2016 19:36:01.350 224 <190>1 2016-12-29T00:36:00.015197+00:00 app web.1 - - :info {:executed-by 1, :card-id 21, :uuid "[deleted]", :query-hash -1931247133, :query-type "native"}}
Are you running this on a free tier DB?
Last I checked, that is maxed out at 10k records, and we typically blow through that on heavy usage. We do thinks like keep track of query executions to figure out average load times for cards, audit trails, etc.
You rock that fixed it! Would be cool to see a warning to that effect when we run out of space.
