Pulse failing after every 4 day run

Our pulse is getting missed after running for 4 consecutive day in a row.
And the error looks like of timezone.

Our database is snowflake which is running in PST.
I updated the Report Timezone to US/Pacific and Default Timezone both in Metabase admin setting.
However in both case no pulse were sent.

I have pasted the metabase configuration and Troubleshoot log from Metabase.

One way I am considering is to stop and rerun the metabase docker with timezone set as ‘US/Pacific’.
however we have 40 users using it and if something goes off it will be big trouble.(basically I am scared)

So before I use the option(which i am not sure if it will work).
I was wondering if there is any other way to resolve the issue.

Error log from troubleshoot screen.

{“status”:“failed”,“exception”:“class clojure.lang.ExceptionInfo”,“message”:“Input to defaulted-timezone does not match schema: \n\n\t \u001b[0;33m [(named (not (instance? metabase.models.card.CardInstance nil)) card)] \u001b[0m \n\n”,“stacktrace”:["–> pulse$fn__60819$defaulted_timezone__60824.invoke(pulse.clj:51)",“pulse$fn__60917.invokeStatic(pulse.clj:165)”,“pulse$fn__60917.invoke(pulse.clj:160)”,“pulse$results__GT_notifications$iter__60947__60951$fn__60952.invoke(pulse.clj:212)”,“pulse$send_notifications_BANG_.invokeStatic(pulse.clj:248)”,“pulse$send_notifications_BANG_.invoke(pulse.clj:247)”,“pulse$send_pulse_BANG_.invokeStatic(pulse.clj:274)”,“pulse$send_pulse_BANG_.doInvoke(pulse.clj:256)”,“task.send_pulses$fn__67957$send_pulses_BANG___67966$fn__67969$fn__67987$fn__67988.invoke(send_pulses.clj:59)”,“models.task_history$fn__41262$do_with_task_history__41267$fn__41268.invoke(task_history.clj:77)”,“models.task_history$fn__41262$do_with_task_history__41267.invoke(task_history.clj:72)”,“task.send_pulses$fn__67957$send_pulses_BANG___67966$fn__67969$fn__67987.invoke(send_pulses.clj:56)”,“task.send_pulses$fn__67957$send_pulses_BANG___67966$fn__67969.invoke(send_pulses.clj:55)”,“task.send_pulses$fn__67957$send_pulses_BANG___67966.invoke(send_pulses.clj:42)”,“task.send_pulses$fn__67957$send_pulses_BANG___67966$fn__67967.invoke(send_pulses.clj:49)”,“task.send_pulses$fn__67957$send_pulses_BANG___67966.invoke(send_pulses.clj:42)”,“task.send_pulses.SendPulses$fn__68023.invoke(send_pulses.clj:100)”,“models.task_history$fn__41262$do_with_task_history__41267$fn__41268.invoke(task_history.clj:77)”,“models.task_history$fn__41262$do_with_task_history__41267.invoke(task_history.clj:72)”,“task.send_pulses.SendPulses.execute(send_pulses.clj:86)”],“ex-data”:{“type”:“schema.core/error”,“schema”:[{“schema”:“class metabase.models.card.CardInstance”,“optional?”:false,“name”:“card”}],“value”:[null],“error”:[“schema.utils.NamedError@2b446b44”]},“original-info”:null}

Metabase configuration
{
“browser-info”: {
“language”: “en-US”,
“platform”: “MacIntel”,
“userAgent”: “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36”,
“vendor”: “Google Inc.”
},
“system-info”: {
“java.runtime.name”: “OpenJDK Runtime Environment”,
“java.runtime.version”: “11.0.3+7”,
“java.vendor”: “AdoptOpenJDK”,
“java.vendor.url”: “https://adoptopenjdk.net/”,
“java.version”: “11.0.3”,
“java.vm.name”: “OpenJDK 64-Bit Server VM”,
“java.vm.version”: “11.0.3+7”,
“os.name”: “Linux”,
“os.version”: “4.14.106-97.85.amzn2.x86_64”,
“user.language”: “en”,
“user.timezone”: “GMT”
},
“metabase-info”: {
“databases”: [
“snowflake”
],
“hosting-env”: “unknown”,
“application-database”: “postgres”,
“run-mode”: “prod”,
“version”: {
“date”: “2019-11-19”,
“tag”: “v0.33.6”,
“branch”: “release-0.33.x”,
“hash”: “be1e0e1”
},
“settings”: {
“report-timezone”: “US/Pacific”
}
}
}

Please note:
The name of the Pulse is
HelloRockerScan + Offer Select| https://@##!!!/browse/CEM-1306 | http://metabase.mycompany.com/dashboard/30 |

I have added the name as well just to ask if having special character in the name might be the reason for the issue.

Please suggest.

Hi @devD

Since Snowflake isn’t one of the database, where Report Time Zone supported, then that’s the reason why you haven’t seen any changes by changing that setting.

I would try setting the container timezone as recommended in the documentation too. You’ll have to do some testing before, so you know the results of your main queries, and then change the timezone, and verify that all results are still the same.

But you should probably open an issue on this, since pulses shouldn’t stop working after 4 days or by some timezone error.
https://github.com/metabase/metabase/issues/new/choose

The only other issue I know about, has something to do with the scheduler stops working, but that seems unrelated to your problem:
https://github.com/metabase/metabase/issues/10088

@flamber

Thanks a lot for quick response.
I have one quick question. I have two container running one is for metabase and one is for postgres (thats the metadata of metabase) and both are running in UTC.

I believe i just need to update the metabase timezone.

Please advise if I am wrong.

@devD

Yes, changing Metabase container should be enough, but consider changing both, so everything would be running PST.

But please remember to make a backup first, so if anything goes wrong or queries are not showing the correct data, then you can revert.

Timezones are really difficult, and handling multiple different ones is even harder. There are a lot of timezone fixes in Metabase the last couple of versions and even more in the upcoming 0.34.0

@flamber
I have restarted the metabase with timezone of US/Pacific. which is actually showing in docker, when i echo $JAVA_TIMEZONE. or when i see from admin page >>
Troubleshooting >> Diagnostic Info . i do see “user.timezone”: “US/Pacific” .

however when I checked date inside container it is still showing UTC.

bash-4.4# date
Tue Dec 10 03:35:52 UTC 2019

I am going to wait till tomorrow afternoon. (Thats the time of scheduled pulse).
If it work i will confirm here.

@devD Yeah, sorry, the container OS is still UTC, but that doesn’t matter.
JAVA_TIMEZONE environment variable makes JVM start with that timezone (like Inception, a JVM inside of a container inside of a host)
You could change the timezone of the container too, but that shouldn’t be a problem, since the JVM handles that change.

Unfortunately, The pulse still not sending, with the same error in Log.
I noticed we are only failing the pulse which use dates variable.

I am going to set a workaround where not to use date variable and see if it resolves the issue.

@devD Interesting. I have a feeling that your issue might be resolved in the next version, since there are some major rewrites of how timezones are handled (which also fixes many other issues). I’m not sure the release date, but 0.34.0 was expected mid-December, though it might come later.

@flamber, just an update, avoiding date variables seems to resolve the issue.
And pulse is getting sent every day on time.

I see there is a new version of metabase but I am not sure if this release will resolve the issue.
I am going to wait for some more version and then will update the metabase.

@devD If you use Google Auth, then upgrade to 0.33.7.3 right now. If you don’t use Google Auth, I would still recommend upgrading.
The major timezone fixes are in the upcoming 0.34.0, which is coming soon (probably early January, is my current guess)

Thanks for confirming.

we don’t use google Auth, but I will upgrade the metabase to latest version.

Meanwhile there is one more issue (may be I will open another ticket, but asking if you have heard about it).

Metrics and Segments are not visible on admin page. however in custom query we can still see and use it. the problem is we are not able to update the old metrics and even if we create a new one it is disappearing.

its look like its a react issue.
as below, i can see error log if inspecting the front end.

Uncaught (in promise) Error: query is a required parameter to format expressions
at app-main.bundle.js?d1dced319abf1b617b66:5
at s (app-main.bundle.js?d1dced319abf1b617b66:5)
at app-main.bundle.js?d1dced319abf1b617b66:5
at Array.map ()
at s (app-main.bundle.js?d1dced319abf1b617b66:5)
at app-main.bundle.js?d1dced319abf1b617b66:5
at Array.map ()
at s (app-main.bundle.js?d1dced319abf1b617b66:5)
at app-main.bundle.js?d1dced319abf1b617b66:5
at Array.map ()

@devD
That’s this issue - I’ve included a workaround:
https://github.com/metabase/metabase/issues/10844 - upvote by clicking :+1: on the first post

@flamber
I am so sorry the pulse failed again today.. with the same error.
only thing is this time it ran for 6 days instead of 4 days before failing.

Error message exactly the same. please and please suggest.

{"status":"failed","exception":"class clojure.lang.ExceptionInfo","message":"Input to defaulted-timezone does not match schema: \n\n\t \u001b[0;33m [(named (not (instance? metabase.models.card.CardInstance nil)) card)] \u001b[0m \n\n","stacktrace":["--> pulse$fn__60819$defaulted_timezone__60824.invoke(pulse.clj:51)","pulse$fn__60917.invokeStatic(pulse.clj:165)","pulse$fn__60917.invoke(pulse.clj:160)","pulse$results__GT_notifications$iter__60947__60951$fn__60952.invoke(pulse.clj:212)","pulse$send_notifications_BANG_.invokeStatic(pulse.clj:248)","pulse$send_notifications_BANG_.invoke(pulse.clj:247)","pulse$send_pulse_BANG_.invokeStatic(pulse.clj:274)","pulse$send_pulse_BANG_.doInvoke(pulse.clj:256)","task.send_pulses$fn__67957$send_pulses_BANG___67966$fn__67969$fn__67987$fn__67988.invoke(send_pulses.clj:59)","models.task_history$fn__41262$do_with_task_history__41267$fn__41268.invoke(task_history.clj:77)","models.task_history$fn__41262$do_with_task_history__41267.invoke(task_history.clj:72)","task.send_pulses$fn__67957$send_pulses_BANG___67966$fn__67969$fn__67987.invoke(send_pulses.clj:56)","task.send_pulses$fn__67957$send_pulses_BANG___67966$fn__67969.invoke(send_pulses.clj:55)","task.send_pulses$fn__67957$send_pulses_BANG___67966.invoke(send_pulses.clj:42)","task.send_pulses$fn__67957$send_pulses_BANG___67966$fn__67967.invoke(send_pulses.clj:49)","task.send_pulses$fn__67957$send_pulses_BANG___67966.invoke(send_pulses.clj:42)","task.send_pulses.SendPulses$fn__68023.invoke(send_pulses.clj:100)","models.task_history$fn__41262$do_with_task_history__41267$fn__41268.invoke(task_history.clj:77)","models.task_history$fn__41262$do_with_task_history__41267.invoke(task_history.clj:72)","task.send_pulses.SendPulses.execute(send_pulses.clj:86)"],"ex-data":{"type":"schema.core/error","schema":[{"schema":"class metabase.models.card.CardInstance","optional?":false,"name":"card"}],"value":[null],"error":["schema.utils.NamedError@201060e8"]},"original-info":null}

@flamber

One another issue, once the pulse fail if i navigate to the failed pulse screen . I dont see the question there again.

@devD Dang it! Well, at you’ve tried. I can only recommend that you open an issue about this on Github. Remember to describe it with details and logs. And even better if you somehow can reproduce the issue with Sample Dataset.
https://github.com/metabase/metabase/issues/new/choose

thanks for suggesting. I created a bug issue 11587.
Its a bottleneck issue now. Hoping if it can be resolved now.

Just for reference:

Thanks a lot

@devD By the way, I noticed that the Diagnostic Info you posted on the issue still says "user.timezone": "GMT", so did you try setting JAVA_TIMEZONE=PST ?

Yes i actually did and its showing as well. Once inside the container if i echo timezone

bash-4.4# echo $JAVA_TIMEZONE
US/Pacific