Pulse failing after every 4 day run

@devD

Yes, changing Metabase container should be enough, but consider changing both, so everything would be running PST.

But please remember to make a backup first, so if anything goes wrong or queries are not showing the correct data, then you can revert.

Timezones are really difficult, and handling multiple different ones is even harder. There are a lot of timezone fixes in Metabase the last couple of versions and even more in the upcoming 0.34.0

@flamber
I have restarted the metabase with timezone of US/Pacific. which is actually showing in docker, when i echo $JAVA_TIMEZONE. or when i see from admin page >>
Troubleshooting >> Diagnostic Info . i do see “user.timezone”: “US/Pacific” .

however when I checked date inside container it is still showing UTC.

bash-4.4# date
Tue Dec 10 03:35:52 UTC 2019

I am going to wait till tomorrow afternoon. (Thats the time of scheduled pulse).
If it work i will confirm here.

@devD Yeah, sorry, the container OS is still UTC, but that doesn’t matter.
JAVA_TIMEZONE environment variable makes JVM start with that timezone (like Inception, a JVM inside of a container inside of a host)
You could change the timezone of the container too, but that shouldn’t be a problem, since the JVM handles that change.

Unfortunately, The pulse still not sending, with the same error in Log.
I noticed we are only failing the pulse which use dates variable.

I am going to set a workaround where not to use date variable and see if it resolves the issue.

@devD Interesting. I have a feeling that your issue might be resolved in the next version, since there are some major rewrites of how timezones are handled (which also fixes many other issues). I’m not sure the release date, but 0.34.0 was expected mid-December, though it might come later.

@flamber, just an update, avoiding date variables seems to resolve the issue.
And pulse is getting sent every day on time.

I see there is a new version of metabase but I am not sure if this release will resolve the issue.
I am going to wait for some more version and then will update the metabase.

@devD If you use Google Auth, then upgrade to 0.33.7.3 right now. If you don’t use Google Auth, I would still recommend upgrading.
The major timezone fixes are in the upcoming 0.34.0, which is coming soon (probably early January, is my current guess)

Thanks for confirming.

we don’t use google Auth, but I will upgrade the metabase to latest version.

Meanwhile there is one more issue (may be I will open another ticket, but asking if you have heard about it).

Metrics and Segments are not visible on admin page. however in custom query we can still see and use it. the problem is we are not able to update the old metrics and even if we create a new one it is disappearing.

its look like its a react issue.
as below, i can see error log if inspecting the front end.

Uncaught (in promise) Error: query is a required parameter to format expressions
at app-main.bundle.js?d1dced319abf1b617b66:5
at s (app-main.bundle.js?d1dced319abf1b617b66:5)
at app-main.bundle.js?d1dced319abf1b617b66:5
at Array.map ()
at s (app-main.bundle.js?d1dced319abf1b617b66:5)
at app-main.bundle.js?d1dced319abf1b617b66:5
at Array.map ()
at s (app-main.bundle.js?d1dced319abf1b617b66:5)
at app-main.bundle.js?d1dced319abf1b617b66:5
at Array.map ()

@devD
That’s this issue - I’ve included a workaround:
https://github.com/metabase/metabase/issues/10844 - upvote by clicking :+1: on the first post

@flamber
I am so sorry the pulse failed again today.. with the same error.
only thing is this time it ran for 6 days instead of 4 days before failing.

Error message exactly the same. please and please suggest.

{"status":"failed","exception":"class clojure.lang.ExceptionInfo","message":"Input to defaulted-timezone does not match schema: \n\n\t \u001b[0;33m [(named (not (instance? metabase.models.card.CardInstance nil)) card)] \u001b[0m \n\n","stacktrace":["--> pulse$fn__60819$defaulted_timezone__60824.invoke(pulse.clj:51)","pulse$fn__60917.invokeStatic(pulse.clj:165)","pulse$fn__60917.invoke(pulse.clj:160)","pulse$results__GT_notifications$iter__60947__60951$fn__60952.invoke(pulse.clj:212)","pulse$send_notifications_BANG_.invokeStatic(pulse.clj:248)","pulse$send_notifications_BANG_.invoke(pulse.clj:247)","pulse$send_pulse_BANG_.invokeStatic(pulse.clj:274)","pulse$send_pulse_BANG_.doInvoke(pulse.clj:256)","task.send_pulses$fn__67957$send_pulses_BANG___67966$fn__67969$fn__67987$fn__67988.invoke(send_pulses.clj:59)","models.task_history$fn__41262$do_with_task_history__41267$fn__41268.invoke(task_history.clj:77)","models.task_history$fn__41262$do_with_task_history__41267.invoke(task_history.clj:72)","task.send_pulses$fn__67957$send_pulses_BANG___67966$fn__67969$fn__67987.invoke(send_pulses.clj:56)","task.send_pulses$fn__67957$send_pulses_BANG___67966$fn__67969.invoke(send_pulses.clj:55)","task.send_pulses$fn__67957$send_pulses_BANG___67966.invoke(send_pulses.clj:42)","task.send_pulses$fn__67957$send_pulses_BANG___67966$fn__67967.invoke(send_pulses.clj:49)","task.send_pulses$fn__67957$send_pulses_BANG___67966.invoke(send_pulses.clj:42)","task.send_pulses.SendPulses$fn__68023.invoke(send_pulses.clj:100)","models.task_history$fn__41262$do_with_task_history__41267$fn__41268.invoke(task_history.clj:77)","models.task_history$fn__41262$do_with_task_history__41267.invoke(task_history.clj:72)","task.send_pulses.SendPulses.execute(send_pulses.clj:86)"],"ex-data":{"type":"schema.core/error","schema":[{"schema":"class metabase.models.card.CardInstance","optional?":false,"name":"card"}],"value":[null],"error":["schema.utils.NamedError@201060e8"]},"original-info":null}

@flamber

One another issue, once the pulse fail if i navigate to the failed pulse screen . I dont see the question there again.

@devD Dang it! Well, at you’ve tried. I can only recommend that you open an issue about this on Github. Remember to describe it with details and logs. And even better if you somehow can reproduce the issue with Sample Dataset.
https://github.com/metabase/metabase/issues/new/choose

thanks for suggesting. I created a bug issue 11587.
Its a bottleneck issue now. Hoping if it can be resolved now.

Just for reference:

Thanks a lot

@devD By the way, I noticed that the Diagnostic Info you posted on the issue still says "user.timezone": "GMT", so did you try setting JAVA_TIMEZONE=PST ?

Yes i actually did and its showing as well. Once inside the container if i echo timezone

bash-4.4# echo $JAVA_TIMEZONE
US/Pacific

The above configuration is before we decided to updated the timezone.
I am pasting the current timezone.

{
“browser-info”: {
“language”: “en-US”,
“platform”: “MacIntel”,
“userAgent”: “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36”,
“vendor”: “Google Inc.”
},
“system-info”: {
“java.runtime.name”: “OpenJDK Runtime Environment”,
“java.runtime.version”: “11.0.3+7”,
“java.vendor”: “AdoptOpenJDK”,
“java.vendor.url”: “https://adoptopenjdk.net/”,
“java.version”: “11.0.3”,
“java.vm.name”: “OpenJDK 64-Bit Server VM”,
“java.vm.version”: “11.0.3+7”,
“os.name”: “Linux”,
“os.version”: “4.14.106-97.85.amzn2.x86_64”,
“user.language”: “en”,
“user.timezone”: “US/Pacific”
},
“metabase-info”: {
“databases”: [
“snowflake”
],
“hosting-env”: “unknown”,
“application-database”: “postgres”,
“run-mode”: “prod”,
“version”: {
“date”: “2019-11-19”,
“tag”: “v0.33.6”,
“branch”: “release-0.33.x”,
“hash”: “be1e0e1”
},
“settings”: {
“report-timezone”: “Canada/Eastern”
}
}
}

I am extremely sorry to paste the old configuration in issue reported.
Just now i updated the issue.

1 Like

@devD Just for the fun of is, could you try a timezone (TZ database name) from this list instead of US/Pacific, which is deprecated along with Canada/Eastern.
https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
I’ve seen some really, really strange issues with timezones - for instance this entire topic: Empty response on SQL-Server queries containing DATETIME

I don’t think it’ll change anything, but just wanna make sure it’s not that.