Date filter "Between" is incorrectly giving me data outside of the date range

image

I'm trying to get someone from Metabase to look at this issue I raised back in May Date filters giving inconsistent results, depends on filter mode · Issue #30454 · metabase/metabase · GitHub

Normally I find the Metabase team to be really responsive so I think there's something about my issue that has made it fall through the cracks. Hoping this can get it in front of someone.

The Metabase "between" filter is incorrectly trying to apply a timezone offset to a SQL datetime which doesn't have a timezone component. This is giving the wrong data. I'm constantly having to explain to users what's going on and helping them adjust their date filters. "after" and "before" filters seem to be working as I'd expect them to, but the UI doesn't let you apply 2 filters on he same field without resorting to a custom expression filter.

For me, showing the wrong data is a pretty significant bug. I'd love to be told that I'm doing something wrong here, just want to get it working correctly.

I've given what I think is a pretty good amount of info about the problem but the issue is still sitting there tagged with "Needs Triage". I've bumped it a couple of times since May in the hope it'll get spotted but no luck.

Thanks

that's most probably a timezone issue. Please post troubleshooting info, timezone configured and database

Current system setup:

{
  "browser-info": {
    "language": "en-NZ",
    "platform": "Win32",
    "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
    "vendor": "Google Inc."
  },
  "system-info": {
    "file.encoding": "Cp1252",
    "java.runtime.name": "OpenJDK Runtime Environment",
    "java.runtime.version": "17.0.2+8",
    "java.vendor": "Eclipse Adoptium",
    "java.vendor.url": "https://adoptium.net/",
    "java.version": "17.0.2",
    "java.vm.name": "OpenJDK 64-Bit Server VM",
    "java.vm.version": "17.0.2+8",
    "os.name": "Windows Server 2012 R2",
    "os.version": "6.3",
    "user.language": "en",
    "user.timezone": "Pacific/Auckland"
  },
  "metabase-info": {
    "databases": [
      "sqlserver",
      "postgres",
      "h2"
    ],
    "hosting-env": "unknown",
    "application-database": "postgres",
    "application-database-details": {
      "database": {
        "name": "PostgreSQL",
        "version": "10.5"
      },
      "jdbc-driver": {
        "name": "PostgreSQL JDBC Driver",
        "version": "42.5.4"
      }
    },
    "run-mode": "prod",
    "version": {
      "date": "2023-09-18",
      "tag": "v0.47.2",
      "branch": "release-x.47.x",
      "hash": "536c24c"
    },
    "settings": {
      "report-timezone": "Pacific/Auckland"
    }
  }
}

DB is SQL Server 2017, fields are all datetime type which has no concept of a timezone component.

Metabase setting of Report timezone has no impact. Setting JVM timezone explicitly at runtime via -Duser.timezone=NZ has not impact. SQL Server timezone is reported as New Zealand Standard Time.

My setup is really simple, all clients, Metabase server, and DB server are in the same timezone. I'm struggling to understand why I'm having this issue.

I agree it's a timezone issue, Metabase is creating a datetimeoffset that has timezone information (correctly for my timezone, +12) to filter my field which is a datetime that has no concept of a timezone. This filter comparison implicitly converts my datetime into a datetimeoffset (precedence) (with a +0 timezone offset). There is a 12 hour difference between these two values. Metabase is forcing a timezone comparison for things that don't have timezones.

declare @my_datetime datetime = '2023-09-19'
select @my_datetime my_datetime
    ,cast(cast(@my_datetime as datetime) as datetimeoffset) my_datetime_as_datetimeoffset
    ,DateTimeOffsetFromParts(2023, 9, 19, 0, 0, 0, 0, 12, 0, 7) metabase_makes_this
my_datetime my_datetime_as_datetimeoffset metabase_makes_this
2023-09-19 00:00:00.000 2023-09-19 00:00:00.0000000 +00:00 2023-09-19 00:00:00.0000000 +12:00

Progress!

I've just tried setting the JVM timezone to UTC via -Duser.timezone=UTC, this gets the date filters working as I expect, this is progresss! But I am concerned that having the JVM timezone different to the SQL and client timezones will cause us other issues. It doesn't feel "correct", and I worry that it will cause issues with date time fields or data sources that do have a timezone.

Any suggestions?

Hi, I saw your comment in Github and wanted to know if setting the timezone via JVM variables fixed it so I can close the GH issue. Thanks for your efforts in fixing this on your own :slight_smile:

I think it fixed it, date handling in the Metabase UI for my SQL Server datetimes now seems to be consistent whichever way I approach it (relative, specific, group by).

I haven't made the change in production yet, I still want to do some checks in test first as I don't understand why I need to set the JVM to UTC and what else this might affect. JVM timezone is a Metabase global setting, but I only want to change the behaviour for SQL Server datetimes.

I guess I just don't understand why Metabase is even considering timezones for a SQL Server datetime, and why it only does it in sometimes. I'll trust your judgement that it's correct and move on :slight_smile: , thanks for prompting me to re-validate my assumptions.

Hi - we are experiencing the same issue, but the fix of changing our Metabase timezone to UTC is not going to work for us, as we have other data that DOES use timezone information and those reports will be broken by changing out system time. Our data is all stored on a Redshift database.

The central issue that the field filter is applying timezone data to a field without timezone data stored. Confirmed the same behaviour is present when using two date filters (start + end date), and also single date filter with equality check.

This is our field filter setup:
image

And the generated query (filter excerpt):

where "utils"."calendar"."date" BETWEEN timestamp with time zone '2024-01-01 00:00:00.000+11:00'
   AND timestamp with time zone '2024-01-31 00:00:00.000+11:00'

The column definition: date DATE NOT NULL ENCODE az64

And we are therefore pulling in data that is outside the field filter's intended range.
image

Diagnostic info is:

  "browser-info": {
    "language": "en-AU",
    "platform": "Win32",
    "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36",
    "vendor": "Google Inc."
  },
  "system-info": {
    "file.encoding": "UTF-8",
    "java.runtime.name": "OpenJDK Runtime Environment",
    "java.runtime.version": "11.0.22+7",
    "java.vendor": "Eclipse Adoptium",
    "java.vendor.url": "https://adoptium.net/",
    "java.version": "11.0.22",
    "java.vm.name": "OpenJDK 64-Bit Server VM",
    "java.vm.version": "11.0.22+7",
    "os.name": "Linux",
    "os.version": "6.1.61-85.141.amzn2023.x86_64",
    "user.language": "en",
    "user.timezone": "GMT"
  },
  "metabase-info": {
    "databases": [
      "redshift"
    ],
    "hosting-env": "unknown",
    "application-database": "postgres",
    "application-database-details": {
      "database": {
        "name": "PostgreSQL",
        "version": "15.5"
      },
      "jdbc-driver": {
        "name": "PostgreSQL JDBC Driver",
        "version": "42.6.0"
      }
    },
    "run-mode": "prod",
    "version": {
      "date": "2024-02-06",
      "tag": "v0.48.5",
      "hash": "dab12cf"
    },
    "settings": {
      "report-timezone": "Australia/Melbourne"
    }
  }
}```

I agree the JVM timezone only works around this for my setup, it still seems wrong in the general case.

I've actually had a similar issue come back in versions v0.48.4 and up, the JVM timezone doesn't make any difference so I'm blocked on v0.48.3.

Head over to Github issues and give them a thumbs up and a comment.

Hopefully we can get some traction.

Hi @notrom and @robswanLL , if you can post reproduction steps with Sample database, or the SQL and DDL with data, that would be great. We are actively investigating this right now.
Thanks!

I haven't been able to reproduce this with the Sample DB.

Date filtering change between v0.48.3 and v0.48.4. looks timezone related · Issue #38248 · metabase/metabase · GitHub gives repro steps that are repeatable for me (UTC +13, MS SQL db).

hi @ignacio

Was not able to reproduce with the sample database, however, repro steps for Redshift database are as follows. Diagnostic info in my previous comment on this thread.

DDL for table + data

CREATE TABLE IF NOT EXISTS utils.calendar_dev
(
	date DATE NOT NULL  ENCODE az64
);

INSERT INTO utils.calendar_dev ("date") 
VALUES('2023-12-30'), ('2023-12-31'), ('2024-01-01'), ('2024-01-02');

New SQL Query

select *
from utils.calendar_dev
where {{date_filter}}

Setup date filter as in previous message (field filter, map to calendar_dev.date, set to Month+Year, January 2024)

Preview generated query to see timezone data being erroneously added.

select *
from utils.calendar_dev
where "utils"."calendar_dev"."date" BETWEEN timestamp with time zone '2024-01-01 00:00:00.000+11:00'
   AND timestamp with time zone '2024-01-31 00:00:00.000+11:00'