Default date filters for Big query partitioning

Feedback and Requests
1

Hi there, and thanks for Metabase!

We’re using Big query along with DBT and most of our tables are partitioned by dates. Dashboards have default date filters, such as a “last week” filter, which is limiting queries to concerned partitions, hence running on a smaller amount of data, at better performance, and lower price : perfect.

Unfortunately, when users are exploring data using questions or databases, the only limitation for the request is the “LIMIT 2000” clause, which doesn’t make use of big query partitioning. This means everytime someone opens a new question, it requests against the whole table.

This is a huge issue for us right now, we sadly can’t go forward with Metabase because of it.

We could use the big query require_partition_filter option on our tables, but everytime someone connects to a table it pops an error, and requires a very repetitive step of adding filters on the partitioned data everytime. This is not a solid solution.

There is an issue opened about this : https://github.com/metabase/metabase/issues/6705
I coulnd’t find more info and i am very surprised that other big query users aren’t mentioning this. How others are working around this? Do you pay for 10 years of data everytime someone requests the last day?

Do you know if this is a known or prioritized issue? Will it be fixed soon?

Thanks again for the amazing product,

Cheers

2

Wow, 3 years ago and yet no answer on this.

We have the same major issue and on top of your perfect explanation I'll add this:
when to Metabase scans a table to create dropdown filters, it's not able to pass a default filter on the partitioning column, resulting in having only blank/null dropdown lists for all tables that have require_partition_filter enabled.

Any news on this?

3

We are having this same issue, and facing high costs due to how Metabase scans Bigquery tables (with no use of partitions, even if a table has a partition column by dates).

We have disabled so far the automatic scans, but still need to run it every time a view is added, and we have big tables which are fully scanned during scans.

If a table has a metadata which specifies that uses a partition on dates, Metabase should use it, or could be an option in the configuration...