After druid connection is created, a lot of big queries are sent continuously to Druid. All of these requests contain 200-years-query (from 1901 to 2100), which will our druid broker can't work without enough memory.
Have you encountered this issue?
BTW, my metabase's version is v0.39.1 . The same issue happened on v0.36.3.
Hi @ShineLiu
I have not seen this problem before. Which broker and version of Druid?
Those queries looks like part of the scanning of field values, which you can disable:
https://www.metabase.com/docs/latest/administration-guide/01-managing-databases.html#choose-when-metabase-syncs-and-scans
You should upgrade to at least 0.39.2, which contains two Druid fixes. Latest release is 0.39.3
https://github.com/metabase/metabase/releases
Thanks a lot!
Our version of Druid is 0.20 for test and 0.18 online. I checked Druid fixes in 0.39.2. It looks they aren't fixing my bug.
BTW, I tried the setting about scanning and disabled it. But there were still a lot of big queries sent continuously to Druid.
@ShineLiu The two Druid fixes wasn't related to your problem, so it won't fix anything in that case.
After you have disabled scan, which queries are you seeing on Druid? I don't have a magic mirror, so you need to post details.
Did you try restarting Metabase, since changing the scan settings, while it's already scanning, will not stop the existing scan queries.
@flamber, After disabled scan, the same queries were sent to Druid. All tables (datasource) in Druid would be queried.
BTW, after repeated tests, we found the scan queries would be sent just one time any way. I didn't try restarting Metabase. And I will try it. But we don't think it will be satisfying, because some queries will be sent as soon as the Druid database is created in Metabase. So I am afraid restarting can't help much.
Thanks for your reply.
@ShineLiu The queries should be different, since 5,000 rows are used during scan, which is done by the schedule defined (or disabled). Metabase will still make select queries to all tables as part of the sync process, but those queries are different. And then there's fingerprinting, which requests 10,000 rows, and is run once during the first sync.
@flamber，I am sure the queries aren't different whether the schedule is disabled or not. Firstly, it seems the setting of scan can't take effect. After this setting was done and saved when the Druid database was created, I found a lot of queries and then went to check this setting. I just found the setting became like this:
But, "Never, I'll do this manually if I need to" had been set when I created this database. So a lot of queries would be still sent and they looked the same. This situation couldn't be approved by our Druid owner. Any big query can't be accepted.
Secondly, I checked "Never, I'll do this manually if I need to" again and saved my change. This time, the setting is checked whenever I come back to the scheduling page. I am not sure whether this setting is taking effect or not. But there are still some queries will be sent. As well, I am not sure whether now these queries is the same as the previous. But our Druid admin doesn't think they can be accepted.
@ShineLiu We are going to work on several sync bugs and features in the upcoming 0.41 a few months out, which should make queries lighter and give more control for bigger instances.
You might want to avoid the UI and just use API or directly manipulating the application database, so you ensure that Metabase does not run scan queries.
Thanks a lot. Look forward to the new version.