@zoove Everything will become slower if you raise the level. If you set it to 100k, then that will cause more load on the database and the network transfers too, besides how the user's browser will handle the many thousands of extra DOM objects.
If you have "infinite" database resource, network speed and unlimited browser memory+CPU, then you could likely set it to 1 million. But that's not the case in the real world.
It would be more interesting to figure out where you are hitting this limitation. Can you give examples?
Well, high-performance graphing libraries draw on canvas instead of using DOM. Please see e.g. what netdata is capable of. Canvas has the drawback that there's much more code to be written though, so I understand that avoiding it is desirable in a general-purpose tool.
Sure, I understand that some limits make sense. I am trying to learn where exactly is the bottleneck at the 2k level, because I doubt it's the database, network speed or back-end. For these components, 2k records should be a breeze.
Right now I'm using custom ggplot-based diagrams for some performance analyses. My goal here is to put them in a form where a person who is not familiar with ggplot can use and modify them for their purposes, as opposed to loading data into a spreadsheet.
I've reproduced a stacked area chart displaying ~30 series of daily measurements over a year, one that was particularly useful in the past. Except at that scale I had to make it weekly to go below 2k data points.
@zoove Great write-up. Though, I have seen several people requesting functionality in Metabase, so it could be used as a tool for various loggers, which gives seconds/minutes data, but that's really not what Metabase is designed for and there are much better tools out there that.
As for DOM vs Canvas. Yes, that's true. And we're definitely feeling some pain-point from DOM and repainting, which we try to optimize as much as possible, but Metabase is a big project, so that's not just something you can fix in a couple of days.
We definitely want to make Metabase more performant and capable, but our priorities will still be on "human" data vs trying to be a replacement for logger graphing tools.
You will get 10k results if you aggregate your results in the GUI questions, so that would still put you above the limit for 30x365, but 27 series would be fine.
Oh yes, I totally understand! I admit I'm already sold on Metabase, I'm just trying to understand more of it before I suggest the team to adopt it, because I'll get to be the person answering all questions for some time.
While I understand that dedicated tools are always better in terms of capabilities and that Metabase aims to be a general-purpose tool for low-density processed data, please allow me to share my perspective on this statement. There's nothing less human in visualising large datasets. Quite the contrary! Human brains have great capabilities in pattern recognition, whereas translating the exact patterns humans recognise into refined metrics is difficult for humans. For example, a high-density time series plot allows people to see not just levels of specific metrics, but also at the same time gauge their variability and recognise high-frequency fluctuations, like weekly cycles in my case. All of that from a single picture.
Right now I have just moved the SQL query from my ggplot code into Metabase to make sure data is the same. I'm definitely going to play with the tool more!
on the first post of each issue.