Providing a file based card and dashboard approach


#1

We are facing two problems with the current Metabase architecture:

We need to put cards/questions in Git so that we can track changes. Yes, Metbase provides a history for cards (but not for dashboards), but it’s much less flexible and methodical that having these in Git. See https://github.com/metabase/metabase/issues/6485.

As we create more dashboards and more questions we find that we need to copy and paste a lot of boilerplate. Part of the problem can be solved using views in the db, but there’s still a lot of code that repeats when a dashboard has 4-5 drop downs and these need to be replicated in each question. touches on this.

Both these issues can be solved by having the cards reside in files rather than in the db. We can then generate the files using some kind of templating engine. There’s a similar idea in https://github.com/metabase/metabase/issues/5579. The advantage of using a file is that we can all use our favorite templating language or use a standard language and its full ecosystem.

In theory, we can implement much of this using the Metabase API: https://github.com/metabase/metabase/blob/master/docs/api-documentation.md
To insert and update cards and dashboards.

Indeed, when I look at two cards that are in the same dashboard, they are different in ways that are consistent with the approach that we’re contemplating. We could write simple scripts to maintain all the cards in a dashboard and generate new ones based on our existing ones.

On the other hand, we’d rather use a supported approach where we don’t need to maintain a custom solution.

Are other people considering the same set of issues? Is there an official plan on providing a templating soltuion? I didn’t see anything in the Github issues.


#2

I too also experience a similar problem. In my case, I generally want to create several different types of visualizations based on the same source data. In most cases, the query behind the source data is the same, only the type of visualization is different. I might wrap the query in a subquery to extract just a specific portion for a “Number” type visualization for example. This leads to a lot of repeated queries with different types like this:
image

Or maybe I’m doing Metabase wrong? It would be useful to have a “base” question and then other questions that work off the source data of the “base” question. This would also improve performance for heavy queries on a dashboard, since the base question could run once and each child question could work off the same data.

(I tried “Analyze the Results of This Query” option but it doesn’t seem to work on SQL Server Native Queries that have declared variables)


#3

Inspiring points from you guys on top of what was already there! (summed up well with @dror’s links)

What do you guys think about this:
(the underlying configuration format for the X-ray feature)

Granted, the X-Ray feature is still quite young - and evolving. But it is already a file based approach to generating both dashboards and cards

Some observations:

  1. It seems to be a higher abstraction level, more coarse grained, format than the API.
  2. The Metabase guys have, understandably, not yet committed to a v1.0 of the API
  3. Currently the .yaml files are baked into the Metabase.jar. I guess it should be easy to add that it picks them up from a folder like plugins.
  4. AFAIK you can represent all json as yaml but not the other way around. So it might even be possible in a glorious future to roundtrip from the specifics of the API back to yaml.
  5. Similar things to what you mention @dror with templates are happening for Grafana. I wonder if there are experiences and ideas to be drawn from there?
  6. I may not be able to - currently - define using X-Ray in a “pixel perfect” way. But if it can save me from just some of the boilerplate I might not care!
  7. If it becomes too “pixel perfect” I’m also getting mostly back to square one like when e.g. the API changes over time :wink:
  8. A textual format for describing dashboards should not necessarily cover the exact same area as a migration tool (for dev, staging, prod) like https://github.com/elevate/elevate.metabase.tools#metabase-exporter