Defining data model metadata as code

I’m new to Metabase and exploring the features. I like the fact that you can add metadata to your data model: primary keys, foreign keys, metics. But I don’t particularly like the UI for adding them. Is it possible to define metadata as code instead of through the frontend, and have metadata pick up these metadata definitions somehow?

Yes, there’s a REST API: https://github.com/metabase/metabase/wiki/Using-the-REST-API
Alternatively, it is open source, so you can build your own UI (not for the faint-hearted)

OK. I guess I was hoping there was a way to do this that didn’t didn’t require working with the REST API (like define the schemas for the table in a config file somewhere and metabase reads from this)

@dwl285 If you don’t like the REST way as Andrew recommended, then you would need to mess around with the sync-code to make it do what you want:
https://github.com/metabase/metabase/tree/master/src/metabase/sync

@sbelak Simon, didn’t you work on something like this, where you could define a yaml file?

The REST stuff isn’t as bad as it first looks. It’s how the standard Metabase GUI interacts with the server. The excellent log shows everything that’s going on behind the scenes.

Part of this defining metrics, segments, and dimensions of interest will be moved to “domain entities” in the coming versions which will be backed by YAML (+ an revamped UI for the data model). PKs and FKs are something we consider to be mostly picked up automatically, so I don’t expect exposing that part more (although once we have the new domain entities machinery in place it might be a case of being trivial enough that we’ll added it if enough interest)

OK, this sounds interesting. When is YAML backed version coming? Can I beta test?

How are PKs and FKs picked up automatically? I’m connecting to a BQ warehouse, and as far as I could tell I had to define these manually

@dwl285 Remember to tag people, since they might not get notified otherwise. Tagging @sbelak :slight_smile:
You can check Simon’s work with domain entities here - he even added a little roadmap on PR 10437 (not sure how up-to-date it is)
https://github.com/metabase/metabase/pulls?utf8=✓&q=is%3Apr+domain+entities

Thanks @flamber. The linked stuff should give you an overview of where we are. Some of it is already merged but not really exposed to the end user. I expect to get some more work done as part of .34 and then have it finished for .35.

We query the DB for information about PKs & FKs during sync. See metabase.sync.sync-metadata.fks. I haven’t done much work on BQ so don’t take this as canonical, but it should work. If not, it’s worth opening an issue (or @flamber can direct us to an existing one, he knows our GH much better than I).

BigQuery doesn’t have a concept of table keys, so I don’t think it would be possible to infer this from the DB automatically. Would love to be corrected if I’m wrong though.

@sbelak It’s sad when you can remember 8k issues/PRs out of the 11k, but great in certain cases :blush:
Yes, there’s an issue open about keys with BigQuery, so it’s something that should be considered, when re-doing the data model UI.
https://github.com/metabase/metabase/issues/4795 - and perhaps 2662 too.

Ok. My experience was that defining the primary/foreign keys in BigQuery worked ok, I just:
A) don’t want to define them in a GUI, I want to define them in something like a YAML file
B) want to be able to version control that YAML file

1 Like