Druid: identifying field as category

metadruid · August 23, 2016, 10:21pm

Using Druid : 0.9.1.1.

I was expecting one my fields which stores type of payment_methods to be classified as "category". This would have helped me use this as a dashboard filter. I have confirmed that this field has 6 unique values so it is well below the limit of 40 that is documented in the other topics .. I tried to use "Sync" button to see if I find any errors in the logs. But I do not. It shows 67% analyzed for this druid table.. in case not being at 100% is a problem.

08-23 18:11:51 INFO sync-database.analyze :: [*********************************·················] 67% Analyzed table 'billdata-1'.

Interestingly, another field "status" in this table which has 3 possible values gets classified as "category"

Lastly, I inspected metabase_fieldvalues table that seems to be keeping list of unique values in the fields marked as category.. I do not see any Druid tables/fields that I have added. Should I see "status" field which was flagged as category in here ?

Any troubleshooting pointers will be a big help.

Thank you.

metadruid · September 1, 2016, 6:32pm

Adding our finding… in case it helps anyone else.

Druid analyzer may be skipping the data analysis as this step may not have been implemented in the driver yet.

github.com

metabase/metabase/blob/7f5072cb51cf307dc29b14ab482e3db97abe78fd/src/metabase/driver.clj#L235


    0
    (int (math/round (/ (->> field-values
                             (map str)
                             (map count)
                             (reduce +))
                        field-values-count))))))




(def IDriverDefaultsMixin
"Default implementations of `IDriver` methods marked *OPTIONAL*."
{:analyze-table                     (constantly nil)
 :date-interval                     (u/drop-first-arg u/relative-date)
 :describe-table-fks                (constantly nil)
 :features                          (constantly nil)
 :humanize-connection-error-message (u/drop-first-arg identity)
 :notify-database-updated           (constantly nil)
 :process-query-in-context          (u/drop-first-arg identity)
 :sync-in-context                   (fn [_ _ f] (f))
 :table-rows-seq                    (constantly nil)})

Status seems to be getting picked up as a “category” because it is defined specifically here –

github.com

metabase/metabase/blob/7f5072cb51cf307dc29b14ab482e3db97abe78fd/src/metabase/models/field.clj#L181


   [#"^lng$"          float       :longitude]
   [#"^long$"         float       :longitude]
   [#"^longitude$"    float       :longitude]
   [#"^name$"         text        :name]
   [#"^postalCode$"   int-or-text :zip_code]
   [#"^postal_code$"  int-or-text :zip_code]
   [#"^rating$"       int-or-text :category]
   [#"^role$"         int-or-text :category]
   [#"^sex$"          int-or-text :category]
   [#"^state$"        text        :state]
   [#"^status$"       int-or-text :category]
   [#"^type$"         int-or-text :category]
   [#"^url$"          text        :url]
   [#"^zip_code$"     int-or-text :zip_code]
   [#"^zipcode$"      int-or-text :zip_code]]))


;; Check that all the pattern tuples are valid
(doseq [[name-pattern base-types special-type] pattern+base-types+special-type]
(assert (instance? java.util.regex.Pattern name-pattern))
(assert (every? (partial contains? base-types) base-types))
(assert (contains? special-types special-type)))

So we can either classify our druid columns as categories in the field.clj or use admin option to mark our Druid column as “Category”.

Lastly, metadata table metabase_fieldvalues will not have values for our Druid category columns, possibly because of un-implemented analyze function. Side effect of that is the dashboard filter on category columns will not show drop down values. So our quick choice (before we attempt to write a druid analyze function) is to manually insert records in metabase_fieldvalues table. Such manually inserted values are displayed ok by our dashboard filters.

Happy to hear about any other ways to address this or any holes with our analysis/work around…

-umesh