I found a issue，
I use Vertica as the physical database, I add a new table to Vertica, and then immediately sync_database, sometimes metabase can not get the new table, need to wait for a period of time to refresh to get it, but sometimes can get the new table immediately, why?
Hi @jesse1
Please post “Diagnostic Info” from Admin > Troubleshooting.
You can follow the sync process in the log - Admin > Troubleshooting > Logs.
It depends on the order of the returned tables/column. And you might need to enable debug logging to see more details of the process - example:
java -Dlog4j.configurationFile="https://log4j.us/v2/templates/metabase?trace=metabase.sync" -jar metabase.jar
I added a new table and hit refresh,it works very well.
#################################################################
# Completed sync on DB_VERTICA_UAT
# Start: 2021-01-20T23:23:29.392-08:00
# End: 2021-01-20T23:25:06.3-08:00
# Duration: 1.6 mins
# ---------------------------------------------------------------
# Completed step 'sync-timezone'
# Start: 2021-01-20T23:23:29.392-08:00
# End: 2021-01-20T23:23:29.423-08:00
# Duration: 31.0 ms
# Found timezone id America/Los_Angeles
# ---------------------------------------------------------------
# Completed step 'sync-tables'
# Start: 2021-01-20T23:23:29.423-08:00
# End: 2021-01-20T23:23:34.272-08:00
# Duration: 4.8 s
# Total number of tables sync'd 305, number of tables updated 1
# ---------------------------------------------------------------
# Completed step 'sync-fields'
# Start: 2021-01-20T23:23:34.272-08:00
# End: 2021-01-20T23:24:55.371-08:00
# Duration: 1.4 mins
# Total number of fields sync'd 6,194, number of fields updated 13
# ---------------------------------------------------------------
# Completed step 'sync-fks'
# Start: 2021-01-20T23:24:55.371-08:00
# End: 2021-01-20T23:25:01.411-08:00
# Duration: 6.0 s
# Total number of foreign keys sync'd 0, 0 updated and 0 tables failed to update
# ---------------------------------------------------------------
# Completed step 'sync-metabase-metadata'
# Start: 2021-01-20T23:25:01.411-08:00
# End: 2021-01-20T23:25:06.3-08:00
# Duration: 4.9 s
#################################################################
But I added a new table and then hit refresh, it did not appear above the log,This is the only log
#################################################################
# Completed analyze on DB_VERTICA_UAT
# Start: 2021-01-20T23:25:06.323-08:00
# End: 2021-01-20T23:25:17.293-08:00
# Duration: 11.0 s
# ---------------------------------------------------------------
# Completed step 'fingerprint-fields'
# Start: 2021-01-20T23:25:06.323-08:00
# End: 2021-01-20T23:25:16.717-08:00
# Duration: 10.4 s
# Fingerprint updates attempted 1,316, updated 13, no data found 1,303, failed 0
# ---------------------------------------------------------------
# Completed step 'classify-fields'
# Start: 2021-01-20T23:25:16.717-08:00
# End: 2021-01-20T23:25:16.967-08:00
# Duration: 250.0 ms
# Total number of fields classified 13, 0 failed
# ---------------------------------------------------------------
# Completed step 'classify-tables'
# Start: 2021-01-20T23:25:16.967-08:00
# End: 2021-01-20T23:25:17.293-08:00
# Duration: 326.0 ms
# Total number of tables classified 306, 0 updated
#################################################################
Is it true that clicking on the next sync_database task before it completes the first one only triggers fingerprint-fields, classified-fields, classified-tables?
Post “Diagnostic Info” from Admin > Troubleshooting.
The sync process is asynchronous. You should let the sync complete before starting a new sync, otherwise it will process several things again.
We are seeing slow sync on our own Vertica tests, which we are looking into:
https://github.com/metabase/metabase/issues/14472
Thanks @flamber
I got it. I observed that metabase took the longest time in sync_field, requiring 3-5min
@jesse1 Yeah, the logging is incorrect - it’s not only doing sync of fields, it’s also fingerprinting, which actually takes up the most time. The logging will become better in upcoming 0.38:
https://github.com/metabase/metabase/pull/14428