Sync issues from Athena to Metabase

Get Help
1

There were 4-5 tables added in AWS Athena 3 days before. We had the configuration setup already in place for Athena DB in metabase through driver. These tables are still not flowing in metabase.
Is there a way to track why these sync is not happening although the sync is set to hourly basis?

2

Hi @abhi1986
Post "Diagnostic Info" from Admin > Troubleshooting, and the version of the driver used.

3

{
"browser-info": {
"language": "en-US",
"platform": "Win32",
"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36",
"vendor": "Google Inc."
},
"system-info": {
"file.encoding": "UTF-8",
"java.runtime.name": "OpenJDK Runtime Environment",
"java.runtime.version": "11.0.10+9",
"java.vendor": "AdoptOpenJDK",
"java.vendor.url": "https://adoptopenjdk.net/",
"java.version": "11.0.10",
"java.vm.name": "OpenJDK 64-Bit Server VM",
"java.vm.version": "11.0.10+9",
"os.name": "Linux",
"os.version": "3.10.0-1127.19.1.el7.x86_64",
"user.language": "en",
"user.timezone": "GMT"
},
"metabase-info": {
"databases": [
"athena",
"h2",
"postgres",
"mongo",
"redshift"
],
"hosting-env": "unknown",
"application-database": "postgres",
"application-database-details": {
"database": {
"name": "PostgreSQL",
"version": "12.3"
},
"jdbc-driver": {
"name": "PostgreSQL JDBC Driver",
"version": "42.2.18"
}
},
"run-mode": "prod",
"version": {
"date": "2021-04-08",
"tag": "v0.38.4",
"branch": "release-x.38.x",
"hash": "057b6bb"
},
"settings": {
"report-timezone": "Asia/Calcutta"
}
}
}

4

Athena driver version: v1.2.0
Metabase version: v0.38

5

@abhi1986 Try starting with debug logging of sync, so you can see what is happening.
Docker run-command would need the following:
... -e JAVA_OPTS="-Dlog4j.configurationFile=https://log4j.us/v2/templates/metabase?trace=metabase.sync" ...

6

Hi,
We still see the sync issues if any new column is added in athena. It takes a lot of time to get refreshed in metabase. Is there a way we can make the refresh in metabase in a real time basis?

7

@abhi1986

Run with debugging enabled like I showed and post the logs.

Metabase syncs based on the settings you have defined:
https://www.metabase.com/docs/latest/administration-guide/01-managing-databases.html#choose-when-metabase-syncs-and-scans

Metabase can not sync real-time unless your database informs Metabase of the changes.
There's an API that can be used to sync specific parts:
https://www.metabase.com/docs/latest/api-documentation.html#post-apinotifydbid

8

Hi,
Still facing the same sync issue. I aaded a new Database in Athena 2 days before but still the same DB is not synced in metabase. Can you please provide a permanent solution for this..How much time does metabae take to sync the new schema from Athena?

9

@abhi1986

  1. Run with debugging enabled like I showed and post the logs. If you do not do this, then I have absolutely no chance of knowing what is happening and what it's doing.
  2. Try latest release - there has been a couple of changes to the sync process.
  3. I cannot tell you how long time it takes to sync/scan - that depends on your data source. If someone has a database with a single table, one column and one row, then Metabase would finish extremely quickly, but if you have thousands of tables, hundreds of columns and millions of rows in each table, then it takes a lot longer.