My Spark SQL database doesn't show all my Hive schemas

Get Help
1

Hi all, I'm using Metabase's Spark SQL to connect to my Spark Thrift Server.

I've successfully connect them but I couldn't see all my schemas on the UI and neither any tables. Can anyone help me on this?

45%20PM
45%20PM2510Ă—235 5.92 KB

Metabase Version : 0.37.2

2

Hi @Jensenity
Please post “Diagnostic Info” from Admin > Troubleshooting.
Try upgrading to latest release, 0.37.7
And then run a manual sync in Admin > Databases > (your-db), and check the log for errors during the sync in Admin > Troubleshooting > Logs.

1 Like
3

I checked the logs before, but there were no error logs. Sync finished but yet no schemas are shown. My Spark Thrift Server has multiple Schemas, and within which schemas, there are many tables.

Diagnostic info :
{
“browser-info”: {
“language”: “en-GB”,
“platform”: “MacIntel”,
“userAgent”: “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36”,
“vendor”: “Google Inc.”
},
“system-info”: {
“file.encoding”: “UTF-8”,
“java.runtime.name”: “OpenJDK Runtime Environment”,
“java.runtime.version”: “11.0.7+10”,
“java.vendor”: “AdoptOpenJDK”,
“java.vendor.url”: “https://adoptopenjdk.net/”,
“java.version”: “11.0.7”,
“java.vm.name”: “OpenJDK 64-Bit Server VM”,
“java.vm.version”: “11.0.7+10”,
“os.name”: “Linux”,
“os.version”: “5.4.0-1025-aws”,
“user.language”: “en”,
“user.timezone”: “GMT”
},
“metabase-info”: {
“databases”: [
“sparksql”,
“athena”,
“postgres”
],
“hosting-env”: “unknown”,
“application-database”: “postgres”,
“application-database-details”: {
“database”: {
“name”: “PostgreSQL”,
“version”: “12.3”
},
“jdbc-driver”: {
“name”: “PostgreSQL JDBC Driver”,
“version”: “42.2.8”
}
},
“run-mode”: “prod”,
“version”: {
“date”: “2020-11-16”,
“tag”: “v0.37.2”,
“branch”: “release-x.37.x”,
“hash”: “25e5f70”
},
“settings”: {
“report-timezone”: null
}
}
}

[] 2021-01-24T21:47:25+09:00 DEBUG metabase.middleware.log GET /api/database/7 200 4.6 ms (3 DB calls) App DB connections: 0/22 Jetty threads: 4/50 (1 idle, 0 queued) (134 total active threads) Queries in flight: 0 (0 queued)
[] 2021-01-24T21:47:27+09:00 DEBUG metabase.middleware.log POST /api/database/7/sync_schema 200 1.9 ms (1 DB calls) App DB connections: 0/22 Jetty threads: 4/50 (1 idle, 0 queued) (134 total active threads) Queries in flight: 0 (0 queued)
[] 2021-01-24T21:47:27+09:00 INFO metabase.sync.util STARTING: Sync metadata for sparksql Database 7 'Spark Thrift Server'
[] 2021-01-24T21:47:27+09:00 INFO metabase.sync.util FINISHED: step 'sync-timezone' for sparksql Database 7 'Spark Thrift Server' (74.0 µs)
[] 2021-01-24T21:47:27+09:00 INFO metabase.sync.util STARTING: step 'sync-tables' for sparksql Database 7 'Spark Thrift Server'
[] 2021-01-24T21:47:27+09:00 INFO metabase.sync.util STARTING: step 'sync-timezone' for sparksql Database 7 'Spark Thrift Server'
[] 2021-01-24T21:47:33+09:00 INFO metabase.sync.util FINISHED: step 'sync-tables' for sparksql Database 7 'Spark Thrift Server' (6.1 s)
[] 2021-01-24T21:47:33+09:00 INFO metabase.sync.util STARTING: step 'sync-fields' for sparksql Database 7 'Spark Thrift Server'
[] 2021-01-24T21:47:33+09:00 INFO metabase.sync.util FINISHED: step 'sync-fields' for sparksql Database 7 'Spark Thrift Server' (1.3 ms)
[] 2021-01-24T21:47:33+09:00 INFO metabase.sync.util STARTING: step 'sync-fks' for sparksql Database 7 'Spark Thrift Server'
[] 2021-01-24T21:47:33+09:00 INFO metabase.sync.util FINISHED: step 'sync-fks' for sparksql Database 7 'Spark Thrift Server' (1.2 ms)
[] 2021-01-24T21:47:33+09:00 INFO metabase.sync.util STARTING: step 'sync-metabase-metadata' for sparksql Database 7 'Spark Thrift Server'
[] 2021-01-24T21:47:33+09:00 INFO metabase.sync.util FINISHED: step 'sync-metabase-metadata' for sparksql Database 7 'Spark Thrift Server' (196.0 ms)
[] 2021-01-24T21:47:33+09:00 INFO metabase.sync.util FINISHED: Sync metadata for sparksql Database 7 'Spark Thrift Server' (6.3 s)
[] 2021-01-24T21:47:33+09:00 INFO metabase.sync.util STARTING: Analyze data for sparksql Database 7 'Spark Thrift Server'
[] 2021-01-24T21:47:33+09:00 INFO metabase.sync.util FINISHED: step 'classify-fields' for sparksql Database 7 'Spark Thrift Server' (24.3 µs)
[] 2021-01-24T21:47:33+09:00 INFO metabase.sync.util FINISHED: step 'classify-tables' for sparksql Database 7 'Spark Thrift Server' (24.6 µs)
[] 2021-01-24T21:47:33+09:00 INFO metabase.sync.util FINISHED: step 'fingerprint-fields' for sparksql Database 7 'Spark Thrift Server' (86.3 µs)
[] 2021-01-24T21:47:33+09:00 INFO metabase.sync.util STARTING: step 'classify-fields' for sparksql Database 7 'Spark Thrift Server'
[] 2021-01-24T21:47:33+09:00 INFO metabase.sync.util STARTING: step 'classify-tables' for sparksql Database 7 'Spark Thrift Server'
[] 2021-01-24T21:47:33+09:00 INFO metabase.sync.util STARTING: step 'fingerprint-fields' for sparksql Database 7 'Spark Thrift Server'
[] 2021-01-24T21:47:33+09:00 INFO metabase.sync.util FINISHED: Analyze data for sparksql Database 7 'Spark Thrift Server' (9.0 ms)
4

@Jensenity Okay, upgrade Metabase and try starting with debug logging to see more information.
Docker - add this environmental variable to the docker run ... command:
…-e JAVA_OPTS="-Dlog4j.configurationFile=https://log4j.us/v2/templates/metabase?trace=metabase.sync"…

5

I added this to my Metabase but after adding it, I couldnt see the logs on Metabase UI. But on my kubernetes pod, the logs are recorded. When I trigger the sync manually, there are not logs in Metabase shown regarding spark sql database.

6

@Jensenity I’m not sure why it isn’t working for you, but try manually creating a log configuration which adds trace/debug to everything:
https://www.metabase.com/docs/latest/operations-guide/log-configuration.html