CSV import taking hours?

I installed the recommended CSV plugin and, in a new H2 installation of Metabase on Amazon Linux, am trying to create a new database using two CSV files that are 7.1MB and 2.3MB respectively.

After about 6 hours, it still has not completed the import apparently. In the logs it shows repeated lines like these (below). When I look at the database in Metabase web UI it says This database doesn't have any tables.

Does it really take this long or is something going wrong? If something is going wrong, what should I do to fix it?

Thanks very muc in advance for any help or suggestions. Logs follow:

[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:18:32-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (62 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:18:34-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 2.0 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (62 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:18:36-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (62 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:18:38-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (62 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:18:40-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.8 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (62 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:18:42-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.6 ms (4 DB calls) App DB connections: 5/13 Jetty threads: 3/50 (4 idle, 0 queued) (62 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:18:44-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:18:46-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:18:48-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:18:50-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 5/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:18:52-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:18:54-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:18:56-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.8 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:18:58-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.8 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.task.sync-databases Starting sync task for Database 2.
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util FINISHED: step 'sync-timezone' for csv Database 2 'UK-EPC' (114.1 µs)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util STARTING: Sync metadata for csv Database 2 'UK-EPC'
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util STARTING: step 'sync-timezone' for csv Database 2 'UK-EPC'
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util STARTING: step 'sync-tables' for csv Database 2 'UK-EPC'
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util FINISHED: step 'sync-tables' for csv Database 2 'UK-EPC' (7.0 ms)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util STARTING: step 'sync-fields' for csv Database 2 'UK-EPC'
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util FINISHED: step 'sync-fields' for csv Database 2 'UK-EPC' (506.8 µs)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util FINISHED: step 'sync-fks' for csv Database 2 'UK-EPC' (214.9 µs)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util STARTING: step 'sync-fks' for csv Database 2 'UK-EPC'
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util STARTING: step 'sync-metabase-metadata' for csv Database 2 'UK-EPC'
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util FINISHED: step 'sync-metabase-metadata' for csv Database 2 'UK-EPC' (4.1 ms)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util FINISHED: Sync metadata for csv Database 2 'UK-EPC' (18.4 ms)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util STARTING: Analyze data for csv Database 2 'UK-EPC'
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util FINISHED: step 'classify-fields' for csv Database 2 'UK-EPC' (62.4 µs)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util FINISHED: step 'classify-tables' for csv Database 2 'UK-EPC' (70.5 µs)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util FINISHED: step 'fingerprint-fields' for csv Database 2 'UK-EPC' (180.6 µs)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util STARTING: step 'classify-fields' for csv Database 2 'UK-EPC'
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util STARTING: step 'classify-tables' for csv Database 2 'UK-EPC'
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util STARTING: step 'fingerprint-fields' for csv Database 2 'UK-EPC'
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 INFO metabase.sync.util FINISHED: Analyze data for csv Database 2 'UK-EPC' (4.3 ms)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:00-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 2.1 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:02-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.8 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:04-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:06-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:08-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:10-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.8 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:12-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 4.3 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:14-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:16-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:18-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.8 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:20-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:22-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:24-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.8 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:26-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 4/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:28-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 2.6 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:30-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.8 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:32-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:34-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.8 ms (4 DB calls) App DB connections: 1/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:36-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.8 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:38-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.8 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:40-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:42-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:44-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:46-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.9 ms (4 DB calls) App DB connections: 3/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:48-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.8 ms (4 DB calls) App DB connections: 2/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:50-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.8 ms (4 DB calls) App DB connections: 3/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:52-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.9 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:19:54-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.7 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:36:37-08:00 DEBUG metabase.server.middleware.log GET /api/database 200 1.7 ms (3 DB calls) App DB connections: 6/13 Jetty threads: 4/50 (2 idle, 0 queued) (58 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:36:54-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 2.1 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 4/50 (3 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:36:54-08:00 DEBUG metabase.server.middleware.log GET /api/database 200 3.2 ms (3 DB calls) App DB connections: 0/13 Jetty threads: 4/50 (3 idle, 0 queued) (60 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:36:56-08:00 DEBUG metabase.server.middleware.log GET /api/automagic-dashboards/database/2/candidates 200 1.9 ms (4 DB calls) App DB connections: 4/13 Jetty threads: 5/50 (3 idle, 0 queued) (62 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:36:56-08:00 DEBUG metabase.server.middleware.log GET /api/collection/root/items 200 3.9 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 4/50 (3 idle, 0 queued) (62 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:36:56-08:00 DEBUG metabase.server.middleware.log GET /api/collection/root 200 1.4 ms (2 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (62 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:36:56-08:00 DEBUG metabase.server.middleware.log GET /api/collection/root/items 200 6.0 ms (4 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (62 total active threads) Queries in flight: 0 (0 queued)
[cc563135-182a-406b-824c-e498a93f89b9] 2023-01-24T16:36:56-08:00 DEBUG metabase.server.middleware.log GET /api/database 200 1.7 ms (3 DB calls) App DB connections: 0/13 Jetty threads: 3/50 (4 idle, 0 queued) (62 total active threads) Queries in flight: 0 (0 queued)

1 Like

What is this "recommended CSV plugin" ?

This one:

I got the data to import by importing the CSV to a database instead, then linking the database.

1 Like

Actually Metabase states in the page that "You install these drivers at your own risk" I would guess the above question would make sense being asked to who wrote that driver

This third party csv plugin is not working in latest 0.46 series. In older series, also this plugin used to have limitations like "join not supported" etc.

Is Metabase developing their own csv plugin? @Luiggi

Yes, in the meantime, ingest the csv with a Postgres db

Currently I am ingesting data into Mysql . But for huge data, it's data duplication and time taking process. Is Postgres DB faster in data ingestion and loading back?

When we can expect Metabase to release their own CSV Plugin?

Ingesting csv via Metabase vs doing so via a native function of a database which is written in C should be always slower, and also the bigger it is, the slower it will be. I would strongly suggest you use Postgres copy function which was developed exactly for what you’re looking for and it’s almost the de facto way to do it nowadays

1 Like