Metabase Initialization FAILED: Unrecognized token

Get Help
1

First off, well done on a glorious piece of software, we get so much use out of it. Thank you.

Unfortunately though, Metabase stopped working and I'm having trouble recovering it.

The server had been set up by someone who's since left the company, so I'm trying to figure it out myself. It seems like he set it up using the default H2 database, which I understand is not recommended for production. Metabase is ran inside a Docker server in this case, so when it stopped working (with an Nginx bad gateway HTTP 502 error), I had a dig around the server and found a shell script called start-metabase-docker.sh which contained the following:

sudo docker run -d -p 3000:3000 -v ~/metabase-data:/metabase-data -e "MB_DB_FILE=/metabase-data/metabase.db" -e "MB_PASSWORD_COMPLEXITY=weak" -e "MB_PASSWORD_LENGTH=12" --name metabase metabase/metabase

That did successfully create a Metabase instance, but a new blank one. So I read some more of the docs and tried to find the previous database file. I found a file called metabase.db.mv.db which was over 30MB, so I figured that might be it.

I then killed the new Docker container I had created, and moved this database file into the /metabase-data/metabase.db directory and ran the shell script again. This time, after quite a bit of progress, I got the following error message:

Metabase Initialization FAILED: Unrecognized token

Any ideas on what might help?

I was thinking about setting up PostgreSQL on the server and trying a migration if that might help. The OS is Ubuntu 16.04.4 LTS (Xenial).

Here's the full output of sudo docker logs metabase (I replaced the actual "token" with XXXXXX in case that shouldn't be posted publicly):

Blockquote
07-16 00:31:47 INFO metabase.core :: Starting Metabase version v0.29.3 (0de4585 release-0.29.3) ...
07-16 00:31:47 INFO metabase.core :: System timezone is 'GMT' ...
07-16 00:31:49 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
07-16 00:31:49 INFO metabase.db :: Verifying h2 Database Connection ...
07-16 00:31:50 INFO metabase.db :: Verify Database Connection ... :white_check_mark:
07-16 00:31:50 INFO metabase.db :: Running Database Migrations...
07-16 00:31:50 INFO metabase.db :: Setting up Liquibase...
07-16 00:31:50 INFO metabase.db :: Liquibase is ready.
07-16 00:31:50 INFO metabase.db :: Checking if Database has unrun migrations...
07-16 00:31:53 INFO metabase.db :: Database Migrations Current ... :white_check_mark:
com.mchange.v2.cfg.DelayedLogItem [ level -> FINE, text -> "The configuration file for resource identifier 'hocon:/reference,/application,/c3p0,/' could not be found. Skipping.", exception -> null]
07-16 00:31:53 INFO db.migrations :: Running all necessary data migrations, this may take a minute.
07-16 00:31:53 INFO db.migrations :: Finished running data migrations.
07-16 00:31:53 INFO metabase.events :: Starting events listener: metabase.events.notifications :ear:
07-16 00:31:53 INFO metabase.events :: Starting events listener: metabase.events.metabot-lifecycle :ear:
07-16 00:31:53 INFO metabase.events :: Starting events listener: metabase.events.dependencies :ear:
07-16 00:31:53 INFO metabase.events :: Starting events listener: metabase.events.activity-feed :ear:
07-16 00:31:53 INFO metabase.events :: Starting events listener: metabase.events.last-login :ear:
07-16 00:31:53 INFO metabase.events :: Starting events listener: metabase.events.revision :ear:
07-16 00:31:53 INFO metabase.events :: Starting events listener: metabase.events.sync-database :ear:
07-16 00:31:53 INFO metabase.events :: Starting events listener: metabase.events.view-log :ear:
07-16 00:31:53 INFO metabase.events :: Starting events listener: metabase.events.driver-notifications :ear:
07-16 00:31:54 INFO metabase.task :: Loading tasks namespace: metabase.task.send-anonymous-stats :calendar:
07-16 00:31:54 INFO metabase.task :: Loading tasks namespace: metabase.task.sync-databases :calendar:
com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'XXXXXX': was expecting ('true', 'false' or 'null')
at [Source: java.io.StringReader@2c2323be; line: 1, column: 24]
at com.fasterxml.jackson.core.JsonParser.constructError(JsonParser.java:1702)
at com.fasterxml.jackson.core.base.ParserMinimalBase.reportError(ParserMinimalBase.java:558)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.reportInvalidToken(ReaderBasedJsonParser.java:2839)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.handleOddValue(ReaderBasedJsonParser.java:1903)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:749)
at cheshire.parse$parse.invokeStatic(parse.clj:88)
at cheshire.parse$parse.invoke(parse.clj:86)
at cheshire.core$parse_string.invokeStatic(core.clj:205)
at cheshire.core$parse_string.invoke(core.clj:191)
at cheshire.core$parse_string.invokeStatic(core.clj:202)
at cheshire.core$parse_string.invoke(core.clj:191)
at metabase.models.interface$json_out.invokeStatic(interface.clj:32)
at metabase.models.interface$json_out.invoke(interface.clj:29)
at clojure.core$comp$fn__4727.invoke(core.clj:2460)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.core$apply.invokeStatic(core.clj:646)
at clojure.core$apply.invoke(core.clj:641)
at clojure.core.memoize$through_STAR$fn__6212.invoke(memoize.clj:81)
at clojure.core.cache$through$fn__6015.invoke(cache.clj:55)
at clojure.core.memoize$through_STAR$fn__6208$fn__6209.invoke(memoize.clj:80)
at clojure.core.memoize.RetryingDelay.deref(memoize.clj:64)
at clojure.core$deref.invokeStatic(core.clj:2228)
at clojure.core$deref.invoke(core.clj:2214)
at clojure.core.memoize$build_memoizer$fn__6263.doInvoke(memoize.clj:167)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.AFunction$1.doInvoke(AFunction.java:29)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$comp$fn__4727.invoke(core.clj:2460)
at toucan.models$apply_type_fns$iter__4033__4037$fn__4038.invoke(models.clj:295)
at clojure.lang.LazySeq.sval(LazySeq.java:40)
at clojure.lang.LazySeq.seq(LazySeq.java:49)
at clojure.lang.RT.seq(RT.java:521)
at clojure.core$seq__4357.invokeStatic(core.clj:137)
at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:24)
at clojure.core.protocols$fn__6738.invokeStatic(protocols.clj:75)
at clojure.core.protocols$fn__6738.invoke(protocols.clj:75)
at clojure.core.protocols$fn__6684$G__6679__6697.invoke(protocols.clj:13)
at clojure.core$reduce.invokeStatic(core.clj:6545)
at clojure.core$into.invokeStatic(core.clj:6610)
at clojure.core$into.invoke(core.clj:6604)
at toucan.models$apply_type_fns.invokeStatic(models.clj:293)
at toucan.models$apply_type_fns.invoke(models.clj:290)
at toucan.models$do_post_select.invokeStatic(models.clj:340)
at toucan.models$do_post_select.invoke(models.clj:335)
at toucan.db$do_post_select$iter__4213__4217$fn__4218.invoke(db.clj:362)
at clojure.lang.LazySeq.sval(LazySeq.java:40)
at clojure.lang.LazySeq.seq(LazySeq.java:49)
at clojure.lang.RT.seq(RT.java:521)
at clojure.lang.LazilyPersistentVector.create(LazilyPersistentVector.java:44)
at clojure.core$vec.invokeStatic(core.clj:377)
at clojure.core$vec.invoke(core.clj:367)
at toucan.db$do_post_select.invokeStatic(db.clj:361)
at toucan.db$do_post_select.invoke(db.clj:352)
at toucan.db$simple_select.invokeStatic(db.clj:383)
at toucan.db$simple_select.invoke(db.clj:372)
at toucan.db$select.invokeStatic(db.clj:640)
at toucan.db$select.doInvoke(db.clj:634)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at metabase.task.sync_databases$task_init.invokeStatic(sync_databases.clj:190)
at metabase.task.sync_databases$task_init.invoke(sync_databases.clj:185)
at clojure.lang.Var.invoke(Var.java:375)
at metabase.task$find_and_load_tasks_BANG.invokeStatic(task.clj:43)
at metabase.task$find_and_load_tasks_BANG.invoke(task.clj:34)
at metabase.task$start_scheduler_BANG_.invokeStatic(task.clj:58)
at metabase.task$start_scheduler_BANG_.invoke(task.clj:50)
at metabase.core$init_BANG_.invokeStatic(core.clj:166)
at metabase.core$init_BANG_.invoke(core.clj:133)
at metabase.core$start_normally.invokeStatic(core.clj:258)
at metabase.core$start_normally.invoke(core.clj:251)
at metabase.core$_main.invokeStatic(core.clj:279)
at metabase.core$_main.doInvoke(core.clj:274)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at metabase.core.main(Unknown Source)
07-16 00:31:54 ERROR metabase.core :: Metabase Initialization FAILED: Unrecognized token 'W1Ar0pkTsUgAv2sZi01cMHH': was expecting ('true', 'false' or 'null')
at [Source: java.io.StringReader@2c2323be; line: 1, column: 24]
07-16 00:31:54 INFO metabase.core :: Metabase Shutting Down ...
07-16 00:31:54 INFO metabase.core :: Metabase Shutdown COMPLETE

2

Hi @mraonghus

Looks like you’re running a fairly old version - 0.29.3 - there has been a lot of changes since, and also stability of H2, in the latest 0.32.9

But from the Docker command, the H2 is located in the volume outside the container - ~/metabase-data
So do you have backups of that directory? Then try restoring that and start Metabase.

Otherwise I would recommend that you try migrating to Postgres - if it is easier for you, then you can use the plain jar-file with Java instead of Docker.

Just note that you should not upgrade and migrate at the same time. In case you would want to upgrade too.

3

Hi @flamber, thanks for the suggestions.

I set up Postgres and ran the migration, which completed successfully, but then when I tried to start metabase with the JAR file, it failed with the same error related to Unrecognized Token, as mentioned in my original post above. Should I try upgrading now since the migration to Postgres seemed to work ok?

Here are the commands I used, as per the docs, to start Metabase having done the PostgreSQL migration.

Blockquote
export MB_DB_TYPE=postgres
export MB_DB_DBNAME=metabase
export MB_DB_PORT=5432
export MB_DB_USER=metabase
export MB_DB_PASS=
export MB_DB_HOST=localhost
java -jar metabase.jar

4

@mraonghus
I’m not sure what token it’s failing on - but I have never encountered that error before.
When I search for “Unrecognized token”, then I only find a couple of old issues, which are connecting to MongoDB.
Make sure to backup, if you want to try to upgrade. There’s many changes since 0.29, like collections, formatting and drivers.

5

Finally figured it out! Thanks for your help @flamber. Here’s what happened:

  1. Started a new instance of the newest version of Metabase (v0.32.9). This gave the same error relating to token, but started up nevertheless.
  2. Found that all the dashboards were giving errors.
  3. Went in to check the databases and they were giving the token error that I receive when starting metabase!
  4. Then I took a poke around the metabase_database table and in the details field I see that very same token that was in the error message.
  5. I then created a new database connection in the Metabase UI for another one of our databases and I could see that the details field has a radically different (plain-text JSON) format.
  6. I was then able to manually change the values in the details field and now everything is working!

Hope this might help someone else. I’ve set up daily backups so as to never get back into this situation. Thanks again @flamber!

1 Like
6

@mraonghus
Awesome, thanks for your detailed resolution!
I’m pretty sure your H2 database might have been corrupted, which lead to the “Unrecognized token” error. Now that you’re using Postgres, you should never encounter that problem again.