Problem after dist upgrade (Ubuntu 16.04 --> 18.04)

Hi,

I upgraded my server where Metabase is running from Ubuntu 16.04 to 18.04.
I’m using MySQL as internal database.

When I try to open the page, I receive this message:

{
"message": "Stringa non valida: deve essere o \"true\" o \"false\" (case-insensitive).",
"type": "class java.lang.Exception",
"stacktrace": [
"--> models.setting$string__GT_boolean.invokeStatic(setting.clj:177)",
"models.setting$string__GT_boolean.invoke(setting.clj:171)",
"models.setting$get_boolean.invokeStatic(setting.clj:187)",
"models.setting$get_boolean.invoke(setting.clj:179)",
"models.setting$get.invokeStatic(setting.clj:229)",
"models.setting$get.invoke(setting.clj:223)",
"models.setting$setting_fn$fn__19024.invoke(setting.clj:454)",
"public_settings$public_settings.invokeStatic(public_settings.clj:214)",
"public_settings$public_settings.invoke(public_settings.clj:209)",
"routes.index$load_entrypoint_template.invokeStatic(index.clj:72)",
"routes.index$load_entrypoint_template.invoke(index.clj:69)",
"routes.index$entrypoint.invokeStatic(index.clj:93)",
"routes.index$entrypoint.invoke(index.clj:88)",
"middleware.exceptions$catch_uncaught_exceptions$fn__64267.invoke(exceptions.clj:104)",
"middleware.exceptions$catch_api_exceptions$fn__64264.invoke(exceptions.clj:92)",
"middleware.log$log_api_call$fn__65919.invoke(log.clj:162)",
"middleware.security$add_security_headers$fn__64230.invoke(security.clj:122)",
"middleware.json$wrap_json_body$fn__65624.invoke(json.clj:63)",
"middleware.json$wrap_streamed_json_response$fn__65642.invoke(json.clj:97)",
"middleware.session$bind_current_user$fn__60792$fn__60793.invoke(session.clj:209)",
"middleware.session$do_with_current_user.invokeStatic(session.clj:184)",
"middleware.session$do_with_current_user.invoke(session.clj:177)",
"middleware.session$bind_current_user$fn__60792.invoke(session.clj:208)",
"middleware.session$wrap_current_user_id$fn__60777.invoke(session.clj:161)",
"middleware.session$wrap_session_id$fn__60762.invoke(session.clj:123)",
"middleware.auth$wrap_api_key$fn__64176.invoke(auth.clj:27)",
"middleware.misc$maybe_set_site_url$fn__65946.invoke(misc.clj:56)",
"middleware.misc$bind_user_locale$fn__65949.invoke(misc.clj:72)",
"middleware.misc$add_content_type$fn__65934.invoke(misc.clj:28)",
"middleware.misc$disable_streaming_buffering$fn__65957.invoke(misc.clj:87)"
]
}

Any idea?

Thanks

Hi @marciuz
Please post Diagnostic Info from Admin > Troubleshooting.
When do you see this error?
Do you see other errors while starting Metabase before this happens?
Did you also upgrade MySQL?

Hi @flamber,

I cannot do nothing, this message appears in all the pages and in the log of the service start. This is the installation: http://metabase.ebookecm.it/.
Here the log: https://gist.github.com/marciuz/30c28828a590e81e3fae6e4c8c2110eb

I think yes, also MySQL has been updated in the dist-update, but it’s still at the 5.7.x version.

I tried also to restore the DB backup of this morning, but it seems the error is the same.

:frowning:

@marciuz I’m not sure what’s going on. Maybe the upgrade has corrupted the MySQL or changed something in the structure. It’s sending bad stuff back to Metabase Unrecognized token 'qr41q2': was expecting ('true', 'false' or 'null')

Try starting just Metabase (without using MySQL) with H2, just as a test - delete the H2 database after the test.
java -DMB_DB_TYPE=h2 -jar metabase.jar

I have this error, in the log:
Nov 19 23:31:33 epeems java[14129]: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'qr41q2': was expecting ('true', 'false' or 'null')

So, I tried to find the string “qr41q2” in a sql dump of the database and it seems in this record:

INSERT INTO metabase_database VALUES (2,‘2018-11-05 12:22:39’,‘2019-10-29 12:21:42’,‘ECM’,NULL,'qr41q2/8t9kXSiFpJ5lmjgMD2wB2h52qbeSPGke8pTNOetSZh […]

the string is the “detail” field of the table metabase_database.

It seems an encrypted string that something in charge of the decryption cannot parse correctly, so the JSON parser raise an error…

@marciuz
Are you using MB_ENCRYPTION_SECRET_KEY (ref) and then forgot to add that again after the upgrade?

No, but this is the key, I presume.

I had this error before,
Nov 19 23:45:15 epeems java[14990]: 11-19 23:45:15 #033[1mWARN util.encryption#033[0m :: Cannot decrypt encrypted string. Have you changed or forgot to set MB_ENCRYPTION_SECRET_KEY? Message seems corrupt or manipulated.

I tried to disable the key, but now I have of course the error "Unrecognized token ‘qr41q2’: … "

So, I de-commented the MB_ENCRYPTION_SECRET_KEY, that it seems to be not recognized anymore!

So, I found an old backup without encryption, I updated the details field in database and disabled the encryption by commenting the MB_ENCRYPTION_SECRET_KEY env var.

Now it starts correctly BUT, and this is very odd, I already have the error in the page at http://metabase.ebookecm.it
Can be some kind of cache?

@marciuz

Okay, I’m fairly sure you’re seeing this issue:
https://github.com/metabase/metabase/issues/11048
I have no idea how to reproduce the cause of this, so I’m not sure what the fix is.

How long is your encryption key (characters)? Do NOT post your full key!

The errors looks similar to your original syslog, so I’m not sure what’s going on, since it starts now. I don’t think it’s a cache.

When this problem is solved, then I would highly recommend that you use a reverse-proxy in front of Metabase. This can be used for many things - even simple free SSL certificates via Let’s Encrypt. You can use any webserver like Nginx, Caddy or Apache. And you can find many great tutorials on https://www.digitalocean.com/community/tutorials?q=reverse+proxy

@marciuz Have you tried manually replacing the metabase_database from the unencrypted backup into the latest encrypted backup (make a new file of course) and import that?
If I remember correctly, only the database information is encrypted.

Yes, it seems this issue. The string is 44 chars long.
I’m already using Apache as reverse Proxy. I can also add a certificate (I have a SSL wildcard on this domain), I have not yet done so because this site was of strictly personal use.

Anyhow, I don’t understand what’s is going on: now I disabled the encryption, I restored the JSON data for the connection in the details field, Metabase starts correctly and syslog is clear and without error…
but I still have the same problem on the website. Why? :weary:

Yes I do!
And it starts apparently well, but I already have the problem on the pages! :expressionless:

@marciuz I’m fairly sure you’re running Metabase directly without Apache, since it’s returning Server: Jetty(9.4.z-SNAPSHOT) and because it’s returning the error output instead of a nice 500 Error Page.
Could it be that you accidentally started multiple Metabase instances? Check running processes:
https://www.cyberciti.biz/faq/what-process-has-open-linux-port/

Uhm, no, I don’t have multiple instances…

It’s strange, do you see this screen?

@marciuz Yes - just looks a little different on Firefox. Unless your Apache configuration is to pipe through everything (also errors, which is not the default), then I’m not sure what’s going on.
If you have a backup of the server, then I would just revert to that - you still have 1,5 years until 16.04 is unsupported - so plenty of time to figure out what’s going on with that upgrade.

I don’t have a backup of the server, only a daily backup of the database.
I re-installed the application on another server, I restored the DB (an old copy before the encryption of the credentials) and it works.

Some conclusions on the experience. Beyond the circumstance or the bug (if it is) that with a version upgrade of Ubuntu the system breaks down, it remains difficult to debug (the system from messages a bit 'vague: string not valid, it expect “true” or “false”… but where, where!) but particularly there is a potentially disastrous behavior with encrypting database credentials.

In fact, if the encrypted key gets corrupted (or if the decrypter somehow changes… I see that you still do not understand), all the backup of a potentially lot of work of the Metabase installation, is likely to be unusable.

A documentation page that explains how to recover a database with encrypted connection details that have become inaccessible (for whatever reason) could be useful.

Bye and thanks for your support.

Marcello

Unfortunately,
after 5 minutes of work, the new installation send the exactly same error.
I don’t have really idea now about how to do :frowning:

@marciuz There’s no simple way to recover if the encryption gets corrupted, as noted in the documentation as well (last paragraph):
https://www.metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
Very interesting that the new installation also starts failing after few minutes. If you re-make that setup, does it run for a few minutes before failing again?

The solution is here: https://github.com/metabase/metabase/issues/11048#issuecomment-644924920

1 Like