We've been informed of another issue affecting Metabase and you should protect your Metabase instance as soon as you can. Please, do the following:
If you are using Metabase Cloud, we have already secured your Metabase until the patch is available. No need to take any action. When the patch is released, we’ll upgrade your Metabase automatically.
If you are self-hosting Metabase, it’s exposed to the Internet, not behind a VPN or isolated, we strongly recommend that you block the following endpoints to your Metabase server until we release the patch and you upgrade your Metabase:
POST /api/database
PUT /api/database/:id
POST /api/setup/validate
Make sure to not block GET requests to those endpoints, or Metabase will not work properly.
If you don’t know how to make these changes to your ingress or web server, it may be better to shut down your instance until the patch is released and you upgrade your instance.
We will update our blog and send another communication once the patch is completed with instructions for how to upgrade, as well as a security advisory and CVE. If you’re running your own fork, we’ll also share instructions on what to patch.
EDIT: we pushed new versions of Metabase and you need to upgrade as soon as you can. Please upgrade to the latest version of the major version you’re in, no matter what. Keep the blocks on the endpoints preventively
EDIT2: we have received reports about botnets using the vulnerability for DDoS and crypto mining. If you see your instance using 100% of its cpu capacity or a huge spike in the traffic, please turn it off and spin up a new Metabase instance in a non compromised server with the latest and patched version
Sorry to hear that another issue has been disclosed but better to find out now and deal with it all at once. From my perspective it's nice to see that everything being dealt with openly
I'm not exactly an Nginx guru so if you notice some problem (like: is the regex for /api/database/:id right?) please let me know so I can fix my own configurations...
That said, I'm getting 405s for all the following commands, so let's hope it'll be enough.
curl -X POST -d "{id:1}" http://my.metabase.local/api/database
curl -X PUT -d "{id:1}" http://my.metabase.local/api/database/3
curl -X POST -d "{id:1}" http://my.metabase.local/api/setup/validate
What would a POST that did something look like? I have a number of them in the logs over the last 24 hours, and they all look pretty much the same:
Logs
INFO 2023-07-28T06:23:12.173207629Z [resource.labels.containerName: metabase] 2023-07-28 06:23:12,173 DEBUG middleware.log :: e[32mGET /api/session/properties 200 7.7 ms (1 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (98 total active threads) Queries in flight: 0 (0 queued)e[0m
INFO 2023-07-28T06:23:12.409986490Z [resource.labels.containerName: metabase] 2023-07-28 06:23:12,409 DEBUG middleware.log :: e[31mPOST /api/setup/validate 400 1.7 ms (0 DB calls)
INFO 2023-07-28T06:23:12.410030482Z [resource.labels.containerName: metabase] {:errors {:token "Token does not match the setup token."}}
INFO 2023-07-28T06:23:12.410036763Z [resource.labels.containerName: metabase] e[0m
---
INFO 2023-07-28T08:06:09.639203914Z [resource.labels.containerName: metabase] 2023-07-28 08:06:09,639 DEBUG middleware.log :: e[32mGET /api/session/properties 200 6.5 ms (1 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (99 total active threads) Queries in flight: 0 (0 queued)e[0m
INFO 2023-07-28T08:06:10.939962188Z [resource.labels.containerName: metabase] 2023-07-28 08:06:10,939 DEBUG middleware.log :: e[31mPOST /api/setup/validate 400 734.3 µs (0 DB calls)
INFO 2023-07-28T08:06:10.940018121Z [resource.labels.containerName: metabase] {:errors {:token "Token does not match the setup token."}}
INFO 2023-07-28T08:06:10.940024292Z [resource.labels.containerName: metabase] e[0m
---
INFO 2023-07-28T18:58:07.290984772Z [resource.labels.containerName: metabase] 2023-07-28 18:58:07,290 DEBUG middleware.log :: e[32mGET /api/session/properties 200 7.0 ms (1 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (99 total active threads) Queries in flight: 0 (0 queued)e[0m
INFO 2023-07-28T18:58:08.060513988Z [resource.labels.containerName: metabase] 2023-07-28 18:58:08,060 DEBUG middleware.log :: e[31mPOST /api/setup/validate 400 687.7 µs (0 DB calls)
INFO 2023-07-28T18:58:08.060579587Z [resource.labels.containerName: metabase] {:errors {:token "Token does not match the setup token."}}
INFO 2023-07-28T18:58:08.060586887Z [resource.labels.containerName: metabase] e[0m
---
INFO 2023-07-28T22:41:09.902216206Z [resource.labels.containerName: metabase] 2023-07-28 22:41:09,902 DEBUG middleware.log :: e[32mGET /api/session/properties 200 6.4 ms (1 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (99 total active threads) Queries in flight: 0 (0 queued)e[0m
INFO 2023-07-28T22:41:12.174958362Z [resource.labels.containerName: metabase] 2023-07-28 22:41:12,174 DEBUG middleware.log :: e[31mPOST /api/setup/validate 400 559.6 µs (0 DB calls)
INFO 2023-07-28T22:41:12.175009340Z [resource.labels.containerName: metabase] {:errors {:token "Token does not match the setup token."}}
INFO 2023-07-28T22:41:12.175015332Z [resource.labels.containerName: metabase] e[0m
---
INFO 2023-07-29T00:25:07.542554277Z [resource.labels.containerName: metabase] 2023-07-29 00:25:07,542 DEBUG middleware.log :: e[32mGET /api/session/properties 200 6.9 ms (1 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (4 idle, 0 queued) (99 total active threads) Queries in flight: 0 (0 queued)e[0m
INFO 2023-07-29T00:25:08.529590785Z [resource.labels.containerName: metabase] 2023-07-29 00:25:08,529 DEBUG middleware.log :: e[31mPOST /api/setup/validate 400 655.1 µs (0 DB calls)
INFO 2023-07-29T00:25:08.529651020Z [resource.labels.containerName: metabase] {:errors {:token "Token does not match the setup token."}}
INFO 2023-07-29T00:25:08.529657426Z [resource.labels.containerName: metabase] e[0m
(We don't use the H2 DB – can this be exploited for Metabase instances using Postgres?)
Good to know, thanks! What's the basis for knowing whether a POST to the vulnerable endpoint did anything or not in @xuheng925's logs? Is it the number of queries executed, the lack of subsequent log messages, or something else?
We’re still analyzing this and we might probably release some guide soon. I would say that you should put the blocks on the /api/setup/validate endpoint along recycling your environment and rotate all your keys. Please don’t waste any time
Am I correct if I say that if my Metabase instances were CREATED with a version older than the ones you listed as affected, then the exploit can't work since the required token isn't available in the /api/session/properties endpoint even if you're running a newer version?
Trying to understand how many of our instances might have been affected and how much (and we might have been VERY lucky if the above is true).
P.S.
Can we clear the setup-token if we find an instance that has it? A simple delete from setting where key='setup-token; should be enough.
From what I've read, if you see that "Token does not match the setup token." error you're likely ok, unless there's another attack vector...?
My logging was set up in such a way I only see the return code 400 and that would be the same for both a failed attempt and a successful one.... how did you get this level of logging?
Update: I have those "Token does not match" messages too, but they're on a separate line (the next in the log usually)) and I didn't see them until now because I was grepping journald
20% 表32'invBinding'
30% 表72'temp_table'
40% 表13'currentInv'
50% 表28'tagMapping'
60% 表68'productPartyVolume'
70% 表9'inventoryHist'
80% 表23'compAlloTransaction'
90% 表64'productMontlySalesStats'