niklas
February 16, 2023, 2:20pm
#1
About 90% of the time when we try to load metabase it responds with a 403 Forbidden.
But when you are logged in there are no more problems.
Better descirption here: forbidden 403 before even getting to the login page · Issue #28359 · metabase/metabase · GitHub
It is hosted in AWS ECS and has an ALB in front of it. Has anyone had a similar experience?
Cheers
niklas
February 16, 2023, 5:41pm
#2
I have a feeling that it might be the /api/ endpoint that returns 403. Could it do that if it is overloaded?
Luiggi
February 16, 2023, 5:58pm
#3
you haven't provided any info at all about what's your metabase version.
the issue is that you're not using the health check endpoint so ECS is killing your containers and spinning up new ones constantly, so you get 403
niklas
February 16, 2023, 6:37pm
#4
We use /api/health
for health checking and it has been running since we started it. So I don't agree with you that that would be the problem.
niklas
February 16, 2023, 6:40pm
#5
{
"browser-info": {
"language": "sv-SE",
"platform": "MacIntel",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/110.0",
"vendor": ""
},
"system-info": {
"file.encoding": "UTF-8",
"java.runtime.name": "OpenJDK Runtime Environment",
"java.runtime.version": "11.0.18+10",
"java.vendor": "Eclipse Adoptium",
"java.vendor.url": "https://adoptium.net/",
"java.version": "11.0.18",
"java.vm.name": "OpenJDK 64-Bit Server VM",
"java.vm.version": "11.0.18+10",
"os.name": "Linux",
"os.version": "4.14.301-224.520.amzn2.x86_64",
"user.language": "en",
"user.timezone": "GMT"
},
"metabase-info": {
"databases": [
"mysql"
],
"hosting-env": "unknown",
"application-database": "postgres",
"application-database-details": {
"database": {
"name": "PostgreSQL",
"version": "13.7"
},
"jdbc-driver": {
"name": "PostgreSQL JDBC Driver",
"version": "42.5.0"
}
},
"run-mode": "prod",
"version": {
"date": "2023-01-27",
"tag": "v0.45.2.1",
"branch": "release-x.45.2.x",
"hash": "1a59de7"
},
"settings": {
"report-timezone": "Europe/Stockholm"
}
}
}
niklas
February 16, 2023, 7:08pm
#6
Is there any logging on the /api endpoint? How do I figure out if the request actually reaches metabase?
Luiggi
February 16, 2023, 7:09pm
#7
do you see that the container is alive when you get the 403's? can you check that in the ECS console?
niklas
February 16, 2023, 7:27pm
#8
It says that it is "RUNNING" and started at "9 hours ago".
Right now I cannot replicate the issue anymore, but when it happened this morning it had been running for approx. 24 hrs. So I'll check tomorrow if the issue is back.