44 on App Engine stopped working even with later versions of jdk

chemdream · September 15, 2022, 5:22pm

Background:
I had an older version of Metabase working perfect on App Engine. The project got pushed back. Now that I'm trying it again, I'm getting errors.
I've searched for this all over. I thought it was an error because of an older version of jdk. Specifically MB44 on openJDK8. But now I'm trying to use the latest version and still getting issues.
Since i had no data yet, I started from a clean DB too.

My starting point was here because that's what worked before:
https://github.com/eddies/metabase-gae

Stack:
App Engine
Postgres
(Adding bigquery once it's actually running again)

app.yaml:

runtime: custom
env: flex

manual_scaling:
  instances: 1

env_variables:
  LANG: C.UTF-8
  LC_ALL: C.UTF-8
  MB_JETTY_PORT: 8080
  MB_DB_TYPE: postgres
  MB_DB_DBNAME: metabase
  MB_DB_PORT: 5432
  MB_DB_USER: metabase
  MB_DB_PASS: randompassword
  MB_DB_HOST: 127.0.0.1
  METABASE_SQL_INSTANCE: instance-name=tcp:5432
  CLOUD_SQL_INSTANCE: my-big-data:us-central1:metabase
#Same results with the below uncommented. I read that jdk11+ doesn't support some of the opts.  
#JAVA_OPTS: "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dfile.encoding=UTF-8"

Dockerfile:

# Dockerfile for Metabase on Google App Engine
#   https://github.com/metabase/metabase/issues/3983
# 

FROM gcr.io/google-appengine/openjdk:latest
ADD https://downloads.metabase.com/latest/metabase.jar ./metabase.jar

# Google Cloud SQL support

ADD https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 ./cloud_sql_proxy

RUN chmod +x ./cloud_sql_proxy

# Expose our default runtime port

EXPOSE 8080

# run it

CMD ./cloud_sql_proxy -instances=${CLOUD_SQL_INSTANCE}=tcp:${MB_DB_PORT} & java -jar ./metabase.jar

I get the same results whether I have the ENV VARs in App.yaml or Dockerfile.
I get the same results if I use a specific metabase version or latest.
I get the same results if I use openjdk, openjdk8 and openjdk:latest

Updating service [default] (this may take several minutes)...failed.

**ERROR:** (gcloud.app.deploy) Error Response: [9] An internal error occurred while processing task /app-engine-flex/flex_await_healthy/flex_await_healthy>2022-09-15T16:56:42.996Z434333.in.1: Start command: /bin/sh -c ./cloud_sql_proxy -instances=${CLOUD_SQL_INSTANCE}=tcp:${MB_DB_PORT} & java -jar ./metabase.jar

2022/09/15 16:56:59 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.

2022/09/15 16:57:01 Listening on 127.0.0.1:5432 for my-big-data:us-central1:metabase

2022/09/15 16:57:01 Ready for new connections

2022/09/15 16:57:01 Generated RSA key in 139.394713ms

Warning: protocol #'java-time.core/Amount is overwriting function abs

WARNING: abs already refers to: #'clojure.core/abs in namespace: java-time.core, being replaced by: #'java-time.core/abs

WARNING: abs already refers to: #'clojure.core/abs in namespace: java-time, being replaced by: #'java-time/abs

Warning: environ value /usr/lib/openjdk for key :java-home has been overwritten with /usr/lib/openjdk/jre

Exception in thread "main" java.lang.Error: Circular loading of installed providers detected

at java.nio.file.spi.FileSystemProvider.installedProviders(FileSystemProvider.java:161)

at java.nio.file.Paths.get(Paths.java:141)

at org.apache.logging.log4j.core.util.Source.toFile(Source.java:55)

at org.apache.logging.log4j.core.util.Source.<init>(Source.java:142)

at org.apache.logging.log4j.core.config.ConfigurationSource.<init>(ConfigurationSource.java:139)

at org.apache.logging.log4j.core.config.ConfigurationSource.getConfigurationSource(ConfigurationSource.java:388)

at org.apache.logging.log4j.core.config.ConfigurationSource.fromResource(ConfigurationSource.java:366)

at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:554)

at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:483)

at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:325)

at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:690)

at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:711)

at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)

at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)

at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)

at org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)

at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:137)

at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:55)

at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:47)

at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:33)

at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:363)

at com.mchange.v2.log.slf4j.Slf4jMLog.getMLogger(Slf4jMLog.java:75)

at com.mchange.v2.log.MLog.getMLogger(MLog.java:398)

at com.mchange.v2.log.MLog.getLogger(MLog.java:281)

at com.mchange.v2.log.MLog.refreshConfig(MLog.java:157)

at com.mchange.v2.log.MLog.<clinit>(MLog.java:52)

at com.mchange.v2.naming.JavaBeanReferenceMaker.<clinit>(JavaBeanReferenceMaker.java:51)

at com.mchange.v2.c3p0.impl.PoolBackedDataSourceBase.<clinit>(PoolBackedDataSourceBase.java:263)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

at clojure.lang.RT.classForName(RT.java:2209)

at clojure.lang.RT.classForName(RT.java:2218)

at metabase.db.connection_pool_setup$fn__20052.<clinit>(connection_pool_setup.clj:20)

at metabase.db.connection_pool_setup__init.load(Unknown Source)

at metabase.db.connection_pool_setup__init.<clinit>(Unknown Source)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

at clojure.lang.RT.classForName(RT.java:2209)

at clojure.lang.RT.classForName(RT.java:2218)

at clojure.lang.RT.loadClassForName(RT.java:2237)

at clojure.lang.RT.load(RT.java:449)

at clojure.lang.RT.load(RT.java:424)

at clojure.core$load$fn__6908.invoke(core.clj:6161)

at clojure.core$load.invokeStatic(core.clj:6160)

at clojure.core$load.doInvoke(core.clj:6144)

at clojure.lang.RestFn.invoke(RestFn.java:408)

at clojure.core$load_one.invokeStatic(core.clj:5933)

at clojure.core$load_one.invoke(core.clj:5928)

at clojure.core$load_lib$fn__6850.invoke(core.clj:5975)

at clojure.core$load_lib.invokeStatic(core.clj:5974)

at clojure.core$load_lib.doInvoke(core.clj:5953)

at clojure.lang.RestFn.applyTo(RestFn.java:142)

at clojure.core$apply.invokeStatic(core.clj:669)

at clojure.core$load_libs.invokeStatic(core.clj:6016)

at clojure.core$load_libs.doInvoke(core.clj:6000)

at clojure.lang.RestFn.applyTo(RestFn.java:137)

at clojure.core$apply.invokeStatic(core.clj:669)

at clojure.core$require.invokeStatic(core.clj:6038)

at clojure.core$require.doInvoke(core.clj:6038)

at clojure.lang.RestFn.invoke(RestFn.java:436)

at metabase.db.connection$loading__6789__auto____20106.invoke(connection.clj:1)

at metabase.db.connection__init.load(Unknown Source)

at metabase.db.connection__init.<clinit>(Unknown Source)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

at clojure.lang.RT.classForName(RT.java:2209)

at clojure.lang.RT.classForName(RT.java:2218)

at clojure.lang.RT.loadClassForName(RT.java:2237)

at clojure.lang.RT.load(RT.java:449)

at clojure.lang.RT.load(RT.java:424)

at clojure.core$load$fn__6908.invoke(core.clj:6161)

at clojure.core$load.invokeStatic(core.clj:6160)

at clojure.core$load.doInvoke(core.clj:6144)

at clojure.lang.RestFn.invoke(RestFn.java:408)

at clojure.core$load_one.invokeStatic(core.clj:5933)

at clojure.core$load_one.invoke(core.clj:5928)

at clojure.core$load_lib$fn__6850.invoke(core.clj:5975)

at clojure.core$load_lib.invokeStatic(core.clj:5974)

at clojure.core$load_lib.doInvoke(core.clj:5953)

at clojure.lang.RestFn.applyTo(RestFn.java:142)

at clojure.core$apply.invokeStatic(core.clj:669)

at clojure.core$load_libs.invokeStatic(core.clj:6016)

at clojure.core$load_libs.doInvoke(core.clj:6000)

at clojure.lang.RestFn.applyTo(RestFn.java:137)

at clojure.core$apply.invokeStatic(core.clj:669)

at clojure.core$require.invokeStatic(core.clj:6038)

at clojure.core$require.doInvoke(core.clj:6038)

at clojure.lang.RestFn.invoke(RestFn.java:457)

at metabase.db$loading__6789__auto____35568.invoke(db.clj:1)

at metabase.db__init.load(Unknown Source)

at metabase.db__init.<clinit>(Unknown Source)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

at clojure.lang.RT.classForName(RT.java:2209)

at clojure.lang.RT.classForName(RT.java:2218)

at clojure.lang.RT.loadClassForName(RT.java:2237)

at clojure.lang.RT.load(RT.java:449)

at clojure.lang.RT.load(RT.java:424)

at clojure.core$load$fn__6908.invoke(core.clj:6161)

at clojure.core$load.invokeStatic(core.clj:6160)

at clojure.core$load.doInvoke(core.clj:6144)

at clojure.lang.RestFn.invoke(RestFn.java:408)

at clojure.core$load_one.invokeStatic(core.clj:5933)

at clojure.core$load_one.invoke(core.clj:5928)

at clojure.core$load_lib$fn__6850.invoke(core.clj:5975)

at clojure.core$load_lib.invokeStatic(core.clj:5974)

at clojure.core$load_lib.doInvoke(core.clj:5953)

at clojure.lang.RestFn.applyTo(RestFn.java:142)

at clojure.core$apply.invokeStatic(core.clj:669)

at clojure.core$load_libs.invokeStatic(core.clj:6016)

at clojure.core$load_libs.doInvoke(core.clj:6000)

at clojure.lang.RestFn.applyTo(RestFn.java:137)

at clojure.core$apply.invokeStatic(core.clj:669)

at clojure.core$require.invokeStatic(core.clj:6038)

at clojure.core$require.doInvoke(core.clj:6038)

at clojure.lang.RestFn.invoke(RestFn.java:3894)

at metabase.core$loading__6789__auto____83943.invoke(core.clj:1)

at metabase.core__init.load(Unknown Source)

at metabase.core__init.<clinit>(Unknown Source)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

at clojure.lang.RT.classForName(RT.java:2209)

at clojure.lang.RT.classForName(RT.java:2218)

at clojure.lang.RT.loadClassForName(RT.java:2237)

at clojure.lang.RT.load(RT.java:449)

at clojure.lang.RT.load(RT.java:424)

at clojure.core$load$fn__6908.invoke(core.clj:6161)

at clojure.core$load.invokeStatic(core.clj:6160)

at clojure.core$load.doInvoke(core.clj:6144)

at clojure.lang.RestFn.invoke(RestFn.java:408)

at clojure.lang.Var.invoke(Var.java:384)

at clojure.lang.Util.loadWithClass(Util.java:251)

at metabase.core.<clinit>(Unknown Source)

flamber · September 15, 2022, 5:30pm

Hi @chemdream
Metabase does not support Java 8 anymore since v44.
https://www.metabase.com/docs/latest/installation-and-operation/java-versions

chemdream · September 15, 2022, 5:44pm

Yes I understand that. But you get the same error if you use jdk:latest. So version 11+..

Would you recommend a different docker source different than gcr.io / google-appengine / openjdk:latest that you know works? (spaces because this kept formatting it at a link..)

flamber · September 15, 2022, 5:57pm

@chemdream The "latest" is still 8. You're have to use another repo like gcr.io/distroless/java:11

chemdream · September 15, 2022, 6:02pm

Ahh. I'll try that now. Thanks!

chemdream · September 15, 2022, 6:03pm

Should I be using any OPTs for JDK11 in AppEngine?

chemdream · September 15, 2022, 6:07pm

Step 4/6 : RUN chmod +x ./cloud_sql_proxy

Step #1: ---> Running in 10d2f2f565fc

Step #1: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/bin/sh": stat /bin/sh: no such file or directory: unknown

Finished Step #1

ERROR

ERROR: build step 1 "Google Cloud console" failed: step exited with non-zero status: 1

flamber · September 15, 2022, 6:28pm

@chemdream It's a different image than what you were using, so you'll need to do a bit of work. You'll need to install bash (I guess, or whatever cloud_sql_proxy script is based on). And perhaps other things too.

chemdream · September 15, 2022, 6:39pm

@flamber thanks for all the replies. (Hopefully) last question, which java OPTS should I use with jdk11+?

Thanks!

flamber · September 15, 2022, 6:51pm

@chemdream None, hopefully the quirks with App Engine doesn't require any on Java 11.