Building custom version - 0.43.2

Hello,

I am trying to built custom version of metabase - I used source from tag v0.43.2.

I did small changes on FE and BE.

I am building it via Gitlab CICD pipeline:
This is my set up of environment:

- mkdir ~/.m2
- apt-get update -yqq
- apt-get install -y curl
- apt-get install -y gettext
- apt-get install -y openjdk-11-jdk
- apt-get install -y maven
- apt-get install -y bash rlwrap
- curl -O https://download.clojure.org/install/linux-install-1.11.1.1124.sh
- chmod +x linux-install-1.11.1.1124.sh
- ./linux-install-1.11.1.1124.sh
- curl -sL https://deb.nodesource.com/setup_14.x | bash -
- apt-get install -y nodejs
- curl -L https://www.npmjs.com/install.sh | sh
- npm install --global yarn
- echo "$MVN_SETTINGS" > ~/.m2/settings.xml
- chmod -R +xr ./bin
- apt-get install -y git-all

And then I run following commands:

- clojure -X:deps prep 
- cd modules/drivers
- clojure -X:deps prep
- cd ../..
- ./bin/build

But my pipeline is stuck on this step:

Can you please help me ?

Thank you very much.

Hi @drydog
I know that some versions of Node can cause strange behavior - have you tried v16, latest LTS?
How many resources are you giving? Could it simply be that it's running low on RAM or CPU?
Have you tried just building via this Dockerfile? https://github.com/metabase/metabase/blob/master/Dockerfile

Hello,
thank you for your response.

I didn't manage to make it work, so I will try to build it in Docker image.

But this dockerfile you provided is using CircleCI and I don't want to use that or should I only focus on this " STAGE 1: builder" part ?

Thank you

@drydog It is not using CircleCI. The image is just build to run on CircleCI. You are welcome to create your own image that does what the stage1 is doing. https://github.com/metabase/metabase-docker-ci

I see, thank you.

I have one last question.

I am running whole build in corporate environment and I need to use our maven repositories.

I am getting following error:

Cloning: https://github.com/clojure/tools.build.git
Error building classpath. Failed to read artifact descriptor for org.clojure:clojure:jar:1.10.3
org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.clojure:clojure:jar:1.10.3
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:255)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:171)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:257)
at clojure.tools.deps.alpha.extensions.maven$eval1161$fn__1163.invoke(maven.clj:112)
at clojure.lang.MultiFn.invoke(MultiFn.java:244)
at clojure.tools.deps.alpha$expand_deps$children_task__836$fn__838$fn__839.invoke(alpha.clj:406)
at clojure.tools.deps.alpha.util.concurrent$submit_task$task__505.invoke(concurrent.clj:34)
at clojure.lang.AFn.call(AFn.java:18)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.clojure:clojure:pom:1.10.3 from/to central (https://repo1.maven.org/maven2/): PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target .

I also tried to set up /home/circleci/.m2/settings.xml to use our repos but as we can see in error message it still uses https://repo1.maven.org/maven2/.

Do you know how to change there repositories or where is "default" maven config used in this "metabase/ci:java-11-clj-1.11.0.1100.04-2022-node-browsers" image ?

@drydog The link I provided in previous comment is for the CI image.
I guess there must be problems in your environment. I'm not sure how to debug your environment.

You would need to adjust the deps.edn in Metabase, since the default is to use the Maven repo:
https://clojure.org/reference/deps_and_cli#_procurers
But this is getting into Clojure land, where you'll likely find better help elsewhere.

Yes, I understand.

I use this dockerfile:

FROM metabase/ci:java-11-clj-1.11.0.1100.04-2022-node-browsers as builder
ARG MB_EDITION=oss

USER circleci
WORKDIR /home/circleci
COPY --chown=circleci settings.xml /home/circleci/.m2/

COPY --chown=circleci . metabase/
RUN cd metabase &&  chmod +x bin/build && INTERACTIVE=false CI=true MB_EDITION=$MB_EDITION bin/build

ENTRYPOINT ["/bin/sh"]

And I am again stuck on this step:

And It will timeout after some time.

Are you sure this docker image is valid?

@drydog If you only need to build, then you can just use metabase/ci:java-11-clj-1.11.0.1100.04-2022-build
As you can see, it's what is running our CI on every commit https://github.com/metabase/metabase/blob/master/.circleci/config.yml

Try doing this locally on your own computer. If it works there, then you know there's something in your CI/CD, which is causing the problem.

Hello,

I am trying it on my own computer.
Shadow-cljs - updating dependencies seems working, but I have another issue:

shadow-cljs - dependencies updated
[:app] Compiling ...
[:app] Build completed. (94 files, 29 compiled, 0 warnings, 20.20s)
$ yarn && webpack --bail
$ echo $npm_execpath | grep -q yarn || echo '\033[0;33mSorry, npm is not supported. Please use Yarn (https://yarnpkg.com/).\033[0m'
[1/5] Validating package.json...
[2/5] Resolving packages...
warning Resolution field "ansi-regex@5.0.1" is incompatible with requested version "ansi-regex@^2.0.0"
warning Resolution field "set-value@4.0.1" is incompatible with requested version "set-value@^2.0.0"
warning Resolution field "set-value@4.0.1" is incompatible with requested version "set-value@^2.0.1"
warning Resolution field "prismjs@1.27.0" is incompatible with requested version "prismjs@~1.25.0"
warning Resolution field "immer@9.0.6" is incompatible with requested version "immer@8.0.1"
warning Resolution field "ansi-regex@5.0.1" is incompatible with requested version "ansi-regex@^2.0.0"
warning Resolution field "nth-check@2.0.1" is incompatible with requested version "nth-check@^1.0.2"
warning Resolution field "ansi-regex@5.0.1" is incompatible with requested version "ansi-regex@^4.0.0"
warning Resolution field "ansi-regex@5.0.1" is incompatible with requested version "ansi-regex@^6.0.1"
warning Resolution field "ansi-regex@5.0.1" is incompatible with requested version "ansi-regex@^2.0.0"
success Already up-to-date.
$ husky install
husky - Git hooks installed
Killed
error Command failed with exit code 137.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 137.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Step failed: Running build steps for Community (OSS) Edition version UNKNOWN: version, translations, frontend, licenses, drivers, uberjar
{:via
[{:type clojure.lang.ExceptionInfo,
:message
"Running build steps for Community (OSS) Edition version UNKNOWN: version, translations, frontend, licenses, drivers, uberjar",
:data {},
:at [metabuild_common.steps$do_step invokeStatic "steps.clj" 88]}
{:type clojure.lang.ExceptionInfo,
:message "Build frontend with MB_EDITION=oss",
:data {},
:at [metabuild_common.steps$do_step invokeStatic "steps.clj" 88]}

It looks like this step failed

yarn && webpack --bail"

and this was last lines of error:

"Killed"
"error Command failed with exit code 137."
"error Command failed with exit code 137."]}}
The command '/bin/sh -c INTERACTIVE=false CI=true MB_EDITION=$MB_EDITION bin/build' returned a non-zero code: 255

@drydog Is it a completely clean environment or your existing development environment?

If you look at our Dockerfiles, then you'll see that we use them on every single PR commit, so you know that everything is working and if you cannot get something to work, then it's likely something in your environment.

Post the output of the following:

# OS and version
sed -nE 's/^PRETTY_NAME="([^"]+)"$/\1/p' /etc/os-release
# Git branch and commit
echo `git status | head -1 && git log -1 --format="commit %H"`
# Java version and architecture
java -version
# NodeJS version
node --version
# Yarn version
yarn --version
# Clojure version
clojure --help | grep Version

Hello,

I used docker image you suggested.

metabase/ci:java-11-clj-1.11.0.1100.04-2022-build

and whole dockerfile:

FROM  metabase/ci:java-11-clj-1.11.0.1100.04-2022-build as builder
ARG MB_EDITION=oss
WORKDIR /home/circleci
COPY --chown=circleci . .
RUN INTERACTIVE=false CI=true MB_EDITION=$MB_EDITION bin/build
ENTRYPOINT [ "/bin/bash" ]

And I am running docker in WSL

@drydog I'm not sure what it isn't working in your environment. You would need to debug that.

So basically, you are saying that I should debug environment in docker image.

Did I get it ?

@drydog Try using this: https://github.com/metabase/metabase/pull/18402/files

Hi,
I tried and still the same problem.

I will write all steps, maybe I am doing something wrong.

  1. git clone https://github.com/metabase/metabase.git
  2. git checkout tags/v0.43.2
  3. create Dockerfile_export_jar
  4. run DOCKER_BUILDKIT=1 docker build -f Dockerfile_export_jar --output target/uberjar/ .

Also here is the full output: https://pastebin.com/kCGs3Bmj

I am really not sure why there is a problem.

@drydog I don't know what the problem is. Try creating a VM and do the build there. If that doesn't work, then I don't know what is wrong, but clearly something with your environment.

Hello, I enabled --verbose option for yarn and I maybe have more info.

Below is shown error.

Do you know how to resolve it?

Thank you

"yarn --verbose && webpack --bail"
"verbose 0.151997 Checking for configuration file "/metabase/.npmrc"."
"verbose 0.153237 Checking for configuration file "/usr/local/share/.npmrc"."
"verbose 0.1535306 Checking for configuration file "/usr/etc/npmrc"."
"verbose 0.1538083 Checking for configuration file "/root/.npmrc"."
"verbose 0.1539597 Checking for configuration file "/metabase/.npmrc"."
"verbose 0.1561243 Checking for configuration file "/metabase/.yarnrc"."
"verbose 0.1564105 Found configuration file "/metabase/.yarnrc"."
"verbose 0.1569674 Checking for configuration file "/usr/local/share/.yarnrc"."
"verbose 0.1573353 Checking for configuration file "/usr/etc/yarnrc"."
"verbose 0.1577712 Checking for configuration file "/root/.yarnrc"."
"verbose 0.1580711 Checking for configuration file "/metabase/.yarnrc"."
"verbose 0.1582808 Found configuration file "/metabase/.yarnrc"."
"verbose 0.1672451 current time: 2022-06-20T11:20:21.995Z"
"$ echo $npm_execpath | grep -q yarn || echo '\033[0;33mSorry, npm is not supported. Please use Yarn (https://yarnpkg.com/).\033[0m'"
"[1/5] Validating package.json..."
"[2/5] Resolving packages..."
"success Already up-to-date."
"$ husky install"
"husky - Git hooks installed"
"assets by status 758 KiB [cached] 27 assets"
"Entrypoint app-main = app-main.bundle.js?99013dff59e2f3bd5ddf 1 auxiliary asset"
"Entrypoint app-public = app-public.bundle.js?f3f875019fe0f2eccb43 1 auxiliary asset"
"Entrypoint app-embed = app-embed.bundle.js?13809c525c15ec961711 1 auxiliary asset"
"Entrypoint styles = styles.bundle.js?a865d7b508a938894f85 styles.css?7f9dfaae955ab6a28a88 19 auxiliary assets"
"modules with errors 117 bytes [errors]"
" ./app-main.js 39 bytes [built] [code generated] [1 error]"
" ./app-public.js 39 bytes [built] [code generated] [1 error]"
" ./app-embed.js 39 bytes [built] [code generated] [1 error]"
"modules by path ./css/*.css 2.68 KiB (javascript) 98.8 KiB (css/mini-extract)"
" ./css/index.css 2.68 KiB [built] [code generated]"
" css ../../../node_modules/css-loader/index.js??ruleSet[1].rules[2].use[1]!../../../node_modules/postcss-loader/dist/cjs.js!./css/index.css 98.8 KiB [code generated]"
""
"ERROR in ./app-embed.js"
"Module build failed (from ../../../node_modules/babel-loader/lib/index.js):"
"Error: Cannot find module '@babel/plugin-proposal-export-default-from'"
"Require stack:"
"- /metabase/node_modules/@babel/core/lib/config/files/plugins.js"
"- /metabase/node_modules/@babel/core/lib/config/files/index.js"
"- /metabase/node_modules/@babel/core/lib/index.js"
"- /metabase/node_modules/babel-loader/lib/index.js"
"- /metabase/node_modules/loader-runner/lib/loadLoader.js"
"- /metabase/node_modules/loader-runner/lib/LoaderRunner.js"
"- /metabase/node_modules/webpack/lib/NormalModule.js"
"- /metabase/node_modules/webpack/lib/ModuleFilenameHelpers.js"
"- /metabase/node_modules/webpack/lib/BannerPlugin.js"
"- /metabase/node_modules/webpack/lib/index.js"
"- /metabase/node_modules/webpack-cli/lib/webpack-cli.js"
"- /metabase/node_modules/webpack-cli/lib/bootstrap.js"
"- /metabase/node_modules/webpack-cli/bin/cli.js"
"- /metabase/node_modules/webpack/bin/webpack.js"
" at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)"
" at resolve (node:internal/modules/cjs/helpers:108:19)"
" at resolveStandardizedName (/metabase/node_modules/@babel/core/lib/config/files/plugins.js:111:7)"
" at resolvePlugin (/metabase/node_modules/@babel/core/lib/config/files/plugins.js:55:10)"
" at loadPlugin (/metabase/node_modules/@babel/core/lib/config/files/plugins.js:63:20)"
" at loadPlugin.next ()"
" at createDescriptor (/metabase/node_modules/@babel/core/lib/config/config-descriptors.js:187:16)"
" at createDescriptor.next ()"
" at step (/metabase/node_modules/@babel/core/node_modules/gensync/index.js:261:32)"
" at evaluateAsync (/metabase/node_modules/@babel/core/node_modules/gensync/index.js:291:5)"
""
"ERROR in ./app-main.js"
"Module build failed (from ../../../node_modules/babel-loader/lib/index.js):"
"Error: Cannot find module '@babel/plugin-proposal-export-default-from'"
"Require stack:"
"- /metabase/node_modules/@babel/core/lib/config/files/plugins.js"
"- /metabase/node_modules/@babel/core/lib/config/files/index.js"
"- /metabase/node_modules/@babel/core/lib/index.js"
"- /metabase/node_modules/babel-loader/lib/index.js"
"- /metabase/node_modules/loader-runner/lib/loadLoader.js"
"- /metabase/node_modules/loader-runner/lib/LoaderRunner.js"
"- /metabase/node_modules/webpack/lib/NormalModule.js"
"- /metabase/node_modules/webpack/lib/ModuleFilenameHelpers.js"
"- /metabase/node_modules/webpack/lib/BannerPlugin.js"
"- /metabase/node_modules/webpack/lib/index.js"
"- /metabase/node_modules/webpack-cli/lib/webpack-cli.js"
"- /metabase/node_modules/webpack-cli/lib/bootstrap.js"
"- /metabase/node_modules/webpack-cli/bin/cli.js"
"- /metabase/node_modules/webpack/bin/webpack.js"
" at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)"
" at resolve (node:internal/modules/cjs/helpers:108:19)"
" at resolveStandardizedName (/metabase/node_modules/@babel/core/lib/config/files/plugins.js:111:7)"
" at resolvePlugin (/metabase/node_modules/@babel/core/lib/config/files/plugins.js:55:10)"
" at loadPlugin (/metabase/node_modules/@babel/core/lib/config/files/plugins.js:63:20)"
" at loadPlugin.next ()"
" at createDescriptor (/metabase/node_modules/@babel/core/lib/config/config-descriptors.js:187:16)"
" at createDescriptor.next ()"
" at step (/metabase/node_modules/@babel/core/node_modules/gensync/index.js:261:32)"
" at evaluateAsync (/metabase/node_modules/@babel/core/node_modules/gensync/index.js:291:5)"
""
"ERROR in ./app-public.js"
"Module build failed (from ../../../node_modules/babel-loader/lib/index.js):"
"Error: Cannot find module '@babel/plugin-proposal-export-default-from'"
"Require stack:"
"- /metabase/node_modules/@babel/core/lib/config/files/plugins.js"
"- /metabase/node_modules/@babel/core/lib/config/files/index.js"
"- /metabase/node_modules/@babel/core/lib/index.js"
"- /metabase/node_modules/babel-loader/lib/index.js"
"- /metabase/node_modules/loader-runner/lib/loadLoader.js"
"- /metabase/node_modules/loader-runner/lib/LoaderRunner.js"
"- /metabase/node_modules/webpack/lib/NormalModule.js"
"- /metabase/node_modules/webpack/lib/ModuleFilenameHelpers.js"
"- /metabase/node_modules/webpack/lib/BannerPlugin.js"
"- /metabase/node_modules/webpack/lib/index.js"
"- /metabase/node_modules/webpack-cli/lib/webpack-cli.js"
"- /metabase/node_modules/webpack-cli/lib/bootstrap.js"
"- /metabase/node_modules/webpack-cli/bin/cli.js"
"- /metabase/node_modules/webpack/bin/webpack.js"
" at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)"
" at resolve (node:internal/modules/cjs/helpers:108:19)"
" at resolveStandardizedName (/metabase/node_modules/@babel/core/lib/config/files/plugins.js:111:7)"
" at resolvePlugin (/metabase/node_modules/@babel/core/lib/config/files/plugins.js:55:10)"
" at loadPlugin (/metabase/node_modules/@babel/core/lib/config/files/plugins.js:63:20)"
" at loadPlugin.next ()"
" at createDescriptor (/metabase/node_modules/@babel/core/lib/config/config-descriptors.js:187:16)"
" at createDescriptor.next ()"
" at step (/metabase/node_modules/@babel/core/node_modules/gensync/index.js:261:32)"
" at evaluateAsync (/metabase/node_modules/@babel/core/node_modules/gensync/index.js:291:5)"
""
"webpack 5.69.0 compiled with 3 errors in 4130 ms"
"verbose 6.3645907 Error: Command failed with exit code 1."
" at ProcessTermError.ExtendableBuiltin (/usr/lib/node_modules/yarn/lib/cli.js:721:66)"
" at ProcessTermError.MessageError (/usr/lib/node_modules/yarn/lib/cli.js:750:123)"
" at new ProcessTermError (/usr/lib/node_modules/yarn/lib/cli.js:790:113)"
" at /usr/lib/node_modules/yarn/lib/cli.js:34406:30"
" at Generator.throw ()"
" at step (/usr/lib/node_modules/yarn/lib/cli.js:310:30)"
" at /usr/lib/node_modules/yarn/lib/cli.js:323:13"
" at processTicksAndRejections (node:internal/process/task_queues:96:5)"
"info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command."
"info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command."],

@drydog It errors on this:

"Module build failed (from ../../../node_modules/babel-loader/lib/index.js):"
"Error: Cannot find module '@babel/plugin-proposal-export-default-from'"

It almost looks like you are using Node18: https://github.com/metabase/metabase/pull/23402
We're working on removing babel, but it's a longer process: https://github.com/metabase/metabase/pull/22409

I am using node 16.

This is the script: https://deb.nodesource.com/setup_lts.x

Should I switch to older version? (maybe 14 ?)

@drydog As you can see, it's working without any problems in our CI, so I cannot tell you what to do, since there's something off in your setup. Try different things, but you'll need to debug yourself.