Skip to content

Conversation

@richardlau
Copy link
Member

@richardlau richardlau commented Jan 23, 2026

(first commit is from #4223)

Refs: nodejs/node#61002 (comment)
Refs: nodejs/node#58730
Refs: nodejs/node#57891
Refs: nodejs/node#57127

Deployment

  • release-digitalocean-rhel8-x64-1
  • release-ibm-rhel8-s390x-1
  • release-ibm-rhel8-x64-1
  • release-ibm-rhel8-x64-2
  • release-osuosl-rhel8-arm64-1
  • release-osuosl-rhel8-ppc64_le-1
  • test-digitalocean-rhel8-x64-1
  • test-digitalocean-rhel9-x64-1
  • test-ibm-rhel8-s390x-1
  • test-ibm-rhel8-s390x-2
  • test-ibm-rhel8-s390x-3
  • test-ibm-rhel8-s390x-4
  • test-ibm-rhel8-x64-1
  • test-ibm-rhel8-x64-2
  • test-ibm-rhel8-x64-3
  • test-ibm-rhel9-x64-1
  • test-ibm-rhel9-x64-2
  • test-linuxonecc-rhel9-s390x-1
  • test-linuxonecc-rhel9-s390x-2
  • test-linuxonecc-rhel9-s390x-3
  • test-linuxonecc-rhel9-s390x-4
  • test-osuosl-rhel8-ppc64_le-1
  • test-osuosl-rhel8-ppc64_le-2
  • test-osuosl-rhel8-ppc64_le-3
  • test-osuosl-rhel8-ppc64_le-4
  • test-osuosl-rhel9-ppc64_le-1
  • test-osuosl-rhel9-ppc64_le-2
  • test-osuosl-rhel9-ppc64_le-3
  • test-osuosl-rhel9-ppc64_le-4

Overwriting `/usr/bin/python3` on RHEL 9 breaks `subscription-manager`
(but does not on RHEL 8).

Instead create and use a Python 3.12 virtual environment for the CI.
@richardlau
Copy link
Member Author

hmm https://ci.nodejs.org/job/node-test-commit-v8-linux/6975/nodes=rhel8-ppc64le,v8test=v8test/console has failed:

[160/2476] python3 ../../build/rust/gni_impl/rustc_print_cfg.py --rust-prefix ../../third_party/rust-toolchain/bin --target powerpc64le-unknown-linux-gnu --output-path gen/build/rust/gni_impl/rustc-print-cfg-powerpc64le-unknown-linux-gnu.txt
FAILED: gen/build/rust/gni_impl/rustc-print-cfg-powerpc64le-unknown-linux-gnu.txt 
python3 ../../build/rust/gni_impl/rustc_print_cfg.py --rust-prefix ../../third_party/rust-toolchain/bin --target powerpc64le-unknown-linux-gnu --output-path gen/build/rust/gni_impl/rustc-print-cfg-powerpc64le-unknown-linux-gnu.txt
Traceback (most recent call last):
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/out.gn/ppc64.release/../../build/rust/gni_impl/rustc_print_cfg.py", line 57, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/out.gn/ppc64.release/../../build/rust/gni_impl/rustc_print_cfg.py", line 52, in main
    capture_rustc_cfg(args.rust_prefix, args.target, args.output_path)
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/out.gn/ppc64.release/../../build/rust/gni_impl/rustc_print_cfg.py", line 43, in capture_rustc_cfg
    subprocess.run(args, stdout=output_file, check=True)
  File "/usr/lib64/python3.11/subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib64/python3.11/subprocess.py", line 1955, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '../../third_party/rust-toolchain/bin/rustc'

I think the --rust-prefix ../../third_party/rust-toolchain/bin is wrong and also the --target powerpc64le-unknown-linux-gnu as well (for RHEL systems).

@richardlau
Copy link
Member Author

richardlau commented Jan 23, 2026

Based on https://github.com/ibmruntimes/v8-build/blob/main/bin/ppc64/release.gn I've tried setting rust_sysroot_absolute=\"/usr\" in tools/make-v8.sh. This got the build further but failed due to missing rustfmt, which @miladfarca warned me about last year and I have forgotten 😅.

Updated to install rust-toolset in preference to cargo and rustc (it includes those and other tools such as rustfmt). Will re-Ansible machines and try again.

@richardlau
Copy link
Member Author

Updated to install rust-toolset in preference to cargo and rustc (it includes those and other tools such as rustfmt).

doh it does not. Need to install rustfmt as a separate package.

@richardlau
Copy link
Member Author

@miladfarca pointed out that just installing clang now installs clang 20 on RHEL 8 and 9, so I've made further updates to pin at clang 19 (and also pinning rust) to the same versions as used in https://github.com/ibmruntimes/v8-build/ for consistency (to minimise differences as I try to fix the build for V8 14.4).

Latest attempt failed https://ci.nodejs.org/job/node-test-commit-v8-linux/6979/nodes=rhel8-ppc64le,v8test=v8test/console

17:28:50 usage: find_std_rlibs.py [-h] --rust-bin-dir RUST_BIN_DIR [--target TARGET]
17:28:50                          --output OUTPUT --depfile DEPFILE --depfile-target
17:28:50                          DEPFILE_TARGET [--extra-libs EXTRA_LIBS]
17:28:50                          [--rustc-revision RUSTC_REVISION]
17:28:50 find_std_rlibs.py: error: argument --rustc-revision: expected one argument

which is likely we need to pass rustc_version to gn (and maybe clang_version as well?).

@richardlau
Copy link
Member Author

@richardlau
Copy link
Member Author

Trying again with richardlau/node-1@85e420e: https://ci.nodejs.org/job/node-test-commit-v8-linux/6980/

Traceback (most recent call last):
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/out.gn/ppc64.release/../../build/rust/gni_impl/run_bindgen.py", line 125, in <module>
    main()
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/out.gn/ppc64.release/../../build/rust/gni_impl/run_bindgen.py", line 107, in main
    subprocess.check_call([args.bindgen_exe, *genargs], env=env)
  File "/usr/lib64/python3.11/subprocess.py", line 408, in check_call
    retcode = call(*popenargs, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/subprocess.py", line 389, in call
    with Popen(*popenargs, **kwargs) as p:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib64/python3.11/subprocess.py", line 1955, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '../../third_party/rust-toolchain/bin/bindgen'

https://github.com/ibmruntimes/v8-build/blob/d0d2c77ee0b2b082b1e4bbf391fb279f5e446957/bin/entry_point.sh#L35

@richardlau
Copy link
Member Author

Need to set RUSTC_BOOTSTRAP=1 since we're not using nightly builds of rust.
Refs: rust-lang/rust#148916
Refs: https://chromium-review.googlesource.com/c/chromium/src/+/7153321

which I've done in richardlau/node-1@5a7bf7e on top of the branch for nodejs/node#61002:

diff --git a/tools/make-v8.sh b/tools/make-v8.sh
index ca3b5729043..6ce591a827a 100755
--- a/tools/make-v8.sh
+++ b/tools/make-v8.sh
@@ -29,10 +29,15 @@ if [ "$ARCH" = "s390x" ] || [ "$ARCH" = "ppc64le" ]; then

   # shellcheck disable=SC2154
   case "$CXX" in
-    *clang*) GN_COMPILER_OPTS="is_clang=true clang_base_path=\"/usr\" clang_use_chrome_plugins=false treat_warnings_as_errors=false use_custom_libcxx=false" ;;
+    *clang*)
+      CLANG_VERSION=$(clang --version | awk '/clang version/ {print $3}' | cut -d. -f1)
+      RUST_VERSION=$(rustc --version | awk '/rustc/ {print $2}')
+      GN_COMPILER_OPTS="is_clang=true clang_base_path=\"/usr\" clang_use_chrome_plugins=false treat_warnings_as_errors=false use_custom_libcxx=false clang_version=\"${CLANG_VERSION}\" rustc_version=\"${RUST_VERSION}\""
+      export RUSTC_BOOTSTRAP=1
+      ;;
     *) GN_COMPILER_OPTS="treat_warnings_as_errors=false use_custom_libcxx=false" ;;
   esac
-  gn gen -v "out.gn/$BUILD_ARCH_TYPE" --args="$GN_COMPILER_OPTS is_component_build=false is_debug=false v8_target_cpu=\"$TARGET_ARCH\" target_cpu=\"$TARGET_ARCH\" v8_enable_backtrace=true $CC_WRAPPER"
+  gn gen -v "out.gn/$BUILD_ARCH_TYPE" --args="$GN_COMPILER_OPTS is_component_build=false is_debug=false v8_target_cpu=\"$TARGET_ARCH\" target_cpu=\"$TARGET_ARCH\" v8_enable_backtrace=true rust_sysroot_absolute=\"/usr\" rust_bindgen_root=\"/usr\" $CC_WRAPPER"
   ninja -v -C "out.gn/$BUILD_ARCH_TYPE" "${JOBS_ARG}" d8 cctest inspector-test
 else
   DEPOT_TOOLS_DIR="$(cd depot_tools && pwd)"

With the above, https://ci.nodejs.org/job/node-test-commit-v8-linux/6983/ passed on rhel8-s390x and is still running on rhel8-ppc64le.

@richardlau richardlau force-pushed the rustrhel branch 3 times, most recently from cf6091d to 54309a6 Compare January 24, 2026 05:02
@richardlau richardlau marked this pull request as draft January 24, 2026 05:19
@richardlau richardlau force-pushed the rustrhel branch 2 times, most recently from 261f10a to af3e78d Compare January 24, 2026 06:36
@richardlau
Copy link
Member Author

Converted to draft. I still need to look at the RHEL x64 machines and arm64 machines (of which the test ones are containers).

I had a working set up, and then I ran the playbook again to check idempotency... and then a rebuild of the V8 CI failed. It turns out this was because the package upgrade role in our playbooks was updating to newer clang (20) and rust (1.88.1).

llvm-toolset and rust-toolset are rolling packages, which means that, e.g. while clang used to be 19.1.7 it is currently 20.1.8 in RHEL 8 and 9. Rust versions are coupled with llvm versions. The V8 CI is much less tolerant of Rust versions (at least on ppc64le and s390x):

  • rust 1.84.1 (based on llvm 19.1.7) works.
  • rust 1.88.1 (based on llvm 20.1.8) does not (cp: cannot stat 'obj/build/rust/std/libadler.rlib': No such file or directory).

It might be possible to get Linux ppc64le and s390x V8 building with newer rust, but we'd still be at the mercy of the next time RHEL decide to update rust (and I'm a little uncomfortable about clang/llvm being updated under us -- the RHEL 9 test machines were bumped to clang 20 when I installed Python 3.12 and I had not noticed it).

I can pin llvm and rust using dnf versionlock, but I also need to pin dependencies otherwise the next dnf update (e.g. via our package upgrade Ansible role or directly on the machine) breaks due to version conflicts.

Pin `clang` to 19.
Add and pin `rust` 1.84 toolchain for Temporal.
Add `bindgen-cli` and `rustfmt` for V8 CI.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants