Allen is the LHCb HLT1 Trigger Software Framework. It is a large and complex codebase, with many dependencies and execution paths. Porting it to a new architecture is an interesting challenge. It does have several backends supported natively, but RISC-V is not one of them. But why not try to see if we can make it work?

So I started this as a simple-sounding experiment:

Can Allen build and run on riscv64?

That question is too vague to be useful. "Run Allen" can mean many things: CUDA, HIP, CPU, standalone, Gaudi-integrated, full HLT1, a tiny sequence, native hardware, cross-compilation, QEMU, monitoring, no monitoring, real input, synthetic input, and so on.

So the first thing I had to do was make the target smaller.

The goal became this:

Build Allen for riscv64 Linux, using the standalone CPU backend, and run a
small real sequence on a small MDF input.

No CUDA. No HIP. No RVV optimization. No throughput claim. No full LHCb stack integration. The point was portability, not performance.

The first milestone was a 10-event velo run inside a riscv64 QEMU system VM, with Allen linked against a ROOT build made natively inside that VM. It read the input MDF, ran the VELO sequence, completed processing, and wrote a ROOT monitoring file.

After that worked, I pushed the same path further and tried the full default HLT1 sequence, hlt1_pp_default. That also ran on riscv64, with one important detail: the Debug build was too slow under qemu-system, but the Release build completed a one-event full-HLT1 smoke test and wrote a ROOT monitoring file.

This post is about the path to get there. Not just the final version, but the debugging trail: what I checked first, what failed, why I changed direction, and what ended up being the reproducible route.

Starting with the smallest useful target

[!NOTE]
All the code and scripts are available in the Allen repository in melashri_riscv branch.

Allen has several possible execution paths, and choosing the wrong first target would make the port much harder than it needed to be.

I did not want the first milestone to include GPU runtimes or vector extensions. If the scalar CPU path does not build, there is no point arguing about RVV yet. So the initial target was:

The sequence I chose was velo.

That was small enough to debug but still meaningful. It uses real Allen configuration and real MDF input. It is not just --help, and it is not a fake unit test that can accidentally avoid the interesting parts of the system.

The rough command I wanted to make work was:

./Allen \
  --sequence velo \
  --mdf input/minbias/mdf/MiniBrunel_2018_MinBias_FTv4_DIGI_retinacluster_v1.mdf \
  -g input/detector_configuration \
  -n 10 \
  -t 1 \
  -r 1 \
  --events-per-slice 10 \
  -m 128 \
  --host-memory 128 \
  -v 4 \
  -p 1

That command changed along the way, but it was the mental model I had in mind. If I could
make that shape of command work on riscv64, then I could claim a real proof of concept.

First rule: get an x86_64 reference before touching RISC-V

It is tempting to start by adding riscv64 to CMake and immediately try a cross build. But I avoided that.

Before changing anything architecture-related, I needed a known-good x86_64 CPU run in the same checkout. Otherwise every later failure would be ambiguous: is this a RISC-V problem, a local dependency problem, a ROOT problem, a broken sequence, or just me invoking Allen incorrectly?

So I configured a standalone CPU build for x86_64:

env PYTHONPATH=$HOME/projects/Allen/artifacts/python-deps/usr/lib/python3/dist-packages \
  ROOTSYS=/home/linuxbrew/.linuxbrew/Cellar/root/6.40.00 \
  ROOT_DIR=/home/linuxbrew/.linuxbrew/Cellar/root/6.40.00/share/root/cmake \
  CMAKE_PREFIX_PATH=/home/linuxbrew/.linuxbrew/Cellar/root/6.40.00 \
  LHCBROOT=$HOME/lhcb-stack/LHCb \
  GAUDIROOT=$HOME/lhcb-stack/Gaudi \
  PARAMFILESROOT=$HOME/lhcb-stack/ParamFiles \
  PKG_CONFIG_EXECUTABLE=/usr/bin/pkg-config \
  cmake -S $HOME/projects/Allen -B build-x86-cpu-system \
  -GNinja \
  -DSTANDALONE=ON \
  -DTARGET_DEVICE=CPU \
  -DSEQUENCES=velo \
  -DPKG_CONFIG_EXECUTABLE=/usr/bin/pkg-config \
  -DPython_EXECUTABLE=/usr/bin/python3 \
  -DCMAKE_BUILD_TYPE=Debug \
  -DBUILD_TESTING=OFF

[!NOTE]
I have ROOT installed on my Local Linux Machine through linuxbrew (Homebrew for Linux). And I had the latest version available at the time.

There are a few details in that command that look annoying, but they mattered.

The host had Linuxbrew in PATH, so pkg-config and python3 could resolve to Linuxbrew versions instead of the system versions. For this build I wanted the apt-installed packages and Python development headers, so I forced:

-DPKG_CONFIG_EXECUTABLE=/usr/bin/pkg-config
-DPython_EXECUTABLE=/usr/bin/python3

I also pointed CMake at local LHCb, Gaudi, and ParamFiles checkouts (although the build can automatically find them if LHCBROOT, GAUDIROOT, and PARAMFILESROOT are set, it is safer to be explicit about the paths, especially that I didn't have CVMFS available on this machine):

LHCBROOT=$HOME/lhcb-stack/LHCb
GAUDIROOT=$HOME/lhcb-stack/Gaudi
PARAMFILESROOT=$HOME/lhcb-stack/ParamFiles

This was a standalone build, but it still needed those paths for headers, configuration, and runtime parameters.

The C++ standard mismatch

The first useful failure was not RISC-V related at all.

The checkout used C++20 library features like std::span and std::source_location, but the ROOT installation advertised C++17. CMake was deriving the standard from ROOT/Gaudi and ended up too low.

The fix was small: after deriving the standard, make sure Allen requires at least C++20. That made the x86_64 CPU build configure and build:

ninja -C build-x86-cpu-system Allen

This is one of those fixes that looks unrelated to the port, but it was important. If the baseline x86_64 CPU build cannot compile, then a RISC-V build failure tells you almost nothing. As we will see later, most of the port effort was actually making sure that we have a ROOT version that plays nice with RISC-V, and that the build system can find it.

If the x86_64 CPU build cannot compile, then you cannot even get to the point of debugging ROOT issues on RISC-V, because you don't have a working reference to compare against.

Sequence generation dependencies

The next issue was Python.

Building the velo sequence required Python modules that were not available in the system Python path on the host (I didn't use the wrapper because I'm not using toolchain from CVMFS, so I didn't have the Python environment from CVMFS either). The missing modules were:

I did not want to install random Python packages globally just to make the build work. So I staged them under the workspace:

mkdir -p artifacts/python-deps artifacts/python-deps-debs

(
  cd artifacts/python-deps-debs
  apt download python3-wrapt python3-pydot python3-mpmath python3-sympy
)

for deb in artifacts/python-deps-debs/*.deb; do
  dpkg-deb -x "$deb" artifacts/python-deps
done

/usr/bin/python3 -m pip install \
  --target "$ALLEN_REPO/artifacts/python-deps/usr/lib/python3/dist-packages" \
  pyeda==0.29.0

export PYTHONPATH="$ALLEN_REPO/artifacts/python-deps/usr/lib/python3/dist-packages"

This is not the prettiest Python environment management story. A uv virtual environment would probably be cleaner. But for a portability PoC, the important thing was that the dependencies were local, explicit, and reproducible enough for the experiment.

There was another small but important detail: at runtime Allen shells out to python3 for dynamic sequence generation. On this machine, python3 in PATH could be the Linuxbrew Python. So the reference run used:

PATH=/usr/bin:/bin:$PATH

That forced runtime sequence generation to use system Python.

The detector configuration was not fully local

The command I wanted to run used:

-g input/detector_configuration

But the tracked detector configuration had a problem on this host: input/detector_configuration/magfield.bin pointed at CVMFS, and CVMFS was not available locally.

The workaround was to make a local copy and replace only the magnetic field
symlink:

ninja -C build-x86-cpu-system generate_magfields

mkdir -p artifacts/detector_configuration_x86_ref
cp -a input/detector_configuration/. artifacts/detector_configuration_x86_ref/
rm -f artifacts/detector_configuration_x86_ref/magfield.bin
ln -s ../../build-x86-cpu-system/local_magfields/magfield.v8r1.down.bin \
  artifacts/detector_configuration_x86_ref/magfield.bin

From that point on, the local runtime commands used:

-g artifacts/detector_configuration_x86_ref

That detail matters because otherwise a "RISC-V runtime failure" could just be a missing magnetic field map file.

The x86_64 reference run

With those pieces in place, the x86_64 CPU reference run completed:

PATH=/usr/bin:/bin:$PATH \
PYTHONPATH=$HOME/projects/Allen/artifacts/python-deps/usr/lib/python3/dist-packages \
./build-x86-cpu-system/Allen \
  --sequence velo \
  --mdf input/minbias/mdf/MiniBrunel_2018_MinBias_FTv4_DIGI_retinacluster_v1.mdf \
  -g artifacts/detector_configuration_x86_ref \
  -n 10 \
  -t 1 \
  -r 1 \
  --events-per-slice 10 \
  -m 128 \
  --host-memory 128 \
  -v 4 \
  -p 1

The important output was:

Read 10 events into 0
n_transposed 10
Processing complete

The sequence was:

initialize_event_lists
initialize_number_of_events
velo_banks
calculate_number_of_retinaclusters_each_sensor_pair
decode_retinaclusters
velo_search_by_triplet
velo_three_hit_tracks_filter
velo_copy_track_hit_number
velo_consolidate_tracks
velo_kalman_filter

That gave me the reference. Now RISC-V work had something to compare against.

Checking the obvious architecture traps

Before adding new files, I searched for x86 assumptions:

rg -n "#include\s*[<\"](immintrin|xmmintrin|emmintrin|smmintrin|cpuid)\.h|__builtin_cpu_supports|_mm_|AVX|SSE|haswell|x86_64_v3|__x86_64__|__amd64__|CPU_ARCH|march|mtune" \
  CMakeLists.txt cmake backend host device main stream configuration checker integration test mdf zmq \
  -g '!build*'

The CPU backend was better than I expected. The explicit CPU feature code lived in backend/include/CPUID.h and backend/src/CPUID.cpp, and the x86-specific parts were already guarded with __x86_64__.

The bigger problems were in build-system assumptions which are:

This was good news. I did not need to start by modifying algorithms or core files. I needed to
make the build system understand the architecture and prevent accidental host dependency leakage.

Adding RISC-V architecture detection

The first RISC-V build-system change was to teach the architecture detection about RISC-V.

The useful preprocessor macros are:

__riscv
__riscv_xlen

So the detection logic can distinguish 32-bit and 64-bit RISC-V. For this PoC, the target was specifically riscv64.

I verified the compiler's view with:

printf '#if defined(__riscv)\n#if defined(__riscv_xlen) && __riscv_xlen == 64\n#error cmake_ARCH riscv64\n#else\n#error cmake_ARCH riscv\n#endif\n#elif defined(__x86_64__)\n#error cmake_ARCH x86_64\n#else\n#error cmake_ARCH unknown\n#endif\n' |
  riscv64-linux-gnu-gcc -x c -c -

The result identified riscv64, which gave the CMake detection something concrete to key off.

Then I added a standalone CPU flag branch for RISC-V.

For x86_64, CPU_ARCH=native commonly turns into something like -march=native. On RISC-V, I did not want to apply x86-style assumptions or invent a tuned ISA string at this stage. The conservative behavior was:

That kept the PoC somehow generic.

The cross toolchain file

The next step was a toolchain file:

cmake/toolchains/riscv64-linux-gnu.cmake

The core idea was simple:

One thing I deliberately did not do was set:

CMAKE_SYSROOT=/usr/riscv64-linux-gnu

That sounds natural, but on this Ubuntu setup it was wrong.

The compiler reported:

riscv64-linux-gnu-g++ -print-sysroot

as:

/

But a trivial default cross link worked:

printf 'int main(){return 0;}\n' |
  riscv64-linux-gnu-g++ -x c++ -o /tmp/riscv64-default-smoke -

But forcing --sysroot=/usr/riscv64-linux-gnu broke the link because the compiler could no longer find the expected libc paths.

So the toolchain file left CMAKE_SYSROOT unset. Instead, it accepted an explicit workspace sysroot for target dependencies:

-DALLEN_RISCV64_SYSROOT=$HOME/projects/Allen/artifacts/riscv64-sysroot

That separation became important later.

First RISC-V configure: the compiler was not the blocker

The initial RISC-V configure looked like this:

env PYTHONPATH=$HOME/projects/Allen/artifacts/python-deps/usr/lib/python3/dist-packages \
  LHCBROOT=$HOME/lhcb-stack/LHCb \
  GAUDIROOT=$HOME/lhcb-stack/Gaudi \
  PARAMFILESROOT=$HOME/lhcb-stack/ParamFiles \
  PKG_CONFIG_EXECUTABLE=/usr/bin/pkg-config \
  cmake -S $HOME/projects/Allen -B build-riscv-cpu-phase2 \
  -GNinja \
  -DCMAKE_TOOLCHAIN_FILE=cmake/toolchains/riscv64-linux-gnu.cmake \
  -DSTANDALONE=ON \
  -DTARGET_DEVICE=CPU \
  -DSEQUENCES=velo \
  -DPKG_CONFIG_EXECUTABLE=/usr/bin/pkg-config \
  -DPython_EXECUTABLE=/usr/bin/python3 \
  -DCMAKE_BUILD_TYPE=Debug \
  -DBUILD_TESTING=OFF

This got through the compiler checks. It did not ask for CUDA. It did not ask for HIP. That was already a useful milestone.

The first real blocker was:

find_package(fmt)

That was exactly the kind of blocker I wanted at this stage. Not "CMake thinks RISC-V is
unknown", not "CPU backend includes x86 intrinsics unconditionally", not "TARGET_DEVICE=CPU still needs CUDA". Just a missing target dependency.

Building a workspace-local riscv64 dependency sysroot

The host had x86_64 development packages installed, but those cannot be used to link a riscv64 executable.

I needed target libraries and headers:

The ordinary packages were available from Ubuntu ports. I created a workspace-local apt state and extracted riscv64 packages into:

artifacts/riscv64-sysroot

The setup started like this:

mkdir -p \
  artifacts/riscv64-apt/lists/partial \
  artifacts/riscv64-apt/cache/archives/partial \
  artifacts/riscv64-apt/sourceparts \
  artifacts/riscv64-debs \
  artifacts/riscv64-sysroot

cat > artifacts/riscv64-apt/sources.list <<'EOF'
deb [arch=riscv64] http://ports.ubuntu.com/ubuntu-ports noble main restricted universe multiverse
deb [arch=riscv64] http://ports.ubuntu.com/ubuntu-ports noble-updates main restricted universe multiverse
deb [arch=riscv64] http://ports.ubuntu.com/ubuntu-ports noble-security main restricted universe multiverse
EOF

Then I used an apt-get wrapper that pointed apt at this local state:

repo_root="$PWD"
apt_riscv() {
  apt-get \
    -o Dir::Etc::sourcelist="${repo_root}/artifacts/riscv64-apt/sources.list" \
    -o Dir::Etc::sourceparts="${repo_root}/artifacts/riscv64-apt/sourceparts" \
    -o Dir::State="${repo_root}/artifacts/riscv64-apt" \
    -o Dir::State::status="${repo_root}/artifacts/riscv64-apt/status" \
    -o Dir::State::lists="${repo_root}/artifacts/riscv64-apt/lists" \
    -o Dir::Cache="${repo_root}/artifacts/riscv64-apt/cache" \
    -o APT::Architecture=riscv64 \
    -o APT::Architectures=riscv64 \
    "$@"
}

touch artifacts/riscv64-apt/status
apt_riscv update

Then I downloaded and extracted the target packages:

(
  cd artifacts/riscv64-debs
  apt_riscv download \
    libfmt-dev:riscv64 libfmt9:riscv64 \
    libzmq3-dev:riscv64 libzmq5:riscv64 \
    libsodium-dev:riscv64 libsodium23:riscv64 \
    cppzmq-dev \
    libkrb5-dev:riscv64 krb5-multidev:riscv64 \
    libgssapi-krb5-2:riscv64 libkrb5-3:riscv64 \
    libk5crypto3:riscv64 libkrb5support0:riscv64 \
    libcom-err2:riscv64 libkeyutils1:riscv64 \
    libpgm-dev:riscv64 libpgm-5.3-0t64:riscv64 \
    libnorm-dev:riscv64 libnorm1t64:riscv64 \
    libbsd-dev:riscv64 libbsd0:riscv64 \
    libmd-dev:riscv64 libmd0:riscv64 \
    libxml2-dev:riscv64 libxml2:riscv64 \
    zlib1g-dev:riscv64 zlib1g:riscv64 \
    python3-dev:riscv64 libpython3-dev:riscv64 \
    python3.12-dev:riscv64 libpython3.12-dev:riscv64 \
    libpython3.12t64:riscv64 python3.12:riscv64 \
    libexpat1-dev:riscv64 libexpat1:riscv64 \
    libfreetype-dev:riscv64 libfreetype6:riscv64 \
    liblz4-dev:riscv64 liblz4-1:riscv64 \
    liblzma-dev:riscv64 liblzma5:riscv64 \
    libzstd-dev:riscv64 libzstd1:riscv64 \
    libpng-dev:riscv64 libpng16-16t64:riscv64 \
    libbrotli-dev:riscv64 libbrotli1:riscv64 \
    libbz2-dev:riscv64 libbz2-1.0:riscv64
)

apt_riscv -y --download-only --no-install-recommends install \
  catch2:riscv64 \
  nlohmann-json3-dev:riscv64 \
  librange-v3-dev:riscv64 \
  libtbb-dev:riscv64 \
  libboost-dev:riscv64 \
  libboost-filesystem-dev:riscv64 \
  libboost-iostreams-dev:riscv64 \
  libboost-thread-dev:riscv64 \
  libboost-regex-dev:riscv64 \
  libboost-serialization-dev:riscv64 \
  libboost-program-options-dev:riscv64 \
  libboost-test-dev:riscv64

find artifacts/riscv64-debs artifacts/riscv64-apt/cache/archives \
  -maxdepth 1 -name '*.deb' \
  -exec dpkg-deb -x {} artifacts/riscv64-sysroot \;

One small merged-usr compatibility fix was needed:

if [ ! -e artifacts/riscv64-sysroot/lib ] && [ -d artifacts/riscv64-sysroot/usr/lib ]; then
  ln -s usr/lib artifacts/riscv64-sysroot/lib
fi

After that, the ordinary target dependency checks passed.

Then CMake reached the next serious blocker:

find_package(ROOT REQUIRED COMPONENTS RIO Core Hist Tree)

Ubuntu ports did not have suitable ROOT packages for riscv64, and the extracted sysroot did not contain ROOTConfig.cmake.

So the problem shifted from "port Allen" to "provide a real riscv64 ROOT".

The tempting no-ROOT branch

At this point I tried a diagnostic path: make ROOT optional for a standalone CPU build.

That branch did work as a diagnostic. It allowed a no-ROOT riscv64 build to compile and Allen -h to run under QEMU. It also forced me to identify ROOT
surfaces in the codebase which are not just monitoring, but also:

But this was not a satisfying final solution.

Disabling ROOT meant the proof of concept avoided real parts of Allen's normal standalone behavior. It did not validate ROOT output. It did not validate ROOT MDF I/O paths. It risked becoming a special "RISC-V demo mode" instead of a proper port (or at least a valid demonstration).

So that branch was later removed.

By then, I decided that I need a real ROOT. By that I mean native riscv64 ELF shared objects built for the target, not host binaries or a fake no-ROOT mode. The next step was to get that.

Cross-building ROOT for riscv64

I added a helper script to build ROOT 6.40.00 (latest at the time) into:

artifacts/root-riscv64

The helper used:

The source tarball was:

artifacts/root-src/root_v6.40.00.source.tar.gz

The correct artifact matters here. ROOT's prebuilt Linux-...-x86_64 tarballs are host binaries. They cannot be used as a riscv64 target dependency.

The cross ROOT build had its own set of fixes:

Eventually the cross-built ROOT installed:

artifacts/root-riscv64/cmake/ROOTConfig.cmake

and:

artifacts/root-riscv64/bin/root-config --version

reported:

6.40.00

The important libraries were real riscv64 ELF shared objects:

That was enough for Allen to configure and link against ROOT.

I added an explicit CMake input:

-DALLEN_ROOT_PREFIX=$HOME/projects/Allen/artifacts/root-riscv64

This was better than relying on ROOTSYS or generic CMAKE_PREFIX_PATH, especially during cross-compilation. A target ROOT prefix should be treated as a target dependency, and not accidentally mixed with the host ROOT.

The RISC-V configure became:

env PYTHONPATH=$HOME/projects/Allen/artifacts/python-deps/usr/lib/python3/dist-packages \
  LHCBROOT=$HOME/lhcb-stack/LHCb \
  GAUDIROOT=$HOME/lhcb-stack/Gaudi \
  PARAMFILESROOT=$HOME/lhcb-stack/ParamFiles \
  PKG_CONFIG_EXECUTABLE=/usr/bin/pkg-config \
  cmake -S $HOME/projects/Allen -B build-riscv-cpu-root-phase3 \
  -GNinja \
  -DCMAKE_TOOLCHAIN_FILE=cmake/toolchains/riscv64-linux-gnu.cmake \
  -DALLEN_RISCV64_SYSROOT=$HOME/projects/Allen/artifacts/riscv64-sysroot \
  -DALLEN_ROOT_PREFIX=$HOME/projects/Allen/artifacts/root-riscv64 \
  -DSTANDALONE=ON \
  -DTARGET_DEVICE=CPU \
  -DSEQUENCES=velo \
  -DPKG_CONFIG_EXECUTABLE=/usr/bin/pkg-config \
  -DPython_EXECUTABLE=/usr/bin/python3 \
  -DPython3_EXECUTABLE=/usr/bin/python3 \
  -DCMAKE_BUILD_TYPE=Debug \
  -DBUILD_TESTING=OFF

Then:

env PYTHONPATH=$HOME/projects/Allen/artifacts/python-deps/usr/lib/python3/dist-packages \
  LHCBROOT=$HOME/lhcb-stack/LHCb \
  GAUDIROOT=$HOME/lhcb-stack/Gaudi \
  PARAMFILESROOT=$HOME/lhcb-stack/ParamFiles \
  PKG_CONFIG_EXECUTABLE=/usr/bin/pkg-config \
  ninja -C build-riscv-cpu-root-phase3 Allen

This completed. It built a riscv64 Allen executable, generated velo.json, and executed the target default_properties helper under QEMU during the build.

That was the first major "this might actually work" point.

Running the riscv64 binary under QEMU user-mode

The first runtime smoke was help output.

One small Allen-specific detail: this executable accepts -h, not --help. It prints usage and exits with 255, so the exit code alone is not a crash signal.

The command looked like this:

timeout 60s env \
  PATH=/usr/bin:/bin:$PATH \
  PYTHONPATH=$HOME/projects/Allen/artifacts/python-deps/usr/lib/python3/dist-packages \
  ROOTSYS=$HOME/projects/Allen/artifacts/root-riscv64 \
  QEMU_LD_PREFIX=/usr/riscv64-linux-gnu \
  qemu-riscv64 -L /usr/riscv64-linux-gnu \
  -E LD_LIBRARY_PATH=$HOME/projects/Allen/build-riscv-cpu-root-phase3:$HOME/projects/Allen/build-riscv-cpu-root-phase3/zmq:$HOME/projects/Allen/artifacts/root-riscv64/lib:$HOME/projects/Allen/artifacts/riscv64-sysroot/usr/lib/riscv64-linux-gnu:$HOME/projects/Allen/artifacts/riscv64-sysroot/lib/riscv64-linux-gnu:$HOME/projects/Allen/artifacts/riscv64-sysroot/usr/lib:/usr/riscv64-linux-gnu/lib \
  -E ROOTSYS=$HOME/projects/Allen/artifacts/root-riscv64 \
  ./build-riscv-cpu-root-phase3/Allen -h

It printed the help text. Good.

Then I ran the velo sequence with monitoring enabled:

timeout 300s env \
  PATH=/usr/bin:/bin:$PATH \
  PYTHONPATH=$HOME/projects/Allen/artifacts/python-deps/usr/lib/python3/dist-packages \
  ROOTSYS=$HOME/projects/Allen/artifacts/root-riscv64 \
  QEMU_LD_PREFIX=/usr/riscv64-linux-gnu \
  qemu-riscv64 -L /usr/riscv64-linux-gnu \
  -E LD_LIBRARY_PATH=$HOME/projects/Allen/build-riscv-cpu-root-phase3:$HOME/projects/Allen/build-riscv-cpu-root-phase3/zmq:$HOME/projects/Allen/artifacts/root-riscv64/lib:$HOME/projects/Allen/artifacts/riscv64-sysroot/usr/lib/riscv64-linux-gnu:$HOME/projects/Allen/artifacts/riscv64-sysroot/lib/riscv64-linux-gnu:$HOME/projects/Allen/artifacts/riscv64-sysroot/usr/lib:/usr/riscv64-linux-gnu/lib \
  -E ROOTSYS=$HOME/projects/Allen/artifacts/root-riscv64 \
  ./build-riscv-cpu-root-phase3/Allen \
  --sequence velo \
  --mdf input/minbias/mdf/MiniBrunel_2018_MinBias_FTv4_DIGI_retinacluster_v1.mdf \
  -g artifacts/detector_configuration_x86_ref \
  -n 10 \
  -t 1 \
  -r 1 \
  --events-per-slice 10 \
  -m 128 \
  --host-memory 128 \
  -v 4 \
  -p 1 \
  --monitoring-filename artifacts/riscv64_root_velo_10_monitoring.root

It exited with code 133.

That usually means SIGTRAP.

I tried disabling monitoring counter registration but keeping a non-empty monitoring filename. It still exited 133.

Then I disabled ROOT monitoring file creation completely:

--register-monitoring-counters 0 \
--monitoring-filename ''

With those options, the riscv64 run completed:

Read 10 events into 0
n_transposed 10
Processing complete

That proved the standalone CPU event-processing path worked on riscv64. But it also showed a real remaining blocker: ROOT TFile creation under QEMU user-mode.

Reducing the ROOT monitoring failure

I did not want to guess whether this was Allen or ROOT.

So I wrote the smallest possible reproducer to test TFile creation under QEMU user-mode with the cross-built ROOT:

#include <TFile.h>
#include <iostream>

int main() {
  TFile f("artifacts/root_tfile_smoke.root", "RECREATE");
  std::cout << "open=" << f.IsOpen() << " zombie=" << f.IsZombie() << "\n";
  f.Close();
  return f.IsZombie() ? 1 : 0;
}

Compiled for riscv64 against the cross-built ROOT:

riscv64-linux-gnu-g++ -x c++ - -std=c++20 \
  -I$HOME/projects/Allen/artifacts/root-riscv64/include \
  -o /tmp/root_tfile_smoke \
  -Wl,--sysroot=$HOME/projects/Allen/artifacts/riscv64-sysroot \
  -L$HOME/projects/Allen/artifacts/root-riscv64/lib \
  -L$HOME/projects/Allen/artifacts/riscv64-sysroot/usr/lib/riscv64-linux-gnu \
  -Wl,-rpath,$HOME/projects/Allen/artifacts/root-riscv64/lib \
  -Wl,-rpath-link,$HOME/projects/Allen/artifacts/root-riscv64/lib \
  -Wl,-rpath-link,$HOME/projects/Allen/artifacts/riscv64-sysroot/usr/lib/riscv64-linux-gnu \
  -lRIO -lCore -lThread -lz -llzma -llz4 -lzstd -pthread -lm -ldl

Then ran it under QEMU user-mode:

timeout 60s env QEMU_LD_PREFIX=/usr/riscv64-linux-gnu \
  qemu-riscv64 -L /usr/riscv64-linux-gnu \
  -E LD_LIBRARY_PATH=$HOME/projects/Allen/artifacts/root-riscv64/lib:$HOME/projects/Allen/artifacts/riscv64-sysroot/usr/lib/riscv64-linux-gnu:$HOME/projects/Allen/artifacts/riscv64-sysroot/lib/riscv64-linux-gnu:$HOME/projects/Allen/artifacts/riscv64-sysroot/usr/lib:/usr/riscv64-linux-gnu/lib \
  -E ROOTSYS=$HOME/projects/Allen/artifacts/root-riscv64 \
  /tmp/root_tfile_smoke

It also exited 133.

The strace was the important part. ROOT/Cling was doing runtime probing through /bin/sh, looking at host paths, mapping libCling.so, and then trapping.

At this point I thought:

QEMU user-mode is probably the wrong environment for this part.

User-mode QEMU with -L gives the target dynamic loader and target libraries, but it is not a full riscv64 filesystem. /bin/sh is still the host shell. Some runtime compiler or interpreter probe can easily cross the streams.

So the next move was a full riscv64 Linux VM.

Building a riscv64 QEMU system VM

I used the Ubuntu 24.04 riscv64 cloud image and created a local QEMU system VM.

The helper scripts eventually became:

scripts/riscv64/prepare-qemu-vm.sh
scripts/riscv64/boot-qemu-vm.sh

Preparation used a manually downloaded cloud image:

QEMU_CLOUD_IMAGE=$HOME/Downloads/ubuntu-24.04-server-cloudimg-riscv64.img \
  scripts/riscv64/prepare-qemu-vm.sh

Booting the VM:

QEMU_MEMORY=16G QEMU_CPUS=8 scripts/riscv64/boot-qemu-vm.sh

The VM setup used:

Inside the VM:

uname -m

reported:

riscv64

That removed the QEMU user-mode host shell problem. Now /bin/sh, ls, the dynamic loader, and the compiler tools were riscv64 guest files.

I reran the tiny TFile smoke inside the VM against the cross-built ROOT. But it still failed with SIGTRAP.

That was useful. It meant the first theory was incomplete. The VM fixed the host-shell leakage, but the cross-built ROOT/Cling prefix still seemed to carry host-triple assumptions. The strace showed probes involving x86_64-unknown-linux-gnu paths before libCling.so trapped.

At that point I stopped trying to patch around the cross-built ROOT runtime. The next reliable move was to build ROOT natively inside the riscv64 VM.

Native ROOT inside the riscv64 VM

This was the slow part.

The native ROOT prefix was:

artifacts/root-riscv64-nativevm

The VM-side build scripts are:

scripts/riscv64/build-root-nativevm.sh
scripts/riscv64/run-native-root-build.sh

The host-side command are:

QEMU_CPUS=8 QEMU_MEMORY=16G ROOT_BUILD_JOBS=8 \
  scripts/riscv64/run-native-root-build.sh

This builds ROOT inside the emulated riscv64 VM and installs it into the mounted Allen workspace, so the host can see the prefix afterwards.

The first native build attempt used a diagnostic Release build with -O0 to try to reduce compile time. That turned out to be a bad mix. The build failed while linking rootcling_stage1, with RISC-V relocation truncation and missing debug-only LLVM symbols.

The likely cause was that unoptimized LLVM/Clang objects became too large for some riscv64 direct-branch/link relaxation assumptions, while NDEBUG removed debug-only definitions that some unoptimized code still referenced.

The fix was to stop being clever and use:

Release -O1 -DNDEBUG

Then restart cleanly:

ROOT_CLEAN_BUILD=1 scripts/riscv64/run-native-root-build.sh

This took 8 hours (to be honest I slept so not sure if it 6 or 8 but definitely not less than 6), but it completed.

The native VM ROOT reported:

linuxriscv64 linux gcc-13 g++-13 cxx20 builtin_clang builtin_cling builtin_llvm builtin_nlohmannjson builtin_pcre builtin_xxhash shared thisroot_scripts

Most importantly, the tiny TFile reproducer now worked inside the VM and gave the expected output:

open=1 zombie=0

That was the turning point for monitoring.

Relinking Allen against native VM ROOT

Now I configured a new riscv64 Allen build against:

artifacts/root-riscv64-nativevm

The configure command:

env PYTHONPATH=$HOME/projects/Allen/artifacts/python-deps/usr/lib/python3/dist-packages \
  LHCBROOT=$HOME/lhcb-stack/LHCb \
  GAUDIROOT=$HOME/lhcb-stack/Gaudi \
  PARAMFILESROOT=$HOME/lhcb-stack/ParamFiles \
  PKG_CONFIG_EXECUTABLE=/usr/bin/pkg-config \
  cmake -S $HOME/projects/Allen -B build-riscv-cpu-root-nativevm-phase4 \
  -GNinja \
  -DCMAKE_TOOLCHAIN_FILE=cmake/toolchains/riscv64-linux-gnu.cmake \
  -DALLEN_RISCV64_SYSROOT=$HOME/projects/Allen/artifacts/riscv64-sysroot \
  -DALLEN_ROOT_PREFIX=$HOME/projects/Allen/artifacts/root-riscv64-nativevm \
  -DSTANDALONE=ON \
  -DTARGET_DEVICE=CPU \
  -DSEQUENCES=velo \
  -DPKG_CONFIG_EXECUTABLE=/usr/bin/pkg-config \
  -DPython_EXECUTABLE=/usr/bin/python3 \
  -DPython3_EXECUTABLE=/usr/bin/python3 \
  -DCMAKE_BUILD_TYPE=Debug \
  -DBUILD_TESTING=OFF

Then:

env PYTHONPATH=$HOME/projects/Allen/artifacts/python-deps/usr/lib/python3/dist-packages \
  LHCBROOT=$HOME/lhcb-stack/LHCb \
  GAUDIROOT=$HOME/lhcb-stack/Gaudi \
  PARAMFILESROOT=$HOME/lhcb-stack/ParamFiles \
  PKG_CONFIG_EXECUTABLE=/usr/bin/pkg-config \
  ninja -C build-riscv-cpu-root-nativevm-phase4 Allen

One subtle issue appeared after removing the old no-ROOT diagnostic branch: the native ROOT prefix installed ROOT's bundled nlohmann/json.hpp, while the target sysroot had the complete distro nlohmann package, including ordered_map.hpp.

Mixing those headers caused an include mismatch.

The fix was to give:

${ALLEN_RISCV64_SYSROOT}/usr/include

precedence for cross builds, so json.hpp and ordered_map.hpp came from the same target package.

That made the clean riscv64 rebuild reproducible.

Running Allen inside the VM

There were three VM-specific runtime details.

First, the host-built executable embeds absolute paths from the host build. In the VM, the repository is mounted at /mnt/Allen, so I needed a compatibility symlink:

HOST_REPO=/path/to/host/Allen
sudo mkdir -p "$(dirname "$HOST_REPO")"
sudo ln -sfn /mnt/Allen "$HOST_REPO"

In my tested setup, HOST_REPO was the absolute path used during host-side configure/build.

Second, I used the generated JSON sequence directly:

--sequence build-riscv-cpu-root-nativevm-phase4/velo.json

I did not use:

--sequence velo

inside the VM. Dynamic sequence generation imports host-side Python modules such as PyConf, and the VM did not have that full host Python stack. The generated JSON had already been produced during the build, so using it directly was the cleaner runtime validation.

Third, I staged a minimal ParamFiles subset:

PARAMFILESROOT=$HOME/lhcb-stack/ParamFiles \
  scripts/riscv64/stage-paramfiles-min.sh

This created:

artifacts/ParamFiles-min

Then the runtime command inside the VM was:

cd /mnt/Allen

export ROOTSYS=/mnt/Allen/artifacts/root-riscv64-nativevm
export LD_LIBRARY_PATH=/mnt/Allen/build-riscv-cpu-root-nativevm-phase4:/mnt/Allen/build-riscv-cpu-root-nativevm-phase4/zmq:${ROOTSYS}/lib:/mnt/Allen/artifacts/riscv64-sysroot/usr/lib/riscv64-linux-gnu:/mnt/Allen/artifacts/riscv64-sysroot/lib/riscv64-linux-gnu:/mnt/Allen/artifacts/riscv64-sysroot/usr/lib:${LD_LIBRARY_PATH:-}

timeout 300s ./build-riscv-cpu-root-nativevm-phase4/Allen \
  --sequence build-riscv-cpu-root-nativevm-phase4/velo.json \
  --params artifacts/ParamFiles-min \
  --mdf input/minbias/mdf/MiniBrunel_2018_MinBias_FTv4_DIGI_retinacluster_v1.mdf \
  -g artifacts/detector_configuration_x86_ref \
  -n 10 \
  -t 1 \
  -r 1 \
  --events-per-slice 10 \
  -m 128 \
  --host-memory 128 \
  -v 4 \
  -p 1 \
  --monitoring-filename artifacts/riscv_nativevm_velo_10_monitoring.root

This time it completed with monitoring enabled. The output file was recognized as a ROOT file:

file artifacts/riscv_nativevm_velo_10_monitoring.root

reported:

ROOT file Version 64000 (Compression: 101)

Native rootls in the VM could read it. For this minimal velo sequence it printed no keys, but the important part was that ROOT could open the file and the run completed.

The final validator

At this point I had a working manual path, but I did not want the proof to live only in shell history.

So I wrote a validator script that codifies the successful path and checks all the important outputs. The script is:

scripts/riscv64/validate-poc.sh

The validator does the reproducibility work:

The important design choice is that it does not silently launch the long native ROOT build. If the native ROOT prefix is missing, it fails fast with
instructions. To intentionally allow the long build, you must opt in:

VALIDATE_BUILD_NATIVE_ROOT=1 scripts/riscv64/validate-poc.sh

Normal validation is:

scripts/riscv64/validate-poc.sh

The validator result:

The x86_64 monitoring run processed 10 events, reported Processing complete,
and wrote artifacts/phase5_x86_velo_10_monitoring.root.

The riscv64 qemu-system native-ROOT monitoring run processed 10 events,
reported Processing complete, and wrote
artifacts/phase5_riscv_nativevm_velo_10_monitoring.root.

Both monitoring outputs are ROOT 6.40 files.

The validator also wrote a manifest with tool versions, ROOT configurations, git state, and checksums for the important artifacts.

Trying the full default HLT1 sequence

At this point the proof of concept was already useful, but it was still a small proof: velo is meaningful, but it is not the full HLT1 default sequence. So the next question was obvious:

Does the same RISC-V path survive hlt1_pp_default?

In this checkout, configuration/python/AllenSequences/hlt1_pp_default.py is a copy of the forward-then-matching plus downstream ParkF-style sequence. It enables UT, downstream reconstruction, and full Kalman filtering. In practice, that means it pulls in a much larger surface than the velo run: UT, SciFi, calo, muon, downstream, RICH-related code, many selection lines, rate validation, routing bits, SelReports, and monitoring.

So I first built and ran it on x86_64 as the control:

cmake -S $HOME/projects/Allen -B build-x86-cpu-hlt1-default \
  -GNinja \
  -DSTANDALONE=ON \
  -DTARGET_DEVICE=CPU \
  -DSEQUENCES=hlt1_pp_default \
  -DPKG_CONFIG_EXECUTABLE=/usr/bin/pkg-config \
  -DPython_EXECUTABLE=/usr/bin/python3 \
  -DCMAKE_BUILD_TYPE=Debug \
  -DBUILD_TESTING=OFF

ninja -C build-x86-cpu-hlt1-default Allen

That generated:

build-x86-cpu-hlt1-default/hlt1_pp_default.json

and the 10-event x86_64 control run completed:

./build-x86-cpu-hlt1-default/Allen \
  --sequence build-x86-cpu-hlt1-default/hlt1_pp_default.json \
  --params artifacts/ParamFiles-min \
  --mdf input/minbias/mdf/MiniBrunel_2018_MinBias_FTv4_DIGI_retinacluster_v1.mdf \
  -g artifacts/detector_configuration_x86_ref \
  -n 10 \
  -t 1 \
  -r 1 \
  --events-per-slice 10 \
  -m 1024 \
  --host-memory 1024 \
  -v 4 \
  -p 1 \
  --monitoring-filename artifacts/hlt1_default_x86_10_monitoring.root

The important part was:

Processing complete

and the monitoring output was a ROOT 6.40 file.

Then I configured the same sequence for RISC-V against the native-VM ROOT prefix:

cmake -S $HOME/projects/Allen -B build-riscv-cpu-root-nativevm-hlt1-default \
  -GNinja \
  -DCMAKE_TOOLCHAIN_FILE=cmake/toolchains/riscv64-linux-gnu.cmake \
  -DALLEN_RISCV64_SYSROOT=$HOME/projects/Allen/artifacts/riscv64-sysroot \
  -DALLEN_ROOT_PREFIX=$HOME/projects/Allen/artifacts/root-riscv64-nativevm \
  -DSTANDALONE=ON \
  -DTARGET_DEVICE=CPU \
  -DSEQUENCES=hlt1_pp_default \
  -DPKG_CONFIG_EXECUTABLE=/usr/bin/pkg-config \
  -DPython_EXECUTABLE=/usr/bin/python3 \
  -DPython3_EXECUTABLE=/usr/bin/python3 \
  -DCMAKE_BUILD_TYPE=Debug \
  -DBUILD_TESTING=OFF

ninja -C build-riscv-cpu-root-nativevm-hlt1-default Allen

This was the first important result: the full default HLT1 sequence built for riscv64. It compiled the broader detector and line stack, generated hlt1_pp_default.json, and linked the final Allen executable.

But the Debug runtime was not practical under qemu-system. A one-event run stayed alive and kept using about one emulated CPU, but it did not finish within an 1800-second timeout. That was not a correctness failure, but it was also not a completed validation.

So I rebuilt the same RISC-V sequence in Release mode:

cmake -S $HOME/projects/Allen -B build-riscv-cpu-root-nativevm-hlt1-default-release \
  -GNinja \
  -DCMAKE_TOOLCHAIN_FILE=cmake/toolchains/riscv64-linux-gnu.cmake \
  -DALLEN_RISCV64_SYSROOT=$HOME/projects/Allen/artifacts/riscv64-sysroot \
  -DALLEN_ROOT_PREFIX=$HOME/projects/Allen/artifacts/root-riscv64-nativevm \
  -DSTANDALONE=ON \
  -DTARGET_DEVICE=CPU \
  -DSEQUENCES=hlt1_pp_default \
  -DPKG_CONFIG_EXECUTABLE=/usr/bin/pkg-config \
  -DPython_EXECUTABLE=/usr/bin/python3 \
  -DPython3_EXECUTABLE=/usr/bin/python3 \
  -DCMAKE_BUILD_TYPE=Release \
  -DBUILD_TESTING=OFF

ninja -C build-riscv-cpu-root-nativevm-hlt1-default-release Allen

Then inside the riscv64 VM:

cd /mnt/Allen

export ROOTSYS=/mnt/Allen/artifacts/root-riscv64-nativevm
export LD_LIBRARY_PATH=/mnt/Allen/build-riscv-cpu-root-nativevm-hlt1-default-release:/mnt/Allen/build-riscv-cpu-root-nativevm-hlt1-default-release/zmq:${ROOTSYS}/lib:/mnt/Allen/artifacts/riscv64-sysroot/usr/lib/riscv64-linux-gnu:/mnt/Allen/artifacts/riscv64-sysroot/lib/riscv64-linux-gnu:/mnt/Allen/artifacts/riscv64-sysroot/usr/lib:${LD_LIBRARY_PATH:-}

timeout 1800s ./build-riscv-cpu-root-nativevm-hlt1-default-release/Allen \
  --sequence build-riscv-cpu-root-nativevm-hlt1-default-release/hlt1_pp_default.json \
  --params artifacts/ParamFiles-min \
  --mdf input/minbias/mdf/MiniBrunel_2018_MinBias_FTv4_DIGI_retinacluster_v1.mdf \
  -g artifacts/detector_configuration_x86_ref \
  -n 1 \
  -t 1 \
  -r 1 \
  --events-per-slice 1 \
  -m 1024 \
  --host-memory 1024 \
  -v 3 \
  -p 0 \
  --monitoring-filename artifacts/hlt1_default_riscv_release_1_monitoring.root

This completed:

Input complete
Processing complete

and the monitoring output was also a ROOT file:

file artifacts/hlt1_default_riscv_release_1_monitoring.root

ROOT file Version 64000 (Compression: 101)

That changed the conclusion of the PoC. The first milestone was not the ceiling. velo was the small sequence that helped debug the port, but the same standalone CPU path can also build and run the full default HLT1 sequence on riscv64, at least as a one-event Release-mode smoke test under qemu-system.

What actually had to change

The successful path needed a surprisingly small amount of Allen source-level
porting.

The important tracked pieces were:

What did not need a big rewrite:

That was one of the nicer findings. Most of the work was not "RISC-V cannot run this C++". Most of the work was "make the build and dependency graph honest about which architecture each thing belongs to".

Mistakes and dead ends that were still useful

The no-ROOT branch was useful, but not worth keeping.

It proved that ROOT availability was the main blocker after the ordinary target dependencies were staged. It also helped locate all the ROOT surfaces. But once native VM ROOT solved monitoring, keeping a no-ROOT mode would make the PoC less clear. The final story should be: use real ROOT.

The cross-built ROOT was also useful, even though it was not the final runtime solution.

It got Allen through configure and link. It proved that the headers and libraries could be made available. It exposed the runtime Cling problem in a controlled way. But for TFile creation and monitoring, the reliable fix was native ROOT inside a real riscv64 userspace.

QEMU user-mode was useful for small smoke tests and build-time target helper execution.

But it was not a complete substitute for a real target environment once ROOT/Cling started probing compilers, shells, loaders, and runtime paths. For that, qemu-system was the right level of emulation.

If I had to reproduce it from scratch

This is the route I would take now.

1. Install host tools

On an Ubuntu-compatible x86_64 host:

sudo apt update
sudo apt install -y \
  build-essential \
  cmake \
  ninja-build \
  git \
  pkg-config \
  python3 \
  python3-dev \
  python3-pip \
  catch2 \
  qemu-user \
  qemu-system-misc \
  u-boot-qemu \
  opensbi \
  cloud-image-utils \
  gcc-riscv64-linux-gnu \
  g++-riscv64-linux-gnu \
  binutils-riscv64-linux-gnu \
  libc6-dev-riscv64-cross \
  libfmt-dev \
  libzmq3-dev \
  cppzmq-dev \
  libsodium-dev \
  libboost-dev \
  libboost-filesystem-dev \
  libboost-iostreams-dev \
  libboost-thread-dev \
  libboost-regex-dev \
  libboost-serialization-dev \
  libboost-program-options-dev \
  libboost-test-dev \
  nlohmann-json3-dev \
  librange-v3-dev \
  libtbb-dev

Make sure host ROOT exists:

root-config --version
root-config --prefix

Then:

export HOST_ROOT_PREFIX="$(root-config --prefix)"
export ROOTSYS="$HOST_ROOT_PREFIX"
export ROOT_DIR="$HOST_ROOT_PREFIX/share/root/cmake"
export CMAKE_PREFIX_PATH="$HOST_ROOT_PREFIX:${CMAKE_PREFIX_PATH:-}"
export PKG_CONFIG_EXECUTABLE=/usr/bin/pkg-config

2. Provide local LHCb, Gaudi, and ParamFiles

export ALLEN_REPO=$HOME/projects/Allen
export LHCB_STACK=$HOME/lhcb-stack

mkdir -p "$LHCB_STACK"
cd "$LHCB_STACK"

git clone https://gitlab.cern.ch/gaudi/Gaudi.git Gaudi
git clone https://gitlab.cern.ch/lhcb/LHCb.git LHCb
git clone https://gitlab.cern.ch/lhcb/ParamFiles.git ParamFiles

export GAUDIROOT="$LHCB_STACK/Gaudi"
export LHCBROOT="$LHCB_STACK/LHCb"
export PARAMFILESROOT="$LHCB_STACK/ParamFiles"

cd "$ALLEN_REPO"

Use compatible revisions for the Allen checkout you are testing.

3. Stage Python dependencies for sequence generation

mkdir -p artifacts/python-deps artifacts/python-deps-debs

(
  cd artifacts/python-deps-debs
  apt download python3-wrapt python3-pydot python3-mpmath python3-sympy
)

for deb in artifacts/python-deps-debs/*.deb; do
  dpkg-deb -x "$deb" artifacts/python-deps
done

/usr/bin/python3 -m pip install \
  --target "$ALLEN_REPO/artifacts/python-deps/usr/lib/python3/dist-packages" \
  pyeda==0.29.0

export PYTHONPATH="$ALLEN_REPO/artifacts/python-deps/usr/lib/python3/dist-packages"

4. Prepare detector configuration

mkdir -p artifacts/detector_configuration_x86_ref
cp -a input/detector_configuration/. artifacts/detector_configuration_x86_ref/
rm -f artifacts/detector_configuration_x86_ref/magfield.bin
ln -s ../../build-x86-cpu-system/local_magfields/magfield.v8r1.down.bin \
  artifacts/detector_configuration_x86_ref/magfield.bin

The symlink target is created by the x86 configure/build path when
generate_magfields runs.

5. Build the riscv64 dependency sysroot

Use the workspace-local Ubuntu ports extraction described earlier, then export:

export ALLEN_RISCV64_SYSROOT="$ALLEN_REPO/artifacts/riscv64-sysroot"

6. Prepare the qemu-system VM

Download the Ubuntu 24.04 riscv64 cloud image, then:

QEMU_CLOUD_IMAGE=$HOME/Downloads/ubuntu-24.04-server-cloudimg-riscv64.img \
  scripts/riscv64/prepare-qemu-vm.sh

Boot it when needed:

QEMU_MEMORY=16G QEMU_CPUS=8 scripts/riscv64/boot-qemu-vm.sh

7. Build native ROOT in the VM

Place:

artifacts/root-src/root_v6.40.00.source.tar.gz

Then run:

QEMU_CPUS=8 QEMU_MEMORY=16G ROOT_BUILD_JOBS=8 \
  scripts/riscv64/run-native-root-build.sh

Expect this to take a long time. The output prefix is:

artifacts/root-riscv64-nativevm

8. Stage the minimal ParamFiles subset

PARAMFILESROOT="$PARAMFILESROOT" scripts/riscv64/stage-paramfiles-min.sh

9. Run the validator

scripts/riscv64/validate-poc.sh

That is the final reproducibility check.

10. Try the full default HLT1 sequence

The validator still focuses on the smaller velo PoC because it is fast enough to run repeatedly. For the broader smoke test, configure a separate Release build:

cmake -S "$ALLEN_REPO" -B build-riscv-cpu-root-nativevm-hlt1-default-release \
  -GNinja \
  -DCMAKE_TOOLCHAIN_FILE=cmake/toolchains/riscv64-linux-gnu.cmake \
  -DALLEN_RISCV64_SYSROOT="$ALLEN_REPO/artifacts/riscv64-sysroot" \
  -DALLEN_ROOT_PREFIX="$ALLEN_REPO/artifacts/root-riscv64-nativevm" \
  -DSTANDALONE=ON \
  -DTARGET_DEVICE=CPU \
  -DSEQUENCES=hlt1_pp_default \
  -DPKG_CONFIG_EXECUTABLE=/usr/bin/pkg-config \
  -DPython_EXECUTABLE=/usr/bin/python3 \
  -DPython3_EXECUTABLE=/usr/bin/python3 \
  -DCMAKE_BUILD_TYPE=Release \
  -DBUILD_TESTING=OFF

ninja -C build-riscv-cpu-root-nativevm-hlt1-default-release Allen

Then run the generated hlt1_pp_default.json inside the VM. I used one event for this smoke test because qemu-system is slow:

./build-riscv-cpu-root-nativevm-hlt1-default-release/Allen \
  --sequence build-riscv-cpu-root-nativevm-hlt1-default-release/hlt1_pp_default.json \
  --params artifacts/ParamFiles-min \
  --mdf input/minbias/mdf/MiniBrunel_2018_MinBias_FTv4_DIGI_retinacluster_v1.mdf \
  -g artifacts/detector_configuration_x86_ref \
  -n 1 \
  -t 1 \
  -r 1 \
  --events-per-slice 1 \
  -m 1024 \
  --host-memory 1024 \
  -v 3 \
  -p 0 \
  --monitoring-filename artifacts/hlt1_default_riscv_release_1_monitoring.root

What this proves, and what it does not

This proves that Allen's standalone CPU backend can be configured, built, and run on riscv64 Linux with real MDF input and real ROOT monitoring output.

There are two levels to that statement:

It does not prove:

The timings in the logs are smoke-test timings only. The VELO validation used 10 events, while the default HLT1 validation used one event in a Release build because the Debug build was too slow under qemu-system. The numbers are useful to confirm that the process ran; they are not performance data.

The main lesson

The port was less about rewriting compute code and more about making every architecture boundary explicit.

The host runs CMake and Python generation. The target runs generated helpers. Some dependencies are host tools. Some dependencies are target libraries. ROOT is both a dependency and a runtime environment with its own compiler/interpreter assumptions. QEMU user-mode is fine until a program expects a whole target userspace. A real target userspace fixes some things, but a cross-built ROOT can still carry host assumptions into runtime Cling behavior.

Once those boundaries were separated, the actual Allen CPU path was fairly portable. That is the part I like about this result. The final command is not magical. It is just the result of removing ambiguity one layer at a time:

  1. Prove x86_64 CPU works
  2. Add RISC-V architecture detection
  3. Cross-compile with target dependency isolation
  4. Provide a real riscv64 ROOT
  5. Stop expecting QEMU user-mode to behave like a full machine
  6. Build ROOT natively in a riscv64 VM
  7. Run the same small sequence on both architectures
  8. Push the same path to hlt1_pp_default
  9. Check that the runs finish and write ROOT files

That is enough for a proof of concept. The next step is to push this into a normal build/test shape: more sequences validation, cleaner
dependency setup, fuller ParamFiles coverage, CI integration, and eventually real hardware tests when available.

But the bridge is there: Allen can run a standalone CPU velo path on riscv64, and it can also run the full default HLT1 sequence in Release mode.