rustbot · GitHub

LatticeArc-Founder pushed a commit to LatticeArc/latticearc that referenced this pull request

LatticeArc Dev Team LatticeArc Dev Team

Four independent breakages, all found while investigating the red build
on  b4b80f8 .
1. FIPS integrity test aborts under cargo build-dir layout v2
---------------------------------------------------------------
`path_looks_like_latticearc_module` required the running binary's parent
directory to be literally `deps`. Layout v2 moved unit test binaries from
    target/<triple>/<profile>/deps/<crate>-<hash>
to
    target/<triple>/<profile>/build/<pkg>/<hash>/out/<crate>-<hash>
so every test binary began failing that check. The rejection failed the
module integrity test, failed the power-up self-test, and hit FIPS
140-3 section 9.1's `process::abort()` - SIGABRT-ing the test runner
before a single result was reported.
This surfaced on the weekly ASan/TSan/LSan jobs, the only ones pinned to
nightly: all three aborted at
`self_test::tests::test_initialize_and_test_sets_flag_succeeds` once
rust-lang/cargo#17258 (merged 2026-07-24) made layout v2 the nightly
default. Layout v2 is stabilized for 1.99.0 by rust-lang/cargo#16807, so
this was days from reaching every `cargo test` on stable - including
downstream consumers whose suites call `initialize_and_test()`.
The check now matches on the two properties cargo does not document as
internal: the `<crate-name>-<16-hex>` file-name shape, and containment
under a `target` ancestor. Cargo explicitly reserves the right to
reshuffle the build-dir layout, so pinning an intermediate directory
name would only queue the same outage up for layout v3.
Consequences of the wider match:
  * `target/<profile>/<crate>-<hex>` (no intervening artifact dir) is
    now accepted. It is a legitimate build-tree artifact.
  * The ancestor walk is bounded at 12 hops rather than 8. The bound
    only stops the walk reaching the filesystem root; it is not a trust
    boundary, and sizing it flush against the deepest layout known today
    is what caused this outage.
Module authenticity is unchanged: it is established by HMAC over the
binary, not by path. The helper's actual job - distinguishing our own
artifact from a Python/Node host process that dlopen'd the library -
still rejects every host-interpreter path.
Verified against a real layout-v2 build (`cargo +nightly test
-Zbuild-dir-new-layout`), which places the binary at
`target/debug/build/latticearc/<hash>/out/latticearc-<hash>`: the
integrity test returns Ok and `test_initialize_and_test_sets_flag_succeeds`
passes instead of aborting. Both layouts are pinned in the unit tests so
a future re-narrowing fails loudly rather than at POST.
2. Feature Flag Powerset never completed, and never went red
---------------------------------------------------------------
Every scheduled run was killed at exactly 6h00m:
  30736018769  cancelled  6h0m22s  2026-08-02
  30191256595  cancelled  6h0m19s  2026-07-26
  29676467815  cancelled  6h0m19s  2026-07-19
  29182750237  cancelled  6h0m21s  2026-07-12
  28733241736  cancelled  6h0m22s  2026-07-05
  28315598345  cancelled  6h0m19s  2026-06-28
Six hours is a hard GitHub limit on job execution time for hosted
runners. `timeout-minutes` can only lower it, never raise it, so the
`timeout-minutes: 720` this workflow carried - added specifically to
escape the cap, per its own header comment - had no effect. And because
the cap terminates a job as `cancelled` rather than `failure`, the
workflow never rendered red: the check silently provided zero coverage
for over a month.
  * Shard the sweep across an 8-way matrix using `cargo hack
    --partition M/8`. The powerset is 5119 combinations (measured via
    `--print-command-list`), so each shard runs ~640. `--partition`
    assigns each run to exactly one shard - contiguous, disjoint blocks
    of `ceil(total/N)` - so the shards cover the sweep exactly once.
  * Set `timeout-minutes: 300`, below the cap, so an overrun is killed
    by us as a visible `failure` rather than silently by GitHub.
  * `fail-fast: false`, so one shard failing does not discard the other
    seven results.
  * Key the cargo cache per shard; eight jobs sharing one key would race
    on save. `restore-keys` lets a cold shard warm from a sibling.
The header comment is rewritten to record why `timeout-minutes` cannot
raise the cap, so the next person to hit a shard timeout raises the
partition count instead of the timeout.
3. One failed commit bricked every later commit
---------------------------------------------------------------
The hook builds its test log with
    mktemp "$TEST_LOG_DIR/pre-commit-test-output.XXXXXX.log"
BSD `mktemp` (macOS, where this hook actually runs) only substitutes a
trailing run of `X`s. Given `...XXXXXX.log` it expands nothing and tries
to create a file named literally `pre-commit-test-output.XXXXXX.log`.
That succeeds the first time - and since the log is deliberately
PRESERVED on failure, every subsequent commit then died with
    mktemp: mkstemp failed on .../pre-commit-test-output.XXXXXX.log: File exists
before running a single test. A single failed commit permanently blocked
all later commits until someone manually deleted the file. A stale log
from 2026-08-02 was doing exactly that.
Moving the `X`s to the end (`pre-commit-test-output.log.XXXXXX`) restores
per-run unique names on both BSD and GNU mktemp; verified that two
successive calls now yield distinct paths. `mktemp -d` at line 436
already had the `X`s trailing and was unaffected. The comment records why
the suffix must not be moved back.
4. Flaky FIPS global-state test
---------------------------------------------------------------
`test_get_fips_validation_result_consistency_succeeds` intermittently
failed at global.rs:375 comparing two consecutive reads of
`FIPS_VALIDATION_RESULT` - caught in a preserved hook log from
2026-08-02, and passing on the very next full run.
`FIPS_GLOBAL_SERIAL` exists precisely to serialize this, and the reader
does hold it across both reads. The hole is on the writer side: of the
three writers of `FIPS_VALIDATION_RESULT` in this module, only one took
the guard.
  * `ensure_initialized_for_test` did a check-then-act on
    `FIPS_INITIALIZED` with no guard, so several tests could all observe
    `false`, each run `validate_module()` (a fresh `validation_id` per
    call) and each `replace()` the global.
  * `test_init_lock_and_store_result_succeeds` replaced the global
    unconditionally on every run, guardless - the guaranteed writer, and
    the most likely source of the observed interleaving.
Both now take `FIPS_GLOBAL_SERIAL`, and `ensure_initialized_for_test`
re-checks the flag under the guard so initialization happens exactly
once. The guard's doc comment now states the actual invariant - every
writer holds it, not just store-then-read tests - and warns that it is a
plain non-reentrant `Mutex`, so a guard holder must not call
`ensure_initialized_for_test`.
Confirmed the globals are `pub(crate)` but touched only in this module,
and that `tests/tests/` merely replicates these code paths in a separate
binary, so there is no cross-module or cross-binary writer left out.
Verified by A/B stress of the module, same binary, same loop:
    pre-fix    600 iterations    7 failures
    post-fix   900 iterations    0 failures
Every pre-fix failure was
`test_get_fips_validation_result_consistency_succeeds`, matching the
2026-08-02 log. The full 510-test binary also passes 20 consecutive
runs, so the added guard introduces no contention or deadlock with the
module's other tests.
Not fixed here: `performance.yml` carries the same layout-v1 assumption
(`find target/{release,debug}/deps -name 'latticearc-*'`) in its two
profiling steps. That workflow is dormant - every trigger is commented
out and it last ran 2026-02-05 - so it is left alone, but it needs the
same treatment before those triggers are re-enabled.

Read the original on github.com ↗