LatticeArc-Founder pushed a commit to LatticeArc/latticearc that referenced this pull request
Four independent breakages, all found while investigating the red build on b4b80f8 . 1. FIPS integrity test aborts under cargo build-dir layout v2 --------------------------------------------------------------- `path_looks_like_latticearc_module` required the running binary's parent directory to be literally `deps`. Layout v2 moved unit test binaries from target/<triple>/<profile>/deps/<crate>-<hash> to target/<triple>/<profile>/build/<pkg>/<hash>/out/<crate>-<hash> so every test binary began failing that check. The rejection failed the module integrity test, failed the power-up self-test, and hit FIPS 140-3 section 9.1's `process::abort()` - SIGABRT-ing the test runner before a single result was reported. This surfaced on the weekly ASan/TSan/LSan jobs, the only ones pinned to nightly: all three aborted at `self_test::tests::test_initialize_and_test_sets_flag_succeeds` once rust-lang/cargo#17258 (merged 2026-07-24) made layout v2 the nightly default. Layout v2 is stabilized for 1.99.0 by rust-lang/cargo#16807, so this was days from reaching every `cargo test` on stable - including downstream consumers whose suites call `initialize_and_test()`. The check now matches on the two properties cargo does not document as internal: the `<crate-name>-<16-hex>` file-name shape, and containment under a `target` ancestor. Cargo explicitly reserves the right to reshuffle the build-dir layout, so pinning an intermediate directory name would only queue the same outage up for layout v3. Consequences of the wider match: * `target/<profile>/<crate>-<hex>` (no intervening artifact dir) is now accepted. It is a legitimate build-tree artifact. * The ancestor walk is bounded at 12 hops rather than 8. The bound only stops the walk reaching the filesystem root; it is not a trust boundary, and sizing it flush against the deepest layout known today is what caused this outage. Module authenticity is unchanged: it is established by HMAC over the binary, not by path. The helper's actual job - distinguishing our own artifact from a Python/Node host process that dlopen'd the library - still rejects every host-interpreter path. Verified against a real layout-v2 build (`cargo +nightly test -Zbuild-dir-new-layout`), which places the binary at `target/debug/build/latticearc/<hash>/out/latticearc-<hash>`: the integrity test returns Ok and `test_initialize_and_test_sets_flag_succeeds` passes instead of aborting. Both layouts are pinned in the unit tests so a future re-narrowing fails loudly rather than at POST. 2. Feature Flag Powerset never completed, and never went red --------------------------------------------------------------- Every scheduled run was killed at exactly 6h00m: 30736018769 cancelled 6h0m22s 2026-08-02 30191256595 cancelled 6h0m19s 2026-07-26 29676467815 cancelled 6h0m19s 2026-07-19 29182750237 cancelled 6h0m21s 2026-07-12 28733241736 cancelled 6h0m22s 2026-07-05 28315598345 cancelled 6h0m19s 2026-06-28 Six hours is a hard GitHub limit on job execution time for hosted runners. `timeout-minutes` can only lower it, never raise it, so the `timeout-minutes: 720` this workflow carried - added specifically to escape the cap, per its own header comment - had no effect. And because the cap terminates a job as `cancelled` rather than `failure`, the workflow never rendered red: the check silently provided zero coverage for over a month. * Shard the sweep across an 8-way matrix using `cargo hack --partition M/8`. The powerset is 5119 combinations (measured via `--print-command-list`), so each shard runs ~640. `--partition` assigns each run to exactly one shard - contiguous, disjoint blocks of `ceil(total/N)` - so the shards cover the sweep exactly once. * Set `timeout-minutes: 300`, below the cap, so an overrun is killed by us as a visible `failure` rather than silently by GitHub. * `fail-fast: false`, so one shard failing does not discard the other seven results. * Key the cargo cache per shard; eight jobs sharing one key would race on save. `restore-keys` lets a cold shard warm from a sibling. The header comment is rewritten to record why `timeout-minutes` cannot raise the cap, so the next person to hit a shard timeout raises the partition count instead of the timeout. 3. One failed commit bricked every later commit --------------------------------------------------------------- The hook builds its test log with mktemp "$TEST_LOG_DIR/pre-commit-test-output.XXXXXX.log" BSD `mktemp` (macOS, where this hook actually runs) only substitutes a trailing run of `X`s. Given `...XXXXXX.log` it expands nothing and tries to create a file named literally `pre-commit-test-output.XXXXXX.log`. That succeeds the first time - and since the log is deliberately PRESERVED on failure, every subsequent commit then died with mktemp: mkstemp failed on .../pre-commit-test-output.XXXXXX.log: File exists before running a single test. A single failed commit permanently blocked all later commits until someone manually deleted the file. A stale log from 2026-08-02 was doing exactly that. Moving the `X`s to the end (`pre-commit-test-output.log.XXXXXX`) restores per-run unique names on both BSD and GNU mktemp; verified that two successive calls now yield distinct paths. `mktemp -d` at line 436 already had the `X`s trailing and was unaffected. The comment records why the suffix must not be moved back. 4. Flaky FIPS global-state test --------------------------------------------------------------- `test_get_fips_validation_result_consistency_succeeds` intermittently failed at global.rs:375 comparing two consecutive reads of `FIPS_VALIDATION_RESULT` - caught in a preserved hook log from 2026-08-02, and passing on the very next full run. `FIPS_GLOBAL_SERIAL` exists precisely to serialize this, and the reader does hold it across both reads. The hole is on the writer side: of the three writers of `FIPS_VALIDATION_RESULT` in this module, only one took the guard. * `ensure_initialized_for_test` did a check-then-act on `FIPS_INITIALIZED` with no guard, so several tests could all observe `false`, each run `validate_module()` (a fresh `validation_id` per call) and each `replace()` the global. * `test_init_lock_and_store_result_succeeds` replaced the global unconditionally on every run, guardless - the guaranteed writer, and the most likely source of the observed interleaving. Both now take `FIPS_GLOBAL_SERIAL`, and `ensure_initialized_for_test` re-checks the flag under the guard so initialization happens exactly once. The guard's doc comment now states the actual invariant - every writer holds it, not just store-then-read tests - and warns that it is a plain non-reentrant `Mutex`, so a guard holder must not call `ensure_initialized_for_test`. Confirmed the globals are `pub(crate)` but touched only in this module, and that `tests/tests/` merely replicates these code paths in a separate binary, so there is no cross-module or cross-binary writer left out. Verified by A/B stress of the module, same binary, same loop: pre-fix 600 iterations 7 failures post-fix 900 iterations 0 failures Every pre-fix failure was `test_get_fips_validation_result_consistency_succeeds`, matching the 2026-08-02 log. The full 510-test binary also passes 20 consecutive runs, so the added guard introduces no contention or deadlock with the module's other tests. Not fixed here: `performance.yml` carries the same layout-v1 assumption (`find target/{release,debug}/deps -name 'latticearc-*'`) in its two profiling steps. That workflow is dormant - every trigger is commented out and it last ran 2026-02-05 - so it is left alone, but it needs the same treatment before those triggers are re-enabled.