- Home
- Research notes
- Predictable code generation needs a preservation contract
RESEARCH NOTE
Why deterministic sampling is not enough, and how observable protected properties and acceptance checks make code-generation behavior testable.
Published Alexey Gavrilov
DIRECT ANSWER
What makes code generation predictable rather than merely controllable?
Predictability requires an observable contract stated before generation: what may change, what must remain stable, how each property will be measured, and when an output will be rejected. A fixed prompt, seed, mask, grammar, or latent code is only a mechanism.
Why the distinction matters
The method and the claimed guarantee need the same observable boundary.
Deterministic generation repeats an output under fixed conditions. Predictable generation makes a declared property testable: for example, an API signature remains unchanged, edits stay inside a region, syntax remains valid, or a test suite continues to pass.
The distinction matters because a control input does not define its effect. Locking a latent position, fixing a seed, or constraining a grammar can make generation easier to inspect, but the protected property must still be measured after decoding.
A practical procedure
Declare protected properties
State the source regions, structures, interfaces, behaviors, or distributions that must remain within a defined tolerance.
Choose a matching control mechanism
Use source reuse, edit masks, formal constraints, latent locking, or proposal-plus-validation according to the property being protected.
Define acceptance before sampling
Specify parsing, compilation, tests, structural checks, locality thresholds, and task-success criteria before inspecting outputs.
Measure repeated behavior
Report acceptance, rejection, stability, diversity, and uncertainty across repeated runs instead of relying on one illustrative result.
Evidence to require
A claim is only as strong as the property measured after generation or decoding.
- Protected and editable properties are stated separately.
- Every claimed guarantee has an observable test or metric.
- Task success is evaluated alongside stability.
- Repeated-run variability and rejection rates are visible.
- Unmeasured properties are explicitly excluded from the claim.
What the linked study reports
- The linked experiment exposes hierarchical discrete positions as one inspectable control surface for short Python functions.
- Locking four top-level positions improved parse rate while leaving high change in unlocked positions and high sample uniqueness.
- The paper reports this as early probabilistic evidence; it does not claim exact AST preservation, semantic equivalence, functional correctness, or repository-scale predictability.
Scope boundary
- Predictability is property-specific: a system can be predictable about syntax and unpredictable about behavior.
- Deterministic decoding can repeat the same wrong or over-broad edit.
- The available site experiment does not establish guarantees beyond its measured short-function setting.
Primary and nearby sources
Use the linked papers for the original methods, measurements, and stated limitations.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.