| Safe Haskell | None |
|---|---|
| Language | GHC2024 |
Shikumi.Optimize.Bootstrap
Description
M2 — bootstrap few-shot. Run a teacher program over the training set, keep
the runs the metric judged correct, and attach those input/output pairs as
demonstrations to the student. This "bootstraps" high-quality demos from the
program's own successful behaviour (DSPy's BootstrapFewShot).
Adapted to the delivered substrate. The plan envisioned recovering a demo
for every internal node by reading EP-7's trace tree, keyed by a per-node
NodePath, via a runProgramTraced. The delivered EP-7
(docsplans7-…replay.md) records LM-call spans by opaque SpanId with the
canonical-request and raw-response JSON, but provides neither a
NodePath↔program-node correlation nor a runProgramTraced, and the recorded
prompt is the rendered wire request, not the structured typed input. So this
plan recovers demos at the program-I/O level: a demo is the pair of the
example's input and the teacher's produced output (recoverDemo), and the kept
demos are attached to every node (the DSPy default for multi-module programs).
This is faithful to bootstrap's user-visible behaviour and is exactly what the
single-node acceptance test (M5) verifies; per-internal-node recovery waits on
EP-7/EP-4 exposing a node-correlated trace. See the plan's Decision Log.
Synopsis
- bootstrapFewShot :: (ToJSON i, ToJSON o) => Program i o -> Budget -> Optimizer i o
- bootstrapFewShotWith :: (ToJSON i, ToJSON o) => BootstrapConfig -> Program i o -> Budget -> Optimizer i o
- bootstrapKeptDemos :: forall i o (es :: [Effect]). (ToJSON i, ToJSON o, LLM :> es, Error ShikumiError :> es, Prim :> es) => BootstrapConfig -> BudgetMeter -> Program i o -> Dataset i o -> Metric o -> Eff es [Demo]
- data BootstrapConfig = BootstrapConfig {}
- defaultBootstrapConfig :: BootstrapConfig
- recoverDemo :: (ToJSON i, ToJSON o) => i -> o -> Demo
Documentation
bootstrapFewShot :: (ToJSON i, ToJSON o) => Program i o -> Budget -> Optimizer i o Source #
Bootstrap few-shot with the default configuration.
Arguments
| :: (ToJSON i, ToJSON o) | |
| => BootstrapConfig | |
| -> Program i o | teacher program whose successful runs supply demos |
| -> Budget | |
| -> Optimizer i o |
Bootstrap few-shot with an explicit configuration. The teacher may be a
stronger or chain-of-thought variant of the student, or the student itself; it
must share the student's input/output types. Each teacher run reserves one
predicted LM completion per teacher predict node before it runs; when the next
teacher run does not fit the Budget, demo recovery stops and the demos found so
far are attached.
bootstrapKeptDemos :: forall i o (es :: [Effect]). (ToJSON i, ToJSON o, LLM :> es, Error ShikumiError :> es, Prim :> es) => BootstrapConfig -> BudgetMeter -> Program i o -> Dataset i o -> Metric o -> Eff es [Demo] Source #
Recover metric-passing demos from teacher runs under a shared budget meter.
data BootstrapConfig Source #
Tunables for a bootstrap search.
Constructors
| BootstrapConfig | |
Fields
| |
Instances
defaultBootstrapConfig :: BootstrapConfig Source #
Keep only perfectly-correct teacher runs; attach at most four demos.
recoverDemo :: (ToJSON i, ToJSON o) => i -> o -> Demo Source #
Recover a demonstration from one teacher run: pair the typed input with the
teacher's produced output, serialized to the JSON Demo the run-time adapter
decodes back into the node's typed demo channel. (The JSON keys are the record
field names, so fromModel round-trips them — see the unit test.)