| Safe Haskell | None |
|---|---|
| Language | GHC2024 |
Shikumi.Optimize.GEPA
Description
GEPA (EP-22): a reflective, evolutionary instruction optimizer. Where greedy
coordinate ascent is blind, GEPA is reflective: it runs the program while
capturing, per node, a short natural-language critique ("feedback") of how that
node performed, then reflects on those critiques to propose a rewritten
instruction. Where greedy search keeps one best program, GEPA keeps a __Pareto
frontier__ (see Shikumi.Optimize.Pareto) of candidates none strictly worse than
another across the per-example score vector, samples a parent from it, mutates one
node by reflection, scores the child, and folds it back in — until a Budget is
spent.
GEPA consumes EP-16's per-node feedback channel (attachFeedback/feedbackFor
keyed by NodePath, with node identity from programNodePaths) and EP-19's
summaries (here via small in-package fallbacks). The Trace/Feedback effects are
discharged internally (via runFeedback against the ambient Prim), so the
public Optimizer row is unchanged (MasterPlan integration point 5). Feedback
is attached at the program level to every node (the DSPy default and the M1
baseline); node-specific critique from per-node sub-traces is a documented deferral.
Output is V1's CompiledProgram via freezeProgram; the
frontier is internal bookkeeping, not part of the returned type. GEPA reuses V1's
Metric/Score plus a critique Text (its FeedbackMetric) rather than a
parallel reward type (MasterPlan integration point #1).
Synopsis
- type FeedbackMetric o = o -> Prediction o -> (Score, Text)
- data ReflectIn = ReflectIn {
- currentInstruction :: !Text
- feedback :: !Text
- programSummary :: !Text
- datasetSummary :: !Text
- fieldSummary :: !Text
- newtype ReflectOut = ReflectOut {}
- reflectiveProposer :: Program ReflectIn ReflectOut
- captureFeedback :: forall (es :: [Effect]) i o. (LLM :> es, Error ShikumiError :> es, Prim :> es) => Dataset i o -> FeedbackMetric o -> Program i o -> Eff es (FeedbackLog, [Double])
- mutateNode :: forall (es :: [Effect]) i o. (LLM :> es, Error ShikumiError :> es) => Program ReflectIn ReflectOut -> Text -> Text -> [NodeFields] -> FeedbackLog -> [NodePath] -> Int -> Program i o -> Eff es (Program i o)
- gepa :: Program ReflectIn ReflectOut -> FeedbackMetric o -> Budget -> Optimizer i o
Documentation
type FeedbackMetric o = o -> Prediction o -> (Score, Text) Source #
A feedback metric: like V1's Metric but also emits a short critique. Reuses
Score (EP-18's reward vocabulary reduces to this) plus a critique Text.
The reflective proposer's input: the node's current instruction, its accumulated critiques, program/dataset summaries, and the node's field names.
Constructors
| ReflectIn | |
Fields
| |
Instances
newtype ReflectOut Source #
Constructors
| ReflectOut | |
Fields | |
Instances
| Generic ReflectOut Source # | |||||
Defined in Shikumi.Optimize.GEPA Associated Types
| |||||
| Show ReflectOut Source # | |||||
Defined in Shikumi.Optimize.GEPA Methods showsPrec :: Int -> ReflectOut -> ShowS # show :: ReflectOut -> String # showList :: [ReflectOut] -> ShowS # | |||||
| ToPrompt ReflectOut Source # | |||||
Defined in Shikumi.Optimize.GEPA Methods toPromptFields :: ReflectOut -> [(Text, Text)] toPrompt :: ReflectOut -> Text imageFields :: ReflectOut -> [Image] imageFieldNames :: ReflectOut -> [Text] | |||||
| FromModel ReflectOut Source # | |||||
Defined in Shikumi.Optimize.GEPA Methods fromModelP :: FieldPath -> Value -> Either ShikumiError ReflectOut | |||||
| ToSchema ReflectOut Source # | |||||
Defined in Shikumi.Optimize.GEPA Methods toSchema :: Proxy ReflectOut -> Value | |||||
| Validatable ReflectOut Source # | |||||
Defined in Shikumi.Optimize.GEPA Methods validate :: ReflectOut -> Either Text ReflectOut | |||||
| type Rep ReflectOut Source # | |||||
Defined in Shikumi.Optimize.GEPA type Rep ReflectOut = D1 ('MetaData "ReflectOut" "Shikumi.Optimize.GEPA" "shikumi-optimize-0.2.1.2-inplace" 'True) (C1 ('MetaCons "ReflectOut" 'PrefixI 'True) (S1 ('MetaSel ('Just "proposedInstruction") 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 Text))) | |||||
reflectiveProposer :: Program ReflectIn ReflectOut Source #
The default reflective proposer: a single predict node that addresses the feedback specifically.
captureFeedback :: forall (es :: [Effect]) i o. (LLM :> es, Error ShikumiError :> es, Prim :> es) => Dataset i o -> FeedbackMetric o -> Program i o -> Eff es (FeedbackLog, [Double]) Source #
Run the program over the whole dataset, attaching the feedback metric's critique
(when non-empty) to every node keyed by its NodePath, and returning the
FeedbackLog alongside the per-example score vector (for the Pareto frontier).
Arguments
| :: forall (es :: [Effect]) i o. (LLM :> es, Error ShikumiError :> es) | |
| => Program ReflectIn ReflectOut | |
| -> Text | program summary |
| -> Text | dataset summary |
| -> [NodeFields] | |
| -> FeedbackLog | |
| -> [NodePath] | |
| -> Int | |
| -> Program i o | |
| -> Eff es (Program i o) |
Reflect on node idx's accumulated feedback and overwrite its instruction with
the proposal. A node with no feedback is left unchanged (nothing to reflect on).
gepa :: Program ReflectIn ReflectOut -> FeedbackMetric o -> Budget -> Optimizer i o Source #
The reflective evolutionary optimizer. Takes its reflective proposer and feedback
metric explicitly (so it is testable under a stub LM) and returns V1's
Optimizer. GEPA gates its seed evaluation before any LM call; if the budget is
too small to score the student once, it returns the student unscored. Each
evolution step reserves a conservative full-step cost before capture, reflection,
and child scoring.