Find a concept

Search Notes and Discovery

Enter a term to search published Notes and Discovery.

    In this topic

    Structured note

    DSPy End to End: Signatures, Modules, and Optimizers

    Build DSPy signatures, modules, metrics, and optimizers, then inspect RAG, bounded ReAct agents, and the fine-tuning handoff.

    Prompt EngineeringAdvanced
    Published
    Reviewed
    Reading time
    27 min

    Prerequisites

    On this page

    DSPy expresses a language-model task as a program: a signature defines the task and fields, a module determines how it runs, and an optimizer searches for better instructions or demonstrations using examples and a metric. The central workflow is to define behavior, measure a baseline, optimize, and evaluate the resulting program.

    The benefit is a repeatable way to improve a pipeline, including retrieval or tool use. It does not remove prompts, establish factual correctness, or make a weak metric reliable. An optimizer can improve the measured score while the application still produces wrong answers.

    This third Prompt Engineering note follows prompt foundations and structured output. Its sole knowledge source is the supplied DSPy_End_to_End_Detailed_Notes.ipynb. The source notebook contains no saved execution outputs. Outputs explicitly marked reproduced offline below come from its deterministic Python examples during page preparation; they do not use DSPy, a provider, or an LLM. LLM calls, optimizer scores, generated reasoning, and training outcomes remain unexecuted examples with no recorded result.

    Read sequentially for the full lifecycle: setup → contracts → execution modules → data and metrics → baseline → optimizers → persistence → RAG → ReAct → fine-tuning handoff. Code blocks follow the notebook’s execution order and share variables; they are not independent scripts. Provider configuration and DSPy APIs are version dependent. The notebook prints an installed version when run, but it records no version or successful provider configuration.

    Execution switches

    RUN_LLM          → normal LLM calls
    RUN_OPTIMIZERS   → Bootstrap / MIPRO / GEPA
    RUN_GPU_SECTION  → optional fine-tuning handoff

    Keep all three False while revising.

    RUN_LLM = False
    RUN_OPTIMIZERS = False
    RUN_GPU_SECTION = False
    
    print("RUN_LLM         =", RUN_LLM)
    print("RUN_OPTIMIZERS  =", RUN_OPTIMIZERS)
    print("RUN_GPU_SECTION =", RUN_GPU_SECTION)

    Output reproduced offline from this deterministic notebook code:

    RUN_LLM         = False
    RUN_OPTIMIZERS  = False
    RUN_GPU_SECTION = False

    With these defaults, the LLM, optimizer, and GPU branches stay disabled. Importing DSPy and defining modules can still occur when the package is installed. Turning on RUN_OPTIMIZERS alone is insufficient: the compile branches also require RUN_LLM and their optimizer imports.

    What DSPy changes

    A handwritten prompt often combines the task, output rules, examples, and execution strategy in one string. DSPy gives those responsibilities names. You still need to decide what counts as a good answer; the framework cannot infer that from the presence of a signature.

    Traditional prompt engineering:

    User Query
        ↓
    Hand-written Prompt
        ↓
    LLM
        ↓
    Answer

    DSPy:

    User Input
        ↓
    DSPy Program
        ├── Signature
        ├── Module
        └── Optimizer
        ↓
    LLM
        ↓
    Output

    Core mental model:

    Replace hand-maintained prompts with programmable LLM pipelines\boxed{\text{Replace hand-maintained prompts with programmable LLM pipelines}}

    DSPy still uses instructions/prompts internally, but you work through program abstractions rather than manually managing every final prompt string.

    The three core pieces

                             DSPy
                               │
              ┌────────────────┼────────────────┐
              │                │                │
          Signature          Module          Optimizer
              │                │                │
            WHAT?             HOW?            IMPROVE?

    Signature

    Defines the task contract.

    • What is the task?
    • What goes in?
    • What comes out?

    Module

    Defines how the Signature executes.

    Examples:

    • Predict
    • ChainOfThought
    • ProgramOfThought
    • ReAct

    Optimizer

    Improves the DSPy program from examples and metrics.

    Examples:

    • BootstrapFewShot
    • MIPROv2
    • GEPA
    Signature = WHAT,Module = HOW,Optimizer = IMPROVE\boxed{\text{Signature = WHAT,\quad Module = HOW,\quad Optimizer = IMPROVE}}
    Three roles in a programmable LM system
    WHATSignatureTask, typed inputs, typed outputs
    HOWModuleExecution: Predict, ChainOfThought, ProgramOfThought
    IMPROVEOptimizerExamples + metric guide program optimization
    Define the program with a Signature and Module. Optimize it separately using examples and a metric. DSPy still uses prompts internally.

    Keep the three questions separate. A sentiment signature can stay unchanged while you compare Predict and ChainOfThought. Later, an optimizer can improve the selected module’s instructions or demonstrations without changing what the application asks the signature to return.

    Setup

    A disabled switch prevents the corresponding example from making calls; it does not configure credentials, install interpreters, or establish a compatible API. Run the examples in order and enable a section only after its prerequisites are ready.

    The notebook removes project-specific helpers and keeps these examples self-contained.

    Install DSPy only if needed:

    pip install -U dspy
    # Uncomment if DSPy is not installed.
    # %pip install -U dspy
    import os
    
    try:
        import dspy
        print("DSPy imported.")
        print("Version:", getattr(dspy, "__version__", "unknown"))
    except Exception as e:
        dspy = None
        print("DSPy is not installed.")
        print("Install with: pip install -U dspy")
        print("Import error:", e)
    # Provider-neutral template.
    # Your installed DSPy/provider setup determines the exact model string
    # and required environment variable(s).
    
    MODEL_NAME = os.getenv("DSPY_MODEL", "openai/gpt-4o-mini")
    lm = None
    
    if RUN_LLM:
        if dspy is None:
            raise RuntimeError("Install DSPy first.")
    
        lm = dspy.LM(MODEL_NAME)
        dspy.configure(lm=lm)
        print("Configured:", MODEL_NAME)
    else:
        print("LLM calls disabled. Set RUN_LLM=True after provider configuration.")

    The model name is a configuration example from the notebook, not a verified deployment choice. DSPY_MODEL lets the environment supply another model string. Configure the credentials required by that provider outside the code; no credential value belongs in an article or notebook output.

    The word contract describes what the program asks for. It should not be read as evidence that every answer is true, that every numeric constraint is enforced, or that every provider handles the contract identically.

    Signatures — the typed contract

    A Signature is the contract between your application and the language model.

    It answers:

    1. What is the task?
    2. What is the input?
    3. What is the output?

    Conceptually:

    Task Description
          ↓
    Typed Input Field(s)
          ↓
    Typed Output Field(s)

    A Signature should stay focused on the actual task. It is not the place to build a huge persona-style prompt.

    if dspy is not None:
        class QuestionAnswering(dspy.Signature):
            """Answer questions with short factual responses."""
    
            question: str = dspy.InputField()
            answer: str = dspy.OutputField(
                desc="A concise factual answer"
            )
    
        print("Signature:", QuestionAnswering.__name__)
        print("Fields:", list(QuestionAnswering.fields.keys()))
    else:
        print("Install DSPy to create the live Signature.")

    Class-based vs inline Signature

    Class-based

    class QuestionAnswering(dspy.Signature):
        """Answer questions with short factual responses."""
    
        question: str = dspy.InputField()
        answer: str = dspy.OutputField()

    Best when you want:

    • typed fields
    • descriptions
    • maintainability
    • reuse

    Inline

    "question -> answer"

    Useful for quick prototyping.

    if RUN_LLM and dspy is not None:
        inline_predictor = dspy.Predict("question -> answer")
        result = inline_predictor(
            question="What is the capital of Odisha?"
        )
        print(result.answer)
    else:
        print("Inline Signature execution skipped.")

    Typed output example

    Input:
        text: str
    
    Outputs:
        sentiment: positive | negative | neutral
        confidence: float

    This makes the output contract explicit.

    from typing import Literal
    
    if dspy is not None:
        class SentimentClassification(dspy.Signature):
            """Classify the sentiment of the supplied text."""
    
            text: str = dspy.InputField()
            sentiment: Literal["positive", "negative", "neutral"] = dspy.OutputField()
            confidence: float = dspy.OutputField(
                desc="Confidence score between 0 and 1"
            )
    
        print("Defined:", SentimentClassification.__name__)
    A Signature names the task and its fields

    Task: Answer questions with short factual responses.

    InputFieldquestion: strThe question supplied by the application.
    OutputFieldanswer: strDescription: “A concise factual answer”.
    The notebook’s QuestionAnswering contract describes short factual answers. The same Signature can be executed by different Modules.

    Literal makes the intended sentiment labels explicit. The confidence field is a float with a description, however: the example does not contain a range validator. A description saying “between 0 and 1” is not an enforced check, and a model’s self-reported confidence is not demonstrated to be calibrated. Validate constraints required by the application separately. No sentiment prediction is recorded in the notebook.

    Modules — how the task executes

    A Signature defines what.

    A Module defines how.

    Signature
        ↓
    Module
        ↓
    LLM
        ↓
    Prediction

    The main execution modules covered here are:

    1. Predict
    2. ChainOfThought
    3. ProgramOfThought
    4. ReAct later

    dspy.Predict

    Predict is the simplest/direct module.

    Input
      ↓
    Signature
      ↓
    Predict
      ↓
    LLM
      ↓
    Output

    Use it when you do not need a more specialized reasoning or tool-execution pattern.

    if RUN_LLM and dspy is not None:
        predictor = dspy.Predict(QuestionAnswering)
        result = predictor(
            question="What is the capital of Odisha?"
        )
        print("Answer:", result.answer)
    else:
        print("Predict execution skipped.")

    dspy.ChainOfThought

    The Signature can stay the same while the Module changes.

    Same Signature
          ↓
    ChainOfThought
          ↓
    Reasoning-oriented execution
          ↓
    Answer

    Trade-off:

    More reasoning
        ↓
    More generated tokens
        ↓
    More latency / cost

    So CoT is useful for multi-step tasks, but should not automatically replace Predict for every simple task.

    if RUN_LLM and dspy is not None:
        cot = dspy.ChainOfThought(QuestionAnswering)
    
        result = cot(
            question="If a train travels 60 km/h for 2.5 hours, how far does it go?"
        )
    
        if hasattr(result, "reasoning"):
            print("Reasoning:")
            print(result.reasoning)
    
        print("Answer:", result.answer)
    else:
        print("ChainOfThought execution skipped.")

    For the train question, distance is speed multiplied by time: 60 × 2.5 = 150 km. This arithmetic explains the target calculation; it is not a captured model answer. Generated reasoning text, if returned, still needs checking against the problem.

    dspy.ProgramOfThought

    ProgramOfThought is useful when the model should generate code and use computation.

    User Question
         ↓
    ProgramOfThought
         ↓
    LLM generates code
         ↓
    Interpreter / sandbox executes
         ↓
    Computed result
         ↓
    Final Answer

    Mental model:

    LLM    → chooses the computational procedure
    Python → performs the calculation

    Example:

    Question:
    "What is the average of 45, 67, 89, and 23?"
    
    Conceptual generated code:
    values = [45, 67, 89, 23]
    average = sum(values) / len(values)
    if RUN_LLM and dspy is not None:
        pot = dspy.ProgramOfThought(QuestionAnswering)
    
        result = pot(
            question="What is the average of 45, 67, 89, and 23?"
        )
    
        print("Answer:", result.answer)
    
        try:
            dspy.inspect_history(n=5)
        except Exception as e:
            print("History inspection unavailable:", e)
    else:
        print("ProgramOfThought execution skipped.")

    The average example has sum 45 + 67 + 89 + 23 = 224, so 224 / 4 = 56. That is a mathematical derivation, not a ProgramOfThought result. The notebook supplies no generated code, execution trace, or final model output. PoT also depends on an available execution environment; selecting the module does not establish that environment’s isolation or safety.

    Quick comparison

    ModuleMain ideaBest fit
    PredictDirect predictionSimple/factual task
    ChainOfThoughtReasoning-oriented executionMulti-step reasoning
    ProgramOfThoughtGenerate + execute codeArithmetic/programmatic task

    Revision shortcut:

    Predict → ANSWER
    
    CoT     → REASON → ANSWER
    
    PoT     → WRITE CODE → EXECUTE → ANSWER

    RLM intuition — when data is larger than the context window

    The RLM material was introduced at a high level.

    Problem:

    Huge Data
       >
    LLM Context Window

    Instead of placing everything into the prompt:

    Huge Data
        ↓
    Keep data external / program-accessible
        ↓
    Inspect / search / filter / summarize
        ↓
    Bring forward useful information
        ↓
    Final Answer

    Revision note:

    RLM = Recursive Language Model

    This is a conceptual data-access strategy, not an implemented RLM module in the notebook. Keeping data external only helps if the inspection/search process can bring forward enough relevant information. The page therefore makes no claim about a working recursive implementation or its performance.

    DSPy compilation / optimization

    Compilation here means preparing an improved DSPy program from data and a scoring rule. In these optimizer examples, it changes prompt configuration rather than training model weights. The later fine-tuning section is a separate handoff.

    The key shift is:

    "Can I write a better prompt manually?"

    to:

    "Can I define a metric and let DSPy improve the program?"

    Common flow:

    Signature + Module
            ↓
    Training Examples
            ↓
    Run Program
            ↓
    Predictions
            ↓
    Metric
            ↓
    Optimizer
            ↓
    Better demos / instructions
            ↓
    Compiled Program
            ↓
    Evaluate on held-out data

    Instruction + Demonstrations

    Two terms become critical.

    Instruction

    The task description.

    Example:

    Translate English to French.

    Demonstration

    A worked input-output example.

    English: cat
    French : chat

    Useful identity:

    Prompt behavior≈Instruction+Demonstrations\boxed{\text{Prompt behavior} \approx \text{Instruction} + \text{Demonstrations}}

    dspy.Example

    Optimization data uses examples such as:

    dspy.Example(
        question="What is RAG?",
        answer="Retrieval-Augmented Generation"
    ).with_inputs("question")

    .with_inputs(...) declares the input fields.
    Other fields can act as target/reference outputs.

    if dspy is not None:
        raw_qa = [
            ("What is DSPy?", "A framework for programming and optimizing language-model pipelines."),
            ("What does RAG stand for?", "Retrieval-Augmented Generation"),
            ("What does CoT stand for?", "Chain of Thought"),
            ("What is few-shot prompting?", "Providing a small number of examples in the prompt"),
            ("What is a metric?", "A scoring function used to evaluate predictions"),
            ("What is a demonstration?", "A worked input-output example used by the program"),
            ("What does MIPROv2 optimize?", "Instructions and demonstrations"),
            ("What is GEPA centered on?", "Reflection, feedback, and Pareto-style prompt evolution"),
            ("What is ReAct?", "A reasoning and tool-action loop"),
            ("What is ProgramOfThought?", "A module that can generate code for computation"),
        ]
    
        examples = [
            dspy.Example(question=q, answer=a).with_inputs("question")
            for q, a in raw_qa
        ]
    
        trainset = examples[:7]
        devset = examples[7:]
    
        print("Trainset:", len(trainset))
        print("Devset  :", len(devset))
        print("Sample  :", trainset[0])
    else:
        trainset = []
        devset = []
        print("Install DSPy to create live dspy.Example objects.")

    The split contains seven training examples and three development examples when DSPy is available. .with_inputs("question") prevents the reference answer from being treated as an ordinary task input. These ten teaching examples are too small to establish broad quality; the three development questions cover only GEPA, ReAct, and ProgramOfThought.

    Metrics — the optimization target

    An optimizer needs a scoring rule.

    Prediction
        ↓
    Metric
     ┌──┴───┐
    Good   Bad
     ↓       ↓
    Keep   Reject / lower score

    Examples:

    TaskPossible metric
    Classificationlabel accuracy
    Extractionfield-level correctness
    QAexact / semantic correctness
    RAGcorrectness + grounding
    Agentanswer quality + tool-use criteria

    Important:

    Weak metric⇒weak or misleading optimization\boxed{\text{Weak metric} \Rightarrow \text{weak or misleading optimization}}
    def normalized_token_overlap(expected: str, predicted: str) -> float:
        """Simple teaching metric, not a universal production metric."""
        expected_words = set(expected.lower().split())
        predicted_words = set(predicted.lower().split())
    
        if not expected_words:
            return 0.0
    
        return len(expected_words & predicted_words) / len(expected_words)
    
    
    def answer_metric(example, prediction, trace=None):
        overlap = normalized_token_overlap(
            example.answer,
            prediction.answer,
        )
        return 1.0 if overlap >= 0.5 else 0.0
    
    
    print(
        normalized_token_overlap(
            "Retrieval-Augmented Generation",
            "Retrieval Augmented Generation"
        )
    )

    Output reproduced offline from this deterministic notebook code:

    0.5

    The metric computes overlap using sets of whitespace-separated, lowercased words:

    R(E,P)=∣E∩P∣∣E∣,m(E,P)={1R(E,P)≥0.50R(E,P)<0.5R(E,P)=\frac{|E\cap P|}{|E|},\qquad m(E,P)=\begin{cases}1 & R(E,P)\geq 0.5\\0 & R(E,P)<0.5\end{cases}

    Here, EE is the set of reference words, PP is the set of predicted words, RR is the reference-word overlap, and mm is the binary score. The code defines an empty reference set to have overlap zero. This is a teaching measure of reference coverage, not a complete semantic-correctness measure.

    For the example, the reference splits into {"retrieval-augmented", "generation"} and the prediction into {"retrieval", "augmented", "generation"}. Only generation matches: 1 / 2 = 0.5, which passes the threshold. Punctuation, hyphens, extra claims, and negation are not understood by this function. A response can mention enough reference words and still be wrong.

    Baseline program

    The same evaluation rule and evaluation examples must be used when comparing the baseline and an optimized program. Otherwise a score difference can reflect a changed measurement rather than an improved program.

    Always establish a baseline before optimization.

    No baseline⇒no evidence of improvement\boxed{\text{No baseline} \Rightarrow \text{no evidence of improvement}}
    if dspy is not None:
        class QA(dspy.Signature):
            """Answer questions with short factual responses."""
    
            question: str = dspy.InputField()
            answer: str = dspy.OutputField()
    
    
        class SimpleQA(dspy.Module):
            def __init__(self):
                super().__init__()
                self.generate = dspy.ChainOfThought(QA)
    
            def forward(self, question):
                return self.generate(question=question)
    
    
        baseline_program = SimpleQA()
        print("Baseline program created.")
    baseline_score = None
    evaluator = None
    
    if RUN_LLM and dspy is not None and devset:
        from dspy.evaluate import Evaluate
    
        evaluator = Evaluate(
            devset=devset,
            metric=answer_metric,
            num_threads=2,
            display_progress=True,
        )
    
        baseline_result = evaluator(baseline_program)
        baseline_score = baseline_result.score
        print("Baseline score:", baseline_score)
    else:
        print("Baseline API evaluation skipped.")

    The notebook stores no baseline score. If the branch runs in a compatible environment, Evaluate scores baseline_program on devset with answer_metric; num_threads=2 permits concurrent evaluation work. The evaluator’s displayed score format depends on the installed version. A printed score is still only evidence about this metric and these examples.

    BootstrapFewShot

    Start with a reasonable task description and ask whether better examples help. This is the smallest optimizer concept in the notebook: retain demonstrations that satisfy the metric, then use them with the existing instruction.

    Core question:

    Which demonstrations should I add?\boxed{\text{Which demonstrations should I add?}}

    Most important rule:

    Instruction → FIXED
    Demos       → SELECTED / OPTIMIZED

    BootstrapFewShot focuses on good examples rather than rewriting the main instruction.

    Bootstrap workflow

    Train Set
       ↓
    Run Original DSPy Program
       ↓
    Collect Predictions
       ↓
    Apply Metric
       ↓
    ┌─────────────┬─────────────┐
    │ Successful  │ Rejected    │
    └──────┬──────┴─────────────┘
           ↓
    Select Top-N successful examples
           ↓
    Use them as demonstrations
           ↓
    Original Instruction + Demos
           ↓
    Compiled Program

    The relevant control is conceptually:

    max_bootstrapped_demos = N

    You normally keep a small demonstration set rather than inserting every successful training sample.

    if dspy is not None:
        try:
            from dspy.teleprompt import BootstrapFewShot
            print("BootstrapFewShot available.")
        except Exception as e:
            BootstrapFewShot = None
            print("BootstrapFewShot import path differs in this version:", e)
    else:
        BootstrapFewShot = None
    optimized_bootstrap = None
    bootstrap_score = None
    
    if (
        RUN_OPTIMIZERS
        and RUN_LLM
        and BootstrapFewShot is not None
        and trainset
    ):
        teleprompter = BootstrapFewShot(
            metric=answer_metric,
            max_bootstrapped_demos=3,
        )
    
        optimized_bootstrap = teleprompter.compile(
            SimpleQA(),
            trainset=trainset,
        )
    
        print("Bootstrap compilation complete.")
    
        if evaluator is not None:
            bootstrap_score = evaluator(optimized_bootstrap).score
            print("Bootstrap score:", bootstrap_score)
    else:
        print("Bootstrap optimization skipped.")

    max_bootstrapped_demos=3 bounds the requested bootstrapped demonstration set. The notebook’s “top-N” diagram is a conceptual selection sketch; it does not demonstrate that the implementation globally ranks all successful examples. Inspect the compiled program and compare evaluation results rather than assuming that every retained trace is desirable.

    No compiled demonstration set, optimizer trace, or BootstrapFewShot score is saved in the source notebook.

    Bootstrap revision summary

    BootstrapFewShot
    
    Input:
    - original DSPy program
    - trainset
    - metric
    
    Learns:
    - useful demonstrations
    
    Instruction:
    - stays fixed
    
    Question answered:
    "Which examples should I show the model?"

    MIPROv2

    If demonstrations are not the only source of failure, instruction wording becomes another search variable. MIPROv2 explores combinations; the notebook’s small candidate and trial counts keep the example bounded, but do not establish an adequate search budget for a real task.

    MIPROv2 asks a broader question:

    Which instruction + demonstration set performs best together?\boxed{\text{Which instruction + demonstration set performs best together?}}

    Difference:

    BootstrapFewShot
    → optimize demos
    
    MIPROv2
    → optimize/search instruction + demos

    Use:

    I+D=prompt configuration\boxed{I + D = \text{prompt configuration}}

    Four-stage mental model

    Stage 1
    Generate candidate demo sets
            ↓
    Stage 2
    Generate candidate instructions
            ↓
    Stage 3
    Create instruction × demo combinations
            ↓
    Stage 4
    Efficient optimization / trial search
            ↓
    Select a strong I + D combination

    Stage 1 — candidate demo sets

    C1 = {D1, D2, D3}
    C2 = {D1, D4, D5}
    C3 = {D2, D3, D6}

    Stage 2 — candidate instructions

    I1 = "Translate English to French."
    I2 = "Return the correct French equivalent."
    I3 = "Convert each English term into French."

    Stage 3 — combinations

    I1 + C1
    I1 + C2
    I2 + C1
    I2 + C2
    ...

    When the combination space is large, trying every pair is expensive.

    The notebook frames MIPROv2 as using Bayesian/trial-based optimization to explore the search space more efficiently.

    With three candidate instructions and three candidate demonstration sets, the conceptual cross-product has 3 × 3 = 9 pairs. This is an illustration of the search space, not a count of actual trials from a recorded MIPRO run. The compile example below requests five trials.

    if dspy is not None:
        try:
            from dspy.teleprompt import MIPROv2
            print("MIPROv2 available.")
        except Exception as e:
            MIPROv2 = None
            print("MIPROv2 import path differs in this version:", e)
    else:
        MIPROv2 = None
    optimized_mipro = None
    mipro_score = None
    
    if (
        RUN_OPTIMIZERS
        and RUN_LLM
        and MIPROv2 is not None
        and trainset
    ):
        mipro = MIPROv2(
            metric=answer_metric,
            auto=None,
            num_candidates=3,
            init_temperature=1.0,
        )
    
        optimized_mipro = mipro.compile(
            SimpleQA(),
            trainset=trainset,
            valset=devset,
            num_trials=5,
            minibatch=False,
        )
    
        print("MIPROv2 compilation complete.")
    
        if evaluator is not None:
            mipro_score = evaluator(optimized_mipro).score
            print("MIPROv2 score:", mipro_score)
    else:
        print("MIPROv2 optimization skipped.")

    Evaluation boundary: this example passes devset to MIPROv2 as valset, then evaluates the compiled program on that same devset. That is useful for development, but the resulting score would be a validation score used during selection, not an untouched test result. Reserve a separate test set before making a generalization claim. No baseline-to-MIPRO improvement is recorded here.

    BootstrapFewShot vs MIPROv2

    DimensionBootstrapFewShotMIPROv2
    Rewrites instructionNoYes
    Selects demosYesYes
    Main questionWhich examples?Which instruction + examples?
    ComplexityLowerHigher
    Relative costLowerHigher
    Good first optimizerYesUsually after baseline/bootstrap

    GEPA

    GEPA’s additional ingredient is feedback about why a candidate failed. A score says how well a candidate performed; natural-language feedback can help propose a different instruction. The feedback itself still needs scrutiny.

    GEPA adds a different style of optimization.

    Instead of only searching candidate combinations:

    Run
      ↓
    Observe success / failure
      ↓
    Reflect in natural language
      ↓
    Generate feedback
      ↓
    Rewrite / mutate prompt
      ↓
    Run again

    Two central ideas:

    1. Reflection-based prompt improvement
    2. Pareto-style retention of multiple useful prompt variants

    Pareto intuition

    Suppose prompts score differently on two abilities:

    PromptMathCoding
    A9570
    B8095
    C9090
    D7560

    Interpretation:

    A → excellent at Math
    B → excellent at Coding
    C → balanced
    D → weaker on both

    D can be discarded.

    A, B, and C each preserve a useful trade-off.

    {A,B,C}=Pareto-style useful set\boxed{\{A,B,C\} = \text{Pareto-style useful set}}

    The point is diversity rather than one universal best prompt.

    prompt_scores = {
        "A": (95, 70),
        "B": (80, 95),
        "C": (90, 90),
        "D": (75, 60),
    }
    
    def dominates(x, y):
        """x dominates y when x is never worse and is better in at least one objective."""
        return (
            all(a >= b for a, b in zip(x, y))
            and any(a > b for a, b in zip(x, y))
        )
    
    pareto = []
    
    for name, score in prompt_scores.items():
        is_dominated = any(
            other_name != name and dominates(other_score, score)
            for other_name, other_score in prompt_scores.items()
        )
    
        if not is_dominated:
            pareto.append(name)
    
    print("Pareto-style retained prompts:", pareto)

    Output reproduced offline from this deterministic notebook code:

    Pareto-style retained prompts: ['A', 'B', 'C']

    For maximization objectives, one score vector dominates another when it is at least as high on every objective and strictly higher on at least one:

    x≻y⟺(∀i, xi≥yi) ∧ (∃j, xj>yj).x \succ y\quad\Longleftrightarrow\quad (\forall i,\ x_i\geq y_i)\ \land\ (\exists j,\ x_j>y_j).

    The components ii and jj index the objectives, here math and coding. A, B, and C each outperform D on both objectives. None of A, B, or C dominates both other retained candidates. The numbers are illustrative scores supplied by the notebook, not model benchmarks. This standalone Pareto calculation explains the idea; it is not an execution of GEPA’s internal selection process.

    Reflection Language Model

    Mental model:

    Original LM
    → task executor / student
    
    Reflection LM
    → reviewer / teacher

    The reflection model can inspect:

    Current Instruction
           +
    Question / Input
           +
    Original Model Answer
           +
    Ground Truth / Desired Answer
           ↓
    Reflection LM
           ↓
    Feedback

    Then:

    Feedback
       ↓
    Generate Better Instruction
       ↓
    Execute Again
       ↓
    Evaluate Again

    GEPA loop

    Current Prompt
          ↓
    Original LM executes
          ↓
    Model Answer
          ↓
    Compare with desired behavior
          ↓
    Reflection LM
          ↓
    Natural-language feedback
          ↓
    Rewrite / mutate instruction
          ↓
    New Prompt
          ↓
    Run again
          ↓
    Repeat within budget

    Repeated prompt evolution is the prompt mutation idea.

    Cost intuition:

    More reflection loops
    → more LLM calls
    → more evaluation
    → more cost
    if dspy is not None:
        try:
            from dspy.teleprompt import GEPA
            print("GEPA available.")
        except Exception as e:
            GEPA = None
            print("GEPA import path differs in this version:", e)
    else:
        GEPA = None

    GEPA-compatible metric shape

    The source notebook used a richer callback signature:

    def metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
        ...

    The extra values allow the optimizer/evaluation flow to carry additional trace context.

    def math_metric(
        gold,
        pred,
        trace=None,
        pred_name=None,
        pred_trace=None,
    ):
        expected = str(gold.answer).strip()
        actual = str(pred.answer).strip()
        return 1.0 if expected in actual else 0.0
    
    
    print("GEPA-compatible metric template created.")

    Output reproduced offline from this deterministic notebook code:

    GEPA-compatible metric template created.

    The extra callback arguments preserve a place for trace context, but this implementation ignores them. It returns only a numeric score and does not construct explanatory feedback. Compatibility must be checked against the installed GEPA interface.

    More seriously, expected in actual is substring matching: reference "30" would accept prediction "130". An empty expected string would match every prediction. The function is a metric-shape template, not a reliable numerical-answer validator.

    gepa_optimized = None
    
    if (
        RUN_OPTIMIZERS
        and RUN_LLM
        and GEPA is not None
        and dspy is not None
    ):
        class MathProblem(dspy.Signature):
            """Solve a math word problem."""
    
            problem: str = dspy.InputField()
            answer: str = dspy.OutputField(
                desc="Final answer"
            )
    
    
        class MathSolver(dspy.Module):
            def __init__(self):
                super().__init__()
                self.solve = dspy.ChainOfThought(MathProblem)
    
            def forward(self, problem):
                return self.solve(problem=problem)
    
    
        math_train = [
            dspy.Example(problem="What is 12 + 18?", answer="30").with_inputs("problem"),
            dspy.Example(problem="What is 9 * 7?", answer="63").with_inputs("problem"),
            dspy.Example(problem="What is 100 / 4?", answer="25").with_inputs("problem"),
            dspy.Example(problem="What is 15 + 5 * 2?", answer="25").with_inputs("problem"),
        ]
    
        gepa = GEPA(
            metric=math_metric,
            auto="light",
            reflection_lm=lm,
        )
    
        gepa_optimized = gepa.compile(
            MathSolver(),
            trainset=math_train,
        )
    
        print("GEPA optimization complete.")
    else:
        print("GEPA run skipped.")
        print("It is disabled by default because it can be expensive.")

    The reflection role and task-execution role are conceptually separate, but this example passes the same configured lm as reflection_lm. It does not require or demonstrate a different reviewer model. The four arithmetic training examples and auto="light" form a teaching configuration. There is no saved GEPA result, retained instruction set, or evaluation score.

    What changes during optimization?
    OptimizerMain targetNotebook mental model
    BootstrapFewShotDemonstrationsKeep the instruction fixed; collect and select metric-approved demonstrations.
    MIPROv2Instruction + demonstrationsSearch candidate instruction and demonstration combinations using evaluation.
    GEPAFeedback-driven prompt variantsReflect on successes and failures, mutate instructions, retain useful trade-offs.
    These optimizers change different parts of the prompt configuration. Establish a baseline first, then choose an approach that fits the task and evaluation budget.

    Optimizer comparison

    DimensionBootstrapFewShotMIPROv2GEPA
    Main targetDemosInstruction + demosFeedback-driven prompt evolution
    Rewrites instructionNoYesYes
    Explicit reflection feedbackNoNoYes
    Pareto-style diversityNot centralNot centralCentral
    Relative complexityLowMediumHigh
    Relative costLowerHigherPotentially high; budget dependent

    A useful study progression:

    Baseline
       ↓
    BootstrapFewShot
       ↓
    MIPROv2
       ↓
    GEPA

    The complexity and cost comparisons are qualitative intuition from the notebook. Actual call count, token count, elapsed time, and spend depend on the model, examples, search budget, and implementation. No measured cost ranking is provided. The progression is a study and troubleshooting guide, not a requirement to run every optimizer.

    Saving and reloading an optimized program

    Expensive optimization and normal inference should be separable.

    Typical pattern:

    optimized.save("optimized_program.json")
    
    loaded = MyProgram()
    loaded.load("optimized_program.json")
    if RUN_OPTIMIZERS and optimized_mipro is not None:
        optimized_mipro.save("optimized_qa.json")
        print("Saved: optimized_qa.json")
    
        loaded = SimpleQA()
        loaded.load("optimized_qa.json")
    
        print("Reloaded optimized program.")
    else:
        print("Save/load example skipped because MIPROv2 was not run.")

    Persistence separates one-time optimization from ordinary inference. The example creates a new SimpleQA instance with the same program structure before loading the saved state. It does not create a trained model or verify deployment compatibility. No optimized JSON artifact was produced by the disabled source example.

    DSPy RAG

    The retrieval example deliberately uses a five-document dictionary rather than a vector database or external service. This makes the handoff from retrieval to generation inspectable, but its search quality is limited by the matching code.

    A RAG pipeline can itself be represented as a DSPy Module.

    Question
       ↓
    Retriever
       ↓
    Relevant Passages
       ↓
    Context
       ↓
    DSPy Generator
       ↓
    Answer

    This matters because the optimizer can improve a program pipeline, not only a standalone one-line prompt.

    documents = {
        "doc_1": "Returns are accepted within 30 days of delivery if the item is unused.",
        "doc_2": "Refunds are processed to the original payment method after inspection.",
        "doc_3": "Damaged items should be reported within 48 hours of delivery.",
        "doc_4": "Standard shipping normally takes 3 to 5 business days.",
        "doc_5": "Support is available by email Monday through Friday.",
    }
    
    
    class SimpleRetriever:
        def __init__(self, corpus, k=3):
            self.corpus = corpus
            self.k = k
    
        def __call__(self, query):
            query_words = query.lower().split()
            scored = []
    
            for doc_id, text in self.corpus.items():
                score = sum(
                    1 for word in query_words
                    if word in text.lower()
                )
                scored.append((doc_id, score))
    
            scored.sort(
                key=lambda item: item[1],
                reverse=True,
            )
    
            return [
                self.corpus[doc_id]
                for doc_id, score in scored[: self.k]
                if score > 0
            ]
    
    
    retriever = SimpleRetriever(documents, k=3)
    
    print(retriever("What is the return policy?"))

    Output reproduced offline from this deterministic notebook code:

    ['Returns are accepted within 30 days of delivery if the item is unused.', 'Refunds are processed to the original payment method after inspection.', 'Support is available by email Monday through Friday.']

    The return-policy query selects extra passages because it scores common words as well as task-specific words. The return passage matches is, the, and return; the refund passage matches the; the support passage matches is. The latter two tie at one match and retain their dictionary insertion order. Matching is substring based in general, but these extra results come from actual common words, not from meaningful return-policy relevance.

    The retriever is useful for seeing every step of the pipeline. It is not evidence of high retrieval accuracy, semantic search, or comprehensive coverage.

    if dspy is not None:
        class GenerateAnswer(dspy.Signature):
            """Answer the question using the supplied context."""
    
            context: str = dspy.InputField()
            question: str = dspy.InputField()
            answer: str = dspy.OutputField()
    
    
        class BasicRAG(dspy.Module):
            def __init__(self, num_passages=3):
                super().__init__()
                self.num_passages = num_passages
                self.generate = dspy.ChainOfThought(GenerateAnswer)
    
            def forward(self, question):
                passages = retriever(question)[: self.num_passages]
                context = "\n".join(passages)
    
                return self.generate(
                    context=context,
                    question=question,
                )
    
    
        print("BasicRAG created.")
    if RUN_LLM and dspy is not None:
        rag = BasicRAG()
    
        result = rag(
            question="What is the return policy?"
        )
    
        print(result.answer)
    else:
        print("RAG LLM execution skipped.")

    BasicRAG retrieves passages, joins them with newlines, and supplies both context and question to the answer generator. num_passages truncates what the shared retriever returns; setting it above the retriever’s k=3 does not increase retrieval coverage. The signature asks the model to use context, but the example has no output citations or enforced grounding policy. No generated RAG answer is stored.

    Follow one question through BasicRAG
    1. QuestionWhat is the return policy?
    2. Retrieveretriever(question)
    3. Build contextJoin the selected passages with newlines.
    4. GenerateChainOfThought(GenerateAnswer) receives context + question.
    5. Return answerPrediction.answer
    BasicRAG retrieves passages, joins them into context, and passes both context and the original question to its generator. The returned answer still needs evaluation.

    RAG validation

    Two useful patterns from the source notebook:

    Hard check

    assert condition, "message"

    Soft check

    if too_long:
        print("Warning...")

    For production RAG, use stronger grounding/faithfulness metrics than simple lexical overlap.

    def basic_grounding_check(answer: str, context: str) -> bool:
        """Lightweight teaching check only."""
        answer_terms = [
            word.strip(".,!?;:").lower()
            for word in answer.split()
            if len(word.strip(".,!?;:")) > 4
        ]
    
        if not answer_terms:
            return False
    
        context_lower = context.lower()
        hits = sum(
            term in context_lower
            for term in answer_terms
        )
    
        return hits / len(answer_terms) >= 0.4
    
    
    context = "\n".join(
        retriever("What is the return policy?")
    )
    
    print(
        basic_grounding_check(
            "Returns are accepted within 30 days if the item is unused.",
            context,
        )
    )

    Output reproduced offline from this deterministic notebook code:

    True

    The grounding check counts answer words longer than four characters whose lowercase text appears somewhere in the context. It does not compare complete claims, interpret negation, or inspect the numerical conditions. The notebook’s correct return sentence passes, but replacing 30 days with 300 days also passes this same lexical check: the numeric terms are excluded by its length filter.

    A hard assertion can stop a program when a condition fails; a warning can leave execution running. Neither makes the condition itself a faithful measure. Decide what must fail closed, then use checks that actually cover those requirements.

    Compare dev scores, then inspect what the metric misses
    Before optimizationBaseline programEvaluate its predictions.
    After optimizationCompiled programEvaluate its predictions.

    Same devset + same metricHeld out from training; reused for MIPROv2 candidate selection.

    CorrectnessDoes the answer satisfy the question?
    GroundingDoes the supplied context support the answer?
    The notebook reuses its devset for MIPROv2 selection and evaluation, so these scores are not untouched test evidence. Its lexical-overlap checks are teaching examples; RAG also needs correctness and grounding checks.

    Multi-hop RAG

    Question
       ↓
    Generate Search Query
       ↓
    Retrieve
       ↓
    Accumulated Context
       ↓
    Generate Next Search Query
       ↓
    Retrieve Again
       ↓
    Final Answer

    Useful when one retrieval step is not enough.

    if dspy is not None:
        class GenerateSearchQuery(dspy.Signature):
            """Generate a search query from the question and current context."""
    
            context: str = dspy.InputField()
            question: str = dspy.InputField()
            search_query: str = dspy.OutputField()
    
    
        class MultiHopRAG(dspy.Module):
            def __init__(self, num_hops=2):
                super().__init__()
                self.num_hops = num_hops
                self.generate_query = dspy.ChainOfThought(
                    GenerateSearchQuery
                )
                self.generate_answer = dspy.ChainOfThought(
                    GenerateAnswer
                )
    
            def forward(self, question):
                collected = []
    
                for _ in range(self.num_hops):
                    query = self.generate_query(
                        context="\n".join(collected),
                        question=question,
                    ).search_query
    
                    collected.extend(
                        retriever(query)
                    )
    
                return self.generate_answer(
                    context="\n".join(collected),
                    question=question,
                )
    
    
        print("MultiHopRAG created.")

    Two hops create a search query from the question and collected context, retrieve, and append passages before final generation. This example defines the module but never invokes it. It also appends duplicate passages without deduplication, has no context-length bound, and does not test whether the next query adds information. More hops can add calls and irrelevant context as well as useful evidence.

    Optimizing RAG

    RAG Program
       +
    Training / Dev Examples
       +
    RAG Metric
       ↓
    MIPROv2
       ↓
    Optimized RAG Program

    A RAG metric should ideally consider:

    • answer correctness
    • completeness
    • grounding / faithfulness
    • optionally retrieval quality

    Optimizing the generator’s prompt cannot establish that a missing fact is present in retrieved evidence. Separate retrieval failures from answer-generation failures while reviewing examples. This section sketches a RAG optimization plan; it supplies no RAG training set, compile call, or measured optimized result.

    DSPy ReAct agents

    A ReAct agent exposes a different kind of execution strategy: it can choose a tool, observe the result, and decide what to do next. The example’s knowledge and weather tools return simulated text. They are not live data sources.

    ReAct repeatedly decides what to do, calls a tool, observes the result, and continues.

    Question
       ↓
    Decision / Thought
       ↓
    Action → Tool Call
       ↓
    Observation
       ↓
    Next Decision
       ↓
    ...
       ↓
    Final Answer

    Always use a bounded iteration count.

    import ast
    import operator as op
    
    _ALLOWED_OPS = {
        ast.Add: op.add,
        ast.Sub: op.sub,
        ast.Mult: op.mul,
        ast.Div: op.truediv,
        ast.Pow: op.pow,
        ast.USub: op.neg,
        ast.UAdd: op.pos,
    }
    
    
    def _eval_ast(node):
        if isinstance(node, ast.Expression):
            return _eval_ast(node.body)
    
        if isinstance(node, ast.Constant) and isinstance(node.value, (int, float)):
            return node.value
    
        if isinstance(node, ast.BinOp) and type(node.op) in _ALLOWED_OPS:
            return _ALLOWED_OPS[type(node.op)](
                _eval_ast(node.left),
                _eval_ast(node.right),
            )
    
        if isinstance(node, ast.UnaryOp) and type(node.op) in _ALLOWED_OPS:
            return _ALLOWED_OPS[type(node.op)](
                _eval_ast(node.operand)
            )
    
        raise ValueError("Unsupported expression")
    
    
    def calculator(expression: str) -> str:
        """Evaluate a basic arithmetic expression."""
        try:
            tree = ast.parse(expression, mode="eval")
            return str(_eval_ast(tree))
        except Exception as e:
            return f"Error: {e}"
    
    
    def search_knowledge(query: str) -> str:
        """Search a small simulated knowledge source."""
        if "india" in query.lower() and "gdp" in query.lower():
            return "Illustrative data: GDP = 3.9 trillion USD; population = 1.4 billion."
    
        return "No matching simulated result."
    
    
    def get_weather(city: str) -> str:
        """Return simulated weather."""
        return f"Simulated weather in {city}: 25°C, sunny"
    
    
    print("15 * 23 =", calculator("15 * 23"))

    Output reproduced offline from this deterministic notebook code:

    15 * 23 = 345

    The calculator accepts a restricted arithmetic AST instead of executing arbitrary Python text. Function calls, names, attribute access, and imports fall into the unsupported-expression branch. However, exponentiation remains allowed without magnitude, depth, or time bounds; bool also satisfies Python’s isinstance(value, (int, float)) check. This is a teaching tool, not a complete resource-limited sandbox.

    The knowledge tool’s GDP and population figures are explicitly illustrative. The weather tool always returns simulated weather. Neither output should be presented as current economic or weather evidence.

    agent = None
    
    if dspy is not None:
        try:
            agent = dspy.ReAct(
                "question -> answer",
                tools=[
                    search_knowledge,
                    calculator,
                    get_weather,
                ],
                max_iters=5,
            )
    
            print("ReAct agent created.")
        except Exception as e:
            print("ReAct API differs in this DSPy version:", e)
    if RUN_LLM and agent is not None:
        question = (
            "Using the supplied illustrative GDP and population data, "
            "estimate GDP per capita for India."
        )
    
        result = agent(question=question)
        print("Answer:", result.answer)
    
        try:
            dspy.inspect_history(n=10)
        except Exception as e:
            print("History inspection unavailable:", e)
    else:
        print("ReAct execution skipped.")

    Using the tool’s illustrative figures, the arithmetic target is 3.9 trillion / 1.4 billion ≈ 2,785.71 USD per person. That is a calculation over simulated inputs, not a factual estimate validated against a data source and not a recorded agent answer.

    max_iters=5 bounds the agent loop shown in the notebook; it does not bound every tool’s runtime or computational demand. Inspecting history can help understand execution, but the notebook records no tool trajectory or history output.

    Optimizing an agent

    The same optimization idea can be applied to a multi-step agent:

    Agent Program
       +
    Examples of desired behavior
       +
    Metric
       ↓
    BootstrapFewShot / other optimizer
       ↓
    Compiled Agent Program

    The important insight:

    The optimized object can be a program, not only one prompt string.

    An agent metric can evaluate answer quality and tool-use criteria. A correct final string alone cannot tell you whether the agent used the expected tools, relied on simulated data, or spent an excessive number of calls. The notebook shows the optimization concept rather than an actual compiled-agent experiment.

    Fine-tuning handoff

    The following section prepares the shape of training records and sketches a possible GPU workflow. It does not collect real model traces, train a model, evaluate a fine-tuned model, or export a trained artifact.

    The notebook treats fine-tuning as a later option after program/prompt optimization.

    Prompt / Program Optimization
    → no model-weight update
    → fast iteration
    → useful when the task changes often
    Fine-tuning
    → model weights change
    → more infrastructure
    → useful for narrow repeated tasks with enough data

    Combined path:

    DSPy Program
        ↓
    Optimize behavior
        ↓
    Generate / curate high-quality training traces
        ↓
    Fine-tune smaller model
        ↓
    Evaluate
        ↓
    Deploy

    Create fine-tuning records

    A simple record:

    {
      "instruction": "What is DSPy?",
      "input": "",
      "output": "..."
    }
    import json
    
    def make_alpaca_record(
        instruction: str,
        output: str,
        input_text: str = "",
    ):
        return {
            "instruction": instruction,
            "input": input_text,
            "output": output,
        }
    
    
    demo_training_data = [
        make_alpaca_record(
            "What is BootstrapFewShot?",
            "A DSPy optimizer that selects useful few-shot demonstrations while keeping the instruction fixed.",
        ),
        make_alpaca_record(
            "What is MIPROv2?",
            "A DSPy optimizer that searches over candidate instructions and demonstrations.",
        ),
    ]
    
    print(
        json.dumps(
            demo_training_data,
            indent=2,
        )
    )

    Output reproduced offline from this deterministic notebook code:

    [
      {
        "instruction": "What is BootstrapFewShot?",
        "input": "",
        "output": "A DSPy optimizer that selects useful few-shot demonstrations while keeping the instruction fixed."
      },
      {
        "instruction": "What is MIPROv2?",
        "input": "",
        "output": "A DSPy optimizer that searches over candidate instructions and demonstrations."
      }
    ]

    These are two manually written demonstration records, not outputs collected from an optimized model. Curating high-quality traces is a prerequisite in the conceptual handoff. Converting text into this JSON shape alone establishes neither training-data quality nor a fine-tuning improvement.

    Optional GPU template

    The notebook describes an Unsloth + LoRA + SFT style pipeline.

    Conceptual flow:

    Load Base Model
       ↓
    Attach LoRA Adapters
       ↓
    Prepare Dataset
       ↓
    SFT Training
       ↓
    Save
       ↓
    Evaluate
       ↓
    Optional Local Export

    This consolidated notebook intentionally does not launch a training job automatically.

    if RUN_GPU_SECTION:
        print("""
    GPU fine-tuning template enabled.
    
    Supplied stack:
    - Unsloth
    - LoRA / PEFT
    - TRL SFTTrainer
    - Transformers TrainingArguments
    
    Run only in a configured GPU environment.
    """)
    else:
        print("GPU section disabled.")

    Output reproduced offline from this deterministic notebook code:

    GPU section disabled.

    Even with RUN_GPU_SECTION=True, this code only prints a template description. It contains no model loading, adapter configuration, dataset preparation, trainer invocation, save call, evaluation, or export. The named stack describes an intended path; it is not a completed GPU training workflow.

    One complete DSPy mental model

                             DSPy
                              │
              ┌───────────────┼────────────────┐
              │               │                │
          SIGNATURE         MODULE         OPTIMIZER
              │               │                │
            WHAT?           HOW?            IMPROVE?
              │               │                │
       task / inputs       Predict       BootstrapFewShot
         / outputs            │                │
                              ├─ CoT            ├─ MIPROv2
                              │                 │
                              ├─ PoT            └─ GEPA
                              │
                              ├─ RAG module
                              │
                              └─ ReAct agent

    Program lifecycle:

    Signature
        ↓
    Module
        ↓
    Baseline DSPy Program
        ↓
    Examples + Metric
        ↓
    Optimizer
        ↓
    Compiled DSPy Program
        ↓
    Evaluate
        ↓
    Save / Deploy

    Optimizer decision guide

    Use BootstrapFewShot first when:

    • the instruction is already reasonable
    • you mainly need useful few-shot examples
    • you want a cheap/simple baseline optimizer

    Move to MIPROv2 when:

    • the instruction itself may need improvement
    • you want to search instruction + demo combinations
    • you can afford more trials

    Move to GEPA when:

    • simple search is not enough
    • failures contain useful natural-language feedback
    • you want reflection-driven rewriting
    • multiple prompt trade-offs matter
    • you can afford a more expensive optimization loop

    Choose the simplest configuration that addresses an observed failure, then measure it. A task that already performs adequately need not advance to a more expensive optimizer. When the metric is weak, improving the metric and examples is more important than expanding the search.

    Common mistakes

    No baseline

    You cannot prove improvement without a baseline.

    Weak metric

    DSPy optimizes what you measure.

    Bad training examples

    Generated examples should be checked before they drive optimization.

    Jumping straight to GEPA

    The notebook suggests starting with simpler configurations:

    Baseline → BootstrapFewShot → MIPROv2 → GEPA

    Using CoT everywhere

    More reasoning can mean more token cost and latency.

    Treating PoT like plain prompting

    PoT depends on generated code + an execution environment.

    Evaluating RAG only by answer match

    Grounding and faithfulness matter too.

    Unbounded agent loops

    Set a maximum iteration/tool budget.

    Three additional boundaries are visible in the actual code: typed descriptions are not range checks; optimizer validation data is not untouched test data; and a lexical grounding score can pass an incorrect numerical claim. Keep these distinctions visible in reports of improvement.

    One-page revision sheet

    DSPy
    ====
    
    CORE IDEA
    ---------
    Replace hand-maintained prompts with programmable LLM pipelines.
    
    
    3 CORE PARTS
    ------------
    Signature → WHAT
    Module    → HOW
    Optimizer → IMPROVE
    
    
    SIGNATURE
    ---------
    Task + Typed Input + Typed Output
    
    
    MODULES
    -------
    Predict
    → direct prediction
    
    ChainOfThought
    → reasoning-oriented execution
    
    ProgramOfThought
    → generate code
    → execute
    → answer
    
    ReAct
    → decision
    → tool action
    → observation
    → repeat
    
    
    RLM
    ---
    Huge Data > Context Window
    → keep data external
    → inspect/search/filter
    → bring forward useful information
    
    
    OPTIMIZATION
    ------------
    Examples + Metric + Baseline + Optimizer
    
    
    PROMPT MENTAL MODEL
    -------------------
    Prompt behavior ≈ Instruction + Demonstrations
    
    
    BOOTSTRAP FEWSHOT
    -----------------
    Instruction = fixed
    Demos       = optimized
    
    Question:
    Which examples should I include?
    
    
    MIPROv2
    -------
    Instruction + Demos
    
    Stage 1 → candidate demo sets
    Stage 2 → candidate instructions
    Stage 3 → combine I × D
    Stage 4 → efficient trial/search
    
    
    GEPA
    ----
    Reflection + Feedback + Prompt Mutation + Pareto
    
    Prompt
    → execute
    → compare
    → reflection
    → feedback
    → rewrite
    → repeat within budget
    
    Keep multiple useful prompt trade-offs when appropriate.
    
    
    RAG
    ---
    Retrieve
    → context
    → DSPy generator
    → answer
    → evaluate correctness + grounding
    
    
    FINE-TUNING HANDOFF
    -------------------
    Optimize DSPy behavior
    → curate outputs
    → training data
    → optional fine-tuning

    Practice exercises

    Exercise 1 — Signature

    Create:

    Input:
        product_review
    
    Outputs:
        sentiment
        confidence

    Exercise 2 — Modules

    Run the same Signature with:

    Predict
    ChainOfThought

    Compare quality, latency, and output verbosity.

    Exercise 3 — BootstrapFewShot

    Create a small classification trainset.

    • define a metric
    • choose max_bootstrapped_demos
    • compare baseline vs optimized

    Exercise 4 — MIPROv2

    Use the same task.

    Compare:

    Baseline
    BootstrapFewShot
    MIPROv2

    Exercise 5 — GEPA

    Use a reasoning task where failures are informative.

    Track:

    • original instruction
    • reflection feedback
    • rewritten instructions
    • retained variants

    Exercise 6 — RAG

    Replace the toy keyword retriever with your actual retriever.

    Evaluate:

    • correctness
    • completeness
    • grounding

    Exercise 7 — ReAct

    Build bounded tools for:

    search
    calculator
    database lookup

    Final takeaway

    DSPy becomes easiest when you separate program definition from program optimization:

    PROGRAM DEFINITION
    Signature + Module
    
    PROGRAM OPTIMIZATION
    Examples + Metric + Optimizer

    The progression is:

    Define behavior→Measure behavior→Optimize behavior\boxed{ \text{Define behavior} \rightarrow \text{Measure behavior} \rightarrow \text{Optimize behavior} }

    That is the central idea connecting the full DSPy material.

    Sources and execution boundary

    This note uses only DSPy_End_to_End_Detailed_Notes.ipynb as its knowledge source. It preserves the notebook’s substantive setup, signatures, modules, metrics, optimizer examples, persistence, RAG, ReAct tools, fine-tuning record preparation, revision guide, and exercises. Repeated summaries reinforce the lifecycle rather than supply independent evidence.

    The notebook has no saved outputs, version report, model answers, optimized prompts, benchmark measurements, or fine-tuning results. The reproduced outputs on this page cover only deterministic Python code. Arithmetic derivations and critiques of the example checks are labeled in the surrounding text. Provider-dependent calls and version-sensitive optimizer interfaces require their own execution and evaluation before their results can be reported.

    Sources

    No external sources are listed for this introductory note.