DSPy expresses a language-model task as a program: a signature defines the task and fields, a module determines how it runs, and an optimizer searches for better instructions or demonstrations using examples and a metric. The central workflow is to define behavior, measure a baseline, optimize, and evaluate the resulting program.
The benefit is a repeatable way to improve a pipeline, including retrieval or tool use. It does not remove prompts, establish factual correctness, or make a weak metric reliable. An optimizer can improve the measured score while the application still produces wrong answers.
This third Prompt Engineering note follows prompt foundations and structured output. Its sole knowledge source is the supplied DSPy_End_to_End_Detailed_Notes.ipynb. The source notebook contains no saved execution outputs. Outputs explicitly marked reproduced offline below come from its deterministic Python examples during page preparation; they do not use DSPy, a provider, or an LLM. LLM calls, optimizer scores, generated reasoning, and training outcomes remain unexecuted examples with no recorded result.
Read sequentially for the full lifecycle: setup → contracts → execution modules → data and metrics → baseline → optimizers → persistence → RAG → ReAct → fine-tuning handoff. Code blocks follow the notebook’s execution order and share variables; they are not independent scripts. Provider configuration and DSPy APIs are version dependent. The notebook prints an installed version when run, but it records no version or successful provider configuration.
Execution switches
RUN_LLM → normal LLM calls
RUN_OPTIMIZERS → Bootstrap / MIPRO / GEPA
RUN_GPU_SECTION → optional fine-tuning handoff
Keep all three False while revising.
RUN_LLM = False
RUN_OPTIMIZERS = False
RUN_GPU_SECTION = False
print("RUN_LLM =", RUN_LLM)
print("RUN_OPTIMIZERS =", RUN_OPTIMIZERS)
print("RUN_GPU_SECTION =", RUN_GPU_SECTION)
Output reproduced offline from this deterministic notebook code:
RUN_LLM = False
RUN_OPTIMIZERS = False
RUN_GPU_SECTION = False
With these defaults, the LLM, optimizer, and GPU branches stay disabled. Importing DSPy and defining modules can still occur when the package is installed. Turning on RUN_OPTIMIZERS alone is insufficient: the compile branches also require RUN_LLM and their optimizer imports.
What DSPy changes
A handwritten prompt often combines the task, output rules, examples, and execution strategy in one string. DSPy gives those responsibilities names. You still need to decide what counts as a good answer; the framework cannot infer that from the presence of a signature.
Traditional prompt engineering:
User Query
↓
Hand-written Prompt
↓
LLM
↓
Answer
DSPy:
User Input
↓
DSPy Program
├── Signature
├── Module
└── Optimizer
↓
LLM
↓
Output
Core mental model:
DSPy still uses instructions/prompts internally, but you work through program abstractions rather than manually managing every final prompt string.
The three core pieces
DSPy
│
┌────────────────┼────────────────┐
│ │ │
Signature Module Optimizer
│ │ │
WHAT? HOW? IMPROVE?
Signature
Defines the task contract.
- What is the task?
- What goes in?
- What comes out?
Module
Defines how the Signature executes.
Examples:
PredictChainOfThoughtProgramOfThoughtReAct
Optimizer
Improves the DSPy program from examples and metrics.
Examples:
BootstrapFewShotMIPROv2GEPA
Keep the three questions separate. A sentiment signature can stay unchanged while you compare Predict and ChainOfThought. Later, an optimizer can improve the selected module’s instructions or demonstrations without changing what the application asks the signature to return.
Setup
A disabled switch prevents the corresponding example from making calls; it does not configure credentials, install interpreters, or establish a compatible API. Run the examples in order and enable a section only after its prerequisites are ready.
The notebook removes project-specific helpers and keeps these examples self-contained.
Install DSPy only if needed:
pip install -U dspy
# Uncomment if DSPy is not installed.
# %pip install -U dspy
import os
try:
import dspy
print("DSPy imported.")
print("Version:", getattr(dspy, "__version__", "unknown"))
except Exception as e:
dspy = None
print("DSPy is not installed.")
print("Install with: pip install -U dspy")
print("Import error:", e)
# Provider-neutral template.
# Your installed DSPy/provider setup determines the exact model string
# and required environment variable(s).
MODEL_NAME = os.getenv("DSPY_MODEL", "openai/gpt-4o-mini")
lm = None
if RUN_LLM:
if dspy is None:
raise RuntimeError("Install DSPy first.")
lm = dspy.LM(MODEL_NAME)
dspy.configure(lm=lm)
print("Configured:", MODEL_NAME)
else:
print("LLM calls disabled. Set RUN_LLM=True after provider configuration.")
The model name is a configuration example from the notebook, not a verified deployment choice. DSPY_MODEL lets the environment supply another model string. Configure the credentials required by that provider outside the code; no credential value belongs in an article or notebook output.
The word contract describes what the program asks for. It should not be read as evidence that every answer is true, that every numeric constraint is enforced, or that every provider handles the contract identically.
Signatures — the typed contract
A Signature is the contract between your application and the language model.
It answers:
1. What is the task?
2. What is the input?
3. What is the output?
Conceptually:
Task Description
↓
Typed Input Field(s)
↓
Typed Output Field(s)
A Signature should stay focused on the actual task. It is not the place to build a huge persona-style prompt.
if dspy is not None:
class QuestionAnswering(dspy.Signature):
"""Answer questions with short factual responses."""
question: str = dspy.InputField()
answer: str = dspy.OutputField(
desc="A concise factual answer"
)
print("Signature:", QuestionAnswering.__name__)
print("Fields:", list(QuestionAnswering.fields.keys()))
else:
print("Install DSPy to create the live Signature.")
Class-based vs inline Signature
Class-based
class QuestionAnswering(dspy.Signature):
"""Answer questions with short factual responses."""
question: str = dspy.InputField()
answer: str = dspy.OutputField()
Best when you want:
- typed fields
- descriptions
- maintainability
- reuse
Inline
"question -> answer"
Useful for quick prototyping.
if RUN_LLM and dspy is not None:
inline_predictor = dspy.Predict("question -> answer")
result = inline_predictor(
question="What is the capital of Odisha?"
)
print(result.answer)
else:
print("Inline Signature execution skipped.")
Typed output example
Input:
text: str
Outputs:
sentiment: positive | negative | neutral
confidence: float
This makes the output contract explicit.
from typing import Literal
if dspy is not None:
class SentimentClassification(dspy.Signature):
"""Classify the sentiment of the supplied text."""
text: str = dspy.InputField()
sentiment: Literal["positive", "negative", "neutral"] = dspy.OutputField()
confidence: float = dspy.OutputField(
desc="Confidence score between 0 and 1"
)
print("Defined:", SentimentClassification.__name__)
Task: Answer questions with short factual responses.
question: strThe question supplied by the application.answer: strDescription: “A concise factual answer”.Literal makes the intended sentiment labels explicit. The confidence field is a float with a description, however: the example does not contain a range validator. A description saying “between 0 and 1” is not an enforced check, and a model’s self-reported confidence is not demonstrated to be calibrated. Validate constraints required by the application separately. No sentiment prediction is recorded in the notebook.
Modules — how the task executes
A Signature defines what.
A Module defines how.
Signature
↓
Module
↓
LLM
↓
Prediction
The main execution modules covered here are:
PredictChainOfThoughtProgramOfThoughtReActlater
dspy.Predict
Predict is the simplest/direct module.
Input
↓
Signature
↓
Predict
↓
LLM
↓
Output
Use it when you do not need a more specialized reasoning or tool-execution pattern.
if RUN_LLM and dspy is not None:
predictor = dspy.Predict(QuestionAnswering)
result = predictor(
question="What is the capital of Odisha?"
)
print("Answer:", result.answer)
else:
print("Predict execution skipped.")
dspy.ChainOfThought
The Signature can stay the same while the Module changes.
Same Signature
↓
ChainOfThought
↓
Reasoning-oriented execution
↓
Answer
Trade-off:
More reasoning
↓
More generated tokens
↓
More latency / cost
So CoT is useful for multi-step tasks, but should not automatically replace Predict for every simple task.
if RUN_LLM and dspy is not None:
cot = dspy.ChainOfThought(QuestionAnswering)
result = cot(
question="If a train travels 60 km/h for 2.5 hours, how far does it go?"
)
if hasattr(result, "reasoning"):
print("Reasoning:")
print(result.reasoning)
print("Answer:", result.answer)
else:
print("ChainOfThought execution skipped.")
For the train question, distance is speed multiplied by time: 60 × 2.5 = 150 km. This arithmetic explains the target calculation; it is not a captured model answer. Generated reasoning text, if returned, still needs checking against the problem.
dspy.ProgramOfThought
ProgramOfThought is useful when the model should generate code and use computation.
User Question
↓
ProgramOfThought
↓
LLM generates code
↓
Interpreter / sandbox executes
↓
Computed result
↓
Final Answer
Mental model:
LLM → chooses the computational procedure
Python → performs the calculation
Example:
Question:
"What is the average of 45, 67, 89, and 23?"
Conceptual generated code:
values = [45, 67, 89, 23]
average = sum(values) / len(values)
if RUN_LLM and dspy is not None:
pot = dspy.ProgramOfThought(QuestionAnswering)
result = pot(
question="What is the average of 45, 67, 89, and 23?"
)
print("Answer:", result.answer)
try:
dspy.inspect_history(n=5)
except Exception as e:
print("History inspection unavailable:", e)
else:
print("ProgramOfThought execution skipped.")
The average example has sum 45 + 67 + 89 + 23 = 224, so 224 / 4 = 56. That is a mathematical derivation, not a ProgramOfThought result. The notebook supplies no generated code, execution trace, or final model output. PoT also depends on an available execution environment; selecting the module does not establish that environment’s isolation or safety.
Quick comparison
| Module | Main idea | Best fit |
|---|---|---|
Predict | Direct prediction | Simple/factual task |
ChainOfThought | Reasoning-oriented execution | Multi-step reasoning |
ProgramOfThought | Generate + execute code | Arithmetic/programmatic task |
Revision shortcut:
Predict → ANSWER
CoT → REASON → ANSWER
PoT → WRITE CODE → EXECUTE → ANSWER
RLM intuition — when data is larger than the context window
The RLM material was introduced at a high level.
Problem:
Huge Data
>
LLM Context Window
Instead of placing everything into the prompt:
Huge Data
↓
Keep data external / program-accessible
↓
Inspect / search / filter / summarize
↓
Bring forward useful information
↓
Final Answer
Revision note:
RLM = Recursive Language Model
This is a conceptual data-access strategy, not an implemented RLM module in the notebook. Keeping data external only helps if the inspection/search process can bring forward enough relevant information. The page therefore makes no claim about a working recursive implementation or its performance.
DSPy compilation / optimization
Compilation here means preparing an improved DSPy program from data and a scoring rule. In these optimizer examples, it changes prompt configuration rather than training model weights. The later fine-tuning section is a separate handoff.
The key shift is:
"Can I write a better prompt manually?"
to:
"Can I define a metric and let DSPy improve the program?"
Common flow:
Signature + Module
↓
Training Examples
↓
Run Program
↓
Predictions
↓
Metric
↓
Optimizer
↓
Better demos / instructions
↓
Compiled Program
↓
Evaluate on held-out data
Instruction + Demonstrations
Two terms become critical.
Instruction
The task description.
Example:
Translate English to French.
Demonstration
A worked input-output example.
English: cat
French : chat
Useful identity:
dspy.Example
Optimization data uses examples such as:
dspy.Example(
question="What is RAG?",
answer="Retrieval-Augmented Generation"
).with_inputs("question")
.with_inputs(...) declares the input fields.
Other fields can act as target/reference outputs.
if dspy is not None:
raw_qa = [
("What is DSPy?", "A framework for programming and optimizing language-model pipelines."),
("What does RAG stand for?", "Retrieval-Augmented Generation"),
("What does CoT stand for?", "Chain of Thought"),
("What is few-shot prompting?", "Providing a small number of examples in the prompt"),
("What is a metric?", "A scoring function used to evaluate predictions"),
("What is a demonstration?", "A worked input-output example used by the program"),
("What does MIPROv2 optimize?", "Instructions and demonstrations"),
("What is GEPA centered on?", "Reflection, feedback, and Pareto-style prompt evolution"),
("What is ReAct?", "A reasoning and tool-action loop"),
("What is ProgramOfThought?", "A module that can generate code for computation"),
]
examples = [
dspy.Example(question=q, answer=a).with_inputs("question")
for q, a in raw_qa
]
trainset = examples[:7]
devset = examples[7:]
print("Trainset:", len(trainset))
print("Devset :", len(devset))
print("Sample :", trainset[0])
else:
trainset = []
devset = []
print("Install DSPy to create live dspy.Example objects.")
The split contains seven training examples and three development examples when DSPy is available. .with_inputs("question") prevents the reference answer from being treated as an ordinary task input. These ten teaching examples are too small to establish broad quality; the three development questions cover only GEPA, ReAct, and ProgramOfThought.
Metrics — the optimization target
An optimizer needs a scoring rule.
Prediction
↓
Metric
┌──┴───┐
Good Bad
↓ ↓
Keep Reject / lower score
Examples:
| Task | Possible metric |
|---|---|
| Classification | label accuracy |
| Extraction | field-level correctness |
| QA | exact / semantic correctness |
| RAG | correctness + grounding |
| Agent | answer quality + tool-use criteria |
Important:
def normalized_token_overlap(expected: str, predicted: str) -> float:
"""Simple teaching metric, not a universal production metric."""
expected_words = set(expected.lower().split())
predicted_words = set(predicted.lower().split())
if not expected_words:
return 0.0
return len(expected_words & predicted_words) / len(expected_words)
def answer_metric(example, prediction, trace=None):
overlap = normalized_token_overlap(
example.answer,
prediction.answer,
)
return 1.0 if overlap >= 0.5 else 0.0
print(
normalized_token_overlap(
"Retrieval-Augmented Generation",
"Retrieval Augmented Generation"
)
)
Output reproduced offline from this deterministic notebook code:
0.5
The metric computes overlap using sets of whitespace-separated, lowercased words:
Here, is the set of reference words, is the set of predicted words, is the reference-word overlap, and is the binary score. The code defines an empty reference set to have overlap zero. This is a teaching measure of reference coverage, not a complete semantic-correctness measure.
For the example, the reference splits into {"retrieval-augmented", "generation"} and the prediction into {"retrieval", "augmented", "generation"}. Only generation matches: 1 / 2 = 0.5, which passes the threshold. Punctuation, hyphens, extra claims, and negation are not understood by this function. A response can mention enough reference words and still be wrong.
Baseline program
The same evaluation rule and evaluation examples must be used when comparing the baseline and an optimized program. Otherwise a score difference can reflect a changed measurement rather than an improved program.
Always establish a baseline before optimization.
if dspy is not None:
class QA(dspy.Signature):
"""Answer questions with short factual responses."""
question: str = dspy.InputField()
answer: str = dspy.OutputField()
class SimpleQA(dspy.Module):
def __init__(self):
super().__init__()
self.generate = dspy.ChainOfThought(QA)
def forward(self, question):
return self.generate(question=question)
baseline_program = SimpleQA()
print("Baseline program created.")
baseline_score = None
evaluator = None
if RUN_LLM and dspy is not None and devset:
from dspy.evaluate import Evaluate
evaluator = Evaluate(
devset=devset,
metric=answer_metric,
num_threads=2,
display_progress=True,
)
baseline_result = evaluator(baseline_program)
baseline_score = baseline_result.score
print("Baseline score:", baseline_score)
else:
print("Baseline API evaluation skipped.")
The notebook stores no baseline score. If the branch runs in a compatible environment, Evaluate scores baseline_program on devset with answer_metric; num_threads=2 permits concurrent evaluation work. The evaluator’s displayed score format depends on the installed version. A printed score is still only evidence about this metric and these examples.
BootstrapFewShot
Start with a reasonable task description and ask whether better examples help. This is the smallest optimizer concept in the notebook: retain demonstrations that satisfy the metric, then use them with the existing instruction.
Core question:
Most important rule:
Instruction → FIXED
Demos → SELECTED / OPTIMIZED
BootstrapFewShot focuses on good examples rather than rewriting the main instruction.
Bootstrap workflow
Train Set
↓
Run Original DSPy Program
↓
Collect Predictions
↓
Apply Metric
↓
┌─────────────┬─────────────┐
│ Successful │ Rejected │
└──────┬──────┴─────────────┘
↓
Select Top-N successful examples
↓
Use them as demonstrations
↓
Original Instruction + Demos
↓
Compiled Program
The relevant control is conceptually:
max_bootstrapped_demos = N
You normally keep a small demonstration set rather than inserting every successful training sample.
if dspy is not None:
try:
from dspy.teleprompt import BootstrapFewShot
print("BootstrapFewShot available.")
except Exception as e:
BootstrapFewShot = None
print("BootstrapFewShot import path differs in this version:", e)
else:
BootstrapFewShot = None
optimized_bootstrap = None
bootstrap_score = None
if (
RUN_OPTIMIZERS
and RUN_LLM
and BootstrapFewShot is not None
and trainset
):
teleprompter = BootstrapFewShot(
metric=answer_metric,
max_bootstrapped_demos=3,
)
optimized_bootstrap = teleprompter.compile(
SimpleQA(),
trainset=trainset,
)
print("Bootstrap compilation complete.")
if evaluator is not None:
bootstrap_score = evaluator(optimized_bootstrap).score
print("Bootstrap score:", bootstrap_score)
else:
print("Bootstrap optimization skipped.")
max_bootstrapped_demos=3 bounds the requested bootstrapped demonstration set. The notebook’s “top-N” diagram is a conceptual selection sketch; it does not demonstrate that the implementation globally ranks all successful examples. Inspect the compiled program and compare evaluation results rather than assuming that every retained trace is desirable.
No compiled demonstration set, optimizer trace, or BootstrapFewShot score is saved in the source notebook.
Bootstrap revision summary
BootstrapFewShot
Input:
- original DSPy program
- trainset
- metric
Learns:
- useful demonstrations
Instruction:
- stays fixed
Question answered:
"Which examples should I show the model?"
MIPROv2
If demonstrations are not the only source of failure, instruction wording becomes another search variable. MIPROv2 explores combinations; the notebook’s small candidate and trial counts keep the example bounded, but do not establish an adequate search budget for a real task.
MIPROv2 asks a broader question:
Difference:
BootstrapFewShot
→ optimize demos
MIPROv2
→ optimize/search instruction + demos
Use:
Four-stage mental model
Stage 1
Generate candidate demo sets
↓
Stage 2
Generate candidate instructions
↓
Stage 3
Create instruction × demo combinations
↓
Stage 4
Efficient optimization / trial search
↓
Select a strong I + D combination
Stage 1 — candidate demo sets
C1 = {D1, D2, D3}
C2 = {D1, D4, D5}
C3 = {D2, D3, D6}
Stage 2 — candidate instructions
I1 = "Translate English to French."
I2 = "Return the correct French equivalent."
I3 = "Convert each English term into French."
Stage 3 — combinations
I1 + C1
I1 + C2
I2 + C1
I2 + C2
...
Stage 4 — search
When the combination space is large, trying every pair is expensive.
The notebook frames MIPROv2 as using Bayesian/trial-based optimization to explore the search space more efficiently.
With three candidate instructions and three candidate demonstration sets, the conceptual cross-product has 3 × 3 = 9 pairs. This is an illustration of the search space, not a count of actual trials from a recorded MIPRO run. The compile example below requests five trials.
if dspy is not None:
try:
from dspy.teleprompt import MIPROv2
print("MIPROv2 available.")
except Exception as e:
MIPROv2 = None
print("MIPROv2 import path differs in this version:", e)
else:
MIPROv2 = None
optimized_mipro = None
mipro_score = None
if (
RUN_OPTIMIZERS
and RUN_LLM
and MIPROv2 is not None
and trainset
):
mipro = MIPROv2(
metric=answer_metric,
auto=None,
num_candidates=3,
init_temperature=1.0,
)
optimized_mipro = mipro.compile(
SimpleQA(),
trainset=trainset,
valset=devset,
num_trials=5,
minibatch=False,
)
print("MIPROv2 compilation complete.")
if evaluator is not None:
mipro_score = evaluator(optimized_mipro).score
print("MIPROv2 score:", mipro_score)
else:
print("MIPROv2 optimization skipped.")
Evaluation boundary: this example passes devset to MIPROv2 as valset, then evaluates the compiled program on that same devset. That is useful for development, but the resulting score would be a validation score used during selection, not an untouched test result. Reserve a separate test set before making a generalization claim. No baseline-to-MIPRO improvement is recorded here.
BootstrapFewShot vs MIPROv2
| Dimension | BootstrapFewShot | MIPROv2 |
|---|---|---|
| Rewrites instruction | No | Yes |
| Selects demos | Yes | Yes |
| Main question | Which examples? | Which instruction + examples? |
| Complexity | Lower | Higher |
| Relative cost | Lower | Higher |
| Good first optimizer | Yes | Usually after baseline/bootstrap |
GEPA
GEPA’s additional ingredient is feedback about why a candidate failed. A score says how well a candidate performed; natural-language feedback can help propose a different instruction. The feedback itself still needs scrutiny.
GEPA adds a different style of optimization.
Instead of only searching candidate combinations:
Run
↓
Observe success / failure
↓
Reflect in natural language
↓
Generate feedback
↓
Rewrite / mutate prompt
↓
Run again
Two central ideas:
- Reflection-based prompt improvement
- Pareto-style retention of multiple useful prompt variants
Pareto intuition
Suppose prompts score differently on two abilities:
| Prompt | Math | Coding |
|---|---|---|
| A | 95 | 70 |
| B | 80 | 95 |
| C | 90 | 90 |
| D | 75 | 60 |
Interpretation:
A → excellent at Math
B → excellent at Coding
C → balanced
D → weaker on both
D can be discarded.
A, B, and C each preserve a useful trade-off.
The point is diversity rather than one universal best prompt.
prompt_scores = {
"A": (95, 70),
"B": (80, 95),
"C": (90, 90),
"D": (75, 60),
}
def dominates(x, y):
"""x dominates y when x is never worse and is better in at least one objective."""
return (
all(a >= b for a, b in zip(x, y))
and any(a > b for a, b in zip(x, y))
)
pareto = []
for name, score in prompt_scores.items():
is_dominated = any(
other_name != name and dominates(other_score, score)
for other_name, other_score in prompt_scores.items()
)
if not is_dominated:
pareto.append(name)
print("Pareto-style retained prompts:", pareto)
Output reproduced offline from this deterministic notebook code:
Pareto-style retained prompts: ['A', 'B', 'C']
For maximization objectives, one score vector dominates another when it is at least as high on every objective and strictly higher on at least one:
The components and index the objectives, here math and coding. A, B, and C each outperform D on both objectives. None of A, B, or C dominates both other retained candidates. The numbers are illustrative scores supplied by the notebook, not model benchmarks. This standalone Pareto calculation explains the idea; it is not an execution of GEPA’s internal selection process.
Reflection Language Model
Mental model:
Original LM
→ task executor / student
Reflection LM
→ reviewer / teacher
The reflection model can inspect:
Current Instruction
+
Question / Input
+
Original Model Answer
+
Ground Truth / Desired Answer
↓
Reflection LM
↓
Feedback
Then:
Feedback
↓
Generate Better Instruction
↓
Execute Again
↓
Evaluate Again
GEPA loop
Current Prompt
↓
Original LM executes
↓
Model Answer
↓
Compare with desired behavior
↓
Reflection LM
↓
Natural-language feedback
↓
Rewrite / mutate instruction
↓
New Prompt
↓
Run again
↓
Repeat within budget
Repeated prompt evolution is the prompt mutation idea.
Cost intuition:
More reflection loops
→ more LLM calls
→ more evaluation
→ more cost
if dspy is not None:
try:
from dspy.teleprompt import GEPA
print("GEPA available.")
except Exception as e:
GEPA = None
print("GEPA import path differs in this version:", e)
else:
GEPA = None
GEPA-compatible metric shape
The source notebook used a richer callback signature:
def metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
...
The extra values allow the optimizer/evaluation flow to carry additional trace context.
def math_metric(
gold,
pred,
trace=None,
pred_name=None,
pred_trace=None,
):
expected = str(gold.answer).strip()
actual = str(pred.answer).strip()
return 1.0 if expected in actual else 0.0
print("GEPA-compatible metric template created.")
Output reproduced offline from this deterministic notebook code:
GEPA-compatible metric template created.
The extra callback arguments preserve a place for trace context, but this implementation ignores them. It returns only a numeric score and does not construct explanatory feedback. Compatibility must be checked against the installed GEPA interface.
More seriously, expected in actual is substring matching: reference "30" would accept prediction "130". An empty expected string would match every prediction. The function is a metric-shape template, not a reliable numerical-answer validator.
gepa_optimized = None
if (
RUN_OPTIMIZERS
and RUN_LLM
and GEPA is not None
and dspy is not None
):
class MathProblem(dspy.Signature):
"""Solve a math word problem."""
problem: str = dspy.InputField()
answer: str = dspy.OutputField(
desc="Final answer"
)
class MathSolver(dspy.Module):
def __init__(self):
super().__init__()
self.solve = dspy.ChainOfThought(MathProblem)
def forward(self, problem):
return self.solve(problem=problem)
math_train = [
dspy.Example(problem="What is 12 + 18?", answer="30").with_inputs("problem"),
dspy.Example(problem="What is 9 * 7?", answer="63").with_inputs("problem"),
dspy.Example(problem="What is 100 / 4?", answer="25").with_inputs("problem"),
dspy.Example(problem="What is 15 + 5 * 2?", answer="25").with_inputs("problem"),
]
gepa = GEPA(
metric=math_metric,
auto="light",
reflection_lm=lm,
)
gepa_optimized = gepa.compile(
MathSolver(),
trainset=math_train,
)
print("GEPA optimization complete.")
else:
print("GEPA run skipped.")
print("It is disabled by default because it can be expensive.")
The reflection role and task-execution role are conceptually separate, but this example passes the same configured lm as reflection_lm. It does not require or demonstrate a different reviewer model. The four arithmetic training examples and auto="light" form a teaching configuration. There is no saved GEPA result, retained instruction set, or evaluation score.
| Optimizer | Main target | Notebook mental model |
|---|---|---|
| BootstrapFewShot | Demonstrations | Keep the instruction fixed; collect and select metric-approved demonstrations. |
| MIPROv2 | Instruction + demonstrations | Search candidate instruction and demonstration combinations using evaluation. |
| GEPA | Feedback-driven prompt variants | Reflect on successes and failures, mutate instructions, retain useful trade-offs. |
Optimizer comparison
| Dimension | BootstrapFewShot | MIPROv2 | GEPA |
|---|---|---|---|
| Main target | Demos | Instruction + demos | Feedback-driven prompt evolution |
| Rewrites instruction | No | Yes | Yes |
| Explicit reflection feedback | No | No | Yes |
| Pareto-style diversity | Not central | Not central | Central |
| Relative complexity | Low | Medium | High |
| Relative cost | Lower | Higher | Potentially high; budget dependent |
A useful study progression:
Baseline
↓
BootstrapFewShot
↓
MIPROv2
↓
GEPA
The complexity and cost comparisons are qualitative intuition from the notebook. Actual call count, token count, elapsed time, and spend depend on the model, examples, search budget, and implementation. No measured cost ranking is provided. The progression is a study and troubleshooting guide, not a requirement to run every optimizer.
Saving and reloading an optimized program
Expensive optimization and normal inference should be separable.
Typical pattern:
optimized.save("optimized_program.json")
loaded = MyProgram()
loaded.load("optimized_program.json")
if RUN_OPTIMIZERS and optimized_mipro is not None:
optimized_mipro.save("optimized_qa.json")
print("Saved: optimized_qa.json")
loaded = SimpleQA()
loaded.load("optimized_qa.json")
print("Reloaded optimized program.")
else:
print("Save/load example skipped because MIPROv2 was not run.")
Persistence separates one-time optimization from ordinary inference. The example creates a new SimpleQA instance with the same program structure before loading the saved state. It does not create a trained model or verify deployment compatibility. No optimized JSON artifact was produced by the disabled source example.
DSPy RAG
The retrieval example deliberately uses a five-document dictionary rather than a vector database or external service. This makes the handoff from retrieval to generation inspectable, but its search quality is limited by the matching code.
A RAG pipeline can itself be represented as a DSPy Module.
Question
↓
Retriever
↓
Relevant Passages
↓
Context
↓
DSPy Generator
↓
Answer
This matters because the optimizer can improve a program pipeline, not only a standalone one-line prompt.
documents = {
"doc_1": "Returns are accepted within 30 days of delivery if the item is unused.",
"doc_2": "Refunds are processed to the original payment method after inspection.",
"doc_3": "Damaged items should be reported within 48 hours of delivery.",
"doc_4": "Standard shipping normally takes 3 to 5 business days.",
"doc_5": "Support is available by email Monday through Friday.",
}
class SimpleRetriever:
def __init__(self, corpus, k=3):
self.corpus = corpus
self.k = k
def __call__(self, query):
query_words = query.lower().split()
scored = []
for doc_id, text in self.corpus.items():
score = sum(
1 for word in query_words
if word in text.lower()
)
scored.append((doc_id, score))
scored.sort(
key=lambda item: item[1],
reverse=True,
)
return [
self.corpus[doc_id]
for doc_id, score in scored[: self.k]
if score > 0
]
retriever = SimpleRetriever(documents, k=3)
print(retriever("What is the return policy?"))
Output reproduced offline from this deterministic notebook code:
['Returns are accepted within 30 days of delivery if the item is unused.', 'Refunds are processed to the original payment method after inspection.', 'Support is available by email Monday through Friday.']
The return-policy query selects extra passages because it scores common words as well as task-specific words. The return passage matches is, the, and return; the refund passage matches the; the support passage matches is. The latter two tie at one match and retain their dictionary insertion order. Matching is substring based in general, but these extra results come from actual common words, not from meaningful return-policy relevance.
The retriever is useful for seeing every step of the pipeline. It is not evidence of high retrieval accuracy, semantic search, or comprehensive coverage.
if dspy is not None:
class GenerateAnswer(dspy.Signature):
"""Answer the question using the supplied context."""
context: str = dspy.InputField()
question: str = dspy.InputField()
answer: str = dspy.OutputField()
class BasicRAG(dspy.Module):
def __init__(self, num_passages=3):
super().__init__()
self.num_passages = num_passages
self.generate = dspy.ChainOfThought(GenerateAnswer)
def forward(self, question):
passages = retriever(question)[: self.num_passages]
context = "\n".join(passages)
return self.generate(
context=context,
question=question,
)
print("BasicRAG created.")
if RUN_LLM and dspy is not None:
rag = BasicRAG()
result = rag(
question="What is the return policy?"
)
print(result.answer)
else:
print("RAG LLM execution skipped.")
BasicRAG retrieves passages, joins them with newlines, and supplies both context and question to the answer generator. num_passages truncates what the shared retriever returns; setting it above the retriever’s k=3 does not increase retrieval coverage. The signature asks the model to use context, but the example has no output citations or enforced grounding policy. No generated RAG answer is stored.
- QuestionWhat is the return policy?
- Retrieveretriever(question)
- Build contextJoin the selected passages with newlines.
- GenerateChainOfThought(GenerateAnswer) receives context + question.
- Return answerPrediction.answer
RAG validation
Two useful patterns from the source notebook:
Hard check
assert condition, "message"
Soft check
if too_long:
print("Warning...")
For production RAG, use stronger grounding/faithfulness metrics than simple lexical overlap.
def basic_grounding_check(answer: str, context: str) -> bool:
"""Lightweight teaching check only."""
answer_terms = [
word.strip(".,!?;:").lower()
for word in answer.split()
if len(word.strip(".,!?;:")) > 4
]
if not answer_terms:
return False
context_lower = context.lower()
hits = sum(
term in context_lower
for term in answer_terms
)
return hits / len(answer_terms) >= 0.4
context = "\n".join(
retriever("What is the return policy?")
)
print(
basic_grounding_check(
"Returns are accepted within 30 days if the item is unused.",
context,
)
)
Output reproduced offline from this deterministic notebook code:
True
The grounding check counts answer words longer than four characters whose lowercase text appears somewhere in the context. It does not compare complete claims, interpret negation, or inspect the numerical conditions. The notebook’s correct return sentence passes, but replacing 30 days with 300 days also passes this same lexical check: the numeric terms are excluded by its length filter.
A hard assertion can stop a program when a condition fails; a warning can leave execution running. Neither makes the condition itself a faithful measure. Decide what must fail closed, then use checks that actually cover those requirements.
Same devset + same metricHeld out from training; reused for MIPROv2 candidate selection.
Multi-hop RAG
Question
↓
Generate Search Query
↓
Retrieve
↓
Accumulated Context
↓
Generate Next Search Query
↓
Retrieve Again
↓
Final Answer
Useful when one retrieval step is not enough.
if dspy is not None:
class GenerateSearchQuery(dspy.Signature):
"""Generate a search query from the question and current context."""
context: str = dspy.InputField()
question: str = dspy.InputField()
search_query: str = dspy.OutputField()
class MultiHopRAG(dspy.Module):
def __init__(self, num_hops=2):
super().__init__()
self.num_hops = num_hops
self.generate_query = dspy.ChainOfThought(
GenerateSearchQuery
)
self.generate_answer = dspy.ChainOfThought(
GenerateAnswer
)
def forward(self, question):
collected = []
for _ in range(self.num_hops):
query = self.generate_query(
context="\n".join(collected),
question=question,
).search_query
collected.extend(
retriever(query)
)
return self.generate_answer(
context="\n".join(collected),
question=question,
)
print("MultiHopRAG created.")
Two hops create a search query from the question and collected context, retrieve, and append passages before final generation. This example defines the module but never invokes it. It also appends duplicate passages without deduplication, has no context-length bound, and does not test whether the next query adds information. More hops can add calls and irrelevant context as well as useful evidence.
Optimizing RAG
RAG Program
+
Training / Dev Examples
+
RAG Metric
↓
MIPROv2
↓
Optimized RAG Program
A RAG metric should ideally consider:
- answer correctness
- completeness
- grounding / faithfulness
- optionally retrieval quality
Optimizing the generator’s prompt cannot establish that a missing fact is present in retrieved evidence. Separate retrieval failures from answer-generation failures while reviewing examples. This section sketches a RAG optimization plan; it supplies no RAG training set, compile call, or measured optimized result.
DSPy ReAct agents
A ReAct agent exposes a different kind of execution strategy: it can choose a tool, observe the result, and decide what to do next. The example’s knowledge and weather tools return simulated text. They are not live data sources.
ReAct repeatedly decides what to do, calls a tool, observes the result, and continues.
Question
↓
Decision / Thought
↓
Action → Tool Call
↓
Observation
↓
Next Decision
↓
...
↓
Final Answer
Always use a bounded iteration count.
import ast
import operator as op
_ALLOWED_OPS = {
ast.Add: op.add,
ast.Sub: op.sub,
ast.Mult: op.mul,
ast.Div: op.truediv,
ast.Pow: op.pow,
ast.USub: op.neg,
ast.UAdd: op.pos,
}
def _eval_ast(node):
if isinstance(node, ast.Expression):
return _eval_ast(node.body)
if isinstance(node, ast.Constant) and isinstance(node.value, (int, float)):
return node.value
if isinstance(node, ast.BinOp) and type(node.op) in _ALLOWED_OPS:
return _ALLOWED_OPS[type(node.op)](
_eval_ast(node.left),
_eval_ast(node.right),
)
if isinstance(node, ast.UnaryOp) and type(node.op) in _ALLOWED_OPS:
return _ALLOWED_OPS[type(node.op)](
_eval_ast(node.operand)
)
raise ValueError("Unsupported expression")
def calculator(expression: str) -> str:
"""Evaluate a basic arithmetic expression."""
try:
tree = ast.parse(expression, mode="eval")
return str(_eval_ast(tree))
except Exception as e:
return f"Error: {e}"
def search_knowledge(query: str) -> str:
"""Search a small simulated knowledge source."""
if "india" in query.lower() and "gdp" in query.lower():
return "Illustrative data: GDP = 3.9 trillion USD; population = 1.4 billion."
return "No matching simulated result."
def get_weather(city: str) -> str:
"""Return simulated weather."""
return f"Simulated weather in {city}: 25°C, sunny"
print("15 * 23 =", calculator("15 * 23"))
Output reproduced offline from this deterministic notebook code:
15 * 23 = 345
The calculator accepts a restricted arithmetic AST instead of executing arbitrary Python text. Function calls, names, attribute access, and imports fall into the unsupported-expression branch. However, exponentiation remains allowed without magnitude, depth, or time bounds; bool also satisfies Python’s isinstance(value, (int, float)) check. This is a teaching tool, not a complete resource-limited sandbox.
The knowledge tool’s GDP and population figures are explicitly illustrative. The weather tool always returns simulated weather. Neither output should be presented as current economic or weather evidence.
agent = None
if dspy is not None:
try:
agent = dspy.ReAct(
"question -> answer",
tools=[
search_knowledge,
calculator,
get_weather,
],
max_iters=5,
)
print("ReAct agent created.")
except Exception as e:
print("ReAct API differs in this DSPy version:", e)
if RUN_LLM and agent is not None:
question = (
"Using the supplied illustrative GDP and population data, "
"estimate GDP per capita for India."
)
result = agent(question=question)
print("Answer:", result.answer)
try:
dspy.inspect_history(n=10)
except Exception as e:
print("History inspection unavailable:", e)
else:
print("ReAct execution skipped.")
Using the tool’s illustrative figures, the arithmetic target is 3.9 trillion / 1.4 billion ≈ 2,785.71 USD per person. That is a calculation over simulated inputs, not a factual estimate validated against a data source and not a recorded agent answer.
max_iters=5 bounds the agent loop shown in the notebook; it does not bound every tool’s runtime or computational demand. Inspecting history can help understand execution, but the notebook records no tool trajectory or history output.
Optimizing an agent
The same optimization idea can be applied to a multi-step agent:
Agent Program
+
Examples of desired behavior
+
Metric
↓
BootstrapFewShot / other optimizer
↓
Compiled Agent Program
The important insight:
The optimized object can be a program, not only one prompt string.
An agent metric can evaluate answer quality and tool-use criteria. A correct final string alone cannot tell you whether the agent used the expected tools, relied on simulated data, or spent an excessive number of calls. The notebook shows the optimization concept rather than an actual compiled-agent experiment.
Fine-tuning handoff
The following section prepares the shape of training records and sketches a possible GPU workflow. It does not collect real model traces, train a model, evaluate a fine-tuned model, or export a trained artifact.
The notebook treats fine-tuning as a later option after program/prompt optimization.
Prompt / Program Optimization
→ no model-weight update
→ fast iteration
→ useful when the task changes often
Fine-tuning
→ model weights change
→ more infrastructure
→ useful for narrow repeated tasks with enough data
Combined path:
DSPy Program
↓
Optimize behavior
↓
Generate / curate high-quality training traces
↓
Fine-tune smaller model
↓
Evaluate
↓
Deploy
Create fine-tuning records
A simple record:
{
"instruction": "What is DSPy?",
"input": "",
"output": "..."
}
import json
def make_alpaca_record(
instruction: str,
output: str,
input_text: str = "",
):
return {
"instruction": instruction,
"input": input_text,
"output": output,
}
demo_training_data = [
make_alpaca_record(
"What is BootstrapFewShot?",
"A DSPy optimizer that selects useful few-shot demonstrations while keeping the instruction fixed.",
),
make_alpaca_record(
"What is MIPROv2?",
"A DSPy optimizer that searches over candidate instructions and demonstrations.",
),
]
print(
json.dumps(
demo_training_data,
indent=2,
)
)
Output reproduced offline from this deterministic notebook code:
[
{
"instruction": "What is BootstrapFewShot?",
"input": "",
"output": "A DSPy optimizer that selects useful few-shot demonstrations while keeping the instruction fixed."
},
{
"instruction": "What is MIPROv2?",
"input": "",
"output": "A DSPy optimizer that searches over candidate instructions and demonstrations."
}
]
These are two manually written demonstration records, not outputs collected from an optimized model. Curating high-quality traces is a prerequisite in the conceptual handoff. Converting text into this JSON shape alone establishes neither training-data quality nor a fine-tuning improvement.
Optional GPU template
The notebook describes an Unsloth + LoRA + SFT style pipeline.
Conceptual flow:
Load Base Model
↓
Attach LoRA Adapters
↓
Prepare Dataset
↓
SFT Training
↓
Save
↓
Evaluate
↓
Optional Local Export
This consolidated notebook intentionally does not launch a training job automatically.
if RUN_GPU_SECTION:
print("""
GPU fine-tuning template enabled.
Supplied stack:
- Unsloth
- LoRA / PEFT
- TRL SFTTrainer
- Transformers TrainingArguments
Run only in a configured GPU environment.
""")
else:
print("GPU section disabled.")
Output reproduced offline from this deterministic notebook code:
GPU section disabled.
Even with RUN_GPU_SECTION=True, this code only prints a template description. It contains no model loading, adapter configuration, dataset preparation, trainer invocation, save call, evaluation, or export. The named stack describes an intended path; it is not a completed GPU training workflow.
One complete DSPy mental model
DSPy
│
┌───────────────┼────────────────┐
│ │ │
SIGNATURE MODULE OPTIMIZER
│ │ │
WHAT? HOW? IMPROVE?
│ │ │
task / inputs Predict BootstrapFewShot
/ outputs │ │
├─ CoT ├─ MIPROv2
│ │
├─ PoT └─ GEPA
│
├─ RAG module
│
└─ ReAct agent
Program lifecycle:
Signature
↓
Module
↓
Baseline DSPy Program
↓
Examples + Metric
↓
Optimizer
↓
Compiled DSPy Program
↓
Evaluate
↓
Save / Deploy
Optimizer decision guide
Use BootstrapFewShot first when:
- the instruction is already reasonable
- you mainly need useful few-shot examples
- you want a cheap/simple baseline optimizer
Move to MIPROv2 when:
- the instruction itself may need improvement
- you want to search instruction + demo combinations
- you can afford more trials
Move to GEPA when:
- simple search is not enough
- failures contain useful natural-language feedback
- you want reflection-driven rewriting
- multiple prompt trade-offs matter
- you can afford a more expensive optimization loop
Choose the simplest configuration that addresses an observed failure, then measure it. A task that already performs adequately need not advance to a more expensive optimizer. When the metric is weak, improving the metric and examples is more important than expanding the search.
Common mistakes
No baseline
You cannot prove improvement without a baseline.
Weak metric
DSPy optimizes what you measure.
Bad training examples
Generated examples should be checked before they drive optimization.
Jumping straight to GEPA
The notebook suggests starting with simpler configurations:
Baseline → BootstrapFewShot → MIPROv2 → GEPA
Using CoT everywhere
More reasoning can mean more token cost and latency.
Treating PoT like plain prompting
PoT depends on generated code + an execution environment.
Evaluating RAG only by answer match
Grounding and faithfulness matter too.
Unbounded agent loops
Set a maximum iteration/tool budget.
Three additional boundaries are visible in the actual code: typed descriptions are not range checks; optimizer validation data is not untouched test data; and a lexical grounding score can pass an incorrect numerical claim. Keep these distinctions visible in reports of improvement.
One-page revision sheet
DSPy
====
CORE IDEA
---------
Replace hand-maintained prompts with programmable LLM pipelines.
3 CORE PARTS
------------
Signature → WHAT
Module → HOW
Optimizer → IMPROVE
SIGNATURE
---------
Task + Typed Input + Typed Output
MODULES
-------
Predict
→ direct prediction
ChainOfThought
→ reasoning-oriented execution
ProgramOfThought
→ generate code
→ execute
→ answer
ReAct
→ decision
→ tool action
→ observation
→ repeat
RLM
---
Huge Data > Context Window
→ keep data external
→ inspect/search/filter
→ bring forward useful information
OPTIMIZATION
------------
Examples + Metric + Baseline + Optimizer
PROMPT MENTAL MODEL
-------------------
Prompt behavior ≈ Instruction + Demonstrations
BOOTSTRAP FEWSHOT
-----------------
Instruction = fixed
Demos = optimized
Question:
Which examples should I include?
MIPROv2
-------
Instruction + Demos
Stage 1 → candidate demo sets
Stage 2 → candidate instructions
Stage 3 → combine I × D
Stage 4 → efficient trial/search
GEPA
----
Reflection + Feedback + Prompt Mutation + Pareto
Prompt
→ execute
→ compare
→ reflection
→ feedback
→ rewrite
→ repeat within budget
Keep multiple useful prompt trade-offs when appropriate.
RAG
---
Retrieve
→ context
→ DSPy generator
→ answer
→ evaluate correctness + grounding
FINE-TUNING HANDOFF
-------------------
Optimize DSPy behavior
→ curate outputs
→ training data
→ optional fine-tuning
Practice exercises
Exercise 1 — Signature
Create:
Input:
product_review
Outputs:
sentiment
confidence
Exercise 2 — Modules
Run the same Signature with:
Predict
ChainOfThought
Compare quality, latency, and output verbosity.
Exercise 3 — BootstrapFewShot
Create a small classification trainset.
- define a metric
- choose
max_bootstrapped_demos - compare baseline vs optimized
Exercise 4 — MIPROv2
Use the same task.
Compare:
Baseline
BootstrapFewShot
MIPROv2
Exercise 5 — GEPA
Use a reasoning task where failures are informative.
Track:
- original instruction
- reflection feedback
- rewritten instructions
- retained variants
Exercise 6 — RAG
Replace the toy keyword retriever with your actual retriever.
Evaluate:
- correctness
- completeness
- grounding
Exercise 7 — ReAct
Build bounded tools for:
search
calculator
database lookup
Final takeaway
DSPy becomes easiest when you separate program definition from program optimization:
PROGRAM DEFINITION
Signature + Module
PROGRAM OPTIMIZATION
Examples + Metric + Optimizer
The progression is:
That is the central idea connecting the full DSPy material.
Sources and execution boundary
This note uses only DSPy_End_to_End_Detailed_Notes.ipynb as its knowledge source. It preserves the notebook’s substantive setup, signatures, modules, metrics, optimizer examples, persistence, RAG, ReAct tools, fine-tuning record preparation, revision guide, and exercises. Repeated summaries reinforce the lifecycle rather than supply independent evidence.
The notebook has no saved outputs, version report, model answers, optimized prompts, benchmark measurements, or fine-tuning results. The reproduced outputs on this page cover only deterministic Python code. Arithmetic derivations and critiques of the example checks are labeled in the surrounding text. Provider-dependent calls and version-sensitive optimizer interfaces require their own execution and evaluation before their results can be reported.