[Bug] dspy.settings.get("assert_failures") inconsistency with program #8544
Description
What happened?
The error is:
dspy/evaluate/evaluate.py", line 163, in process_item
program._assert_failures += dspy.settings.get("assert_failures")
TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType'
I do not understand the root reason, but I guess that settings.get("assert_failures") is None. Which are the default settings. But do not understand what a program is and why it required this setting to be set.
Same with suggest_failures
And in the same environment MIPROv2 works fine. Booted 72% to 79.5%. The only change is removing num_candidate_programs arg
Steps to reproduce
Not the full code, sadly. But there not much here. LLM-as-a-judge for summarization.
class MessagesModel(BaseModel):
participant: ParticipantEnum
text: str
class SummaryJudgeSignature(Signature):
"""Judge if the answer is factually correct based on the context and target summary."""
messages: list[MessagesModel] = InputField(
description="Last turn in conversation between client and operator.",
)
generated_summary: str | Literal["None"] = InputField(
description="Summary for a turn in conversation or 'None'",
)
golden_summary: str | Literal["None"] = InputField(
description="Target summary to compare with.",
)
is_similar: bool = OutputField()
class SummaryJudgeModule(Module):
def __init__(self) -> None:
super().__init__()
self.is_similar_module = ChainOfThought(SummaryJudgeSignature)
def forward(
self,
messages: list[MessagesModel],
generated_summary: str | Literal["None"],
golden_summary: str | Literal["None"],
):
return self.is_similar_module(
messages=messages,
generated_summary=generated_summary,
golden_summary=golden_summary,
)
def answer_exact_match(example, pred, trace=None) -> bool:
if isinstance(example.is_similar, bool):
return pred.is_similar == example.is_similar
return False
optimizer = BootstrapFewShotWithRandomSearch(
metric=answer_exact_match,
num_candidate_programs=n_few_shots,
max_bootstrapped_demos=n_few_shots,
max_errors=1,
)
compiled_pe = optimizer.compile(judge_module, trainset=examples)DSPy version
3.0.0b2