Skip to content

[Bug] dspy.settings.get("assert_failures") inconsistency with program #8544

@NikitaSemenovAiforia

Description

@NikitaSemenovAiforia

What happened?

The error is:

dspy/evaluate/evaluate.py", line 163, in process_item
program._assert_failures += dspy.settings.get("assert_failures")
TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType'

I do not understand the root reason, but I guess that settings.get("assert_failures") is None. Which are the default settings. But do not understand what a program is and why it required this setting to be set.

Same with suggest_failures

And in the same environment MIPROv2 works fine. Booted 72% to 79.5%. The only change is removing num_candidate_programs arg

Steps to reproduce

Not the full code, sadly. But there not much here. LLM-as-a-judge for summarization.

class MessagesModel(BaseModel):
    participant: ParticipantEnum
    text: str


class SummaryJudgeSignature(Signature):
    """Judge if the answer is factually correct based on the context and target summary."""

    messages: list[MessagesModel] = InputField(
        description="Last turn in conversation between client and operator.",
    )
    generated_summary: str | Literal["None"] = InputField(
        description="Summary for a turn in conversation or 'None'",
    )
    golden_summary: str | Literal["None"] = InputField(
        description="Target summary to compare with.",
    )
    is_similar: bool = OutputField()


class SummaryJudgeModule(Module):
    def __init__(self) -> None:
        super().__init__()
        self.is_similar_module = ChainOfThought(SummaryJudgeSignature)

    def forward(
        self,
        messages: list[MessagesModel],
        generated_summary: str | Literal["None"],
        golden_summary: str | Literal["None"],
    ):
        return self.is_similar_module(
            messages=messages,
            generated_summary=generated_summary,
            golden_summary=golden_summary,
        )

def answer_exact_match(example, pred, trace=None) -> bool:
    if isinstance(example.is_similar, bool):
        return pred.is_similar == example.is_similar
    return False

optimizer = BootstrapFewShotWithRandomSearch(
            metric=answer_exact_match,
            num_candidate_programs=n_few_shots,
            max_bootstrapped_demos=n_few_shots,
            max_errors=1,
        )

compiled_pe = optimizer.compile(judge_module, trainset=examples)

DSPy version

3.0.0b2

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions