Add LLM-as-a-judge to eval suite - #2324
Conversation
Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
|
||
| DataFormat = Literal["jsonl", "parquet"] | ||
| FilterOperator = Literal["eq", "ne", "gt", "gte", "lt", "lte", "in", "not_in"] | ||
| _FILTER_OPERATORS = {"eq", "ne", "gt", "gte", "lt", "lte", "in", "not_in"} |
There was a problem hiding this comment.
I think we have some operator typing in audio somewhere... if yes, we should make it a generic util (if it is not already).
| if not isinstance(execution, dict): | ||
| raise ValueError("'execution' must be a mapping when provided.") | ||
| mode = execution.get("mode", "single_stage") | ||
| if mode not in {"single_stage", "multi_stage"}: |
There was a problem hiding this comment.
I am not sure if splitting up each judge task into its own NDD stage is worth it, or if it should all stay within the same NDD stage. Both are possible. Not sure it matters though.
| files_per_partition: int | None, | ||
| ) -> Pipeline: | ||
| """Build one streaming Curator reader → optional language gate → NDD stages → filters → writer pipeline.""" | ||
| # TODO: Add an optional TokenLengthFilter stage before NDD stages so prompts |
There was a problem hiding this comment.
Big TODO. Right now I just do string truncation in the jinja file. But we should decide the best course of action.
Also, since some extractors might return null while others return content, we should decide whether those examples are evaluated or not? Right now I force the empty string and still give it to the judge.
|
|
||
| Use this example when you have JSONL or Parquet records and want an LLM to evaluate fields in each record with your own prompt and rubric. You supply the input data, a YAML configuration, and one or more small Jinja prompt files. | ||
|
|
||
| The runner starts a local Curator Dynamo/vLLM `InferenceServer`, runs NeMo Data Designer (NDD) judge columns, and writes the original records plus judge output. The bundled example compares `raw_text`, `justext_text`, and `trafilatura_text`. Copy and adapt it to judge a PDF parser result against raw PDF text, assess whether two documents are semantic duplicates, or evaluate another text-processing result. |
There was a problem hiding this comment.
Very specific example to the data generated by cc_extract.py. But everything is supposed to be generic as long as the user provides the input path, YAML, and Jinja file(s).
| # Synthetic Data Generation (SDG) Dependencies | ||
| sdg_cpu = [ | ||
| "data-designer==0.5.5", | ||
| "data-designer==0.9.1", |
There was a problem hiding this comment.
Using this version for my tests. It is only like 2 commits behind NDD main right now.
| @@ -0,0 +1,94 @@ | |||
| # User-owned evaluation semantics: replace prompts, rubrics, models, and execution mode for another task. | |||
| models: | |||
| - alias: judge | |||
There was a problem hiding this comment.
It can handle as many models as we want to serve (assuming the GPU math works out). Each should have a unique alias.
| extract_reasoning_content: false | ||
| scores: | ||
| - name: best_extraction | ||
| description: Select the candidate that is most useful as clean document text. |
There was a problem hiding this comment.
I had Codex generate all the Jinja, scoring, etc. prompts. Just a POC for now.
Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
1 participant
Things to consider:
eval/llm_judgedirectory directly.