[Proposal] New Task: Turkish Clinical Source Support (clinical_safety group)

[goktugozkanmd]

Proposal: Turkish Clinical Source Support Evaluation Task

Summary

I propose adding a new evaluation task — turkish_clinical_source_support — under a new clinical_safety group in lm-evaluation-harness. This task evaluates whether LLMs can identify evidence-based clinical recommendations in Turkish and reject unsafe medical advice.

Motivation

• Language gap: There are no Turkish-language clinical safety evaluation tasks in lm-eval currently.
• Clinical AI safety: As LLMs are increasingly used for health information, evaluating their ability to distinguish safe, guideline-supported answers from dangerous advice is critical — especially in non-English languages.
• Medical expertise: Authored by a practicing internal medicine specialist (Dr. Goktug Ozkan, Turkey).

Task Details

• Group: clinical_safety
• Task name: turkish_clinical_source_support
• Format: Multiple choice (4 options, 1 correct)
• Language: Turkish
• Samples: 10 questions
• Output type: multiple_choice

Clinical Topics Covered

1. Diabetes insulin management during acute illness
2. Warfarin-TMP/SMX drug interaction monitoring
3. COPD exacerbation steroid dosing (REDUCE trial)
4. Contrast nephropathy prevention in AKI
5. Febrile neutropenia empiric antibiotics
6. Hyperkalemia — calcium gluconate indications
7. Metformin and lactic acidosis risk assessment
8. Perioperative DOAC bridging (apixaban)
9. Opioid-induced constipation prophylaxis
10. ACE inhibitor potassium monitoring frequency

Boundary Statement (included in README)

• No real patient data — synthetic clinical questions only
• No clinical validation — research benchmark, not a clinical tool
• No model ranking for clinical deployment
• Turkish language only
• Guidelines reflect evidence as of 2025

File Structure

lm_eval/tasks/clinical_safety/
├── _clinical_safety.yaml              # Group config
├── turkish_clinical_source_support.yaml  # Task config
├── turkish_clinical_source_support.jsonl # Dataset (10 questions)
├── utils.py                           # Processing/formatting utilities
└── README.md                          # Documentation + boundary statement

Checklist

• YAML-based task with multiple_choice output type
• Local JSONL dataset included
• Python utility functions for doc formatting
• Group config for future clinical_safety tasks
• Comprehensive README with boundary statement
• Follows new task guide conventions

I'd be happy to open a PR if this proposal is accepted. The complete implementation is ready in my fork.

Author: Dr. Goktug Ozkan (@v0id-lab)

[goktugozkanmd]

Quick follow-up on this proposal — the full implementation is ready in a fork and I'm happy to open a PR whenever it's convenient for the maintainers. If there's any feedback on the design or scope, happy to adjust.

cc @goktugozkanmd

[goktugozkanmd]

Opened an initial implementation PR here: #3918

It adds medfailbench as a zero-shot generate-until task using the public HF prompt dataset goktugozkanmd/medfailbench-v02-prompts. Validation passed locally with lm-eval validate --tasks medfailbench and a one-sample dummy run.