OpenEnv Environment Submission - Issue Tracking

[rycerzes]

COMPLETED ITEMS

Core Implementation

• Environment: QED Math - Mathematical proof generation and evaluation
• openenv.yaml - Full spec with grader_model, prompt_name, verifier settings, metrics
• models.py - Typed Pydantic models (QEDMathAction, QEDMathObservation, SubmitProof, ProblemObservation, ProofSubmissionObservation)
• server/qed_math_environment.py - Full MCPEnvironment implementation
• server/rubric.py - LLM-judge rubric (0-7 scale, normalized to [0,1])
• server/app.py - FastAPI app with /healthz endpoint
• server/Dockerfile - Multi-stage build with openenv-base

MCP Tools (3 tools)

• get_problem - Return current problem statement and metadata
• submit_proof - Submit proof for LLM-based rubric grading
• get_grading_guidelines - Return the rubric/marking scheme

Features

• LLM-Judge Rubric - Grades proofs on 0-7 scale with normalized rewards
• Answer-mode verification - Uses math_verify for fast \boxed{} checking
• Reward shaping - Discount factor, length penalty, optional score thresholding
• Flexible datasets - Local JSONL/JSON, HuggingFace Hub, or built-in

Inference

• examples/qed_math_inference.py - Baseline inference script with OpenAI client
• client.py - QEDMathEnv client for interaction
• Proper logging with START/STEP/END format

Documentation

• README.md - Setup instructions, features, usage examples

---

IN PROGRESS / NEEDS UPDATE

Environment Variables

• API_BASE_URL - The API endpoint for the LLM (defined in inference.py)
• MODEL_NAME - The model identifier (defined as MODEL in inference.py)
• HF_TOKEN - Your Hugging Face / API key

Required Updates

• Move examples/qed_math_inference.py to root as inference.py
• Update inference.py to use MODEL_NAME variable name (not MODEL)
• Add log_start(), log_step(), log_end() functions with [START], [STEP], [END] format
• Verify environment variables are properly documented

---

PENDING ITEMS

Pre-Submission Checklist

• HF Space deploys - Automated ping to Space URL - must return 200 and respond to reset()
• OpenEnv spec compliance - Validate openenv.yaml, typed models, step()/reset()/state() endpoints
• Dockerfile builds - Automated docker build on the submitted repo
• Baseline reproduces - Run the submitted inference script - must complete without error and produce scores
• 3+ tasks with graders - Enumerate tasks, run each grader, verify scores in 0.0–1.0 range

Scoring Breakdown

• Real-world utility - Does the environment model a genuine task?
• Task & grader quality - Are tasks well-defined with clear objectives?
• Environment design - Clean state management, sensible action/observation spaces?
• Code quality & spec compliance - Follows OpenEnv spec?
• Creativity & novelty - Novel problem domain?

Evaluation Criteria

• Phase 1: Automated Validation (pass/fail gate)
• Phase 2: Agentic Evaluation (baseline agent re-run, Nemotron 3 Super)
• Phase 3: Human Review

Infra Restrictions

• Runtime of inference script should be less than 20min
• Make sure env and inference can run on vcpu=2, memory=8gb

---

DETAILED REQUIREMENTS CHECKLIST

Parameter | Weight | Description

• Real-world utility - 30%

  • 0–5: Toy/artificial problem with no practical application
  • 6–15: Valid domain but shallow modeling of the real task
  • 16–25: Good domain modeling, would be useful for agent evaluation
  • 26–30: Excellent - fills a real gap, immediate value for the RL/agent community

• Task & grader quality - 25%

  • 3+ tasks with difficulty range? (proof, answer, multi_step problem types)
  • Graders produce scores between 0.0–1.0? (normalized score/7)
  • Graders deterministic and reproducible? (uses LLM judge with parse_schema)
  • Hard task genuinely challenges frontier models?

• Environment design - 20%

  • reset() produces clean state? (implemented in qed_math_environment.py)
  • Action/observation types well-designed and documented? (models.py)
  • Reward function provides useful varying signal (not just sparse)? (0-1 with partial progress)
  • Episode boundaries sensible? (done after proof submission)

• Code quality & spec compliance - 15%

  • openenv validate passes?
  • docker build works? (Dockerfile present)
  • HF Space deploys and responds?
  • Baseline script exists (needs update for spec compliance)

• Creativity & novelty - 10%

  • Domain we haven't seen in OpenEnv before? (mathematical proof evaluation)
  • Reward design has interesting properties? (LLM judge + math_verify)
  • Clever mechanics that make the environment engaging?

---

Disqualification Criteria

• Environment does not deploy or respond
• Plagiarized or trivially modified existing environments
• Graders that always return the same score
• No baseline inference script

---

Mandatory Additional Instructions

• Before submitting, ensure the following variables are defined in your environment configuration:

  • API_BASE_URL - The API endpoint for the LLM.
  • MODEL_NAME - The model identifier to use for inference.
  • HF_TOKEN - Your Hugging Face / API key.

• The inference script must be named inference.py and placed in the root directory of the project

• Participants must use OpenAI Client for all LLM calls using above variables

• Participants must emit structured stdout logs strictly following the [START], [STEP], and [END] format

---

Pre Validation Script

• Run the validate-submission.sh script before submitting

[rycerzes]

currently our env has been submitted successfully to the openenv hackathon