improve_plan_gemini.md

Model Improvement Plan (Gemini)

1. Analysis Plan (分析計畫)

To understand why the current score (0.1090) is low compared to top teams (0.37+), we need to analyze specific failure points.

1.1 Baseline Validation

• Goal: Establish a local baseline using train.csv (train_solution.csv).
• Action: Run python main.py --mode train with current config.py settings.
• Metrics: Precision, Recall, F1, IoU distribution.

1.2 Component Analysis

We will analyze the pipeline in three stages:

1. Retrieval Phase (Recall Analysis)

   • Hypothesis: Relevant segments are not being retrieved in the Top-K.
   • Experiment: Check if Ground Truth (GT) segments are present in the top_k_candidates before LLM filtering.
   • Metric: Recall @ K (before LLM).

2. LLM Filtering Phase (Precision/Recall Trade-off)

   • Hypothesis: LLM confidence_threshold (7) is too high, rejecting correct matches (False Negatives), or prompt is not specific enough.
   • Experiment:
     • Compare "LLM Score < 7" items against GT to see how many valid matches were rejected.
     • Compare "LLM Score >= 7" items against GT to see False Positives.

3. Bounding Box Merging (IoU Analysis)

   • Hypothesis: BBoxes are too fragmented or too large, leading to low IoU even if the text match is correct.
   • Experiment: Analyze IoU distribution for "Code Correct" predictions. If many are < 0.5, the merge logic needs tuning.

2. Improvement Plan (改善計畫)

Based on the analysis, we will execute the following improvements.

2.1 Parameter Tuning (Grid Search)

Run experiments on train.json varying:

• retrieval.score_threshold: [0.5, 0.55, 0.6]
• llm.confidence_threshold: [6, 7, 8]
• bbox.max_merged_ratio: [0.3, 0.5, 0.7]

2.2 Ensemble Strategy (Model Fusion)

Use widely different configurations to capture different signals, then merge them.

• Model A (High Recall): Low retrieval threshold, Low LLM threshold.
• Model B (High Precision): High retrieval threshold, High LLM threshold.
• Fusion: Union of A and B, or Majority Vote.

2.3 Prompt Engineering

• Optimize the system prompt to include "Negative Constraints" (what is not a match) to reduce False Positives.
• Add specific "Few-Shot" examples if possible.

3. Experimental Setup (完整實驗設置)

To enable iteration, we will create a dedicated experiment runner script.

3.1 Script: run_experiments.py

This script will:

1. Accept a list of configurations.
2. Iteratively update config.py (or pass args).
3. Run main.py --mode train.
4. Parse the output logs for F1, Precision, Recall.
5. Save a summary report experiment_results.csv.

3.2 Target Metric

• Primary: F1 Score > 0.4 on Local Train Set.
• Secondary: F1 Score > 0.4 on Leaderboard (Test Set).

4. Execution Steps

1. Baseline: Run main.py --mode train and record F1.
2. Debug: Inspect logs for "LLM BLOCKED" or "Matches found but low score".
3. Optimize: Run run_experiments.py with parameter grid.
4. Ensemble: Create a script merge_submissions.py to combine best results.