Skip to content

Commit d0182c5

Browse files
committed
feat: add unified core thesis pipeline
1 parent 6b0d51b commit d0182c5

21 files changed

Lines changed: 276 additions & 997 deletions

File tree

README.md

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -77,17 +77,16 @@ Here is a short video:
7777
5. **Workflow Generator**: Automatically generates customizable CI/CD workflows for Python repositories,
7878
including unit tests, code formatting, PEP 8 compliance checks, and PyPI publication.
7979

80-
6. **Thesis (VKR) check**: Evaluates a repository against a set of formal criteria (
81-
non-empty README, license file, etc.). It also extracts claims (unique entities such as preprocessing type, model
82-
architecture, etc.) from the thesis (VKR) text and matches them against the repository's code.
80+
6. **Thesis (VKR) repository-quality score**: Evaluates a repository against formal criteria such as a non-empty
81+
README and license file. This score is a component of the canonical thesis-analysis pipeline.
8382

8483
7. **Standalone paper claims pipeline**: Extracts technical claims from PDF papers through the reusable
8584
`paper_claims` operation and batch utilities. This pipeline is available as a separate module and is not registered
8685
in the scheduler yet.
8786

88-
8. **Thesis repository analysis**: Composes the VKR repository-quality score, the typed paper-claims pipeline, and
89-
batched claim-to-code verification into a canonical JSON/text artifact. Run it with
90-
`python -m osa_tool.tools.thesis_analysis --help`.
87+
8. **Thesis repository analysis**: The sole CLI-supported thesis pipeline composes the VKR repository-quality score,
88+
typed paper-claim extraction, and batched claim-to-code verification into canonical JSON/text artifacts. Run it
89+
with `python -m osa_tool.tools.thesis_analysis --help`.
9190

9291
---
9392

docs/paper-claims/index.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22

33
The paper claims pipeline extracts verifiable technical claims from PDF papers. It is a reusable single-document
44
operation under `osa_tool.operations.analysis.paper_claims` and is not registered in the legacy scheduler or agent graph.
5+
Use `python -m osa_tool.tools.thesis_analysis` when those claims must be verified against a repository together with
6+
the formal VKR quality score.
57

68
The current flow is:
79

@@ -11,9 +13,9 @@ PDF → physical PDF chunks → Marker Markdown → structured sections → extr
1113

1214
## Status
1315

14-
This is the first half of the paper-claims workflow. It focuses on conversion, section parsing, claim extraction, and
15-
local evaluation utilities. Downstream comparison or repository-specific integration can be built on top of the typed
16-
result objects.
16+
This standalone extraction stage focuses on conversion, section parsing, claim extraction, and
17+
local evaluation utilities. The `thesis_analysis` CLI supplies the standard repository-specific integration for the
18+
typed result objects.
1719

1820
## Runtime behavior
1921

docs/thesis-analysis/index.md

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
# Thesis Repository Analysis
22

3-
`thesis_analysis` is the canonical OSA operation for evaluating a thesis and its repository together. It deliberately
4-
keeps presentation layers such as OSA.Edu Streamlit, leaderboard data, and bilingual PDF layouts outside the core
5-
operation.
3+
`thesis_analysis` is OSA's sole CLI-supported pipeline for evaluating a thesis and its repository together. It
4+
deliberately keeps legacy experiment-reproducibility validation, OSA.Edu Streamlit, leaderboard data, and bilingual
5+
PDF layouts outside the core operation.
66

77
## Pipeline
88

@@ -19,6 +19,9 @@ claims JSON ───────────────────^
1919
excluded from the implementation rate. Both decisions are recorded in the result.
2020
- Verification is performed in batches of at most 50 claims. Each model result must cover every requested claim index
2121
exactly once.
22+
- A root `thesis_analysis.json` and `thesis_analysis.txt` are written only after scoring and verification succeed.
23+
A completed PDF extraction is exported under `paper_claims/`, so it can be supplied to a later run with
24+
`--claims-json` if verification must be retried.
2225

2326
## CLI
2427

@@ -42,3 +45,9 @@ Use `--include-low-verifiability` or `--include-low-confidence` only when the de
4245

4346
The command writes `thesis_analysis.json` and `thesis_analysis.txt`. PDF and UI renderers should consume this canonical
4447
JSON artifact rather than duplicate verification logic.
48+
49+
## Migration from the removed VKR claim flow
50+
51+
`VkrScorer` now calculates repository quality only. Its former PDF parser and claim extractor/verifier were removed;
52+
use this CLI for all thesis claim analysis. The `paper_claims` module's `claims_legacy.json` export remains accepted as
53+
an input adapter for staged runs, but it does not activate the removed VKR flow.
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
[prompts]
2+
verify_system = """You are a code reviewer checking which technical claims from a research paper are implemented in the provided repository source code.
3+
For each claim determine whether the code contains evidence of its implementation.
4+
OUTPUT: a JSON array with one object per claim (same order, 0-based index):
5+
[
6+
{"index": 0, "implemented": true, "confidence": "high", "evidence_file": "train.py", "explanation": "Adam optimizer set at line 42"},
7+
...
8+
]
9+
confidence: 'high' = directly visible, 'medium' = inferable, 'low' = uncertain.
10+
implemented=false when no evidence found. Return ONLY the JSON array."""

osa_tool/config/prompts/vkr_scoring.toml

Lines changed: 0 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -104,51 +104,3 @@ Respond with JSON in this exact format:
104104
}}
105105
106106
If no experiment or visualization files are found, return {{"experiment_files": []}}."""
107-
108-
filter_system = """You are an academic document structure filter. Process a list of extracted section headings from a research paper and return only those relevant for downstream technical claim extraction.
109-
RULES:
110-
1. EXCLUDE: Related Work, Literature Review, Limitations, Conclusion, Introduction, Acknowledgments, References, Appendices, Supplementary Material.
111-
2. RETAIN: sections covering methods, algorithms, system design, experiments, results, theoretical analysis, applications.
112-
3. When in doubt, prefer retention.
113-
4. Preserve original sequential order.
114-
OUTPUT: ONLY a valid JSON array of strings. No markdown, no explanations.
115-
Example: ["Methodology", "Experimental Setup", "Results"]"""
116-
117-
extract_system = """You are an expert technical reviewer for machine learning research. Extract verifiable factual claims from a section of a research paper.
118-
DEFINITION: A 'verifiable claim' is a specific, concrete statement that can be confirmed or refuted by inspecting the code repository, configuration files, or execution logs.
119-
RULES:
120-
- EXTRACT only specific, technical statements about datasets, models, training procedures, metrics, or infrastructure.
121-
- EXCLUDE vague descriptions, motivations, references to prior work, subjective evaluations.
122-
GOOD EXAMPLES:
123-
- 'ResNet-50 was used as the backbone'
124-
- 'The dataset was split 80/10/10 for train/val/test'
125-
- 'Adam optimizer with lr=0.001 was used'
126-
- 'The model was trained for 50 epochs'
127-
BAD EXAMPLES (DO NOT EXTRACT):
128-
- 'a deep learning model was used' (vague)
129-
- 'we chose this approach because it is efficient' (motivation)
130-
OUTPUT SCHEMA (every object must have exactly these fields):
131-
- 'claim': string, self-contained sentence
132-
- 'original_text': string, verbatim fragment <=30 words
133-
- 'category': one of [dataset, model_architecture, training_procedure, evaluation_metric, numerical_result, baseline_comparison, data_preprocessing, infrastructure]
134-
- 'value': string or null — the specific value if present
135-
- 'verifiability': one of [high, medium, low]
136-
OUTPUT FORMAT: ONLY a valid JSON array. No markdown, no preamble. Return [] if nothing found."""
137-
138-
dedup_system = """You are a technical claim deduplication engine. Process a list of ML paper claims: merge duplicates and flag factual contradictions.
139-
RULES:
140-
1. MERGE DUPLICATES: consolidate into the most specific version.
141-
2. FLAG CONTRADICTIONS: if claims disagree on the same fact, keep both and set 'contradiction': true.
142-
3. PRESERVE all claims even if they seem incorrect — accuracy is checked later.
143-
SCHEMA: every object must have: claim, original_text, category, value, verifiability, contradiction (bool, default false).
144-
OUTPUT: ONLY a valid JSON array. No markdown, no preamble. Return [] if empty."""
145-
146-
verify_system = """You are a code reviewer checking which technical claims from a research paper are implemented in the provided repository source code.
147-
For each claim determine whether the code contains evidence of its implementation.
148-
OUTPUT: a JSON array with one object per claim (same order, 0-based index):
149-
[
150-
{"index": 0, "implemented": true, "confidence": "high", "evidence_file": "train.py", "explanation": "Adam optimizer set at line 42"},
151-
...
152-
]
153-
confidence: 'high' = directly visible, 'medium' = inferable, 'low' = uncertain.
154-
implemented=false when no evidence found. Return ONLY the JSON array."""

osa_tool/operations/analysis/thesis_analysis/__init__.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,13 +7,15 @@
77
ThesisAnalysisRequest,
88
ThesisAnalysisResult,
99
)
10+
from .data_context import CsvAnalyzer
1011
from .pipeline import ThesisAnalysisOperation
1112
from .verifier import ClaimVerifier
1213

1314
__all__ = [
1415
"ClaimSelection",
1516
"ClaimVerificationResult",
1617
"ClaimVerificationStats",
18+
"CsvAnalyzer",
1719
"ClaimVerifier",
1820
"ThesisAnalysisOperation",
1921
"ThesisAnalysisRequest",
Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
"""Lightweight tabular-data context for thesis claim verification."""
2+
3+
from __future__ import annotations
4+
5+
import csv
6+
import io
7+
import math
8+
from typing import Any
9+
10+
11+
class CsvAnalyzer:
12+
"""Summarize raw CSV or TSV text for repository-claim verification."""
13+
14+
def __init__(self, content: str, filename: str = "") -> None:
15+
self._content = content
16+
self._filename = filename
17+
18+
def analyze(self) -> dict[str, Any]:
19+
"""Return schema, missing-value, sample, and numeric statistics."""
20+
result: dict[str, Any] = {
21+
"filename": self._filename,
22+
"row_count": 0,
23+
"column_count": 0,
24+
"columns": [],
25+
"column_stats": {},
26+
"error": None,
27+
}
28+
try:
29+
sample = self._content[:4096]
30+
try:
31+
dialect = csv.Sniffer().sniff(sample, delimiters=",;\t|")
32+
except csv.Error:
33+
dialect = csv.excel # type: ignore[assignment]
34+
rows = list(csv.reader(io.StringIO(self._content), dialect))
35+
except Exception as exc:
36+
result["error"] = str(exc)
37+
return result
38+
39+
if not rows:
40+
return result
41+
42+
columns = [column.strip() for column in rows[0]]
43+
data_rows = rows[1:]
44+
result["columns"] = columns
45+
result["column_count"] = len(columns)
46+
result["row_count"] = len(data_rows)
47+
for column_index, column_name in enumerate(columns):
48+
values = [row[column_index].strip() if column_index < len(row) else "" for row in data_rows]
49+
missing_count = sum(1 for value in values if not value)
50+
non_empty = [value for value in values if value]
51+
dtype = self._infer_type(values)
52+
stats: dict[str, Any] = {
53+
"dtype": dtype,
54+
"missing_count": missing_count,
55+
"missing_pct": round(missing_count / len(values) * 100, 1) if values else 0.0,
56+
"unique_count": len(set(non_empty)),
57+
"sample_values": list(dict.fromkeys(non_empty))[:5],
58+
}
59+
if dtype == "numeric":
60+
stats.update(self._numeric_stats(values))
61+
result["column_stats"][column_name] = stats
62+
return result
63+
64+
@staticmethod
65+
def _infer_type(values: list[str]) -> str:
66+
non_empty = [value for value in values if value.strip()]
67+
if not non_empty:
68+
return "empty"
69+
numeric_count = 0
70+
for value in non_empty:
71+
try:
72+
float(value.replace(",", "."))
73+
numeric_count += 1
74+
except ValueError:
75+
pass
76+
return "numeric" if numeric_count / len(non_empty) >= 0.8 else "categorical"
77+
78+
@staticmethod
79+
def _numeric_stats(values: list[str]) -> dict[str, Any]:
80+
numbers: list[float] = []
81+
for value in values:
82+
try:
83+
if value.strip():
84+
numbers.append(float(value.replace(",", ".")))
85+
except ValueError:
86+
pass
87+
if not numbers:
88+
return {}
89+
numbers.sort()
90+
count = len(numbers)
91+
mean = sum(numbers) / count
92+
variance = sum((number - mean) ** 2 for number in numbers) / count
93+
return {
94+
"min": numbers[0],
95+
"max": numbers[-1],
96+
"mean": round(mean, 4),
97+
"std": round(math.sqrt(variance), 4),
98+
"median": numbers[count // 2] if count % 2 else (numbers[count // 2 - 1] + numbers[count // 2]) / 2,
99+
}
100+
101+
@staticmethod
102+
def format_for_prompt(stats: dict[str, Any]) -> str:
103+
"""Render statistics as a compact LLM prompt block."""
104+
lines = [
105+
f"File: {stats['filename']}",
106+
f" Rows (data): {stats['row_count']}",
107+
f" Columns ({stats['column_count']}): {', '.join(stats['columns'])}",
108+
]
109+
if stats.get("error"):
110+
lines.append(f" ERROR: {stats['error']}")
111+
return "\n".join(lines)
112+
113+
lines.append(" Column details:")
114+
for column, column_stats in stats.get("column_stats", {}).items():
115+
missing = f"missing={column_stats['missing_pct']}%" if column_stats["missing_count"] else "complete"
116+
unique = f"unique={column_stats['unique_count']}"
117+
if column_stats["dtype"] == "numeric":
118+
numeric = (
119+
f"min={column_stats.get('min')}, max={column_stats.get('max')}, "
120+
f"mean={column_stats.get('mean')}, std={column_stats.get('std')}"
121+
)
122+
lines.append(f" {column}: numeric, {missing}, {unique}, {numeric}")
123+
else:
124+
samples = ", ".join(repr(value) for value in column_stats.get("sample_values", [])[:3])
125+
lines.append(f" {column}: {column_stats['dtype']}, {missing}, {unique}, samples=[{samples}]")
126+
return "\n".join(lines)

osa_tool/operations/analysis/thesis_analysis/verifier.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,10 @@
77
from pathlib import Path
88
from typing import Any, Callable
99

10-
from osa_tool.operations.analysis.vkr_scoring.csv_analyzer import CsvAnalyzer
1110
from osa_tool.utils.prompts_builder import PromptLoader
1211
from osa_tool.utils.response_cleaner import JsonProcessor
1312

13+
from .data_context import CsvAnalyzer
1414
from .models import ClaimSelection, ClaimVerificationResult, ClaimVerificationStats
1515

1616
Progress = Callable[[str, float], None] | None
@@ -216,7 +216,7 @@ def _verify_batches(
216216
parsed = self._model_handler.send_and_parse(
217217
prompt,
218218
lambda raw: self._parse_verification_batch(raw, expected_indices),
219-
self._prompts.get("vkr_scoring.verify_system"),
219+
self._prompts.get("thesis_analysis.verify_system"),
220220
)
221221
verification_by_index.update({item["index"]: item for item in parsed})
222222

osa_tool/operations/analysis/vkr_scoring/checks.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@
3131

3232
@dataclass
3333
class VkrConfig:
34-
"""Thin config object threaded through all VKR check/claim functions."""
34+
"""Thin config object threaded through all VKR repository-quality checks."""
3535

3636
clone_dir: str # absolute path to the already-cloned repository
3737
repo_url: str # original URL — used only in report metadata

0 commit comments

Comments
 (0)