[week-02] 26512076

submissions/26512076/week-02/REPORT.md

@@ -0,0 +1,45 @@
+# Week 02 — Harness A/B Report
+
+## 1. Variant definition
+
+이번 실험에서는 동일한 태스크와 도구를 사용하여 ReAct와 Plan-then-Execute 하네스를 비교했습니다. 최종 비교 실행(13–18번)의 Provider는 Groq이며, 두 하네스 모두 명시적인 단일 모델 ID인 `openai/gpt-oss-20b`를 사용했습니다. `tools_shared.py`에서 모델을 고정하고 Groq API를 OpenAI 호환 SDK로 호출했습니다. API 키는 파일에 저장하지 않고 `GROQ_API_KEY` 환경변수로 전달했습니다.
+
+공통 도구는 `read_file(path)`와 `count_pattern(path, pattern)`입니다. 태스크는 변경하지 않은 `app.log`에서 ERROR가 가장 많이 발생한 시간대(HH:00)를 찾는 것입니다. `TASK.md`의 기대 문자열 `14:00`이 최종 답변에 포함되면 성공으로 판정했습니다. 실행 명령은 `python run_ab.py --runs 3`입니다. 이 명령은 각 하네스를 3회 실행하고 결과와 실행별 로그를 기존 파일에 추가합니다.
+
+ReAct는 모델의 판단, 도구 호출, 관찰 결과를 단계별로 반복하며 전체 대화 기록을 다음 호출에 전달합니다. 최종 답변을 내거나 최대 단계 수에 도달하면 종료합니다. 도구 실행 중 발생한 오류는 관찰 결과로 전달하여 다음 단계에서 대응할 수 있습니다.
+
+Plan-then-Execute는 먼저 도구 없이 JSON 리스트 형식의 계획을 생성하고, 실행기가 각 계획 단계를 순서대로 처리합니다. 한 단계의 도구 호출은 최대 3라운드이며, `OFF_PLAN`이 발생하면 최대 1회 재계획할 수 있습니다. 두 하네스는 컨텍스트를 구성하는 방식, 종료 조건, 오류 복구 방식이 다릅니다. 도구 종류와 도구의 작업 단위는 같으며, 사람의 승인 단계는 두 하네스 모두 없습니다.
+
+## 2. Measurements
+
+`results.csv`의 모든 실행을 아래에 보존했습니다. 1–6번은 기존 `openrouter/free` 실행입니다. 이 ID는 호출마다 실제 무료 모델을 선택할 수 있어 고정 모델 A/B 비교에는 사용하지 않았습니다. 7–12번은 Groq 모델을 고정한 뒤의 첫 실행이며, 10–12번에서 계획 생성 중 API 오류가 발생했습니다. 계획 단계 지시를 수정한 다음 실행한 **13–18번이 최종 A/B 비교 대상**입니다.
+
+| run | harness | success | tokens | iters | interventions | note |
+|---:|---|:---:|---:|---:|---:|---|
+| 1 | react | O | 15395 | 5 | 0 | |
+| 2 | react | O | 11782 | 4 | 0 | |
+| 3 | react | O | 8838 | 4 | 0 | |
+| 4 | plan_exec | O | 15382 | 7 | 0 | replans=1 |
+| 5 | plan_exec | X | 225 | 1 | 0 | replans=0 |
+| 6 | plan_exec | X | 738 | 1 | 0 | replans=0 |
+| 7 | react | O | 2581 | 2 | 0 | |
+| 8 | react | O | 2355 | 2 | 0 | |
+| 9 | react | O | 2096 | 2 | 0 | |
+| 10 | plan_exec | X | | | | API 400: tool_use_failed |
+| 11 | plan_exec | X | | | | API 400: tool_use_failed |
+| 12 | plan_exec | X | | | | API 400: tool_use_failed |
+| 13 | react | O | 2183 | 2 | 0 | |
+| 14 | react | O | 2056 | 2 | 0 | |
+| 15 | react | O | 3495 | 2 | 0 | |
+| 16 | plan_exec | O | 18143 | 8 | 0 | replans=0 |
+| 17 | plan_exec | O | 33126 | 13 | 0 | replans=0 |
+| 18 | plan_exec | O | 33465 | 10 | 0 | replans=0 |
+
+최종 비교(13–18번)에서 두 하네스 모두 3/3회 성공했고 사람의 개입은 0회였습니다. ReAct는 평균 2,578토큰과 2회 호출을 사용했습니다. Plan-then-Execute는 평균 약 28,245토큰과 10.33회 호출을 사용했습니다. `iters`는 도구 호출 횟수가 아니라 모델 호출 횟수입니다.
+
+## 3. Interpretation
+
+최종 실행에서는 성공률이 같았지만 ReAct가 더 적은 토큰과 모델 호출로 답을 구했습니다. ReAct 로그(13–15번)에서는 `app.log`를 읽은 뒤 두 번째 모델 호출에서 모두 `14:00`을 답했습니다. Plan-then-Execute 로그(16–18번)에서는 먼저 계획을 만들고 각 단계를 별도 호출로 실행했습니다. 특히 17번은 파일 읽기, 전체 줄 수 확인, ERROR 줄 수 확인, 시간 추출, 시간대별 집계 등을 여러 단계로 나누어 13회 호출했습니다. 이 순차 실행 방식이 이번 짧은 태스크에서 토큰과 반복 횟수를 늘린 것으로 해석됩니다. 세 실행 모두 재계획은 사용하지 않았습니다.
+
+이전 실패도 하네스의 동작을 보여줍니다. 5·6번은 계획을 유효한 JSON 리스트로 파싱하지 못해 실행 단계에 들어가지 못했습니다. 10–12번은 고정 모델의 첫 설정에서 계획 생성 중 도구 호출을 시도하여 API가 `tool_use_failed`를 반환한 사례입니다. 이 실행들의 빈 토큰·반복 횟수 칸은 `run_ab.py`가 예외 발생 시 계측값을 기록하지 못한 결과이며, 0을 뜻하지 않습니다. 이후 계획 단계에서는 도구를 사용할 수 없다는 지시를 명확히 하여 13–18번을 다시 실행했습니다. 따라서 서로 다른 모델이 선택될 수 있었던 1–6번이나 계획 지시가 달랐던 7–12번을 최종 성능 평균에 섞지 않았습니다. 최종 결과는 각 하네스 3회에 대한 관찰이므로, 다른 태스크에서도 같은 효율 차이가 난다고 일반화할 수는 없습니다.
+

submissions/26512076/week-02/TASK.md

@@ -0,0 +1,15 @@
+# Task
+
+The same task for both harnesses. `run_ab.py` reads the `task:` and
+`expected:` lines below. Write the success criterion before you run
+anything, and do not change it after you see the results.
+
+task: In app.log, which hour (HH:00) has the most ERROR lines? Answer with the hour in HH:00 form.
+
+## Success criterion
+
+A run succeeds when the final answer contains the hour with the most ERROR
+lines in `app.log`, written as HH:00. `app.log` is the reference input; the
+graded runs use it unchanged.
+
+expected: 14:00

submissions/26512076/week-02/app.log

@@ -0,0 +1,60 @@
+2026-09-01 09:07:27 INFO  job finished in 320ms
+2026-09-01 09:11:56 ERROR NullReference in QuizService.score
+2026-09-01 09:13:14 INFO  health check ok
+2026-09-01 09:26:45 INFO  health check ok
+2026-09-01 09:30:17 INFO  health check ok
+2026-09-01 09:32:05 INFO  request served /api/quiz
+2026-09-01 10:07:50 INFO  session started
+2026-09-01 10:12:15 INFO  request served /api/quiz
+2026-09-01 10:18:22 ERROR db connection refused
+2026-09-01 10:23:48 WARN  retrying upstream call
+2026-09-01 10:53:22 WARN  slow query 1.8s on rooms
+2026-09-01 10:54:07 ERROR NullReference in QuizService.score
+2026-09-01 10:57:10 INFO  cache hit for room 4412
+2026-09-01 11:23:35 ERROR payment webhook signature mismatch
+2026-09-01 11:36:12 INFO  cache hit for room 4412
+2026-09-01 11:39:20 INFO  request served /api/quiz
+2026-09-01 11:40:27 WARN  slow query 1.8s on rooms
+2026-09-01 11:44:02 INFO  health check ok
+2026-09-01 11:56:13 INFO  request served /api/quiz
+2026-09-01 12:08:13 INFO  request served /api/quiz
+2026-09-01 12:28:08 ERROR upstream timeout after 5000ms
+2026-09-01 12:32:19 INFO  health check ok
+2026-09-01 12:49:54 ERROR db connection refused
+2026-09-01 12:50:06 WARN  token near expiry
+2026-09-01 12:53:48 ERROR upstream timeout after 5000ms
+2026-09-01 12:59:50 WARN  token near expiry
+2026-09-01 13:24:27 INFO  cache hit for room 4412
+2026-09-01 13:26:02 INFO  session started
+2026-09-01 13:28:15 INFO  request served /api/quiz
+2026-09-01 13:42:13 ERROR db connection refused
+2026-09-01 13:43:37 WARN  slow query 1.8s on rooms
+2026-09-01 13:59:54 ERROR upstream timeout after 5000ms
+2026-09-01 14:04:06 ERROR NullReference in QuizService.score
+2026-09-01 14:07:11 ERROR NullReference in QuizService.score
+2026-09-01 14:15:56 ERROR upstream timeout after 5000ms
+2026-09-01 14:19:13 INFO  health check ok
+2026-09-01 14:26:19 INFO  request served /api/quiz
+2026-09-01 14:30:20 ERROR db connection refused
+2026-09-01 14:43:37 INFO  job finished in 320ms
+2026-09-01 14:47:03 ERROR upstream timeout after 5000ms
+2026-09-01 14:54:53 ERROR upstream timeout after 5000ms
+2026-09-01 15:04:23 INFO  cache hit for room 4412
+2026-09-01 15:33:16 INFO  request served /api/quiz
+2026-09-01 15:34:38 ERROR payment webhook signature mismatch
+2026-09-01 15:39:25 INFO  cache hit for room 4412
+2026-09-01 15:41:14 ERROR NullReference in QuizService.score
+2026-09-01 15:45:36 INFO  cache hit for room 4412
+2026-09-01 15:48:41 WARN  token near expiry
+2026-09-01 16:11:35 INFO  request served /api/quiz
+2026-09-01 16:16:04 INFO  cache hit for room 4412
+2026-09-01 16:24:31 WARN  slow query 1.8s on rooms
+2026-09-01 16:44:46 INFO  session started
+2026-09-01 16:51:30 WARN  retrying upstream call
+2026-09-01 16:53:34 ERROR NullReference in QuizService.score
+2026-09-01 17:17:14 ERROR NullReference in QuizService.score
+2026-09-01 17:27:48 WARN  token near expiry
+2026-09-01 17:37:46 INFO  health check ok
+2026-09-01 17:48:38 WARN  token near expiry
+2026-09-01 17:55:29 INFO  job finished in 320ms
+2026-09-01 17:56:05 INFO  cache hit for room 4412

submissions/26512076/week-02/harness_plan_execute.py

@@ -0,0 +1,107 @@
+"""Week 02 starter — the Plan-then-Execute harness.
+
+One call produces the whole plan as a JSON list. Then each step is executed
+in order with tools. If a step reports OFF_PLAN, the plan is rebuilt once
+(max_replan=1): that number is the flexibility cap, and it is explicit.
+"""
+import json
+import re
+import sys
+
+from tools_shared import Chat, Meter, Reply
+
+SYSTEM_PLAN = (
+    "You are a planner. In this planning phase you cannot call tools. "
+    "Describe actions for a later executor. Reply with a JSON list of "
+    "short step strings and nothing else. No prose, no code fences."
+)
+
+SYSTEM_EXEC = (
+    "You execute exactly the current step of a plan using the provided tools. "
+    "Do not perform later steps or give the final answer during a step. "
+    "After completing the current step, report its result briefly. "
+    "If the current step cannot be done, start your reply with 'OFF_PLAN:' "
+    "and explain why. Only when explicitly asked for the final answer, "
+    "start your reply with 'Answer:'."
+)
+
+def parse_plan(text: str):
+    """Return a list of step strings, or None if the model did not give JSON."""
+    text = re.sub(r"^```(?:json)?|```$", "", text.strip(), flags=re.M).strip()
+    try:
+        plan = json.loads(text)
+    except json.JSONDecodeError:
+        return None
+    if isinstance(plan, list) and all(isinstance(s, str) for s in plan):
+        return plan
+    return None
+
+
+def run_plan_execute(task: str, max_replan: int = 1,
+                     max_tool_rounds: int = 3, log=print):
+    meter = Meter()
+
+    # 1) PLAN: the whole plan in one call, no tools
+    planner = Chat(SYSTEM_PLAN, meter, tools=False)
+    planner.add_user(
+       f"Task: {task}\n"
+       "The later executor can read text files and count lines matching "
+       "a regular expression. Describe the steps; do not perform them."
+    )
+    raw = planner.send().text
+    plan = parse_plan(raw)
+    if plan is None:                              # a parse failure is one failure mode
+        log(f"[plan] not valid JSON: {raw.strip()[:300]!r}")
+        return "plan parse failed", meter, 0
+    log(f"[plan] {plan}")
+
+    # 2) EXECUTE: each step in order
+    executor = Chat(SYSTEM_EXEC, meter)
+    executor.add_user(f"Task: {task}\nPlan: {json.dumps(plan)}")
+    replans = 0
+    i = 0
+    while i < len(plan):
+        executor.add_user(
+            f"Execute only step {i + 1}: {plan[i]}. "
+            "Use no more tools than needed, then briefly report this step's result."
+        )
+        reply = executor.send()
+        rounds = 0
+        while reply.tool_calls:                   # tool calls inside one step
+            executor.run_tools(reply, log)
+            reply = executor.send()
+            rounds += 1
+            if rounds >= max_tool_rounds and reply.tool_calls:
+                executor.run_tools(reply, log)    # answer every call so the transcript stays valid
+                reply = Reply("OFF_PLAN: step exceeded the tool-call budget", [])
+                break
+        log(f"[step {i + 1}] {reply.text.strip()[:300]}")
+
+        if reply.text.strip().startswith("OFF_PLAN") and replans < max_replan:
+            replans += 1                          # flexibility cap
+            planner.add_user(f"Step {i + 1} ({plan[i]}) failed: {reply.text.strip()[:300]}\n"
+                             f"Reply with a JSON list of the remaining steps.")
+            raw = planner.send().text
+            new_steps = parse_plan(raw)
+            if new_steps is None:
+                log(f"[replan] not valid JSON: {raw.strip()[:300]!r}")
+                break
+            plan = plan[:i] + new_steps
+            log(f"[replan] {plan}")
+            continue
+        i += 1
+
+    executor.add_user("Give the final answer now, starting with 'Answer:'.")
+    final = executor.send()
+    if final.tool_calls:                          # the model tried to keep going
+        executor.run_tools(final, log)
+        final = executor.send()
+    return final.text, meter, replans
+
+
+if __name__ == "__main__":
+    task = sys.argv[1] if len(sys.argv) > 1 else \
+        "In app.log, which hour (HH:00) has the most ERROR lines? Answer with the hour in HH:00 form."
+    answer, m, replans = run_plan_execute(task)
+    print(answer)
+    print(f"tokens={m.tokens} iters={m.iters} interventions={m.interventions} replans={replans}")

submissions/26512076/week-02/harness_react.py

@@ -0,0 +1,60 @@
+"""Week 02 starter — the ReAct-style harness.
+
+Every step: the model writes a Thought, calls a tool (Action), sees the result
+(Observation), and decides again. The five axes from the lecture are marked.
+"""
+import sys
+
+from tools_shared import Chat, Meter
+
+SYSTEM = (
+    "You solve tasks with the tools you are given. Before every tool call, "
+    "write one line that starts with 'Thought:' saying what you know and what "
+    "you will do next. When the task is complete, reply with a line that starts "
+    "with 'Answer:' and make no tool call."
+)
+
+# [axis 5] calls that must be approved by a human before they run.
+# The starter tools are read-only, so this set is empty and interventions
+# stay at 0. Add a tool that writes or deletes, and put its name here.
+IRREVERSIBLE = set()
+
+
+def ask_human(call) -> bool:
+    answer = input(f"approve {call.name}({call.args})? [y/N] ").strip().lower()
+    return answer == "y"
+
+
+def run_react(task: str, max_steps: int = 8, log=print):
+    meter = Meter()
+    chat = Chat(SYSTEM, meter)                    # [axis 1] context: full history, every call
+    chat.add_user(task)
+
+    for step in range(max_steps):                 # [axis 3] termination: iteration cap
+        reply = chat.send()
+        if reply.text.strip():
+            log(f"[step {step + 1}] {reply.text.strip()}")
+
+        if not reply.tool_calls:                  # [axis 3] the model chose to finish
+            return reply.text, meter
+
+        approved = []
+        for call in reply.tool_calls:
+            if call.name in IRREVERSIBLE and not ask_human(call):
+                meter.interventions += 1          # [axis 5] intervention point
+                chat.add_tool_result(call, "denied: human did not approve")
+                continue
+            approved.append(call)
+        reply.tool_calls = approved
+        chat.run_tools(reply, log)                # [axis 2] granularity lives in tools_shared
+                                                  # [axis 4] errors come back as Observations
+
+    return "MAX_STEPS reached: incomplete", meter
+
+
+if __name__ == "__main__":
+    task = sys.argv[1] if len(sys.argv) > 1 else \
+        "In app.log, which hour (HH:00) has the most ERROR lines? Answer with the hour in HH:00 form."
+    answer, m = run_react(task)
+    print(answer)
+    print(f"tokens={m.tokens} iters={m.iters} interventions={m.interventions}")

submissions/26512076/week-02/logs/plan_exec-04.txt

@@ -0,0 +1,16 @@
+[plan] ['Read app.log', 'For each hour 00-23, count lines matching "^HH:00.*ERROR"', 'Select hour with highest count', 'Return that hour in HH:00 format']
+  [tool] read_file({'path': 'app.log'}) -> 2026-09-01 09:07:27 INFO  job finished in 320ms | 2026-09-01 09:11:56 ERROR NullReference in QuizService.score | 2026-09-01 09:13:14 INFO  health check ok | 2026-09-01 09:26:45 INFO  health check ok | 2026-09
+  [tool] count_pattern({'path': 'app.log', 'pattern': '^2026-09-01 00:00.*ERROR'}) -> 0
+  [tool] count_pattern({'path': 'app.log', 'pattern': '^2026-09-01 09:.*ERROR'}) -> 1
+  [tool] count_pattern({'path': 'app.log', 'pattern': '^2026-09-01 10:.*ERROR'}) -> 2
+  [tool] count_pattern({'path': 'app.log', 'pattern': '^2026-09-01 11:.*ERROR'}) -> 1
+  [tool] count_pattern({'path': 'app.log', 'pattern': '^2026-09-01 12:.*ERROR'}) -> 3
+  [tool] count_pattern({'path': 'app.log', 'pattern': '^2026-09-01 13:.*ERROR'}) -> 2
+  [tool] count_pattern({'path': 'app.log', 'pattern': '^2026-09-01 14:.*ERROR'}) -> 6
+  [tool] count_pattern({'path': 'app.log', 'pattern': '^2026-09-01 15:.*ERROR'}) -> 2
+  [tool] count_pattern({'path': 'app.log', 'pattern': '^2026-09-01 16:.*ERROR'}) -> 1
+  [tool] count_pattern({'path': 'app.log', 'pattern': '^2026-09-01 17:.*ERROR'}) -> 1
+[step 1] OFF_PLAN: step exceeded the tool-call budget
+[replan] not valid JSON: 'User Safety: safe\nResponse Safety: safe'
+[final] Answer: 14:00
+[judge] expected='14:00' -> O (95.2s)

submissions/26512076/week-02/logs/plan_exec-05.txt

@@ -0,0 +1,3 @@
+[plan] not valid JSON: '{"analysis": "I need to find which hour has the most ERROR lines in app.log. I\'ll first read the file to understand its format, then count ERROR lines per hour.", "plan": "1. Read app.log to see the format and identify ERROR lines.\\n2. Use count_pattern to count ERROR lines for each hour.\\n3. Determ'
+[final] plan parse failed
+[judge] expected='14:00' -> X (1.1s)

submissions/26512076/week-02/logs/plan_exec-06.txt

@@ -0,0 +1,3 @@
+[plan] not valid JSON: 'User Safety: safe'
+[final] plan parse failed
+[judge] expected='14:00' -> X (2.9s)

submissions/26512076/week-02/logs/plan_exec-10.txt

@@ -0,0 +1,4 @@
+[model] openai/gpt-oss-20b
+crash: BadRequestError: Error code: 400 - {'error': {'message': 'Tool choice is none, but model called a tool', 'type': 'invalid_request_error', 'code': 'tool_use_failed', 'failed_generation': '{"name": "python", "arguments": import re, textwrap, json, sys, math, os, typing, collections, pprint, itertools, statistics}'}}
+[final] 
+[judge] expected='14:00' -> X (0.7s)

submissions/26512076/week-02/logs/plan_exec-11.txt

@@ -0,0 +1,4 @@
+[model] openai/gpt-oss-20b
+crash: BadRequestError: Error code: 400 - {'error': {'message': 'Tool choice is none, but model called a tool', 'type': 'invalid_request_error', 'code': 'tool_use_failed', 'failed_generation': '{"name": "read_file", "arguments": {"path":"app.log"}}'}}
+[final] 
+[judge] expected='14:00' -> X (0.4s)

submissions/26512076/week-02/logs/plan_exec-12.txt

@@ -0,0 +1,4 @@
+[model] openai/gpt-oss-20b
+crash: BadRequestError: Error code: 400 - {'error': {'message': 'Tool choice is none, but model called a tool', 'type': 'invalid_request_error', 'code': 'tool_use_failed', 'failed_generation': '{"name": "read_file", "arguments": {"path":"app.log"}}'}}
+[final] 
+[judge] expected='14:00' -> X (1.2s)

submissions/26512076/week-02/logs/plan_exec-16.txt

@@ -0,0 +1,19 @@
+[model] openai/gpt-oss-20b
+[plan] ['Read app.log line by line.', 'For each line containing "ERROR", extract the hour (HH) from the timestamp.', 'Increment a counter for that hour.', 'After all lines are processed, find the hour with the maximum count.', 'Print the hour in "HH:00" format.']
+  [tool] read_file({'path': 'app.log'}) -> 2026-09-01 09:07:27 INFO  job finished in 320ms | 2026-09-01 09:11:56 ERROR NullReference in QuizService.score | 2026-09-01 09:13:14 INFO  health check ok | 2026-09-01 09:26:45 INFO  health check ok | 2026-09
+[step 1] Step 1 completed: Read app.log line by line. (First 4000 characters of the file have been retrieved.)
+[step 2] Step 2 completed: Extracted the hour (HH) for each ERROR line. The extracted hours are: 09, 10, 10, 11, 12, 12, 12, 13, 13, 14, 14, 14, 14, 14, 14, 15, 15, 16, 17.
+[step 3] Step 3 completed: Counter for each hour:
+- 09: 1
+- 10: 2
+- 11: 1
+- 12: 3
+- 13: 2
+- 14: 6
+- 15: 2
+- 16: 1
+- 17: 1
+[step 4] Step 4 completed: The hour with the maximum ERROR count is **14** with 6 errors.
+[step 5] Step 5 completed: The hour with the most ERROR lines is **14:00**.
+[final] Answer: 14:00
+[judge] expected='14:00' -> O (136.3s)