feat(benchmark): per-k latency sweep infrastructure + GPU/CPU report

[nmrenyi]

Summary

• Adds a runtime --retrieve-k override so the latency benchmark can sweep retrieval depth without rebuilding the APK. One build + install, then loop the Python wrapper across k values.
• Refactors BenchmarkActivity → new BenchmarkForegroundService with a PARTIAL_WAKE_LOCK and sticky notification, so multi-hour sweeps survive screen-off + device-lock. (OPPO ColorOS also requires the per-app battery-optimization whitelist set in Settings — documented in the report.)
• Captures per-run retrieved chunks + full response text in the benchmark JSON, not just lengths — enables downstream answer-quality analysis at any k.
• --rag-only flag to skip the redundant No-RAG mode during k-sweeps (No-RAG is k-invariant; only needs to be measured once).
• Fixes a pre-existing ANR (Thread.sleep → delay()), a stale gemma-3n model-file check in the pre-flight script, and a hard-coded "backend":"CPU" JSON-metadata bug that mislabeled all GPU runs.
• New evaluation/aggregate_k_sweep.py aggregation script; renders evaluation/reports/latency_report_v2.md — full per-k GPU↔CPU comparison.

Headline findings

On OPPO Snapdragon 8 Elite + Gemma 4 E4B + LiteRT-LM 0.11.0 (16 canonical 54-run benchmarks):

• GPU is 13–19× faster on TTFT than CPU and 2–4× faster on total latency. Decode is memory-bandwidth-bound (within ~1.4× across backends); the GPU win is entirely in compute-heavy prefill.
• The 4096-token context window is the binding upper limit, not latency. At k=20, the same 24 of 54 runs (same 8 queries × 3 reps) fail identically on both GPU and CPU with `Input token ids are too long … >= 4096`. The cap is a property of the `gemma-4-E4B-it.litertlm` artifact, not a runtime config and not backend-dependent. k_max ≈ 17–18.
• All GPU totals at k ≤ 15 stay between 13 s (no-RAG) and 25 s — well under any reasonable UX budget. CPU is unusable past k≈3 for a 60 s budget; at k=15 CPU p95 hits 113 s.

Test plan

• Build default (flutter build apk --release) → CPU. With -PuseGpuForLlm=true → GPU. Verify the [BACKEND] *** LLM running on … *** log line on startup matches.
• python evaluation/benchmark_latency.py --filter medium_01 --repeats 1 --rag-only --retrieve-k 5. Verify the JSON output has num_retrieved_docs == 5, populated retrieved_chunks array with chunk text/source/page/chars, and populated response_text. Filename contains `_k5`.
• Screen-off survival: launch a multi-query benchmark, lock the device after init, confirm runs continue. Requires Settings → Battery → App Battery Management → MAM-AI → "Allow background activity" on OPPO/Xiaomi devices (otherwise OplusHansManager freezes the process at the OS level).
• python evaluation/aggregate_k_sweep.py regenerates evaluation/reports/latency_report_v2.md from the JSONs in evaluation/latency_results/.

Commits (10)

```
4daf626 analysis: add CPU k=20 — confirms 4096-token wall is backend-invariant
ede273f analysis: k-sweep latency report (GPU + CPU on Snapdragon 8 Elite)
ef96538 fix(benchmark): record actual backend (GPU/CPU) in config metadata
7ac3b36 refactor(benchmark): move benchmark to a foreground service
12fd358 fix(benchmark): keep CPU alive through screen-off (PARTIAL_WAKE_LOCK + Default dispatcher)
795ac84 fix(benchmark): use suspending delay() instead of Thread.sleep()
197a7bc feat(benchmark): add --rag-only flag to skip No-RAG mode
33604df fix(benchmark): update model-file check to match production stack
8848bee feat(benchmark): capture retrieved chunks + response text per run
fd85cd7 feat(benchmark): add --retrieve-k override for per-k latency sweep
```

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds infrastructure for per-k retrieval-depth latency sweeps on the Android RAG app: a runtime --retrieve-k override, a foreground-service-based benchmark harness that survives screen-off, richer per-run JSON output, and a Python aggregator that emits a GPU↔CPU markdown report. Also fixes a few latent bugs (Thread.sleep ANR, hard-coded "backend":"CPU", stale model filename in pre-flight).

Changes:

• New BenchmarkForegroundService (with PARTIAL_WAKE_LOCK + sticky notification, in :benchmark process); BenchmarkActivity is now a thin shim that forwards Intent extras and finishes.
• RagPipeline.generateResponse gains an optional retrieveKOverride; benchmark JSON now records full retrieved_chunks/response_text and the actual GPU/CPU backend.
• New evaluation/aggregate_k_sweep.py aggregator + generated evaluation/reports/latency_report_v2.md; benchmark_latency.py adds --retrieve-k, --rag-only, k-suffixed output filenames, and updated model filename.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
app/android/app/src/main/AndroidManifest.xml	Adds WAKE_LOCK permission and BenchmarkForegroundService declaration in :benchmark process.
app/android/app/src/main/kotlin/com/example/app/BenchmarkActivity.kt	Reduced to ~60-line shim that forwards extras to the new service.
app/android/app/src/main/kotlin/com/example/app/BenchmarkForegroundService.kt	New service hosting the entire benchmark loop, wake lock, notification, and JSON output.
app/android/app/src/main/kotlin/com/example/app/RagPipeline.kt	Adds retrieveKOverride param to generateResponse for per-call top_k override.
evaluation/benchmark_latency.py	Adds --retrieve-k/--rag-only flags, k-suffixed filename, updated model file check.
evaluation/aggregate_k_sweep.py	New aggregator that groups runs by (backend, k) and renders the v2 report.
evaluation/reports/latency_report_v2.md	Generated GPU↔CPU latency-sweep report.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (3)

app/android/app/src/main/kotlin/com/example/app/BenchmarkForegroundService.kt:99

• retrieveKOverride is gated only with takeIf { it >= 0 }, so passing --ei retrieve_k 0 is forwarded as a 0 to RagPipeline.generateResponse, which sets effectiveTopK = 0 and calls RetrievalConfig.create(0, …). The Python wrapper also has no validation on --retrieve-k (it's a plain int arg). The semantically correct way to "retrieve nothing" in this codebase is --no-retrieval / useRetrieval=false; a top_k=0 retrieval call has undefined behavior in MediaPipe's RetrievalConfig. Recommend rejecting non-positive values either in the Python CLI (type=positive_int) or in the service (takeIf { it > 0 }).

        val retrieveKOverride: Int? = intent?.getIntExtra("retrieve_k", -1)?.takeIf { it >= 0 }

app/android/app/src/main/kotlin/com/example/app/BenchmarkForegroundService.kt:240

• loopStart is computed but never read after the activity-side ETA reporting was removed. Dead code — safe to delete.

        val loopStart = System.currentTimeMillis()

app/android/app/src/main/kotlin/com/example/app/BenchmarkForegroundService.kt:234

• The KDoc says "skipRetrieval and ragOnly are mutually exclusive (skipRetrieval wins)," and the Python wrapper enforces this with parser.error(...). However, the service itself silently chooses skipRetrieval over ragOnly if both extras are passed via am start directly. A user invoking am start by hand could end up surprised. Consider logging a warning (or refusing to start) when both flags are true rather than silently picking one.

        // skipRetrieval and ragOnly are mutually exclusive (skipRetrieval wins).
        val retrievalModes = when {
            skipRetrieval -> listOf(false)
            ragOnly -> listOf(true)
            else -> listOf(true, false)
        }

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 7 comments.

Comments suppressed due to low confidence (4)

evaluation/aggregate_k_sweep.py:110

• For canonical 54-run files this p95 index int(len(s) * 0.95) evaluates correctly (51 → 52nd of 54), but for runs with errors (e.g. the k=20 set with only 30 successes) it yields int(30*0.95)=28 → the 29th of 30 sorted, which is the 96.7th percentile, not the 95th. math.ceil(0.95*n)-1 (or statistics.quantiles) would give the standard nearest-rank p95 across all sample sizes.

        s = sorted(vs)
        out[c] = {
            "n": len(vs),
            "median": int(statistics.median(vs)),
            "p95": int(s[min(len(s) - 1, int(len(s) * 0.95))]),
        }
    return out


def aggregate_overall(d: dict, key: str) -> dict:
    vs = [r[key] for r in d["results"] if not r.get("error")]
    if not vs:
        return {}
    s = sorted(vs)
    return {
        "n": len(vs),
        "median": int(statistics.median(vs)),
        "p95": int(s[min(len(s) - 1, int(len(s) * 0.95))]),

evaluation/reports/latency_report_v2.md:115

• The "Wall-clock comparison" table mixes runs that completed all 54 queries with runs that errored out partway: at k=20, 24 of 54 runs failed-fast with the context-overflow error before any decode, so the wall time (22.8 / 58.6 min) is artificially low compared to k=15 and is not a meaningful CPU÷GPU comparison. This row should either be flagged in the table (e.g. an asterisk + footnote) or excluded from the CPU÷GPU column.

| 15 | 32.4 | 90.8 | 2.80× |
| 20 | 22.8 | 58.6 | 2.57× |

evaluation/aggregate_k_sweep.py:259

• fmt_ms(ge) / fmt_ms(ce) are reused here just to render an integer count of failed runs. The helper is named fmt_ms and its docstring/sibling fmt_s strongly suggest it is for milliseconds. Either rename fmt_ms to a generic int-or-dash helper, or call str(ge) if ge is not None else "—" inline so the table's intent is clear.

        ge = sum(1 for r in gpu_run["data"]["results"] if r.get("error")) if gpu_run else None
        ce = sum(1 for r in cpu_run["data"]["results"] if r.get("error")) if cpu_run else None
        label = "**0 (no-RAG)**" if k == 0 else str(k)
        md.append(f"| {label} | {fmt_ms(ge)} | {fmt_ms(ce)} |")

evaluation/reports/latency_report_v2.md:37

• The headline methodology says "Reported as median across the 54 runs unless noted (p95 in tables marked p95)", but the very next "Headline — Median total query latency" table breaks results down by category (short / medium / long) — i.e. medians over ~18 runs each, not over the full 54. The methodology line should mention the per-category breakdown so readers don't misinterpret the 13.1 / 12.6 / 17.3 values as 54-run statistics.

- `TTFT` excludes retrieval — measured from end-of-retrieval to first generated token.
- `decode` is first-token to last-token.
- `total_query` is everything: `retrieval + TTFT + decode`.
- Reported as median across the 54 runs unless noted (p95 in tables marked `p95`).

## Headline — Median total query latency (seconds)

| k | doc_chars med | GPU short / med / long | CPU short / med / long | CPU÷GPU |
|---:|---:|---:|---:|---:|
| **0 (no-RAG)** | 0 | 12.9 / 15.6 / 16.1 | 27.2 / 26.9 / 29.8 | 1.94× |
| 1 | 561 | 13.1 / 12.6 / 17.3 | 29.3 / 31.9 / 30.3 | 2.14× |
| 3 | 2098 | 18.6 / 18.6 / 21.0 | 37.3 / 44.5 / 42.5 | 2.24× |
| 5 | 3547 | 18.2 / 20.0 / 21.4 | 54.8 / 60.7 / 63.0 | 3.07× |
| 7 | 5139 | 21.3 / 23.2 / 22.8 | 61.4 / 62.3 / 60.4 | 2.72× |
| 10 | 7482 | 22.5 / 20.5 / 20.4 | 61.8 / 70.6 / 77.9 | 3.10× |
| 15 | 11297 | 25.3 / 24.0 / 22.4 | 84.8 / 80.8 / 89.7 | 3.48× |
| 20 | 14520 | 23.9 / 20.5 / 18.5 | 88.7 / 95.6 / 95.6 | 4.46× |

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.