Skip to content

Commit 107c41c

Browse files
docs(readme): describe the landed AI stack, not the stub it replaced
Chat streaming, typed extraction, the shared InjectableNativeLlm seam, BYOK settings/desktop and the toolkit 2.0.0 pin had all merged but the README still described the old separate seams, an untyped OCR scrape, and stale toolkit/test/screenshot counts.
1 parent 9cbe0ec commit 107c41c

1 file changed

Lines changed: 50 additions & 19 deletions

File tree

README.md

Lines changed: 50 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -71,11 +71,13 @@ Doori doesn't stand alone. Its Gradle convention plugins live in a separate, reu
7171
[**kmp-build-logic**](https://github.com/darkpandawarrior/kmp-build-logic), pulled in as a Gradle
7272
`includeBuild`, so the AGP/Kotlin/Compose/test setup isn't copy-pasted per project but shared across
7373
my KMP work. Its shared *libraries* increasingly come from the same place too:
74-
[**kmp-toolkit**](https://github.com/darkpandawarrior/kmp-toolkit), a 36-module MIT Kotlin
75-
Multiplatform toolkit vendored here as a git submodule. Doori consumes **ten** of its modules
76-
`:mvi-core`, `:result`, `:common`, `:location`, `:offline-outbox`, `:security`, `:app-shell`,
77-
`:network`, `:settings` and the on-device `:ai` seam (multimodal + streaming), rather than
78-
hand-rolling them, the "extract the reusable core the moment a second app needs it, then consume it"
74+
[**kmp-toolkit**](https://github.com/darkpandawarrior/kmp-toolkit), a 40-module MIT Kotlin
75+
Multiplatform toolkit vendored here as a git submodule, pinned to **2.0.0**. Doori consumes
76+
**thirteen** of its modules: `:mvi-core`, `:result`, `:common`, `:location`, `:offline-outbox`,
77+
`:security`, `:app-shell`, `:network`, `:settings`, the on-device `:ai` seam (multimodal +
78+
streaming, `InjectableNativeLlm`), `:ai-testing`, `:llm-chat` (the BYOK provider chain) and
79+
`:designsystem` (the shared `AiSettingsSection`), rather than hand-rolling them, the "extract the
80+
reusable core the moment a second app needs it, then consume it"
7981
philosophy in practice: Doori is both the flagship *and* a consumer. Its sibling,
8082
[**PaymentsLab-KMP**](https://github.com/darkpandawarrior/PaymentsLab-KMP), goes deep on the payments/UPI
8183
slice the same way this repo goes deep on location and offline-first. All three sit under the same
@@ -116,13 +118,22 @@ slice the same way this repo goes deep on location and offline-first. All three
116118
- 🔥 **Ember theme, four platforms from one KMP core.** A warm amber/red dark theme (replacing an
117119
earlier phosphor-green look) skins Android/iOS phone, Wear OS, watchOS and Compose Desktop, all
118120
from the same `commonMain` architecture.
119-
- 📄 **On-device document intelligence.** A capture-to-form pipeline combines on-device AI, text
120-
recognition and heuristics, OCR field-fill, doc-type classification and duplicate detection, on
121-
device where the platform supports it, degrading gracefully everywhere else.
122-
- 🤖 **On-device LLM assistant.** The expense chat runs against a real on-device model behind a
123-
shared `LlmGateway`, ML Kit GenAI on Android, Apple Foundation Models on iOS via a Swift bridge
124-
(`xcodebuild`-gated, not device-verified), degrading to the offline retrieval engine wherever no
125-
model is available. Not a stub response generator.
121+
- 📄 **On-device document intelligence, typed extraction.** A capture-to-form pipeline combines
122+
on-device AI, text recognition and heuristics; the AI tier returns a typed `AiResult<AiExtraction>`
123+
(schema-typed field extraction via `structuredOutput<...>()`, untrusted OCR text guarded through
124+
`PromptGuard`) instead of a regex scrape, feeding one unified `FormFieldWithSuggestions` prefill
125+
across every call site, plus doc-type classification and duplicate detection, on device where the
126+
platform supports it, degrading gracefully everywhere else.
127+
- 🤖 **On-device LLM assistant, one shared seam, streaming.** The expense chat streams real
128+
per-token replies behind `LlmGateway`, ML Kit GenAI (Gemini Nano) on Android, Apple Foundation
129+
Models on iOS through one shared Swift bridge, kmp-toolkit's `InjectableNativeLlm`/`OnDeviceLlm`
130+
seam that document extraction above now runs on too, degrading to the offline retrieval engine
131+
wherever no model is available. AI is visible in Settings and on desktop: consent, on-device model
132+
download/pause/delete, and a BYOK cloud key (Anthropic/OpenAI/Gemini, tested and stored via
133+
`SecureKeyStore`) as the fallback tier everywhere, the *only* answer path on desktop, which has no
134+
on-device model at all. Neither on-device tier is device-verified here: no Gemini-Nano-class
135+
hardware (Pixel 8+/AICore-eligible) and no Apple Foundation Models device in this environment, so
136+
both are compile-verified only.
126137

127138
## Screenshots
128139

@@ -421,7 +432,7 @@ Doori/
421432
| Charts | Canvas-only (no MPAndroidChart / Vico) |
422433
| Theming | MaterialKolor **5.0.0** |
423434
| Capture | Peekaboo (KMP camera/gallery) |
424-
| On-device AI | ML Kit GenAI (Android) / Apple Foundation Models (iOS, Swift-bridge) behind a shared `LlmGateway`, text recognition + barcode scanning, degrading to an offline heuristic engine where a model isn't available |
435+
| On-device AI | kmp-toolkit's `InjectableNativeLlm`/`OnDeviceLlm` seam, ML Kit GenAI (Gemini Nano, Android) / Apple Foundation Models (iOS, one shared Swift bridge) behind `LlmGateway` (streaming chat) and `DocumentAiAnalyzer` (typed extraction); a BYOK cloud fallback (Anthropic/OpenAI/Gemini via `SecureKeyStore`) on every platform including desktop, which has no on-device tier at all; text recognition + barcode scanning; degrading to an offline heuristic engine where no model is available. Neither on-device tier is device-verified here (no Gemini Nano / Foundation Models hardware) |
425436
| Testing | JUnit, MockK, Turbine, Robolectric, Koin-Test, **Roborazzi 1.68.0** screenshots |
426437
| Quality | detekt **2.0.0-alpha.5**, ktlint, Kover, dependency-guard |
427438
| SDK | compileSdk **37**, minSdk **30**, JDK 21 |
@@ -540,12 +551,12 @@ hoisting, iOS parity, the AI assistant rebuild, etc.). Progress is tracked per i
540551

541552
## Testing and quality
542553

543-
**2,510 `@Test` methods across 370 test classes in 30 modules**, plus 159 host-rendered screenshots.
544-
Numbers you can reproduce:
554+
**2,925 `@Test` methods across 397 test classes in 33 modules**, plus 368 host-rendered screenshots.
555+
Numbers you can reproduce (excluding the vendored `external/` toolkit, which ships its own suite):
545556

546557
```bash
547-
grep -rho '@Test' --include='*.kt' . | wc -l # 2510
548-
ls docs/screenshots/*.png | wc -l # 159
558+
grep -rho '@Test' --include='*.kt' --exclude-dir=external . | wc -l # 2925
559+
ls docs/screenshots/*.png | wc -l # 368
549560
```
550561

551562
| Layer | What runs it | Gates a merge? |
@@ -609,8 +620,10 @@ roadmap reflects direction rather than commitments.
609620
- [x] **AI assistant / "agent" feature (V20).** Offline, retrieval-grounded chat over real local
610621
trip/expense/card data; Room-backed persistent history + 5-minute session resume; on-device
611622
voice I/O (STT/TTS); feedback, export and real-usage popular-question ranking; full
612-
`commonMain` + iOS parity. (A dedicated Popular/Unanswered analytics screen and persisted
613-
unanswered-question submission are still open, tracked as backlog.)
623+
`commonMain` + iOS parity. The dedicated Popular/Unanswered analytics screen (`PopularTab`,
624+
`UnansweredTab` in `AgentChatScreen`) has since shipped; persisted unanswered-question
625+
submission has not, `AgentRepository.unansweredQuestions` still reads from
626+
`AgentMockData`, not Room, tracked as backlog.
614627
- [x] **On-device LLM backing (post-V25).** `LlmGateway` swaps the assistant onto a real on-device
615628
model, ML Kit GenAI on Android, Apple Foundation Models on iOS (Swift bridge,
616629
`xcodebuild`-gated, not yet device-verified), degrading to the offline retrieval engine
@@ -686,6 +699,20 @@ roadmap reflects direction rather than commitments.
686699
- [x] **iOS launch-crash fix (V33).** `CADisableMinimumFrameDurationOnPhone` added to `Info.plist`
687700
(a Compose Multiplatform `PlistSanityCheck` requirement), the iOS app builds
688701
(`xcodebuild`-green) and launches correctly on device.
702+
- [x] **AI stack unification (V35).** Chat and document extraction share ONE on-device seam now:
703+
kmp-toolkit's `InjectableNativeLlm`/`OnDeviceLlm` (`MlKitGenAiOnDeviceLlm` on Android,
704+
`FoundationModelsOnDeviceLlm` over a single Swift `FoundationModelsBridge` registered once in
705+
`AppDelegate.swift` on iOS), replacing this app's own separate seams. `feature:agent`'s
706+
`LlmAssistantEngine` streams real per-token output through it; `core:ai`'s
707+
`DocumentAiAnalyzer` returns a typed `AiResult<AiExtraction>` via `structuredOutput<...>()`
708+
instead of a hand-rolled regex scrape, and OCR-to-form prefill is unified on one
709+
`FormFieldWithSuggestions` component. AI is now visible in Settings (consent, on-device model
710+
download/pause/delete, BYOK cloud-key entry and test) on Android and on the Compose Desktop
711+
target, which has no on-device model at all and always answers through the BYOK cloud
712+
fallback (Anthropic/OpenAI/Gemini keys via `SecureKeyStore`). **Not device-verified:**
713+
Gemini-Nano-class hardware (Pixel 8+/AICore-eligible) and an Apple Foundation Models device
714+
are both unavailable in this environment, so both on-device tiers remain compile-verified
715+
only, same caveat the V25 bullet above already carried.
689716

690717
**Exploring**
691718

@@ -703,6 +730,9 @@ roadmap reflects direction rather than commitments.
703730
`NetworkBackendFlags.useRealBackend` is a compile-time `false` with no debug toggle wiring it
704731
on, so the authenticated path has never run in an actual app process, that wiring, plus the
705732
remaining PLAN_V33.1 routes beyond miles/location/events, is the next step
733+
- [ ] On-device LLM hardware verification: Gemini Nano (Pixel 8+/AICore-eligible) and Apple
734+
Foundation Models are both compile/build-verified only here, no eligible device in this
735+
environment
706736

707737
## iOS, Wear OS and watchOS
708738

@@ -747,6 +777,7 @@ roadmap reflects direction rather than commitments.
747777
| iOS WidgetKit + Live Activity/Dynamic Island |`xcodebuild -scheme MilewayWidgets build` | ✅ host-rendered screenshots (WidgetScreenshotTests) | ✅ widgets + Live Activity captured |
748778
| iOS App Intents / Siri Shortcuts | ✅ compiles, `AppShortcutsProvider` registered |, | ⏸ Siri phrase invocation needs a device/simulator with Siri running |
749779
| Compose Desktop dashboard |`:desktopApp:desktopMain` compiles |`desktopTest` (host-rendered screenshot) |, (pure-JVM, no separate device verification needed) |
780+
| On-device LLM (chat streaming + document extraction) | ✅ compiles both platforms | ✅ unit-tested (`LlmAssistantEngineTest`, `DocumentIntelligenceTest`) | ⏸ Gemini Nano and Apple Foundation Models both need eligible hardware, none available here; the BYOK cloud fallback is the only tier ever exercised live |
750781
| Accessibility sweep (Android + iOS/watchOS surfaces) | ✅ compiles |, | ⏸ manual VoiceOver/TalkBack walkthrough documented inline; no automated a11y audit target yet |
751782

752783
## The location engine

0 commit comments

Comments
 (0)