Skip to content

Commit 76fc623

Browse files
Updated PromptFormatter and LlamaChatSessionImpl to support explicit assistant turn lifecycles and thinking-block extraction.
* **SDK**: * Enhanced `PromptFormat` and `PromptFormatter` to handle assistant prefixes/suffixes separately and support `<think>` tags. * Updated `LlamaChatSessionImpl` to leverage native EOG token decoding for turn closing, reducing Kotlin-side logic. * Removed `systemPromptReserve` from `DecodeConfig` and `LlamaSession` in favor of simplified context management. * Fixed `PromptFormatter` to conditionally add BOS tokens based on the model's chat template. * Reduced default `contextSize` to 2048. * Added Android logging utilities to the native C++ layer. * **App**: * Redesigned `ChatScreen` UI with a new message input style, copy-to-clipboard functionality, and improved markdown rendering. * Added support for displaying thinking blocks in the chat history. * Updated `AppNavigation` to reactively handle model loading states. * Modified `ChatViewModel` system prompt and message mapping.
1 parent 8ca98d7 commit 76fc623

19 files changed

Lines changed: 445 additions & 180 deletions

File tree

CLAUDE.md

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## Project Overview
6+
7+
**Llama Bro** is an Android SDK for on-device LLM inference, wrapping [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI. It consists of two modules:
8+
- **`:sdk`** — reusable Android library (published to JitPack)
9+
- **`:app`** — demo application showcasing the SDK
10+
11+
The llama.cpp engine is vendored as a git submodule at `sdk/src/main/cpp/external/llama.cpp`.
12+
13+
## Build Commands
14+
15+
```bash
16+
# Initialize submodules (required after clone)
17+
git submodule update --init --recursive
18+
19+
# Build SDK AAR
20+
./gradlew :sdk:assembleRelease
21+
22+
# Build demo app (debug)
23+
./gradlew :app:assembleDebug
24+
25+
# Install debug app on connected device
26+
./gradlew :app:installDebug
27+
28+
# Run unit tests (SDK only; no instrumentation tests exist)
29+
./gradlew :sdk:test
30+
31+
# Run a single test class
32+
./gradlew :sdk:test --tests "com.suhel.llamabro.sdk.util.PromptFormatterTest"
33+
34+
# Publish SDK to local Maven repository (used by JitPack)
35+
./gradlew :sdk:publishToMavenLocal
36+
37+
# Clean
38+
./gradlew clean
39+
```
40+
41+
**NDK requirement:** NDK 29.0.14206865 and CMake 3.22.1 must be installed via the Android SDK Manager. The project only builds for `arm64-v8a` — x86_64 emulators are not supported.
42+
43+
## Architecture
44+
45+
### Layer Stack
46+
47+
```
48+
UI (Jetpack Compose)
49+
50+
ViewModels (MVVM, Hilt-injected)
51+
52+
Repositories (ChatRepository, ModelRepository)
53+
54+
SDK Public API ─────────────────────────────────────────────
55+
LlamaEngine → LlamaSession → LlamaChatSession
56+
57+
JNI Bridge (llama_engine_jni.cpp, llama_session_jni.cpp)
58+
59+
Native C++ (session.cpp, engine.cpp → llama.cpp)
60+
```
61+
62+
### SDK Module
63+
64+
Three tiers of API, each building on the previous:
65+
66+
1. **`LlamaEngine`** — loads a GGUF model file from disk; creates sessions. Use `LlamaEngine.createFlow(modelConfig)` for reactive loading that emits `ResourceState<LlamaEngine>`.
67+
68+
2. **`LlamaSession`** — low-level token control. Call `setSystemPrompt()`, then loop `prompt()` + `generate()` to produce tokens. Wrap using `createChatSession()` to get the high-level API.
69+
70+
3. **`LlamaChatSession`** — high-level conversational API. `completion(message)` returns `Flow<Completion>`. Handles prompt template formatting, thinking-block extraction (`<think>...</think>`), and `OverflowStrategy`.
71+
72+
**`ResourceState<T>`** is the lifecycle ADT used throughout. It has subtypes `Loading(progress)`, `Success(value)`, `Failure(error)` and rich Flow extension operators (`flatMapResource`, `filterSuccess`, etc.) for composing resource loads.
73+
74+
**`PromptFormat`** / **`PromptFormats`** — chat template definitions. Built-in formats: `Llama3`, `Gemma3`, `ChatML` (Qwen/Yi), `Mistral`. Each model in `ModelZoo` references one of these.
75+
76+
**`LlamaError`** — sealed error hierarchy (`ModelNotFound`, `ModelLoadFailed`, `ContextOverflow`, `DecodeFailed`, `Cancelled`, `NativeException`, etc.).
77+
78+
### App Module
79+
80+
Standard MVVM with Hilt DI:
81+
82+
- **`ModelRepository`** — singleton managing model download/load/eject lifecycle. Download state is a FSM: `NotDownloaded → Downloading → Downloaded`. The currently loaded engine is exposed as `currentInferenceContextFlow: StateFlow<CurrentInferenceContext?>`.
83+
- **`ChatRepository`** — Room-backed CRUD for conversations and messages.
84+
- **`ModelZoo`** — hardcoded list of 6 pre-curated GGUF models (SmolLM2 135M–1.7B, Qwen2.5 0.5B, Llama-3.2 1B, DeepSeek-R1 1.5B) with download URLs and default configs.
85+
- Navigation uses type-safe `Route` sealed class with Jetpack Navigation Compose.
86+
87+
### JNI / Native
88+
89+
- `sdk/src/main/cpp/jni/` — JNI entry points that convert Kotlin calls to C++ and map C++ exceptions to `LlamaError` subtypes via `NativeErrorMapper`.
90+
- `sdk/src/main/cpp/session.cpp` — C++ session implementation wrapping `llama_context`.
91+
- OpenCL/GPU is intentionally disabled (causes UI stalls on mobile). OpenMP multi-threading is enabled.
92+
93+
## Key Configuration Classes
94+
95+
| Class | Purpose |
96+
|---|---|
97+
| `ModelConfig` | Model path, `PromptFormat`, MMAP/MLOCK flags, thread count |
98+
| `SessionConfig` | Context size, `OverflowStrategy`, `InferenceConfig`, `DecodeConfig` |
99+
| `InferenceConfig` | Temperature, top-p/k, min-p, repeat penalty |
100+
| `DecodeConfig` | Batch sizes for performance tuning |
101+
| `PromptFormat` | Per-role prefix/suffix tokens, BOS/EOS, `<think>` tag markers |
102+
103+
## Testing
104+
105+
Unit tests live in `sdk/src/test/` — currently only `PromptFormatterTest` covering chat template formatting. There are no instrumentation tests. New SDK behavior should be covered in this test source set.

0 commit comments

Comments
 (0)