Skip to content

Commit 3f2e31a

Browse files
Merge pull request #408 from appdevforall/feat/ADFA-5094-box-tree-import
ADFA-5094: offline topic tree with sizes — box serves Studio-shaped subtrees
2 parents c9fbf85 + 021f49f commit 3f2e31a

5 files changed

Lines changed: 175 additions & 13 deletions

File tree

‎controller/app/src/main/java/org/iiab/controller/kolibri/data/LocalTreeSource.java‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -46,8 +46,10 @@ public final class LocalTreeSource implements TreeSource {
4646

4747
private static final String TAG = "K2Go-Kolibri";
4848

49-
/** Endpoint under the box API base: {@code /k2go-api/kolibri/tree/<nodeId>}. */
50-
private static final String TREE_PATH = "/kolibri/tree/";
49+
// Endpoint under the box API base: /k2go-api/kolibri/subtree/<nodeId>. Deliberately NOT
50+
// /kolibri/tree/:channelId — that one is the web wizard's granular tree (counts, no bytes).
51+
// This one returns Studio-shaped JSON with byte sizes, so StudioCatalogMapper parses it (ADFA-5094).
52+
private static final String TREE_PATH = "/kolibri/subtree/";
5153

5254
/** Box API base ({@code …:8085/k2go-api}); overridable so a test can point at a mock. */
5355
private static final String DEFAULT_BASE = BoxEndpoints.API;

‎controller/app/src/test/java/org/iiab/controller/kolibri/data/LocalTreeSourceTest.java‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@
3030
* the same treatment {@link StudioTreeSourceTest} gives the internet source.
3131
*
3232
* <p>Two things are specific to the local source and worth pinning: it addresses
33-
* the box's {@code /k2go-api/kolibri/tree/} path, and a 404 — the ordinary case
33+
* the box's {@code /k2go-api/kolibri/subtree/} path, and a 404 — the ordinary case
3434
* where a channel's metadata is not on the box — comes back as {@code null} so
3535
* {@link FallbackTreeSource} moves on to Studio, rather than as an error. The
3636
* JSON parsing itself is Studio's mapper and is exercised there; here the focus
@@ -84,7 +84,7 @@ public void readsATreeAndHitsTheBoxPath() throws Exception {
8484
assertEquals(800L, t.subtreeBytes());
8585

8686
RecordedRequest req = server.takeRequest();
87-
assertEquals("/k2go-api/kolibri/tree/" + CH, req.getPath());
87+
assertEquals("/k2go-api/kolibri/subtree/" + CH, req.getPath());
8888
assertEquals("GET", req.getMethod());
8989
assertEquals("application/json", req.getHeader("Accept"));
9090
}
@@ -125,6 +125,6 @@ public void aBaseUrlWithoutATrailingSlashWorksToo() throws Exception {
125125
String noSlash = server.url("/k2go-api").toString();
126126
server.enqueue(json(treeJson()));
127127
assertNotNull(new LocalTreeSource(noSlash).fetchTree(CH));
128-
assertEquals("/k2go-api/kolibri/tree/" + CH, server.takeRequest().getPath());
128+
assertEquals("/k2go-api/kolibri/subtree/" + CH, server.takeRequest().getPath());
129129
}
130130
}

‎controller/docs/ADR-5094-offline-tree-depth.md‎

Lines changed: 17 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -25,11 +25,22 @@ So the tree can live on the box at a fraction of the content's size.
2525
onto the box without downloading content. The box then knows the whole tree offline; the
2626
content is fetched later, subtree by subtree, as it is today.
2727

28-
**D2 — Box serves the tree, Studio-shaped (PR3).** The box answers
29-
`GET /k2go-api/kolibri/tree/:nodeId` with the **same JSON as Studio's `contentnode_tree`**.
30-
Not a new shape: `StudioCatalogMapper.tree(JSONObject)` then parses box and Studio
31-
responses with no branching, and the device side stays a thin read. The box owns the
32-
translation from Kolibri's own model into that shape.
28+
**D2 — Box serves the tree, Studio-shaped, from the local DB (PR3).** The box answers
29+
`GET /k2go-api/kolibri/subtree/:nodeId` with the **same JSON as Studio's `contentnode_tree`**,
30+
built from the local Kolibri content database. `StudioCatalogMapper.tree(JSONObject)` then
31+
parses box and Studio responses with no branching, and the device side stays a thin read.
32+
33+
*Corrected after reading the box (PR3 exploration):*
34+
- **The path is `/kolibri/subtree/:nodeId`, not `/kolibri/tree/:nodeId`.** A
35+
`GET /kolibri/tree/:channelId` already exists and serves the web wizard a *granular* tree
36+
(`contentnode_granular`: resource counts, camelCase, **no byte sizes**). Reusing it would
37+
collide and give the app the wrong shape, so the app path is a distinct, nodeId-keyed one.
38+
- **Byte sizes come from raw SQL, not a Kolibri API.** The granular endpoint carries no
39+
bytes, and `importexportsizeview` reports only a selection's *outstanding* transfer, not
40+
per-node structural size. So the box sums `content_localfile.file_size` over each node's
41+
MPTT (`lft`/`rght`) range and emits it in Studio's `files[].file_size` field. This is what
42+
gives the picker byte sizes at every level offline — the reason for choosing this over an
43+
app-side cache of the granular tree.
3344

3445
**D3 — App routes local-first (app, PR2, this PR).** `fetchTree` prefers the box and falls
3546
back to Studio. Concretely: a `TreeSource` seam with two implementations — `LocalTreeSource`
@@ -39,7 +50,7 @@ Every source keeps the "never throws, null on failure" contract, so a box miss
3950
channel not imported — falls through to Studio silently.
4051

4152
**D4 — Two independent PRs.** PR2 is the app side (this). PR3 is the box/dashboard side
42-
(D1, D2). They are independent: with only PR2 merged the box has no `/tree` endpoint, so
53+
(D1, D2). They are independent: with only PR2 merged the box has no `/subtree` endpoint, so
4354
`LocalTreeSource` always misses and behaviour is exactly what it was before — the tree
4455
comes from Studio. Nothing to gate, nothing to break; PR3 lights the local path up.
4556

‎static/dashboard/routes.ts‎

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,8 @@ import { searchCatalog, listLibrary, removeBook, listLanguages, getCalibreSessio
1313
import { parseBox, parseEstimate } from './sockets/maps.socket';
1414
import {
1515
preflight, listInstalledChannels, browseRemoteChannels, resolveIdentifier,
16-
browseChannelTree, estimateSelection, deleteChannel, getKolibriTask, verifyCredentials,
17-
ChannelNotInstalledError,
16+
browseChannelTree, buildLocalSubtree, estimateSelection, deleteChannel, getKolibriTask,
17+
verifyCredentials, ChannelNotInstalledError,
1818
} from './sockets/kolibri.query';
1919
import { checkReadiness, KolibriAuthError, KolibriApiError, login as kolibriLogin } from './sockets/kolibri.session';
2020
import {
@@ -472,6 +472,26 @@ apiRouter.get('/kolibri/tree/:channelId', async (req: Request, res: Response): P
472472
}
473473
});
474474

475+
// Studio-shaped subtree WITH byte sizes, built from the local DB (ADFA-5094). Distinct from
476+
// /kolibri/tree/:channelId above (granular, for the web wizard, no bytes): the app's
477+
// LocalTreeSource parses this with the same mapper it uses for Studio, so the picker shows
478+
// sizes at every level offline. Node-id keyed; 404 when the channel's metadata is not imported,
479+
// which the app reads as "fall back to Studio".
480+
apiRouter.get('/kolibri/subtree/:nodeId', (req: Request, res: Response): void => {
481+
const nodeId = String(req.params.nodeId);
482+
if (!/^[0-9a-f]{32}$/.test(nodeId)) {
483+
res.status(400).json({ error: 'invalid node id' });
484+
return;
485+
}
486+
try {
487+
const tree = buildLocalSubtree(nodeId);
488+
if (!tree) { res.status(404).json({ error: 'channel not imported' }); return; }
489+
res.json(tree);
490+
} catch (e: any) {
491+
res.status(500).json({ error: e?.message || 'subtree build failed' });
492+
}
493+
});
494+
475495
// What a selection still has to transfer, plus free space, for the consent step.
476496
// Body: { channelId, nodeIds?, excludeNodeIds? }
477497
apiRouter.post('/kolibri/estimate', async (req: Request, res: Response): Promise<void> => {

‎static/dashboard/sockets/kolibri.query.ts‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -226,6 +226,135 @@ export async function browseChannelTree(
226226
return toTreeNode(node);
227227
}
228228

229+
// ---- Offline subtree, Studio-shaped, with byte sizes (ADFA-5094) ------------------
230+
// The app's LocalTreeSource reads this and parses it with the SAME mapper it uses for
231+
// Studio, so the wire shape here is Studio's public contentnode_tree — not the granular
232+
// TreeNode above (which the web wizard uses and which carries no bytes).
233+
//
234+
// Why raw SQL and not a Kolibri API: the granular endpoint has no byte sizes, and
235+
// importexportsizeview reports only a selection's OUTSTANDING transfer, never per-node
236+
// structural size. The per-node bytes live in content_localfile.file_size, so they are
237+
// summed here over each node's MPTT (lft/rght) range. That is what lets the picker show
238+
// sizes at every level with no network.
239+
//
240+
// NOTE (verification): the SQL runs against a live Kolibri database, which cannot be
241+
// exercised in CI — it is validated on a box with a metadata-imported channel. If a column
242+
// or the schema differs, this throws, the route returns 500, and LocalTreeSource falls back
243+
// to Studio: a wrong query degrades to today's behaviour, it does not break the picker.
244+
245+
interface StudioFile { file_size: number; preset: string; }
246+
interface StudioNode {
247+
id: string;
248+
title: string;
249+
kind: string;
250+
is_leaf: boolean;
251+
lft: number;
252+
rght: number;
253+
files: StudioFile[];
254+
children: { more: unknown | null; results: StudioNode[] };
255+
}
256+
257+
interface CnRow {
258+
id: string;
259+
parent_id: string | null;
260+
title: string;
261+
kind: string;
262+
lft: number;
263+
rght: number;
264+
}
265+
interface FileRow { node_id: string; preset: string; file_size: number; }
266+
267+
// A generous ceiling that keeps the JSON well under LocalTreeSource's 8 MB cap while
268+
// covering ordinary channels whole. A subtree larger than this is truncated and its root
269+
// is marked incomplete, so the mapper reports the size as unknown rather than under-counting.
270+
const SUBTREE_NODE_CAP = 20000;
271+
272+
/**
273+
* One channel subtree rooted at {@code nodeId}, in Studio's contentnode_tree shape, built
274+
* from the local content database. Returns {@code null} when the node is not present — i.e.
275+
* the channel's metadata has not been imported — which the app reads as "fall back to Studio".
276+
*
277+
* The channel is scoped by the root node's own channel_id + MPTT range, so a node id alone
278+
* is enough (as with Studio), and no channelId has to be threaded from the caller.
279+
*/
280+
export function buildLocalSubtree(nodeId: string): StudioNode | null {
281+
if (!fs.existsSync(MAIN_DB)) return null;
282+
const db = new Database(MAIN_DB, { readonly: true });
283+
try {
284+
const root = db.prepare(
285+
`SELECT id, parent_id, title, kind, lft, rght, channel_id
286+
FROM content_contentnode WHERE id = ?`,
287+
).get(nodeId) as (CnRow & { channel_id: string }) | undefined;
288+
if (!root) return null;
289+
290+
const rows = db.prepare(
291+
`SELECT id, parent_id, title, kind, lft, rght
292+
FROM content_contentnode
293+
WHERE channel_id = ? AND lft >= ? AND rght <= ?
294+
ORDER BY lft
295+
LIMIT ?`,
296+
).all(root.channel_id, root.lft, root.rght, SUBTREE_NODE_CAP) as CnRow[];
297+
298+
// Real content files only: thumbnails and supplementary files are excluded so the
299+
// sizes match a normal import, the same exclusion StudioCatalogMapper makes.
300+
// No LIMIT on the subquery (unlike the rows query): it must cover every node the rows
301+
// query kept — an ORDER-BY-less LIMIT here could select a different subset and leave a
302+
// built node without its files, under-counting its size. Files for nodes past the cap
303+
// are simply never looked up.
304+
const files = db.prepare(
305+
`SELECT f.contentnode_id AS node_id, f.preset AS preset, lf.file_size AS file_size
306+
FROM content_file f
307+
JOIN content_localfile lf ON lf.id = f.local_file_id
308+
WHERE f.thumbnail = 0 AND f.supplementary = 0
309+
AND f.contentnode_id IN (
310+
SELECT id FROM content_contentnode
311+
WHERE channel_id = ? AND lft >= ? AND rght <= ?)`,
312+
).all(root.channel_id, root.lft, root.rght) as FileRow[];
313+
314+
const filesByNode = new Map<string, StudioFile[]>();
315+
for (const f of files) {
316+
const arr = filesByNode.get(f.node_id) ?? [];
317+
arr.push({ file_size: f.file_size ?? 0, preset: f.preset ?? '' });
318+
filesByNode.set(f.node_id, arr);
319+
}
320+
321+
const byId = new Map<string, StudioNode>();
322+
for (const r of rows) {
323+
byId.set(r.id, {
324+
id: r.id,
325+
title: r.title ?? '',
326+
kind: r.kind ?? '',
327+
is_leaf: r.kind !== 'topic',
328+
lft: r.lft,
329+
rght: r.rght,
330+
files: filesByNode.get(r.id) ?? [],
331+
children: { more: null, results: [] },
332+
});
333+
}
334+
for (const r of rows) {
335+
if (r.id === root.id || !r.parent_id) continue;
336+
byId.get(r.parent_id)?.children.results.push(byId.get(r.id)!);
337+
}
338+
339+
const rootNode = byId.get(root.id) ?? null;
340+
// Truncation correctness. Rows are ordered by lft and capped, so the dropped nodes are
341+
// exactly those with lft beyond the last one kept. Any node whose range (lft..rght)
342+
// extends past that cutoff therefore has a missing descendant, so mark it incomplete —
343+
// the mapper then reports its subtree size as unknown instead of silently under-counting.
344+
// Over-marking is safe; under-marking is the bug. Nodes fully within the cutoff keep a
345+
// complete child set and an exact size.
346+
if (rows.length >= SUBTREE_NODE_CAP) {
347+
const cutoffLft = rows[rows.length - 1].lft;
348+
for (const node of byId.values()) {
349+
if (node.rght > cutoffLft) node.children.more = { cursor: 'truncated' };
350+
}
351+
}
352+
return rootNode;
353+
} finally {
354+
db.close();
355+
}
356+
}
357+
229358
export interface SelectionSize {
230359
resourceCount: number;
231360
fileSize: number;

0 commit comments

Comments
 (0)