Skip to content

Commit d678132

Browse files
GottZclaude
andcommitted
feat(graph): load-all — flacher Voll-Korpus-Seed + SPA-Button
GET /api/graph/all liefert JEDEN sichtbaren Block (kanonische Visibility-Triple + Ego-Node-Filter) newest-first mit id-Tiebreak, dazu induzierte Kanten beider Datenklassen und sichtbare Degrees. Keine Traversal: das Node-Set ist prädikat-definiert, nicht erreichbarkeits-definiert — verlinkungsfreie Blöcke und getrennte Komponenten erscheinen mit, was der Ego-Pfad strukturell nie liefern kann. Edges/Degrees laufen durch die GETEILTEN SQL-Stages (egoSQLEdges = Q2/Q2s/Q3 wörtlich): keine zweite Visibility-Wahrheit. Budgets: limit UND edge_limit defaulten auf ihre Decken (1500 bewiesen per G39-Bench; 20000 = harte Client-Kante evict §6.6) — ein 4000-Edge-Default würde exakt den Korpus kappen, den der Button zeigen soll (live heute ~3.8k Links). Ceilings bleiben 400, nie geklemmt; bei Korpus > Decke bleibt der Endpoint eine budgetierte Sicht (stats.truncated + budget_report), kein unbegrenzter Dump @1m. Envelope = Ego-Envelope mit focus:"" (params.hops/per_node_cap echoen 0 = nicht anwendbar); mergeEgo toleriert den leeren Focus (Seed am Ursprung). SPA: "load all"-Button in der Meta-Row — gleicher Merge/Settle-Pfad wie expand, Kamera-Reset (Korpus-Sicht, keine Nachbarschaft), disconnected Nodes lesen hop=Infinity und stehen damit automatisch vorn in der evict-farthest-first-Ordnung. Tests: FullGraph-Integrationssuite (Scope-Sichtbarkeit, Kanten nur zwischen sichtbaren Endpunkten, deterministische newest-first-Kappung, T07/T6 fail-closed), parseAllParams-Unit-Pins (Ceiling-Defaults, keine Klemmung, GB5-Partition), fetchGraphAll-Serialisierungs-Pins. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 82695d7 commit d678132

9 files changed

Lines changed: 497 additions & 7 deletions

File tree

docs/api.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ All endpoints under `/api/*`. Auth via `X-Context-Key` header or `Authorization:
1010
| `POST /api/store` | Upsert (embedding async via scheduler). Optional `sensitivity` (`credentials`\|`personal`\|`internal`\|`public`) classifies the block manually (`sensitivity_source='manual'`); absent ⇒ settings key `pool.default_block_sensitivity` (fail-closed `credentials`). On an upsert conflict an explicit value applies upgrade-only — downgrades go through `manage update` with `confirm_sensitivity_downgrade`. A credentials pattern in the content forces `credentials` upgrade-only regardless of the requested level (G40 detector, `sensitivity_source='pattern'`). Optional `type` validates against the registry and sets `type_source='manual'`. |
1111
| `POST /api/search` | Lightweight search (no LLM). Results carry the type axes (`type`/`lifecycle_state`/`type_source`). Optional `types` / `types_exclude` arrays are server-side opt-in type filters (bind parameters; `block_roles_exclude` = legacy alias, both ⇒ union). |
1212
| `GET /api/graph/ego` | Scope-filtered k-hop ego subgraph over dream links (read-only, no LLM — see [Graph API](#graph-api)). |
13+
| `GET /api/graph/all` | Flat load-all seed: every visible block up to the budget ceilings (newest-first, deterministic), induced edges of both data classes, degrees. Same envelope as ego with `focus:""`; no traversal params (see [Load-all](#load-all-flat-corpus-seed)). |
1314
| `GET /api/graph/overview` | Scope-pure Louvain cluster supergraph ("landkarte"); reads precomputed scope-partitioned aggregates, gated on `graph_overview.enabled` (off → 404). Read-only, no LLM. |
1415
| `POST /api/synthesize/daily` | Manual trigger of the daily synthesis (Tagesbericht) for the caller's `home_scope` — the same `GenerateDailyReport` path the 03:00 scheduler iteration runs (digest role, constant internal). Concurrency cap 1 per principal (429 fail-fast), dispatch-interactive (429 + `Retry-After` on capacity rejection). Optional `?date=YYYY-MM-DD` (M103) re-synthesizes the report titled that day over its historical window `[date-1 03:00 UTC, date 03:00 UTC)` — the backfill path; the `(category, title, scope)` upsert replaces the old report in place (same block id, embedding regenerates). A malformed date or a window that has not closed yet ⇒ 400. `{ok, block_id, scope[, date][, reason:"no_activity"]}`. |
1516
| `GET /api/whoami` | Calling key's identity: `label`, `home_scope`, `read_scopes`, the server-global `admin` tier flag, plus the Model-C tenant identity `tenant_id` + per-tenant `role` (`owner`/`admin`/`member`), and the feature-flag bag `capabilities` (`{workflow: true}` since v4.3.0 — switches the SPA's issues/board surface visible; the field is data, so a later per-tenant gate only changes the value, not the SPA). The SPA login gate probes it, derives its read-only degradation from `admin`, and tells server-admin from tenant-admin. |
@@ -214,6 +215,12 @@ Since the graph-budget wave the envelope also carries `budget_report` — the ty
214215

215216
**Security semantics.** The visibility triple (not archived, block type on the registry visibility allowlist, scope readable by the key) is applied inside every hop AND inside the per-node cap legs — a node reachable only through a foreign private bridge is never delivered, and invisible edges never consume cap slots. `degree` counts only visible neighbors (scan budget 1000 raw edges/direction). "Does not exist" and "not visible" answer with an identical `404` (no existence oracle); only successful calls write an access-log row (`action='graph'`, `block_id=NULL`).
216217

218+
### Load-all (flat corpus seed)
219+
220+
`GET /api/graph/all` returns the flat "load all" seed behind the SPA button of the same name: **every visible block** (canonical visibility triple + the ego node filters), newest-first with an id tiebreak so truncation is deterministic, plus the induced edges of both data classes and the visible degrees. Nothing is traversed — the node set is predicate-defined, not reachability-defined, so blocks without any link (and disconnected components) appear too.
221+
222+
Params are the ego set MINUS `block`/`hops`/`per_node_cap`: `limit` (default **1500** = the ceiling — the button's whole point), `edge_limit` (default **20000** = the ceiling, matching the client's hard edge budget), `min_confidence`, `link_class` (same unified GB5 channel), `category`, `created_after`/`created_before`. Ceilings are still enforced, never clamped; the corpus growing past a ceiling reports as `stats.truncated` + `budget_report` limits (`node_limit_reached`/`edge_limit_reached`) — the endpoint stays a budgeted view at 1M+, never an unbounded dump. Response: the ego envelope verbatim with `focus:""` and `params.hops`/`params.per_node_cap` echoed as 0 (not applicable). Same auth, same `graph` rate-limit bucket; there is no focus, hence no 404 arm.
223+
217224
### Overview (cluster "landkarte")
218225

219226
`GET /api/graph/overview` returns the cluster supergraph: a few hundred meta-nodes (precomputed Louvain communities over the dream-link graph) with `size`, `top_categories`, a representative block, and aggregated inter-cluster meta-edges. The Louvain rebuild runs offline in the scheduler (`internal/overview`, gonum); the endpoint only reads precomputed tables. Since WF T6 the node set is policy-cut: a block becomes a Louvain node only if its type is on the registry visibility allowlist AND carries `overview.include=true`.

go/cmd/ctxd/server.go

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -229,6 +229,8 @@ func NewRouter(ctx context.Context, pool *pgxpool.Pool, cfgStore *config.Store,
229229
r.Post("/api/search", searchH.HandleSearch)
230230
// Graph — scope-filtered k-hop ego subgraph (read-only, no LLM)
231231
r.Get("/api/graph/ego", graphH.HandleEgo)
232+
// Graph load-all — flat visible-corpus seed (SPA "load all" button)
233+
r.Get("/api/graph/all", graphH.HandleAll)
232234
// Graph overview — scope-pure Louvain cluster supergraph (F5-W6, gated)
233235
r.Get("/api/graph/overview", overviewH.HandleOverview)
234236
// Graph category-hue overrides (AM-2, U02-W5): GET is member-tier (the

go/internal/handler/graph.go

Lines changed: 100 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -80,11 +80,11 @@ func (h *GraphHandler) egoCache(cfg *config.Config) store.EgoCache {
8080

8181
// Parameter ceilings (out-of-range → 400, never silently clamped).
8282
const (
83-
egoDefaultHops = 1
84-
egoMaxHops = 3
85-
egoDefaultCap = 25
86-
egoMaxCap = 100
87-
egoDefaultLimit = 500
83+
egoDefaultHops = 1
84+
egoMaxHops = 3
85+
egoDefaultCap = 25
86+
egoMaxCap = 100
87+
egoDefaultLimit = 500
8888
// egoMaxLimit was lowered from 5000 to 1500 by the G39/F5-W5 1M-synthetic
8989
// benchmark: at 5000 nodes the worst-case pipeline (hops=3/cap=100, a dense
9090
// region with ≥5 degree-10^4 hubs) ran p95 ~850ms — dominated by Q3 (per-node
@@ -156,6 +156,65 @@ type egoStats struct {
156156
ElapsedMs int64 `json:"elapsed_ms"`
157157
}
158158

159+
// HandleAll processes GET /api/graph/all requests — the flat "load all" seed
160+
// (store.FullGraph): every visible block up to limit, induced edges, degrees.
161+
// Same auth/rate-limit/registry discipline as HandleEgo; no focus, so there is
162+
// no 404 arm and the access log carries an empty focus.
163+
func (h *GraphHandler) HandleAll(w http.ResponseWriter, r *http.Request) {
164+
ctx := r.Context()
165+
reqID := RequestIDFromContext(ctx)
166+
167+
authResult := AuthResultFromContext(ctx)
168+
if authResult == nil || !authResult.IsValid {
169+
writeJSON(w, http.StatusUnauthorized, map[string]any{"success": false, "error": "unauthorized"})
170+
return
171+
}
172+
173+
cfgSnap := h.cfg.SnapshotForRequest(ctx)
174+
175+
// Same "graph" action bucket as ego — a load-all IS a graph read.
176+
if limit := cfgSnap.Query.RateLimitRead; limit > 0 {
177+
readCount, err := store.CheckRateLimitByAction(ctx, h.pool, authResult.ApiKeyID, "graph")
178+
if err != nil {
179+
slog.Error("graph: read rate limit check error", "error", err, "request_id", reqID)
180+
writeJSON(w, http.StatusInternalServerError, map[string]any{"success": false, "error": "Internal server error"})
181+
return
182+
}
183+
if readCount >= limit {
184+
writeJSON(w, http.StatusTooManyRequests, map[string]any{
185+
"success": false,
186+
"error": fmt.Sprintf("Rate limit exceeded: max %d graph reads per 60 seconds", limit),
187+
})
188+
return
189+
}
190+
}
191+
192+
snap := h.blocktypes.SnapshotForRequest(ctx)
193+
params, rawLinkClass, err := parseAllParams(r.URL.Query(), snap.StructuralClasses())
194+
if err != nil {
195+
writeJSON(w, http.StatusBadRequest, map[string]any{"success": false, "error": err.Error()})
196+
return
197+
}
198+
199+
start := time.Now()
200+
// grantedBlockIDs nil — T40a parity with HandleEgo (grant resolution is a
201+
// later wave); the type allowlist comes from the registry snapshot (T6).
202+
result, err := store.FullGraph(ctx, h.pool, params, authResult.ReadScopes, nil, snap.VisibleTypes())
203+
if err != nil {
204+
slog.Error("graph: full query error", "error", err, "request_id", reqID)
205+
writeJSON(w, http.StatusInternalServerError, map[string]any{"success": false, "error": "Internal server error"})
206+
return
207+
}
208+
elapsedMs := time.Since(start).Milliseconds()
209+
210+
// Best effort, same bucket as ego (focus "" = load-all in the metadata).
211+
if err := store.LogGraphAccess(ctx, h.pool, authResult.ApiKeyID, "", 0, params.Limit, len(result.Nodes), len(result.StructEdges)); err != nil {
212+
slog.Error("graph: access log error", "error", err, "request_id", reqID)
213+
}
214+
215+
writeJSON(w, http.StatusOK, buildEgoResponse(result, params, rawLinkClass, elapsedMs))
216+
}
217+
159218
// HandleEgo processes GET /api/graph/ego requests.
160219
func (h *GraphHandler) HandleEgo(w http.ResponseWriter, r *http.Request) {
161220
ctx := r.Context()
@@ -350,6 +409,42 @@ func parseEgoParams(q url.Values, structVocab []string) (store.EgoParams, []stri
350409
return p, rawLinkClass, nil
351410
}
352411

412+
// parseAllParams validates the /api/graph/all query: the ego params MINUS
413+
// block/hops/per_node_cap (no focus, no traversal). limit defaults to the
414+
// CEILING — "load all" wants everything the benchmark-proven envelope allows
415+
// (egoMaxLimit, G39); the truncation flag tells the client what was cut.
416+
// Same ceilings-not-clamps discipline and the same link_class partition.
417+
func parseAllParams(q url.Values, structVocab []string) (store.EgoParams, []string, error) {
418+
p := store.EgoParams{}
419+
420+
var err error
421+
if p.Limit, err = egoIntParam(q, "limit", egoMaxLimit, 1, egoMaxLimit); err != nil {
422+
return p, nil, err
423+
}
424+
// edge_limit defaults to the CEILING too (not egoDefaultEdgeLimit): the
425+
// max equals the client's hard edge budget (evict, design 05-§6.6), and a
426+
// 4000-default would truncate the very corpus the button exists to show.
427+
if p.EdgeLimit, err = egoIntParam(q, "edge_limit", egoMaxEdgeLimit, 1, egoMaxEdgeLimit); err != nil {
428+
return p, nil, err
429+
}
430+
if p.MinConfidence, err = egoFloatParam(q, "min_confidence", 0, 0, 1); err != nil {
431+
return p, nil, err
432+
}
433+
var rawLinkClass []string
434+
if p.LinkClasses, p.StructClasses, rawLinkClass, err = egoLinkClassPartition(q.Get("link_class"), structVocab); err != nil {
435+
return p, nil, err
436+
}
437+
p.Categories = egoCSV(q.Get("category"))
438+
if p.CreatedAfter, err = egoTimeParam(q, "created_after"); err != nil {
439+
return p, nil, err
440+
}
441+
if p.CreatedBefore, err = egoTimeParam(q, "created_before"); err != nil {
442+
return p, nil, err
443+
}
444+
445+
return p, rawLinkClass, nil
446+
}
447+
353448
// egoIntParam parses an optional integer query parameter with inclusive
354449
// bounds. Absent or empty → def.
355450
func egoIntParam(q url.Values, name string, def, minVal, maxVal int) (int, error) {

go/internal/handler/graph_test.go

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -260,3 +260,60 @@ func TestEgoResponse_EmptyCollectionsAreArrays(t *testing.T) {
260260
}
261261
}
262262
}
263+
264+
// parseAllParams (load-all): no block/hops/per_node_cap; limit AND edge_limit
265+
// default to their ceilings (the button's whole point is "everything the
266+
// proven envelope allows"); the shared filter/partition pieces behave exactly
267+
// like ego.
268+
func TestParseAllParams(t *testing.T) {
269+
t.Run("Defaults", func(t *testing.T) {
270+
p, raw, err := parseAllParams(egoQuery(t, ""), nil)
271+
if err != nil {
272+
t.Fatalf("unexpected error: %v", err)
273+
}
274+
if p.Limit != egoMaxLimit || p.EdgeLimit != egoMaxEdgeLimit {
275+
t.Errorf("defaults wrong: limit=%d edge_limit=%d (want ceilings %d/%d)", p.Limit, p.EdgeLimit, egoMaxLimit, egoMaxEdgeLimit)
276+
}
277+
if p.Focus != "" || p.Hops != 0 || p.PerNodeCap != 0 {
278+
t.Errorf("traversal params must stay zero: %+v", p)
279+
}
280+
if raw != nil || p.LinkClasses != nil || p.Categories != nil {
281+
t.Error("optional filters must default to nil")
282+
}
283+
})
284+
285+
t.Run("CeilingsNotClamped", func(t *testing.T) {
286+
bad := []string{
287+
"limit=0", "limit=1501",
288+
"edge_limit=0", "edge_limit=20001",
289+
"min_confidence=1.5",
290+
"link_class=topical,unknown",
291+
"created_after=gestern",
292+
}
293+
for _, raw := range bad {
294+
if _, _, err := parseAllParams(egoQuery(t, raw), nil); err == nil {
295+
t.Errorf("query %q: expected error, got nil", raw)
296+
}
297+
}
298+
})
299+
300+
t.Run("SharedFilterPieces", func(t *testing.T) {
301+
p, raw, err := parseAllParams(
302+
egoQuery(t, "limit=10&min_confidence=0.5&link_class=topical,references&category=infra"),
303+
[]string{"references"},
304+
)
305+
if err != nil {
306+
t.Fatalf("unexpected error: %v", err)
307+
}
308+
if p.Limit != 10 || p.MinConfidence != 0.5 {
309+
t.Errorf("scalars wrong: %+v", p)
310+
}
311+
if len(p.LinkClasses) != 1 || p.LinkClasses[0] != "topical" ||
312+
len(p.StructClasses) != 1 || p.StructClasses[0] != "references" {
313+
t.Errorf("link_class partition wrong: dream=%v struct=%v", p.LinkClasses, p.StructClasses)
314+
}
315+
if len(raw) != 2 || len(p.Categories) != 1 || p.Categories[0] != "infra" {
316+
t.Errorf("raw echo / category wrong: raw=%v cat=%v", raw, p.Categories)
317+
}
318+
})
319+
}

go/internal/store/graph_all.go

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
// FullGraph — the flat "load all" seed behind GET /api/graph/all (SPA
2+
// load-all button). NO traversal: the node set is simply every visible block
3+
// (canonical visibility triple + the ego node filters), newest-first with an
4+
// id tiebreak so truncation is deterministic. Edges and degrees then run
5+
// through the SHARED SQL stages (egoSQLEdges = Q2/Q2s/Q3 verbatim) — scope
6+
// safety holds by the Q2 invariant (both endpoints passed the triple), and a
7+
// second copy of any visibility check would be a second truth; there is none.
8+
//
9+
// The bridge-leak / cap-starvation machinery of EgoGraph does not apply here:
10+
// nothing is traversed THROUGH, no per-node cap slot is granted — the node set
11+
// is predicate-defined, not reachability-defined. Deliberately SQL-only: the
12+
// W05.5 cache arm answers frontier walks, a flat listing gains nothing from it.
13+
package store
14+
15+
import (
16+
"context"
17+
"errors"
18+
"fmt"
19+
20+
"github.com/GottZ/ctx/internal/graphcache"
21+
"github.com/jackc/pgx/v5/pgxpool"
22+
)
23+
24+
// FullGraph returns up to p.Limit visible blocks (created_at DESC, id) plus
25+
// the induced edges of both data classes and the visible degrees. p.Focus,
26+
// p.Hops and p.PerNodeCap are ignored — EgoResult.Focus is "" (no focus
27+
// exists). Truncated reports node OR edge budget exhaustion, exactly like
28+
// EgoGraph.
29+
func FullGraph(ctx context.Context, pool *pgxpool.Pool, p EgoParams, readScopes, grantedBlockIDs, visibleTypes []string) (*EgoResult, error) {
30+
if err := RequireScopes(readScopes); err != nil { // T07 fail-closed (design/01 §5.4)
31+
return nil, err
32+
}
33+
if len(visibleTypes) == 0 {
34+
return nil, errors.New("store: empty visible-types allowlist (block-type registry not wired?)")
35+
}
36+
if grantedBlockIDs == nil {
37+
grantedBlockIDs = []string{} // deterministic '{}'::uuid[], never NULL
38+
}
39+
normalizeClassFilters(&p)
40+
41+
// Flat seed: the ego NODE filters (category, created window) apply; edge
42+
// filters (min_confidence, link_class) belong to the edge stages below.
43+
// LIMIT reads one row beyond the budget so the truncation flag is exact.
44+
q := fmt.Sprintf(
45+
`SELECT b.id::text, left(b.title, 120), b.category, b.scope::text, b.created_at
46+
FROM context_blocks b
47+
WHERE %s
48+
AND ($5::text[] IS NULL OR b.category = ANY($5))
49+
AND ($6::timestamptz IS NULL OR b.created_at >= $6)
50+
AND ($7::timestamptz IS NULL OR b.created_at < $7)
51+
ORDER BY b.created_at DESC, b.id
52+
LIMIT $4`,
53+
VisibilityPredicate("b", "$3", "$1", "$2"),
54+
)
55+
rows, err := pool.Query(ctx, q,
56+
readScopes, // $1
57+
grantedBlockIDs, // $2 block-grant OR-arm (T40a; nil-wired like ego)
58+
visibleTypes, // $3 registry type allowlist (T6)
59+
p.Limit+1, // $4 one extra row → exact truncation flag
60+
nilIfEmpty(p.Categories), // $5
61+
p.CreatedAfter, // $6
62+
p.CreatedBefore, // $7
63+
)
64+
if err != nil {
65+
return nil, fmt.Errorf("store: graph full seed query: %w", err)
66+
}
67+
defer rows.Close()
68+
69+
var nodes []GraphNode
70+
for rows.Next() {
71+
// Hop 1 for every node: there is no hop-0 focus, and the client seeds
72+
// new nodes on a hop-scaled ring — hop 0 would stack the whole corpus
73+
// on one point before the layout worker untangles it.
74+
n := GraphNode{Hop: 1}
75+
if err := rows.Scan(&n.ID, &n.Title, &n.Category, &n.Scope, &n.CreatedAt); err != nil {
76+
return nil, fmt.Errorf("store: graph full seed scan: %w", err)
77+
}
78+
nodes = append(nodes, n)
79+
}
80+
if err := rows.Err(); err != nil {
81+
return nil, fmt.Errorf("store: graph full seed rows: %w", err)
82+
}
83+
84+
budget := graphcache.NewBudgetReport(graphcache.SourceSQL)
85+
truncated := false
86+
if len(nodes) > p.Limit {
87+
nodes = nodes[:p.Limit]
88+
truncated = true
89+
budget.Add(graphcache.TravNodeLimitReached)
90+
}
91+
92+
ids := make([]string, len(nodes))
93+
index := make(map[string]int, len(nodes))
94+
for i := range nodes {
95+
ids[i] = nodes[i].ID
96+
index[nodes[i].ID] = i
97+
}
98+
99+
sql := &egoSQLEdges{pool: pool, p: p, readScopes: readScopes, grantedBlockIDs: grantedBlockIDs, visibleTypes: visibleTypes}
100+
es, err := sql.edges(ctx, ids, index)
101+
if err != nil {
102+
return nil, err
103+
}
104+
if es.DreamTrunc {
105+
truncated = true
106+
budget.Add(graphcache.TravEdgeLimitReached)
107+
}
108+
if es.StructTrunc {
109+
truncated = true
110+
budget.Add(graphcache.TravEdgeLimitReached)
111+
}
112+
if err := sql.degrees(ctx, ids, nodes); err != nil {
113+
return nil, err
114+
}
115+
116+
return &EgoResult{
117+
Focus: "",
118+
Rels: GraphRels,
119+
StructRels: es.StructRels,
120+
Origins: es.Origins,
121+
Nodes: nodes,
122+
Edges: es.Dream,
123+
StructEdges: es.Struct,
124+
Truncated: truncated,
125+
Budget: budget,
126+
}, nil
127+
}

0 commit comments

Comments
 (0)