Refactor/per module compile - #1
Merged
Merged
Conversation
…esolution Phase 0 of per-module compilation refactor. All AST resolution types now use QualifiedName for top-level references instead of global counter-based IDs (VariableId, TypeId, RequestId, ConstructorId, ModuleId). - VariableRef resolution: Maybe VariableId → Maybe VariableResolution (ResolvedTopLevel QualifiedName | ResolvedLocal LocalVarId) - TypeRef: Maybe TypeId → Maybe QualifiedName - ModuleRef: Maybe ModuleId → Maybe Text - RequestRef: Maybe RequestId → Maybe QualifiedName - ConstructorRef: Maybe ConstructorId → Maybe QualifiedName Internal maps (IdentifierResult, type environment, etc.) still use legacy ID types as keys with reverse-lookup maps for conversion. This is the foundation for eliminating global state and enabling fully parallel per-module compilation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…Phase 1) - add topologicalSort to ImportGraph (level-grouped for parallelism) - add scanExportNames for lightweight pre-pass (no ID allocation) - add identifyModule: identifies one module independently - rewrite identify as sequential identifyModule loop (stdlib first, then user modules in topo order) - remove monolithic buildExports/buildTopLevels/resolveModule orchestrators - all 730 tests pass Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- add ModuleInterface type (exportedTypes per module) - add generateConstraintsForModule (single-module CG with imported type injection) - rewrite compile to typecheck modules in topological order - each module's CG injects upstream resolved types as concrete (no cross-module type vars) - Solver/Zonker unchanged (receive per-module constraint sets) - all 730 tests pass Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- add AgentGraph.hs: compute intra-module agent call graph SCCs - add generateConstraintsForSCC: CG scoped to one SCC only - rewrite per-module typecheck to iterate over agent SCCs in topo order - each SCC's resolved types are available to downstream SCCs - non-recursive agents are singleton SCCs (smallest possible solver input) - all 730 tests pass Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- add CompileLog type with per-phase/per-module/per-SCC log entries - add compileLogs field to CompileResult - accumulate logs during typecheck loop (module name, SCC index/total) - document speculative lowering insertion point (par/pseq) - verify: IR suppression only on Error-level diagnostics (not warnings) - all 730 tests pass Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- add ModuleCache type (source hash, interface, identified/zonked AST) - add cache field to CompileInput, updatedCache to CompileResult - skip recompilation when source hash matches and upstream cache is valid - build fresh cache entries on recompile for future runs - cache is in-memory only (disk persistence is caller's responsibility) - all 730 tests pass Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…types - remove TypeRef/ConstructorRef/RequestRef edges from agent call graph (data/request types are fully declared, no inference needed) - pre-compute non-agent types (data, request, external, prim) before the agent SCC loop so they're available as known facts - SCC graph now contains only value-level call edges - all 730 tests pass Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace VariableId/TypeId/RequestId/ConstructorId/ModuleId with QualifiedName throughout the internal data structures: - IdentifierResult/State: all maps keyed by QualifiedName or Text - SymbolEntry: slots use QualifiedName/VariableResolution/Text - SemanticType: SemanticTypeData(QualifiedName), SemanticRequestElementConcrete(QualifiedName) - ConstraintGenerator: type environment keyed by VariableResolution - Solver: request substitution uses Set QualifiedName - Zonker: ZonkResult maps keyed by Text/VariableResolution - Render: renderSemanticType/renderSemanticRequest no longer need name maps - Remove reverse-lookup maps — redundant with QualifiedName keys - Fix duplicate detection for QualifiedName-keyed maps - Fix test helpers for VariableResolution-based lookups Identifier is now fully parallel-ready: no global counters, no shared mutable state between modules. All 730 tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lowering: - restructure lowerProgramM to process per-module internally - extract lowerOneDeclaration as top-level function - remove dead lowerAllDeclarations Parallel execution: - add parallel dependency - typecheck modules within same topo level via parMap rseq - add ModuleTypecheckResult for independent per-module results - merge results after each parallel level LSP in-memory cache: - add wsCompileCache to WorkspaceState - thread updatedCache from CompileResult back into next compile - unchanged modules skip typechecking across recompiles All 730 tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Update Cache module to define project-local cache paths under .katari/ - Replace defaultCachePaths with projectCachePaths to support project-specific cache locations - Ensure cache directories are created on demand in the build and check commands - Introduce CompileCache module for managing disk-backed compile cache - Modify build and check commands to utilize compile cache for improved performance - Update .gitignore to include .katari/ directory - Add CompileCache functionality to load and save cache entries for modules
- Removed unused imports and unnecessary code in Katari.Id, Katari.Lowering, Katari.Schema, Katari.SemanticType.Render, Katari.Typechecker.ConstraintGenerator, Katari.Typechecker.Identifier, Katari.Typechecker.Solver, Katari.Typechecker.Zonker, and Katari.LSP handlers. - Improved code readability by restructuring some function definitions and using qualified imports consistently. - Updated the pre-commit script to format Haskell files using ormolu before committing.
- Updated import statements in various files to use qualified imports for better clarity and to avoid name clashes. - This change enhances code readability and maintains consistency in the codebase.
…stive and Identifier modules
These ID types were made unnecessary by the QualifiedName-based per-module compile refactor. TypeData, RequestData, ConstructorData no longer carry int IDs; IdentifierState no longer tracks counters for them. Lowering uses Set QualifiedName for membership checks instead of Map QualifiedName Id. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ole-program identify Remove `identify`, `identifyIncremental`, and `CachedIdentifierData` from Identifier module. Compile.hs now orchestrates per-module identification directly via `identifyModule`, unifying fresh and cached paths in a single `runIdentify` function. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Remove generateConstraintsForModule from CG exports. Tests now use Compile.generateConstraintsAll (which delegates to CG.generateConstraints) instead of importing generateConstraints directly from CG. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…t/compileLogs compile now takes a (CompileLog -> IO ()) callback for real-time progress logging. CompileResult no longer carries solverResult or compileLogs fields. Tests use compileSync (unsafePerformIO wrapper) for backward compatibility. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The zonkedModuleNames field was always Map.mapWithKey const, making every lookup return the input key unchanged. Removed the field and simplified Lowering.lowerModuleM to use moduleName directly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add Compile.renderCompileLog and wire the CLI commands (check, build, apply) to print per-phase progress to stderr — matching the convention of GHC and cargo (stdout reserved for build artefacts). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CompileLog constructors now carry the module name. Each phase (parsing, identifying, typechecking, lowering, schema) emits one line per cache-miss module. Typechecking is collapsed from per-SCC to per-module to keep the output readable. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Change ZonkResult.zonkedTypeEnvironment from a flat 'Map VariableResolution ...' to a per-module 'Map Text (Map VariableResolution ...)'. This isolates each module's local 'ResolvedLocal LocalVarId' entries so they no longer collide across modules — a prerequisite for per-module identify that can reset LocalVarId counters without state threading. Add two helpers in Katari.Typechecker.Zonker for downstream consumers: - lookupTopLevelType: qualified-name lookup (uses qn.module_) - lookupTypeInModule: explicit-module lookup for ResolvedLocal Update Lowering, Schema, Exhaustive, Query, and Query.Completion to thread the current module's name through lookup sites, replacing direct flat-map lookups. zonk now takes the module name as its first argument so its output already has the correct per-module key. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move the per-module CG -> Solve -> Zonk orchestration out of Compile.hs
into a new Katari.Typechecker module exporting:
- ModuleTypecheckResult { zonkedModule, localTypeEnv, moduleInterface,
diagnostics }
- typecheckModule :: IdentifierResult -> importedTypes -> moduleName
-> ModuleTypecheckResult
Compile.hs now delegates the heavy lifting to typecheckModule and
keeps only the cross-module accumulation and cache plumbing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the whole-program checkExhaustive with checkExhaustiveModule plus an ExhaustiveEnv carrying just the data the checker needs: constructors and top-level types reachable from this module, and the module's own local type environment. Internal walk functions drop IdentifierResult / ZonkResult arguments and thread the env instead. The Compile orchestrator builds the env once and reuses it per module by swapping in each module's localTypeEnv. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace lowerModule's IdentifierResult / ZonkResult arguments with: - LowerContext: cross-module info (topLevelTypes, dataDefs, requestNames, constructorNames) that the orchestrator builds once and reuses across modules. - moduleName: the module being lowered. - moduleLocalTypeEnv: this module's local type environment for ResolvedLocal lookups. Drops Lowering's dependency on IdentifierResult and ZonkResult types entirely. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the whole-program buildSchemas with: - buildModuleSchemas: takes a SchemaContext (cross-module data: dataDefs, topLevelTypes, requestData) and a module's own variable map. - buildDataDefs: now takes per-module pieces (constructor map, topLevelTypes, annotation map) rather than IdentifierResult/ZonkResult. - collectDataAnnotations: exposed helper that extracts data parameter annotations from a single Zonked module's AST. Compile.hs builds the merged DataDefs once from per-module pieces and reuses it across modules' lowering / schema generation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Query and Query.Completion now take a 'QuerySnapshot' instead of IdentifierResult / ZonkResult directly. The snapshot wraps both fields and is built once by the caller (LSP / tests) from a CompileResult and reused across calls. LSP handlers (Hover, Definition, References, Completion, Document) all switch to building a QuerySnapshot from compile output before delegating to the Query API. Internal helpers inside Query still consume the unwrapped maps, but the public surface no longer depends on phase-output types. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the two phase-output fields on CompileResult with a single 'querySnapshot :: QuerySnapshot' field that LSP / tests use as their dedicated query input. The orchestrator builds the snapshot via Query.buildQuerySnapshot from the same internal results. CompileResult is now down to compile artifacts (IR, schema, diag, cache) plus the opaque QuerySnapshot for downstream query consumers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the per-level parMap typecheck loop with an Async-based per-module pipeline. Each module's typecheck task waits for its direct imports' tasks via Async.wait, reads their exportedTypes, and runs Typechecker.typecheckModule. Modules with no shared dependencies run their typecheck concurrently. Lower/Schema still run as a parMap batch after all typechecks complete (they need the full cross-module DataDefs). CompileLogTypechecking is emitted from inside each task, so the log appears as the task actually starts — interleaved with sibling modules' parsing/identifying logs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop full Identified/Zonked ASTs and scope frames from the cache: - cacheIdentifiedAST → cacheImports (just the import list, used to rebuild the dep graph) - cacheZonkedModule, cacheZonkedTypeEnv → removed; downstream phases only need cacheInterface.exportedTypes (already kept) - cacheScopeFrames → removed For DataDefs reconstruction on the downstream side, pre-extract data parameter annotations at cache write time and store them in cacheDataAnnotations. LSP queries on cache-hit modules no longer find the module's AST in QuerySnapshot, so hover / scope queries on them return Nothing until the module is recompiled (e.g. on next file change). Tradeoff accepted. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Updated EnvUpsertDialog to streamline description rendering and placeholder text. - Modified SchemaViewer to enhance indentation styles and ensure consistent type badge rendering. - Adjusted ValueViewer for better layout and spacing in nested structures. - Refined AnyField, ArrayField, ObjectField, TupleField, and UnionField components for consistent border styling. - Improved Dialog component structure for better readability and maintainability. - Enhanced MarkdownContent for clearer paragraph and link styling. - Simplified AgentDetailPage by removing redundant text in optional fields. - Introduced TestSupport module in Haskell for better testing capabilities, including compile sugar and parser helpers. - Implemented AsyncLocalStorage in katari-port for improved delegation context management.
- Adjusted import order and formatting in `endpoints.ts`, `env.ts`, `ffi.ts`, `index.ts`, `orchestrator.ts`, `recovery.ts`, `types.ts`, `secret-crypto.ts`, `bundle-loader.ts`, `mock-sidecar.ts`, `sidecar-manager.ts`, `store.ts`, `subprocess-sidecar.ts`, `value-codec.ts`, `value-secret-codec.ts`, `index.ts`, `raw-value.ts`, and `schema.ts`. - Consolidated and simplified type exports in `katari-types`. - Improved readability by reducing line breaks and aligning function signatures. - Ensured consistent formatting for error messages and function definitions.
…ror handling - Replaced placeholder span with emptySourceSpan in diagnostic functions to improve clarity and maintainability. - Removed IdentifierResult from the Identifier module and restructured related code to enhance modularity. - Updated ConstraintGenerator to handle single module results instead of a map, simplifying the constraint generation process. - Refactored Zonker to utilize emptySourceSpan for error reporting, ensuring consistent source span attribution. - Enhanced Orchestrator's transaction handling by implementing AsyncLocalStorage for better concurrency management. - Adjusted various test files to align with the new structure and ensure proper imports.
The incremental cache judged identify/typecheck by source-hash but typecheck
by dependency state. A module whose dependency changed (but whose own source
did not) was therefore re-typechecked as an *empty* module — its diagnostics
and types silently vanished (breaks LSP / `katari check`). Reproduced with a
new CompileSpec test, then fixed by splitting the cache cleanly:
* identify / typecheck run for every module every time (cheap, now parallel)
* only the heavy lowering / schema are cached, invalidated transitively via
the import graph (invalidClosure)
This removes the "empty module" path entirely and makes LSP hover/diagnostics
always current.
Other changes folded in:
- identify: each module runs from fresh per-module state (LocalVarId resets to
0 per module), dropping the global IdentifierState threading. identifyProgram
is shared by the compiler and the test suite, removing TestSupport's
duplicated orchestration (test path now exercises production code).
- identify parallelised per topological level (parMap); typecheck stays
per-module Async; executables gain -threaded -with-rtsopts=-N.
- ModuleCache slimmed 15 -> 4 fields (lowering/schema only); LSP in-memory and
CLI disk caches now agree, no SemanticType serialization needed.
- Compile simplified: 5-tuple accumulator removed, stdlib level computation
unified into compilationLevels / compilationOrder.
- Lowering: registerDecl / lowerOneDeclaration deduplicated via lowerWrapperCallable.
- Schema: exclude secret-typed callables from the AI tool-calling bundle
(recursively, through data fields); golden snapshots updated.
- Remove dead code (Id.VariableId/ModuleId, scanExportNames, declaredNames,
CG.walkModule), fix stale comments, merge Zonker's two type helpers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e structure The parMap/Async parallelism added earlier does not actually parallelise: rseq/wait only force results to WHNF, so the heavy identify/typecheck/lower work stays as thunks the main thread evaluates serially. Forcing it properly needs NFData, but the AST is a phase-indexed GADT/type-family that cannot derive Generic (and NFData is not stock-derivable), so it would take ~72 hand-written rnf instances — not worth it until profiling justifies it. The -threaded -N flags would then only add parallel-GC overhead for no gain. Revert to sequential: identify (level-by-level map), typecheck (dependency- ordered fold), lowering (map); drop -threaded -with-rtsopts=-N. The per-module independence (fresh state, level structure, per-module interfaces) is kept and documented as "parallelisable with parMap/Async once NFData exists", so the design isn't lost — only the non-paying parallel execution. Also removes dead code surfaced here and by case B: registerAllModules / registerModule / emitImportCycleError (superseded by per-module identify + importCycleErrors) and an unused Data.Foldable import. All 680 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s with direct calls
yukikurage
added a commit
that referenced
this pull request
Jul 1, 2026
Address the xhigh review of the http/io/env phase. Runtime + compiler. Shared external-call reactor (review #7 / #9): - Extract the ffi/http callee-call lifecycle into a new `ExternalCallReactor` base (react / complete / afterCommit / persist / load / reset, the running → cancelling → awaitingAnswer state machine, and the caller the reply routes to). FfiReactor and HttpReactor become thin: a per-call payload plus how to dispatch / abort and read / write their ext row. The core/api reactors keep their engines, so this is a call-reactor base, not folded into the root. - The caller reactor is now owned + persisted uniformly by the base (http gains a `caller_reactor` column, migration 0012); HttpReactor no longer hardcodes `core` on recovery. - HttpReactor lifts a result through the shared `jsonToValue` codec (drops the duplicated, coercing `httpResponseValue`). Cancelling recovery, uniform + correct (review #8 / #2 / #11): - Recovery is uniform in the base: a running call re-dispatches, a cancelling call re-aborts, an awaitingAnswer call waits. http no longer reaches a terminateAck via a redispatch error — it aborts, like ffi. - A transport `abort` with no live request now synthesises a `cancelled` (FetchHttpTransport, SnapshotFfiTransport, InProcessFfiTransport), so a cancelling call recovered after a crash is confirmed instead of hanging. Smaller fixes: - http.fetch sends an explicit empty body for body-carrying methods (#4). - env get_all / readPublic build records on a null-prototype map so an env key named `__proto__` is a real field, not a silent drop (#6). - env.set refreshes `updated_at` on overwrite (onConflictDoUpdate does not fire the column's `$onUpdate`). - Compiler rejects an unknown `from "reactor"` name (K3018) instead of a silent runtime fallback to ffi (#1). - `asEffectMetavar` excludes io, so a bare effect metavar is matched exactly (#10). - `signatureValueScheme` sets io through a `withIo` helper, not a hand-written record update that re-derives the lattice join (#12). - `reactivate` loads the reactors' disjoint state concurrently (#13). Restore FetchHttpTransport unit coverage (request building, GET no body, empty body sent, non-2xx = result, at-most-once redispatch, abort → cancelled) and add http cancelling + cancelling-recovery e2e tests. compiler 490, runtime 95 green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.