Skip to content

Latest commit

 

History

History
1353 lines (980 loc) · 32.3 KB

File metadata and controls

1353 lines (980 loc) · 32.3 KB

LangForge Tool Improvement Roadmap

Document id: lang-forge-tool-improvement-roadmap-v1

Status: active

Last updated: 2026-07-10

Owner: Project maintainers

Scope: Forward-looking public roadmap for LangForge usability, diagnostics, generated APIs, editor tooling, and production-readiness

Document purpose: Capture suggested improvements to LangForge as a parser/scanner/compiler tooling project.
Audience: LangForge maintainers, contributors, and AI coding agents working on future development.
Scope: Core generator features, grammar ergonomics, typed semantic values, diagnostics, IDE/editor support, runtime maturity, and production-readiness.
Date: 2026-06-29


This roadmap is directional rather than a commitment list. The implementation backlog remains the source of truth for accepted work, while this document explains the larger improvement themes and helps contributors understand which ideas should be shaped into backlog items or ADRs next.

1. Current Position

LangForge already has a strong foundation as a modern Lex/Yacc-style parser-generator tool. Its current strengths include:

  • combined .lf specifications;
  • scanner generation;
  • parser-table construction;
  • LR-family algorithms such as SLR, LALR(1), IELR(1), and canonical LR(1);
  • conflict reporting;
  • Go, C#, C, and C++ output;
  • reducer-based semantic hooks;
  • deterministic manifests;
  • named RHS labels and target-specific nonterminal type declarations;
  • generated Go, C#, C, and C++ typed reducer contexts/adapters plus reducer coverage validation where the target exposes reducer maps;
  • deterministic cross-target semantic action manifests;
  • generated example projects.

The examples show the intended model well:

.lf grammar
  -> generated scanner/parser
  -> handwritten reducer
  -> AST or semantic model
  -> compiler/interpreter/renderer/report

The next stage should focus on making LangForge easier, safer, more diagnosable, and more useful for both beginners and production users.


2. Strategic Direction

The most valuable long-term direction is:

less runtime casting
better diagnostics
more grammar tooling
cleaner generated APIs
better editor integration
stronger production/runtime story

LangForge should continue to serve three levels of users:

  1. Beginners building small DSLs and learning compiler construction.
  2. Application developers embedding parsers in real products.
  3. Advanced users building compilers, language servers, transpilers, validators, or editor tooling.

3. Highest-Value Addition: Typed Semantic Values

Problem

The baseline reducer model exposes boxed values:

Target Current semantic value style
Go any boxed fallback; generated typed contexts when the action contract is complete
C# object? boxed fallback; generated internal typed contexts when the action contract is complete
C void* boxed fallback plus generated typed argument contexts in parser_typed.h
C++ std::any boxed fallback plus generated typed contexts/adapters in parser_typed.hpp

This gives flexibility, but it pushes casts into user code:

SemanticAction.Add => (double)ctx.Values[0]! + (double)ctx.Values[2]!
return std::any_cast<double>(ctx.values.at(index));
return calc_number(demo, error, calc_value_as_number(ctx->values[0]) + calc_value_as_number(ctx->values[2]));

This creates several problems:

  • reducer code becomes fragile after grammar changes;
  • type errors appear at runtime;
  • examples teach scattered casts;
  • diagnostics vary by target;
  • code becomes harder to audit.

Implemented Foundation

LangForge now accepts target-specific nonterminal result types:

%semantic go type Expr float64
%semantic csharp type Expr double
%semantic c type Expr double
%semantic cpp type Expr double

Terminals remain generated Lexeme values so scanner text and source spans are not discarded. Go, C#, C, and C++ now generate typed contexts or adapters:

parser.TypedAdd(func(ctx parser.AddReduction) (float64, error) {
	return ctx.Left + ctx.Right, nil
})
[SemanticAction.Add] = SemanticReducerContexts.TypedAdd(
    static ctx => ctx.Left + ctx.Right)

Named RHS labels are implemented in the grammar:

Expr : left=Expr Plus right=Term {go: add}

All backends emit the labels and declared types in langforge.actions.json. C and C++ also emit companion typed headers, while preserving boxed reducer APIs for gradual migration.

Current Acceptance Snapshot

  • Go, C#, C, and C++ reducers can avoid or validate direct positional ctx.Values[n] access through generated typed contexts/adapters and label-aware reductions.
  • Go, C#, and C++ ReducerMap coverage validation detects missing and unknown handlers before input-dependent parser execution; C typed reducers validate required handler pointers before parsing.
  • Portable action labels are preserved in manifests and runtime action names; C and C++ derive deterministic target-safe identifiers such as PREFIX_ACTION_RUN_OBJECTS_JOB and SemanticAction::RunObjectsJob.
  • C and C++ calc plus the maintained mini-compiler and library-dsl templates demonstrate direct typed reducers; boxed-to-typed adapters remain available for migration, and --boxed stays as an explicit boxed/debug path where examples expose it.
  • Boxed reducer mode remains available while direct typed reducers are preferred.
  • The CLI now has optional stderr-only verbosity for validate, inspect, and generate: level 1 reports major build stages, level 2 reports lexer, grammar, semantic-action, and parser-table decisions, and level 3 reports DFA and parser state rows for small-grammar tracing.

4. Generate CST, Parse Trees, Visitors, and Listeners

Problem

Reducer mode is compact and good for examples, but many users expect generated parse trees that can be walked independently from semantic actions.

Recommendation

Add optional parse-tree generation.

Possible options:

%tree cst
%visitor true
%listener true

Generated output could include:

ParseTree
NodeKind
Visit(node)
Walk(listener)

Benefits

  • Beginners can inspect parse results.
  • Tool builders can use concrete syntax trees.
  • Formatters and refactoring tools can preserve source structure.
  • Compiler authors can explicitly lower CST to AST.
  • Reducer mode remains available for compact semantic evaluation.

Acceptance Criteria

  • Users can choose reducer-only or CST mode.
  • CST nodes include source spans.
  • Generated visitors/listeners are idiomatic per target.
  • Parse-tree generation is optional to avoid overhead for production parsers.

5. Structured Diagnostics and Error Recovery

Problem

Production parsers need useful errors, not just parse failure.

Recommendation

Introduce a target-neutral diagnostic model.

Example JSON shape:

{
  "severity": "error",
  "code": "LF_PARSE_001",
  "message": "expected expression after '+'",
  "file": "input.calc",
  "range": {
    "start": { "line": 1, "column": 4 },
    "end": { "line": 1, "column": 5 }
  },
  "expected": ["Number", "LParen", "Minus"]
}

Generated APIs should expose:

Diagnostic
SourceRange
ExpectedToken
RecoveryAction

Error Recovery

Add grammar-level recovery support.

Current status: the first conservative recovery level is implemented. The reserved error symbol, explicit synchronization productions, aliases, groups, hidden expected tokens, structured cross-target results, and non-looping tests are available. Phrase/editor recovery heuristics and lookahead-correction refinement remain future work.

Possible syntax:

Statement : error Semi {go: recover.statement}

Possible recovery modes:

%recovery panic
%recovery phrase
%recovery editor

Recovery Levels

Mode Purpose
panic Skip tokens until synchronizing token.
phrase Recover inside a known grammar phrase.
editor Try to continue and produce partial trees for IDE use.

Acceptance Criteria

  • Parser can return multiple diagnostics.
  • Parser can optionally continue after syntax errors.
  • Recovery behavior is grammar-controllable.
  • Error messages include source spans and expected tokens.

6. Add an Incremental / Editor-Facing Mode

Problem

Compiler parsers can fail fast. IDE parsers must handle incomplete and invalid input continuously.

Recommendation

Add separate generation modes:

lang-forge generate --target go --mode compiler
lang-forge generate --target go --mode editor

Compiler Mode

Optimized for:

  • strict parsing;
  • fast failure;
  • AST/reducer output;
  • production compilation.

Editor Mode

Optimized for:

  • partial parse results;
  • multiple diagnostics;
  • syntax-error recovery;
  • hidden-token/trivia preservation;
  • stable node ranges;
  • future incremental reparsing.

First Milestone

Before full incremental parsing, implement:

  • tolerant parse;
  • partial CST;
  • stable source spans;
  • multiple diagnostics;
  • expected-token information.

Later Milestone

Add incremental parsing support:

previous tree + text edit -> updated tree

Acceptance Criteria

  • Compiler mode remains fast and strict.
  • Editor mode can produce useful output for invalid input.
  • Future LSP/editor features can build on editor mode.

7. Improve Grammar Ergonomics

7.1 Grammar Imports and Modules

Add:

%import common.tokens
%import expressions.grammar

Useful for:

  • shared expression grammars;
  • shared lexer definitions;
  • large language specifications;
  • reusable token sets.

7.2 EBNF Sugar

Allow concise grammar forms:

Arguments : Expr (Comma Expr)* ;
Block : LBrace Statement* RBrace ;
ParameterList : Ident (Comma Ident)* ;

Internally lower to LR-compatible productions.

7.3 Named RHS Labels

Add:

FunctionDecl :
    Func name=Ident LParen params=ParameterList RParen body=Block
    {go: functionDecl}
;

This improves:

  • reducers;
  • diagnostics;
  • generated CST/AST names;
  • readability.

7.4 Precedence and Associativity

Support, if not already available:

%left Plus Minus
%left Star Slash
%right UnaryMinus

This is expected by users familiar with yacc/bison-like tools.

7.5 Lexer Modes / States

Add lexer modes:

%mode default
%mode string
%mode interpolation

Use cases:

  • strings;
  • multiline comments;
  • template languages;
  • XML/HTML-like syntax;
  • interpolated expressions.

7.6 Unicode Categories

Support readable Unicode categories:

LETTER = \p{L};
DIGIT = \p{Nd};
IDENT = LETTER (LETTER | DIGIT | "_")*;

This would make Unicode-aware lexers much easier to write.


8. Add Optional GLR or Generalized Parsing

Problem

Some real grammars are hard to express as deterministic LR without awkward refactoring.

Recommendation

Keep LALR/IELR/canonical LR as the default, but add an advanced generalized parser mode later.

Possible syntax:

%parser glr
%ambiguity report
%ambiguity keep
%ambiguity resolve preferShift

Use Cases

  • ambiguous DSLs;
  • split-file grammars;
  • natural-language-like inputs;
  • complex expression syntaxes;
  • IDE parsing;
  • grammars where ambiguity should be inspected rather than rejected.

Roadmap

  1. Improve conflict explanation first.
  2. Add ambiguity reporting.
  3. Add optional GLR mode.
  4. Add ambiguity-resolution hooks.

Acceptance Criteria

  • Deterministic LR remains the default.
  • GLR is clearly documented as advanced.
  • Ambiguities are inspectable and controllable.

9. Conflict Diagnosis and Grammar Explainability

Problem

Conflict reports are necessary, but users also need to understand why conflicts happen.

Recommendation

Add commands:

lang-forge explain --spec grammar.lf --conflicts
lang-forge explain --spec grammar.lf --state 42
lang-forge visualize --spec grammar.lf --format html

Useful Outputs

  • minimal token sequence that reaches the conflict;
  • competing parse paths;
  • shift/reduce or reduce/reduce explanation;
  • FIRST/FOLLOW sets;
  • state-machine graph;
  • lookahead propagation;
  • LALR merge explanation;
  • suggestions such as “try IELR” or “try canonical LR”.

Acceptance Criteria

  • Users can debug conflicts without reading raw table dumps.
  • Reports are useful for both beginners and experts.
  • CI can produce conflict reports as artifacts.

10. Add Grammar Formatter and Linter

Recommendation

Add:

lang-forge fmt grammar.lf
lang-forge lint grammar.lf

Lint Rules

Potential lint warnings:

  • unused tokens;
  • unused lexer macros;
  • unreachable parser rules;
  • tokens shadowed by earlier lexer rules;
  • duplicate semantic action labels;
  • nullable cycles;
  • empty productions that cause conflicts;
  • rules that only forward values unnecessarily;
  • inconsistent action naming;
  • token/nonterminal naming convention issues;
  • hidden ambiguity risks.

Acceptance Criteria

  • fmt produces stable output.
  • lint can run in CI.
  • Warnings have stable codes.
  • Users can suppress intentional warnings.

11. Add Project Scaffolding

Current status: examples already provide source-clean starter material under examples/templates/{go,csharp,c,cpp}/mini-compiler and examples/templates/{go,csharp,c,cpp}/library-dsl. C# and C++ also have examples/templates/{csharp,cpp}/layered-compiler starters. The C# version shows Ast/, Semantics/, Parsing/, isolated Generated/*.g.cs, a public IMiniCompilerParser, domain ParseResult<T>, and DI-friendly semantic policy injection. The C++ version provides public headers, isolated generated output, direct typed reducers, intentional ownership, source-based parsing, and CMake. The next usability step is a CLI bootstrap command that copies and rewrites those templates into a new project so developers do not have to assemble the generated/handwritten boundary by hand. This is tracked internally as W-071 and LF-128.

The public Handwritten Integration Guide now defines the code users are expected to write beside generated recognizers: reducers, parser facades, domain models, diagnostics, C# dependency-injection adapters, C++ semantic policy interfaces, and multi-parser layouts. Follow-up template work should turn those patterns into scaffolded files rather than leaving every project to recreate them manually.

Recommendation

Add:

lang-forge init calc --target go
lang-forge init mini-compiler --target csharp
lang-forge init dsl --target cpp --template compiler
lang-forge init draw-like --target go --template interpreter --out ./draw-like
lang-forge init expression --target csharp --with-facade --with-di
lang-forge init multi-dsl --target cpp --template multi-parser

Templates

Suggested templates:

calc
expression
config-file
mini-compiler
library-dsl
layered-compiler
tree-walker
repl
language-server

Generated Project Structure

A new project should include:

grammar.lf
Makefile or build script
README.md
sample input
parser adapter
typed reducer helpers
AST
tests
target build file
clean generated-output policy

The generated starter should include a .lf file with named RHS labels, target-specific semantic type declarations where useful, handwritten reducer helpers, README usage, shared Makefile-style commands, and .gitignore entries for generated/build output.

Acceptance Criteria

  • New users can start a working parser project in one command.
  • Generated starter projects follow best practices.
  • Templates are kept in sync with examples.
  • Bootstrap output validates, generates, runs, tests, and cleans without manual edits.
  • The command supports Go, C#, C, and C++ targets with target-idiomatic filenames and reducer boundaries.

12. Add an LSP or VS Code Extension for .lf

Recommendation

Add editor support for grammar authoring.

Useful features:

  • syntax highlighting;
  • format-on-save;
  • token/rule navigation;
  • go to action implementation;
  • inline conflict diagnostics;
  • preview generated tokens/rules;
  • preview parse tree for sample input;
  • FIRST/FOLLOW/state inspection;
  • warnings for unused tokens/rules.

Acceptance Criteria

  • .lf files become comfortable to edit.
  • Conflicts and lint warnings show inline.
  • The extension can call the CLI for validation.

13. Add a Parse Playground

Recommendation

Add a local playground:

lang-forge playground --spec grammar.lf

Features

  • paste sample input;
  • show token stream;
  • show parse tree;
  • show reductions;
  • show diagnostics;
  • show generated action IDs;
  • compare LALR vs IELR vs canonical behavior;
  • export minimal conflict reproductions.

Acceptance Criteria

  • Useful for demos, docs, and debugging.
  • Can run locally without external services.
  • Can produce shareable reports.

14. Add Performance and Size Reports

Current status: examples/benchmarks now provides an optional benchmark workflow outside normal CI. The Go suite uses the standard benchmark runner to measure generated scanner throughput, calc-large source parsing versus pre-tokenized parsing, typed versus boxed reducer dispatch, DRAW large-source parsing, source recovery versus pre-tokenized recovery, and allocations. The C# suite uses BenchmarkDotNet with memory diagnostics for comparable scanner, calc, DRAW, and recovery paths. Static generated artifact sizes and parser table metrics are emitted as Markdown/JSON reports under dist/benchmarks instead of timed benchmark rows. The current reporting layer writes compact Go and C# Markdown summaries, keeps raw outputs beside them, labels quick versus stable modes, and keeps C# summary paths repository-relative. C and C++ dedicated benchmark harnesses remain future work.

Recommendation

Add:

lang-forge inspect --spec grammar.lf --metrics
lang-forge bench --spec grammar.lf --input sample.txt

Metrics

Report:

lexer states
parser states
action table size
goto table size
conflicts
nullable rules
generation time
parse throughput
allocations
largest DFA state
largest parser state

Acceptance Criteria

  • Users can evaluate parser size and performance.
  • CI can track generated parser growth.
  • Benchmarks are reproducible.

14.5. Prefer Streaming Lexeme-Source Parser APIs

Problem

Generated parsers currently support token collections as a visible API shape:

input/source
  -> Tokenize all tokens
  -> Parse token collection

That is useful for tests, debugging, examples, and token inspection, but it forces production callers to materialize every token before parsing. Larger inputs and long-running tools are better served by a lazy scanner-to-parser pipeline.

Recommendation

Status: implemented for Go, C#, C, and C++ generated backends, with collection APIs retained as debugging and token-inspection wrappers.

Make the preferred generated parser path pull lexemes directly from a scanner or lexeme source:

input/source
  -> scanner.Next()
  -> parser consumes lexeme source
  -> reducer/semantic actions run during parsing
  -> final value / AST / result

Keep existing collection APIs such as Tokenize, All, and Parse(tokens, ...) as debugging and token-inspection helpers. Where practical, those helpers should adapt their collection into the source-based parser core so parser behavior has one implementation path.

Target direction:

Go:      LexemeSource interface with Next()
C#:      ILexemeSource with synchronous Next/TryRead
C:       lexeme-source struct with next callback and user context
C++:     LexemeSource abstraction with scanner-backed parse overloads

This is synchronous pull-based streaming, not async parsing. Do not introduce parser-core goroutines, channels, queues, tasks, threads, or producer/consumer machinery.

Acceptance Criteria

  • Generated Go, C#, C, and C++ parsers can parse directly from generated scanners or lexeme sources.
  • Collection APIs remain source-compatible and keep working for tests, examples, and token inspection.
  • Existing parse behavior remains equivalent for valid input, lexical errors, syntax errors, recovery diagnostics, reducer errors, hidden/skipped tokens, explicit EOF handling, and normal EOF synthesis.
  • Examples show source parsing as the production path and collection parsing as a debugging, teaching, or token-inspection path.
  • Tests prove source and collection parity across EOF, scanner errors, syntax errors, recovery, reducer failures, hidden/skipped tokens, source spans, and reducer coverage validation.
  • Generated parser cores remain synchronous and pull-based, with static checks against async, channel, queue, worker-thread, or producer/consumer parsing machinery.

14.6. Add Reader/Stream-Backed Scanner Inputs

Problem

The lexeme-source parser path is implemented, but generated scanners still primarily accept complete in-memory source strings or buffers. That means a caller often has to do this before parsing:

file/stdin/network/editor source
  -> read whole source into string
  -> scanner.Next()
  -> parser consumes lexeme source

That is fine for examples and many small DSLs, but production tools often want to parse from io.Reader, TextReader, Stream, FILE-like callbacks, std::istream, stdin, pipes, virtual files, or editor buffers without forcing the entire source into one string first.

Recommendation

Status: generated-runtime support is implemented for Go, C#, C, and C++; the cross-target calc examples now demonstrate reader/stream-backed parsing, and the optional benchmark suite includes reader/TextReader rows for scanner and calc parser measurements.

Add generated streaming scanner variants below the existing parser lexeme-source API:

reader/stream/callback/source
  -> generated scanner pulls enough input to recognize next lexeme
  -> generated parser pulls lexemes from the scanner
  -> reducers run during parsing

Implemented target API direction:

Go:      NewScanner(string), NewReaderScanner(io.Reader, ...ReaderScannerOption), TokenizeFromReader
C#:      Scanner.FromTextReader(TextReader), Scanner.FromStream(Stream, Encoding, TextReaderScannerOptions)
C:       *_stream_scanner with read callback, user pointer, buffer limits, dispose
C++:     InputStreamScanner over std::istream with read-buffer and max-lexeme limits

The existing string/buffer scanner path remains the simple and fastest path. The streaming path must be synchronous and pull-based; it must not introduce async, channels, queues, worker threads, or producer/consumer parser machinery.

The important implementation concerns are:

  • maximal-munch lexing across chunk boundaries;
  • UTF-8/scalar sequences split across chunks;
  • source spans based on absolute offsets and line/column positions;
  • read errors reported as scanner/lexeme-source failures;
  • configurable maximum buffered lexeme length;
  • lexeme text ownership, especially for C and C++ where lexemes currently borrow from caller-owned input.

Acceptance Criteria

  • Implemented: generated Go, C#, C, and C++ scanners can read from target-idiomatic synchronous stream/reader/callback inputs.
  • Implemented: existing string scanner APIs remain source-compatible.
  • Implemented in generated integration tests: valid input, lexical errors, read failures, syntax recovery, source spans, buffer limits, and UTF-8 split behavior where target input encoding makes that applicable.
  • Implemented: C and C++ stream scanners own copied visible-lexeme text and document scanner-lifetime/disposal rules in generated APIs and public docs.
  • Implemented in examples: calc uses Go NewReaderScanner, C# Scanner.FromStream/Scanner.FromTextReader, C *_stream_scanner, and C++ InputStreamScanner.
  • Implemented in benchmarks: Go and C# benchmark rows compare in-memory source parsing with reader/TextReader-backed scanner parsing.
  • Follow-up: update more reusable templates to expose reader/stream facade overloads where that improves copyability.

15. Improve Runtime Packaging and Versioning

Problem

Multi-target generators need consistent runtime and generated-code versioning.

Recommendation

Track:

lang-forge CLI version
generated manifest version
runtime API version
target backend version
spec hash

Example manifest:

{
  "langforgeVersion": "x.y.z",
  "target": "go",
  "backendVersion": "x.y.z",
  "runtimeApi": "v1",
  "specHash": "...",
  "generatedAt": null,
  "deterministic": true
}

Add:

lang-forge verify-generated

Acceptance Criteria

  • Stale generated code can be detected.
  • Tool/runtime mismatches are clear.
  • CI can verify generated output.

16. Add Optional Shared Runtime Packages

Recommendation

Support two runtime modes:

--runtime embedded
--runtime package

Embedded Runtime

Benefits:

  • no external dependency;
  • simple examples;
  • easy vendoring.

Package Runtime

Benefits:

  • smaller generated code;
  • shared diagnostics;
  • easier bug fixes;
  • stable target APIs.

Possible packages:

github.com/digixoil/langforge/runtime/go
LangForge.Runtime.CSharp
langforge_runtime_c
langforge_runtime_cpp

Acceptance Criteria

  • Existing embedded-output behavior remains available.
  • Package runtime is optional.
  • Generated manifests record runtime mode.

17. Better Source Span and Trivia Support

Recommendation

Generated tokens and parse nodes should optionally track:

byte start/end
rune start/end
line/column start/end
file name
leading trivia
trailing trivia
hidden tokens

Add options:

%trivia preserve
%trivia discard

Use Cases

  • formatters;
  • refactoring tools;
  • language servers;
  • documentation generators;
  • code generators that preserve comments.

Acceptance Criteria

  • Compiler-focused users can discard trivia.
  • Editor/tooling users can preserve trivia.
  • CST nodes can carry full source ranges.

18. Semantic Action Validation

Recommendation

Generate an action manifest and validate reducer coverage.

Status: implemented across Go, C#, C, and C++ generation as deterministic langforge.actions.json. Go ReducerMap additionally exposes ValidateCoverage, and ParseWithReducer performs the check automatically.

Example manifest:

{
  "actions": [
    {
      "id": 1,
      "name": "add",
      "typed": true,
      "rules": [
        {
          "id": 2,
          "lhs": "Expr",
          "returnType": "float64",
          "rhs": [
            {"position": 1, "symbol": "Expr", "label": "left", "type": "float64"},
            {"position": 2, "symbol": "Plus", "type": "Lexeme"},
            {"position": 3, "symbol": "Term", "label": "right", "type": "float64"}
          ]
        }
      ]
    }
  ]
}

Checks:

  • action label declared but reducer missing;
  • reducer implements action not present in grammar;
  • rule has no action and default reduce is ambiguous;
  • action naming inconsistent;
  • action return type mismatch when typed semantics exist.

Acceptance Criteria

  • Users can test reducer coverage.
  • Missing semantic actions are easy to find.
  • Large grammars become easier to maintain.

19. Optional AST Generation

Recommendation

Add optional AST generation for simple grammars.

Possible syntax:

%ast generate

or:

Expr :
    left=Expr Plus right=Term {node: BinaryExpr}
  | Number                  {node: NumberExpr}
;

Benefits

  • beginners can get started quickly;
  • DSL authors can avoid boilerplate;
  • generated visitors can operate on AST nodes;
  • examples become easier to explain.

Caution

Generated AST should remain optional. Serious compilers often need handwritten ASTs.

Acceptance Criteria

  • AST generation is opt-in.
  • Generated AST nodes are idiomatic per target.
  • Users can still use manual reducers.

20. Language-Server-Oriented Output

Recommendation

Emit optional metadata useful for editor tooling:

tokens.json
grammar.json
nodes.json
highlights.scm
folds.scm
indents.scm
symbols.json

Use Cases

  • syntax highlighting;
  • folding;
  • symbol extraction;
  • outline views;
  • language-server indexing;
  • grammar documentation.

Acceptance Criteria

  • Metadata output is deterministic.
  • Editor tooling can consume it without parsing generated code.
  • Metadata aligns with generated parser behavior.

21. Strict Mode, Teaching Mode, and Production Mode

Teaching Mode

%mode teaching

Prioritizes:

  • readable generated code;
  • verbose comments;
  • beginner diagnostics;
  • examples in output.

Production Mode

%mode production

Prioritizes:

  • compact tables;
  • fewer comments;
  • fewer allocations;
  • performance;
  • stable API.

IDE Mode

%mode ide

Prioritizes:

  • recovery;
  • partial trees;
  • trivia;
  • source spans;
  • incremental parsing support.

Acceptance Criteria

  • Users can choose output style by purpose.
  • Defaults remain simple.
  • Mode differences are documented.

22. Suggested Development Roadmap

Phase 1: Polish Current Strengths

  1. typed reducer helpers;
  2. named RHS labels;
  3. streaming lexeme-source parser APIs;
  4. structured diagnostics;
  5. generated action manifest;
  6. lang-forge fmt;
  7. lang-forge lint;
  8. clean reusable examples.

Phase 2: Improve Developer Workflow

  1. lang-forge init;
  2. watch mode;
  3. generated-code verification;
  4. grammar visualization;
  5. conflict explainer;
  6. performance metrics;
  7. reusable runtime option.

Phase 3: Improve Generated Structure

  1. CST generation;
  2. visitor/listener generation;
  3. optional AST generation;
  4. source-span/trivia preservation;
  5. semantic action coverage checks.

Phase 4: Editor and Tooling Support

  1. .lf language server;
  2. VS Code extension;
  3. parse playground;
  4. tolerant parse mode;
  5. multiple-error reporting;
  6. editor metadata output.

Phase 5: Advanced Parsing

  1. ambiguity reporting;
  2. optional GLR mode;
  3. ambiguity-resolution hooks;
  4. incremental parsing;
  5. grammar imports/modules at scale.

23. Top 10 Recommendations

If only ten improvements are chosen, prioritize:

  1. Typed semantic values or typed reducer accessors.
  2. Named RHS symbols instead of positional ctx.Values[n].
  3. Streaming lexeme-source parser APIs with collection wrappers preserved.
  4. Structured diagnostics with source ranges and expected tokens.
  5. lang-forge lint for grammar quality.
  6. lang-forge fmt for grammar consistency.
  7. Conflict explainer with state, lookahead, and minimal repro.
  8. Project templates via lang-forge init.
  9. CST plus visitor/listener generation.
  10. Source-only reusable examples with golden tests.

24. Recommended First Implementation Tasks

Task 1: Named RHS Labels

Add grammar support for:

Expr : left=Expr Plus right=Term {go: add}

Generate metadata exposing RHS labels.

Task 2: Typed Accessor Helpers

Generate or document target-specific helpers:

Arg<T>
LexemeArg
TextArg
NodeArg

Task 3: Structured Diagnostics

Introduce a target-neutral diagnostic shape.

Task 4: Token-Source Parser Runtime

Introduce generated scanner/lexeme-source parser APIs for Go, C#, C, and C++. Keep collection parsing as wrappers for debugging and token inspection.

Status: implemented. Follow-up work should focus on additional examples, performance measurements for very large inputs, and any target-specific naming polish discovered by downstream projects.

Task 5: Grammar Linter

Add warnings for unused tokens, unreachable rules, shadowed lexer rules, duplicate actions, and nullable cycles.

Task 6: Conflict Explainer

Produce human-readable conflict reports and minimal examples.

Task 7: Template Scaffolding

Add:

lang-forge init mini-compiler --target go

Then mirror it across C#, C, and C++.


25. Final Direction

LangForge should not try to clone any single existing parser generator. Its opportunity is to combine:

Lex/Yacc-style determinism
modern multi-target code generation
clear reducer semantics
strong diagnostics
typed generated APIs
clean templates
editor-aware parsing

The core is already promising. The next major improvement is to make the tool feel safer and more guided:

grammar authors get linting and explanations
application developers get typed APIs and templates
compiler authors get diagnostics and performance reports
IDE/tooling authors get CSTs, spans, recovery, and metadata

That would make LangForge useful across the full range of needs: learning, DSL embedding, production parsing, compiler construction, and language tooling.