Skip to content

learning-path.md

Latest commit

 

History

History
272 lines (198 loc) · 9.07 KB

learning-path.md

File metadata and controls

272 lines (198 loc) · 9.07 KB
Raw

Learning Path

Document id: lang-forge-learning-path-v1

Status: active

Last updated: 2026-07-11

Owner: Project maintainers

Scope: A guided path for learning compiler tooling through LangForge

LangForge is meant to be useful software and readable compiler-learning material. You should be able to move from a tiny calculator grammar to real generated parser tables, semantic reducer hooks, DSL examples, and migration fixtures without needing to reverse-engineer the whole repository at once.

What You Can Learn Here

LangForge currently demonstrates:

  • how Lex/Yacc-style specifications are structured;
  • how regular expressions become lexical automata;
  • how overlapping character classes become deterministic scanner alphabets;
  • how longest-match and rule-priority tokenization works;
  • how grammar rules become LR parser states;
  • why SLR, LALR(1), IELR(1), and canonical LR(1) differ;
  • how generated parsers and semantic reducers fit into a larger compiler or DSL tool;
  • how to keep generated code deterministic and testable;
  • how split Lex/Yacc-style inputs can be represented without copying tool-specific assumptions.

Suggested Order

1. Run The Smallest Thing

Start with the calculator:

go run ./cmd/lang-forge validate --spec examples/go/calc/calc.lf
go run ./cmd/lang-forge inspect --spec examples/go/calc/calc.lf --format text
make -C examples/go/calc run

Then open:

Goal: understand the boundary between lexer rules, parser rules, generated Go parser output, and reducer-backed expression evaluation. In particular, learn that {go: add} is a reducer label, not built-in arithmetic code. Then use the handwritten integration guide to see which reducer, facade, library, and test files a LangForge user normally writes beside the .lf file.

If a term is unfamiliar, keep Glossary open nearby.

2. Learn The Compiler Pipeline

Read Automata And Driving Tables, then Compiler Pipeline. The first page gives a visual map of scanner boxes, parser boxes, and generated tables. The second explains how source text moves through these stages:

.lf or .l/.y
  -> spec model
  -> lexer DFA
  -> grammar model
  -> parser table
  -> generated code
  -> reducer hooks or handwritten semantic layer

Goal: connect repository packages such as internal/spec, internal/lex, and internal/parse to the compiler concepts they implement, without treating the generated tables as black boxes.

3. Inspect The Tables

Generate machine-readable inspection output:

go run ./cmd/lang-forge inspect \
  --spec examples/go/calc/calc.lf \
  --format json > /tmp/calc.inspect.json

Look for:

  • lexer states and accepting rules;
  • parser actions and gotos;
  • normalized grammar rules;
  • parser algorithm.

Goal: see generated automata as data, not magic.

4. Compare LR Algorithms

Run the parser-algorithm fixtures:

make -C examples/parser-algorithms test

Then read Parser Algorithms.

Goal: understand why a grammar can be LR(1) and LALR(1) but still fail under SLR.

5. Recover From Syntax Errors

Run the recovery fixture. Start with Go, then compare the same fixture in the other generated targets:

make -C examples/go/parser-recovery run
make -C examples/csharp/parser-recovery run
make -C examples/c/parser-recovery run
make -C examples/cpp/parser-recovery run

Then read Parser Error Recovery.

Goal: understand the reserved error symbol, synchronization terminals, expected-token diagnostics, cascade suppression, and the parser's progress guarantee.

6. Study A Real DSL Flow

Run the DataKeeper example:

make -C examples/go/datakeeper run

Open:

Goal: see a generated parser reducer build an AST, lower it to stack-machine instructions, and run a mock execution.

7. Study A Visual DSL

Run the DRAW renderer:

make -C examples/go/draw run

Open:

Goal: see a generated parser reducer build a visual DSL AST that becomes an interpreted model and visible output.

8. Study A Migration-Shaped DSL

Run the vehicle report example:

make -C examples/go/vehicle-report run

Open:

Goal: see how a Flex/Bison-style exercise language can become a modern LangForge .lf spec with generated parser reductions, AST construction, and report output.

9. Explore Legacy Inspiration

Read UCDT Reference, then validate a split fixture:

go run ./cmd/lang-forge validate \
  --lex testdata/ucdt/draw/draw.l \
  --yacc testdata/ucdt/draw/draw.y

Goal: understand which Lex/Yacc-style ideas informed LangForge and how source fixtures can guide regression tests without becoming public behavior promises.

Concept Map

Concept Public doc Code to read Example
Vocabulary Glossary names across packages all examples
Spec syntax Specification internal/spec examples/go/calc/calc.lf
Automata and driving tables Automata And Driving Tables generated scanner/parser tables calc inspect output
Lexer automata Compiler Pipeline internal/lex calc lexer rules
Parser tables Parser Algorithms internal/parse parser algorithm fixtures
Generated Go Usage internal/codegen/golang examples/go/calc
Semantic layer Examples generated reducers and example-local packages calc, DataKeeper, DRAW, and vehicle report
UCDT reference UCDT Reference internal/spec split parsing testdata/ucdt

Exercises

Try these in order:

  1. Add a modulo operator to examples/go/calc/calc.lf.
  2. Add a keyword token before an identifier token in a small test grammar and observe longest-match plus rule-priority behavior.
  3. Change the parser algorithm fixture from %type lalr to %type slr and inspect the conflict.
  4. Add a harmless comment syntax to the DataKeeper lexer and send it to a hidden channel.
  5. Add a new DRAW command that maps cleanly to one new AST node and renderer operation.
  6. Add one vehicle feature field to examples/go/vehicle-report/sample.vehicle and verify the report changes.
  7. Save inspect --format json output before and after a grammar change and compare state counts.

Learning-Friendly Coding Practices

LangForge code should prefer:

  • small packages with clear compiler-stage ownership;
  • named concepts that match compiler literature where practical;
  • deterministic output so examples are repeatable;
  • focused tests that show the reason a behavior exists;
  • public docs that explain how to use a feature before explaining internals;
  • comments near non-obvious algorithms, not comment noise around simple code.

When adding a feature, try to leave three breadcrumbs:

  1. A small example or fixture.
  2. A test that protects the edge case.
  3. A short doc note explaining when a user should care.

Best Practices For Users

  • Keep .lf specs as the source of truth.
  • Validate early and often.
  • Use LALR(1) by default.
  • Use IELR(1) when LALR reports a false merge conflict.
  • Use canonical LR(1) to diagnose deeper conflicts.
  • Keep generated output in ignored generated directories unless you are intentionally creating golden fixtures.
  • Use target-specific semantic action labels for rule-local semantics, and keep larger domain behavior in ordinary target-language code. Use %semantic <target> import to record handwritten semantic dependencies, and reserve inline mode for target-specific generated reductions that truly need to call APIs directly.
  • Use inspect --format json to reason about state counts, conflicts, and table shape.

Quality And Performance Expectations

Learning material should not make the tool slow or sloppy. LangForge aims for:

  • deterministic generation without timestamp churn;
  • compact table-driven scanner and parser runtimes;
  • reentrant generated APIs;
  • validation that rejects non-progressing scanners and malformed grammars early;
  • tests for edge cases before examples rely on them;
  • generated code that reads like normal target-language code.