Skip to content

refactor JSON tokenizer state - #724

Draft
bobzhang wants to merge 1 commit into
mainfrom
codex/json-tokenizer-state
Draft

refactor JSON tokenizer state#724
bobzhang wants to merge 1 commit into
mainfrom
codex/json-tokenizer-state

Conversation

@bobzhang

@bobzhang bobzhang commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

What changed

  • add a private JsonState enum for normal and block-comment modes
  • carry the typed state through the JSON lexer loop
  • confine the existing TokenizerState("n") / TokenizerState("c") representation to encode/decode boundary helpers
  • preserve unknown-state fallback behavior and the public tokenizer API

Why

This isolates JSON's language-specific state semantics from the shared, language-neutral TokenizerState carrier. It provides a small prototype for evaluating a typed state representation without committing the public API to a closed cross-language enum.

Validation

  • JSON lexer checks with warning 73 enabled on JS, native, and Wasm
  • JSON lexer tests: 8/8 on JS, native, and Wasm
  • complete editor suite: 1,023/1,023 tests
  • moon info: no public API change

The root just check, just test, and just build gates cannot resolve this checkout because the workspace member desktop/lepus/config is absent; they fail before compiling the change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant