Optimize Unicode predicate checks

[mizchi]

Motivation

This PR is intended as another concrete example of what the core library can do if the V128 type is available to core implementations. Unicode and UTF-8 routines are common byte-oriented hot paths, and they are good candidates for target-specific fast paths hidden behind stable package APIs.

The new UTF-8 validity API is allocation-free on non-JS targets and keeps the same strict contract as decode. It does not use V128 yet because the natural input type is BytesView; a zero-copy way to load V128 from Bytes / BytesView would make SIMD UTF-8 scanning practical for wasm-gc as well. The char predicate changes are included as helper-level fast paths around the same Unicode boundary.

Summary

• Add @encoding/utf8.is_valid(BytesView, ignore_bom?) for strict UTF-8 validation without decoding on non-JS targets.
• Reuse a shared malformed-byte finder for the JS decode error suffix path.
• Add small integer fast paths for helper char predicates: Char::is_ascii_whitespace, Char::is_whitespace, and Char::is_numeric.
• Add boundary tests, README docs, generated interface updates, and focused benchmarks.

UTF-8 Validation

The new @encoding/utf8.is_valid API follows the same strict UTF-8 contract as @encoding/utf8.decode: it rejects overlong sequences, UTF-16 surrogates, code points above U+10FFFF, lone continuation bytes, truncated sequences, and invalid leading bytes.

The non-JS path uses a BytesView scalar walker with an ASCII-prefix skip. I did not use V128 here because the current public V128 load APIs take FixedArray[Byte], while this API naturally works on BytesView; using V128 for wasm-gc would need a zero-copy way to load from Bytes/BytesView.

There was no previous is_valid API, so the benchmark compares validation against decode as the existing way to validate well-formed UTF-8.

Command:

moon bench --target wasm-gc --release -p encoding/utf8 -f is_valid_bench_test.mbt --no-parallelize
moon bench --target wasm --release -p encoding/utf8 -f is_valid_bench_test.mbt --no-parallelize
moon bench --target js --release -p encoding/utf8 -f is_valid_bench_test.mbt --no-parallelize

wasm-gc --release

Benchmark	is_valid	decode
ASCII n=4096	818.26 ns	3.23 us
Mixed UTF-8 n=4096	2.02 us	7.38 us

wasm --release

Benchmark	is_valid	decode
ASCII n=4096	842.26 ns	499.46 ns
Mixed UTF-8 n=4096	1.97 us	7.27 us

js --release

Benchmark	is_valid	decode
ASCII n=4096	328.05 ns	308.77 ns
Mixed UTF-8 n=4096	6.75 us	6.80 us

Char Helper Predicate Benchmarks

The char predicate changes are helper-level fast paths for common low-codepoint checks used around Unicode handling. Baseline was measured on upstream/main with only the new benchmark file added. After was measured on this branch.

Command:

moon bench --target wasm-gc --release -p builtin -f char_predicate_bench_test.mbt --no-parallelize
moon bench --target wasm --release -p builtin -f char_predicate_bench_test.mbt --no-parallelize

wasm-gc --release

Benchmark	Before	After	Change
Char::is_ascii_whitespace ascii n=62000	47.29 us	34.57 us	-26.9%
Char::is_whitespace ascii n=62000	85.31 us	60.05 us	-29.6%
Char::is_whitespace unicode n=22000	31.50 us	24.13 us	-23.4%
Char::is_numeric ascii n=62000	2.03 ms	84.97 us	-95.8%
Char::is_numeric unicode n=22000	559.82 us	289.51 us	-48.3%

wasm --release

Benchmark	Before	After	Change
Char::is_ascii_whitespace ascii n=62000	55.15 us	45.29 us	-17.9%
Char::is_whitespace ascii n=62000	91.75 us	57.35 us	-37.5%
Char::is_whitespace unicode n=22000	33.23 us	25.37 us	-23.7%
Char::is_numeric ascii n=62000	2.11 ms	96.14 us	-95.4%
Char::is_numeric unicode n=22000	581.70 us	296.73 us	-49.0%

Validation

moon fmt
moon info
moon check --target wasm-gc
moon check --target wasm
moon check --target js
moon test --target wasm-gc -p encoding/utf8
moon test --target wasm -p encoding/utf8
moon test --target js -p encoding/utf8
moon test -p char
moon test -p builtin
moon test --enable-coverage

moon test --enable-coverage passed with 6594 tests.

[Copilot]

Pull request overview

This PR extends the encoding/utf8 package with a strict, allocation-free (non-JS) UTF-8 validation API and refactors shared malformed-byte detection to support JS error reporting, while also adding small fast paths for common Char Unicode predicate helpers to improve hot-path performance in core Unicode handling.

Changes:

• Added @encoding/utf8.is_valid(BytesView, ignore_bom?) -> Bool with target-specific implementations (JS via TextDecoder(fatal: true), non-JS via a shared scalar malformed-byte finder).
• Centralized strict malformed-byte scanning in encoding/utf8/decode.mbt and reused it for the JS decode malformed-suffix path.
• Added small integer fast paths for Char::is_ascii_whitespace, Char::is_whitespace, and Char::is_numeric, plus targeted tests/docs and benchmark coverage.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
encoding/utf8/README.mbt.md	Documents the new is_valid API with a short example.
encoding/utf8/pkg.generated.mbti	Exposes is_valid in the generated public interface.
encoding/utf8/moon.pkg	Adds target gating for new is_valid_* files and imports bench for benchmarks.
encoding/utf8/is_valid_nonjs.mbt	Implements non-JS is_valid via BOM handling + shared malformed scan.
encoding/utf8/is_valid_js.mbt	Implements JS is_valid using TextDecoder with fatal: true.
encoding/utf8/is_valid_bench_test.mbt	Adds benchmarks comparing is_valid vs decode across ASCII/mixed inputs.
encoding/utf8/decode.mbt	Adds shared strict malformed-byte finder with an ASCII fast-skip.
encoding/utf8/decode_test.mbt	Adds validation tests for is_valid and BOM-related cases.
encoding/utf8/decode_js.mbt	Switches JS malformed-suffix computation to reuse the shared malformed finder.
char/char_test.mbt	Adds boundary/edge-case assertions for updated numeric/whitespace predicates.
builtin/char.mbt	Adds fast paths for whitespace/numeric helpers using integer codepoint checks.
builtin/char_predicate_bench_test.mbt	Adds benchmarks for the updated Char predicate helpers.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[bobzhang]

Benchmark review of the builtin/char.mbt predicate changes: most of the win is achievable with patterns

Thanks for the thorough PR! I re-ran the char-predicate part of this PR on native, js, and wasm-gc (this PR's tables only cover wasm/wasm-gc), using this PR's own char_predicate_bench_test.mbt and methodology (moon bench --release, current main as baseline), and also measured a third variant for each function: the same early-exit control flow expressed with char range patterns (self is ('x'..='y')) instead of .to_int() arithmetic. Findings, per function:

1. is_ascii_whitespace — the integer arithmetic is a net loss; a range pattern wins

main spells this as six singleton alternatives. Rewriting it as one range plus space is both the most readable form and the fastest on 2 of 3 backends:

pub fn Char::is_ascii_whitespace(self : Self) -> Bool {
  self is ('\u{09}'..='\u{0D}' | ' ')
}

ascii n=62000	main (6 singletons)	this PR (int arith)	range pattern
native	13.6 µs	28.1 µs	13.4 µs
js	224.1 µs	212.3 µs	109–118 µs
wasm-gc	78.3 µs	58.8 µs	68.7 µs

On native the arithmetic is 2.1x slower than the pattern; on js the pattern is ~1.9x faster than the arithmetic. wasm-gc is the only backend where the arithmetic wins (by ~14%). Recommendation: take the range pattern, not the arithmetic.

2. is_numeric — 100% of the −95% win is the early-exit structure, and it's expressible in patterns

The gain comes from not entering the ~140-line table for ASCII, not from integers. This form keeps the fast path readable and the table untouched:

pub fn Char::is_numeric(self : Self) -> Bool {
  if self is ('0'..='9') {
    return true
  }
  if self < '\u{B2}' {
    return false
  }
  self
  is ('\u{B2}' // ... table unchanged, leading '0'..='9' range removed
  )
}

	main	this PR (int arith)	pattern fast path
native ascii	882.8 µs	80.1 µs	75.8 µs
native unicode	246.1 µs	152.5 µs	150.3 µs
js ascii	1.46 ms	216.3 µs	215.6 µs
js unicode	750.1 µs	395.5 µs	402.8 µs
wasm-gc ascii	2.56 ms	153.9 µs	161.6 µs
wasm-gc unicode	830.8 µs	444.7 µs	457.1 µs

Statistical tie with the arithmetic on all three backends (11.6x / 6.8x / 15.8x over main on ASCII). Recommendation: take the pattern form.

3. is_whitespace — the one place the arithmetic still earns its keep, and it exposes a codegen gap

I tried several readable shapes. Two are interesting:

// (a) same control flow as this PR, in char terms
if self <= '\u{20}' {
  return self is ('\u{09}'..='\u{0D}' | ' ')
}
if self < '\u{85}' || self > '\u{3000}' {
  return false
}
self
is ('\u{85}' | '\u{A0}' | '\u{1680}' | '\u{2000}'..='\u{200A}'
| '\u{2028}' | '\u{2029}' | '\u{202F}' | '\u{205F}' | '\u{3000}')

// (b) one ordered match
match self {
  '\u{09}'..='\u{0D}' | ' ' => true
  '\u{00}'..='\u{84}' => false
  '\u{85}' | '\u{A0}' | '\u{1680}' | '\u{2000}'..='\u{200A}'
  | '\u{2028}' | '\u{2029}' | '\u{202F}' | '\u{205F}' | '\u{3000}' => true
  _ => false
}

ascii / unicode	main	this PR (int arith)	(a) char compares	(b) ordered match
native	66.9 / 26.4	32.2 / 17.2	32.0 / 17.1	58.0 / 37.1
js	243.5 / 93.6	227.2 / 88.2	224.9 / 87.0	224.3 / 87.2
wasm-gc	146.0 / 53.0	102.2 / 52.4	177.7 / 72.9	102.0 / 49.7

Every cell is reproducible (re-runs within ~1%). The strange part: (a) exactly ties the arithmetic on native and js but is 1.7x slower on wasm-gc — slower than main, even — while (b) exactly ties the arithmetic on wasm-gc and js but is 1.8x slower on native. Two more data points on the sensitivity: pattern-based guards (if self is ('\u{00}'..='\u{20}') then guard self is ('\u{85}'..='\u{3000}')) land at 159.5 µs on wasm-gc, and adding a '\u{3001}'..='\u{10FFFF}' => false arm to (b) swings wasm-gc from 102 → 169 µs.

So for is_whitespace specifically, no single source-level pattern form matches the integer arithmetic on all three backends today — but the per-backend ties show the machine code is achievable from patterns; which shape wins is a backend codegen accident (char Compare vs pattern lowering on wasm-gc, match vs if-chains on native). That looks like a moonc issue worth filing: char range/or patterns and char comparisons should lower to the same decision tree regardless of surface syntax. The is_numeric table is further evidence — even after the ASCII exit, the ~80-range table costs ~7 ns/char (a linear-ish scan); a compiler-emitted binary search over the ranges would beat every variant measured here.

Suggested resolution for this PR

• is_ascii_whitespace: use self is ('\u{09}'..='\u{0D}' | ' ') — faster than the arithmetic on native and js, and more readable than both current forms.
• is_numeric: use the pattern-based fast path — ties the arithmetic everywhere, keeps the table.
• is_whitespace: either keep the integer arithmetic with a comment noting it's a workaround for backend-dependent pattern codegen, or take form (a)/(b) and accept one backend's regression until the codegen gap is fixed upstream.

(The UTF-8 is_valid part of this PR is not affected by any of this. Measurements: moon bench --release per backend on darwin/arm64; baseline = current main, which this PR currently conflicts with — the three predicates are unchanged on main, so the comparison holds.)

Benchmarks and analysis by Claude Code on behalf of @bobzhang.

[bobzhang]

done in #4112