feat: Add opt-in PCRE2 regex engine and Java-regex compatibility test suite

[baibaichen]

Proposal

Add PCRE2 as a second regex engine in Velox, enabled at compile time via a new CMake option (default OFF). The existing RE2 path is unchanged; users that need fuller java.util.regex semantics — primarily Spark / Presto front-ends running through Gluten — can opt in at build time.

The PR also ships:

• a Java-regex-syntax translator library that also benefits RE2 (lifts RE2's OpenJDK 17 corpus pass-rate from 528/860 → 757/860, +27 pp), and
• a multi-backend test harness that uses an embedded JVM as Java-semantic oracle.

Why a second engine, not a replacement

Past discussion ( #9897, #9823 ) repeatedly raised the RE2 vs java.util.regex gap and equally raised legitimate concerns about every proposed alternative. PCRE2 is qualitatively different from Oniguruma / Hyperscan, and this PR addresses each historical objection directly:

Five concerns, one by one

ReDoS / non-linear runtime — @mbasmanova on #9823: "It is not realistic to expect users to 'check the provided patterns carefully'."

PCRE2 has built-in LIMIT_MATCH / LIMIT_DEPTH / LIMIT_HEAP that the engine itself enforces. Our Pcre2Regex wrapper sets conservative defaults and exposes them as Options fields. Hitting a limit surfaces a typed error rather than running forever. This is the same protection Java 9+ added to java.util.regex. Oniguruma and Hyperscan do not have this.

Oniguruma trades reliability for coverage — @mbasmanova on #9897.

We do not ship Oniguruma.

Hyperscan effectively abandoned — @philo-he on #9823.

We do not ship Hyperscan. PCRE2 by contrast is actively maintained (PHP, Apache HTTPD, nginx, git all depend on it; releases roughly every 6 months).

Velox SQL is single-pattern per row; multi-pattern engines don't pay off — @mbasmanova on #9823.

PCRE2 is a single-pattern engine; matches that constraint.

User must opt in if accepting any risk — @FelixYBW on #9823: "In Gluten we use a config to enable re2 offload. Gluten user needs to take the risk if they enable."

New CMake option VELOX_ENABLE_PCRE2_BACKEND=OFF by default; identical opt-in pattern. Default cmake ... adds zero new dependencies.

What's in the PR

1. PCRE2 engine wrapper — velox/external/regex_compat/Pcre2Regex.{h,cpp}

Production-quality wrapper around PCRE2 with a re2::RE2-shaped surface so call sites that opt in can switch with minimal churn:

• JIT-compiles on construction (falls back to interpreter where unavailable).
• Honors Options.{caseSensitive, dotNl, oneLine, ...} matching Velox's existing Re2Regex options.
• Sets PCRE2's built-in match / depth / heap limits to safe defaults; exposed as Options fields so callers can tune them. Hitting a limit surfaces a typed error, not a hang.
• Integrates the translator below: every Java pattern goes through toPcre2Pattern before pcre2_compile, so Java-specific syntax (\p{InGreek}, \p{javaXxx}, (?U) flag inversion, char-class intersection, …) just works.

2. Java→PCRE2/RE2 translator — velox/functions/lib/java_pcre2_translator/

Reusable C++ library that rewrites a java.util.regex pattern into PCRE2- or RE2-compatible syntax:

• 1:1 port of pcre4j PR (feat) regex: java.util.regex compatibility translator + :compat-test harness (68% -> 93.7%) alexey-pelykh/pcre4j#606. I authored that pcre4j PR (wrote the org.pcre4j.regex.translate module from scratch and contributed it upstream); pcre4j does not require a CLA, so I retain copyright on that contribution and am dual-licensing the C++ port here under Apache-2.0. Per-file headers carry the attribution; LICENSE-NOTICE.md documents provenance.
• Adds toRe2Pattern that reuses the same parser/evaluator/property map and additionally rejects features RE2 fundamentally cannot represent (lookaround, backref, possessive, atomic, (?U) semantic inversion) — throws EvaluationFailedException rather than silently changing semantics.
• Builds unconditionally (it's a functions/lib/ member); production call sites can adopt it independently of the PCRE2 backend.

Headline result for RE2: when the existing Re2Regex is wired through this translator, RE2's OpenJDK 17 corpus pass-rate jumps from 528/860 (61.40%) to 757/860 (88.02%) without changing the engine. That benefit alone — for anyone who wants to stay on RE2 — is the second reason to merge.

3. Test harness — velox/external/regex_compat/tests/

Three-backend GTest framework (Re2Regex, Pcre2Regex, JavaRegex via embedded JVM as oracle), parameterised via TYPED_TEST_SUITE. Runs:

• OpenJDK 17 corpus (4 files, 860 + 5 cases) fetched at CMake configure time, SHA-pinned to jdk-17.0.13-ga.
• 96 ported tests from OpenJDK RegExTest.java.
• 178 translator unit tests (1:1 from pcre4j).

A TestMain guard asserts the Java backend remains 100% and loudly warns otherwise — Java is java.util.regex, so any failure there means our JNI bridge / adapter is wrong, not an engine difference. The harness is gated behind VELOX_ENABLE_REGEX_COMPAT_TESTS=OFF with a further sub-option VELOX_ENABLE_REGEX_COMPAT_JAVA_BACKEND=OFF that auto-disables if no JDK is found.

Default-build impact

Zero. With the default cmake ... invocation:

• No PCRE2 download or link.
• No JNI / JDK probe.
• No new code in any existing target.

# default — nothing changes
cmake -S . -B build

# enable PCRE2 engine for production use (adds PCRE2 10.47 via FetchContent)
cmake -S . -B build -DVELOX_ENABLE_PCRE2_BACKEND=ON

# also run the compatibility test suite
cmake -S . -B build -DVELOX_ENABLE_PCRE2_BACKEND=ON \
                    -DVELOX_ENABLE_REGEX_COMPAT_TESTS=ON

# include the Java JNI oracle in the test suite (needs JDK >= 17)
cmake -S . -B build -DVELOX_ENABLE_PCRE2_BACKEND=ON \
                    -DVELOX_ENABLE_REGEX_COMPAT_TESTS=ON \
                    -DVELOX_ENABLE_REGEX_COMPAT_JAVA_BACKEND=ON

Motivation — gap is documented, recurring, and previously unmeasured

• Support Oniguruma-based regex functions. #9897 (open) — Support Oniguruma-based regex functions. @spershin lists 8 concrete RE2-vs-Java semantic discrepancies seen in production Presto workloads. Status: prototype only, no shipped fix in either direction.
• why velox supports regex function using re2 instead of hyperscan. #9823 (closed) — repeated public acknowledgement that "re2 patterns and results aren't 100% compatible with Spark" ( @FelixYBW ), with maintainers explicitly noting that no proposed alternative met Velox's reliability bar.
• fix(regexp_replace): Move regex preprocessing to functions/lib for Spark reuse and fix backslash handling #10981 (merged) — moved prepareRegexpReplacePattern / prepareRegexpReplaceReplacement into functions/lib for Spark reuse. That helper covers one rewrite ((?<name>) → (?P<name>)); everything else (char-class intersection, \p{InGreek}, \p{javaXxx}, (?U) semantic inversion, \Q\E edges, …) silently falls on the floor.
• fix(spark): Fix regexp_replace with backslash-letter replacements #17578 (merged) / Spark's regexp_replace drops LF bytes when input is a NativeScan string column #17577 (closed) — most recent silent-data-correctness regression in regexp_replace. Root cause: the one-shot preprocessing introduced in fix(regexp_replace): Move regex preprocessing to functions/lib for Spark reuse and fix backslash handling #10981 had no compatibility regression set behind it. Exactly the class of bug a Java-corpus regression suite catches up front.
• ExpressionFuzzer fails on like: Max number of regex reached #7824 / kMaxCompiledRegexes leads to non-deterministic SparkSQL regexp_replace behavior #8438 / fix: Use EvictingCacheMap for compiled regular expressions #15953 — recurring kMaxCompiledRegexes / cache-eviction issues. No shared regression corpus existed to validate fixes; this PR provides one.
• refactor: Replace boost regex with re2 #15134 / fix: Replace boost::regex with RE2 in parse_duration function #15124 / fix: Replace boost::regex with RE2 regex in url_extract_parameter and fb_url_extract_parameter Presto function in Velox #14681 / build: Fix TpcdsConnector error "'this' pointer is null" #14878 / fix: Compile RE2 regex once in isColumnNameRequiringEscaping #16973 — the long stream of "replace boost::regex with RE2" migrations; each PR needed ad-hoc per-call-site verification because no shared compatibility harness existed.

Validation

OpenJDK 17 corpus (860 cases)             Java   860 / 860  100.00%
                                          Pcre2  860 / 860  100.00%   ←  drop-in Java parity
                                          Re2    757 / 860   88.02%   (98.18% on translatable subset *)
                                          Re2 raw (no translator)  528 / 860  61.40%

OpenJDK RegExTest.java (96 ported)        Java    96 /  96  100.00%
                                          Pcre2   94 /  96   97.92%
                                          Re2     84 /  96   87.50%   (90.32% on translatable subset *)

Translator unit tests                                        178 / 178 passing

* translatable subset = the test set excluding patterns the RE2 translator deliberately rejects as engine-impossible (lookaround, backref, possessive quantifiers, atomic groups, (?U) semantic inversion). RE2's algorithm guarantees linear time and intentionally does not support these constructs.

Net takeaway for users that opt in to PCRE2: effective Java parity (100% on the OpenJDK corpus, 97.92% on RegExTest), with PCRE2's own resource-cap protection enforcing predictable runtime.

Net takeaway for users that stay on RE2: the translator alone, applied to RE2, picks up 229 corpus cases (+27 pp) without touching the engine or its linear-time guarantee.

Out of scope

• We do not switch any existing production call site to PCRE2 in this PR. That belongs in separate per-call-site PRs with their own justification, perf data, and ReDoS-limits review.
• We do not change RE2's defaults or its kMaxCompiledRegexes behaviour. Issues ExpressionFuzzer fails on like: Max number of regex reached #7824 / kMaxCompiledRegexes leads to non-deterministic SparkSQL regexp_replace behavior #8438 / fix: Use EvictingCacheMap for compiled regular expressions #15953 are out of scope; this PR's corpus is the regression baseline a future fix can use.
• \b{g} (grapheme cluster boundary) and CANON_EQ full canonical-equivalence expansion are PCRE2 engine ceilings; documented in code comments. They are the only meaningful items not at 100% in the PCRE2 column above.

File-level overview

velox/external/regex_compat/                 ~3.0 kLOC   PCRE2 backend + test harness (default OFF)
  Re2Regex / Pcre2Regex / JavaRegex          three parallel concrete classes, re2::RE2-shaped API
  JavaMatcherAdapter                         re-builds java.util.regex.Matcher state machine
  BackendTestBase, TestMain                  TYPED_TEST plumbing + Java=100% guard
  tests/                                     OpenJDK corpus + ported pcre4j tests + RegExTest port

velox/functions/lib/java_pcre2_translator/   ~2.4 kLOC   reusable translator library
  PropertyMap / JdkPropertyExpander          \p{InGreek} / \p{javaXxx} / \p{IsXxx} rewriting
  ClassBodyParser / Evaluator / ClassRenderer  char-class AST + intersection resolution
  RangeSet                                   code-point range set algebra
  JavaRegexTranslator                        toPcre2Pattern / toRe2Pattern pipelines
  tests/                                     178 unit tests, 1:1 from pcre4j PR #606

Total: ~5.4 kLOC added; 0 lines changed in any existing source file outside CMake wiring.

Licensing

The translator port derives from pcre4j PR alexey-pelykh/pcre4j#606, which I authored (the entire org.pcre4j.regex.translate module was a from-scratch contribution by me to upstream pcre4j). pcre4j does not require a Contributor License Agreement, so copyright on my contribution remains mine, and I am dual-licensing it under Apache-2.0 for inclusion in this Apache-2.0 project.

The pcre4j upstream files carry a Copyright (C) 2024-2026 Oleksii PELYKH header — that is pcre4j's project-wide convention applied to all merged contributions, not a copyright assignment from contributors. The actual authorship of the translate module is mine and the Apache-2.0 grant here is therefore valid.

Per-file attribution headers in this PR retain the pcre4j provenance; LICENSE-NOTICE.md in the translator directory documents the full chain.

[netlify]

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	4f13785
🔍 Latest deploy log	https://app.netlify.com/projects/meta-velox/deploys/6a1f90d0f85b6d0008501f24

[github-actions]

Build Impact Analysis

Full build recommended. Files outside the dependency graph changed:

• velox/external/regex_compat/CMakeLists.txt
• velox/external/regex_compat/JavaRegex.cpp
• velox/external/regex_compat/JavaRegex.h
• velox/external/regex_compat/JvmFixture.cpp
• velox/external/regex_compat/JvmFixture.h
• velox/external/regex_compat/Pcre2Regex.cpp
• velox/external/regex_compat/Pcre2Regex.h
• velox/external/regex_compat/README.md
• velox/external/regex_compat/Re2Regex.cpp
• velox/external/regex_compat/Re2Regex.h
• ... and 18 more

These directories are not fully covered by the dependency graph. A full build is the safest option.

cmake --build _build/release

---

Slow path • Graph generated from PR branch

[mbasmanova]

Thank you for the thorough proposal and for addressing each of the historical concerns from #9897 and #9823. The approach is reasonable — PCRE2 with built-in limits is a better answer than Oniguruma or Hyperscan, and the translator improving RE2's corpus pass-rate from 61% to 88% is valuable on its own.

However, 12K lines in a single PR is not reviewable. This contains three independent components that should be separate PRs:

1. Java→PCRE2/RE2 translator (velox/functions/lib/java_pcre2_translator/) — builds unconditionally, benefits RE2 immediately, no new dependencies.
2. PCRE2 engine wrapper (velox/external/regex_compat/) — the new engine, gated behind VELOX_ENABLE_PCRE2_BACKEND.
3. Test harness (velox/external/regex_compat/tests/) — the JVM-oracle-based test framework.

Items 2 and 3 need broader discussion before we commit to them. Adding a second regex engine is a significant maintenance burden — two code paths to test, debug, and keep compatible. The opt-in CMake flag means CI either tests both paths (doubling regex test matrix) or only tests RE2 (leaving PCRE2 undertested). We'd like to discuss the long-term ownership and maintenance plan before merging the engine itself.

Please split accordingly. The translator PR alone would be a significant contribution and can be reviewed independently. The PCRE2 wrapper and test harness can follow once the translator lands.

For the translator PR: please add documentation (README or header-level docs) explaining what transformations it performs, what it doesn't handle, and the testing strategy. The 178 unit tests are there, but a reader needs to understand what coverage they provide — are these ported 1:1 from pcre4j? Do they cover the transformations that matter most for Spark/Presto patterns?

Also: before the translator PR, we'd like to understand the integration plan. How will existing Velox functions (regexp_extract, regexp_replace, regexp_like, like, Spark split) use the translator and/or PCRE2? Is this a per-function opt-in, a global config, or a per-query setting? The translator library is useful, but the review needs to understand how it connects to the rest of the system.