Skip to content

Commit fa26511

Browse files
GaaraZhuclaude
andcommitted
feat(gate2): improve AU/NZ phone number detection coverage
- Add AU/NZ landline patterns (+61 [2378] / +64 [34679], local and international) - Split AU/NZ phone confidence: international prefix (+61/+64) scores 0.85 (auto-redacts in any column); local format scores 0.70 (needs key context) - Fix +610/+640 stray leading zero after country code silently skipped - Strip ASCII whitespace from values before regex matching so variants like "+ 640220831619" and "+6 40220831619" are caught - Add phone column synonyms: phnumber, ph_number, cell, cell_number Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent ff70779 commit fa26511

5 files changed

Lines changed: 202 additions & 8 deletions

File tree

CHANGELOG.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,16 @@
1+
## [Unreleased]
2+
3+
### 🐛 Bug Fixes
4+
5+
- Fix AU/NZ international phone numbers not matched when `+` is separated from digits by whitespace (e.g. `+ 640220831619`, `+6 40220831619`) — values are now whitespace-stripped before regex matching
6+
- Fix `+610`/`+640` (stray leading zero after country code) silently skipped by AU/NZ phone patterns
7+
8+
### ✨ Improvements
9+
10+
- Add AU/NZ landline patterns: `+61 [2378]`/`0[2378]` (NSW/ACT, VIC/TAS, QLD, WA/SA/NT) and `+64 [34679]`/`0[34679]` (South Island, Wellington, Taranaki, Waikato, Auckland)
11+
- Split AU/NZ phone confidence: international-prefix numbers (`+61`/`+64`) score 0.85 and auto-redact in any column; local-format numbers score 0.70 and require a PII-named column
12+
- Add phone column synonyms: `phnumber`, `ph_number`, `cell`, `cell_number`
13+
114
## [0.8.7] - 2026-05-25
215

316
### 🚀 Features

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -216,7 +216,7 @@ Stats are collected by default and written to a local JSONL log on disk — they
216216
- **Commands not in `tools:`.** The AI can invoke them freely; their output is never inspected.
217217
- **Non-JSON tool output.** Plain text, CSV, and other formats pass through unchanged. Configure tools to emit JSON.
218218
- **Encoded or obfuscated PII.** Base64-encoded emails, URL-encoded values, or deliberately spaced strings (`a l i c e @ e x a m p l e . c o m`) are not detected.
219-
- **Non-US PII by value alone.** The built-in SSN regex requires dashes and the phone pattern is US-centric. AU/NZ identifiers are covered at the value layer: ABN (mod-89 checksum), Medicare (mod-10 checksum), formatted TFN and IRD numbers (mod-11, separators required), NZ NHI (alpha-prefix regex), and NZ bank account numbers. Bare/unformatted TFN and IRD strings without separators are not detected by value alone — column-name matching remains the safety net for those. Other non-AU/NZ formats rely solely on column-name matching — extend `pii.column_names` or `pii.patterns` for your region.
219+
- **Non-US PII by value alone.** The built-in SSN regex requires dashes. AU/NZ phone numbers are caught by value — mobile (`04XX`/`02X` local, `+61 4XX`/`+64 2X` international) and landline (`0[2378]`/`0[34679]` local, `+61 [2378]`/`+64 [34679]` international) — including the common `+610`/`+640` stray-leading-zero variant and arbitrary whitespace in the number. International-prefix numbers (`+61`/`+64`) auto-redact regardless of column name; local-format numbers require a PII-named column. Other AU/NZ identifiers are also covered at the value layer: ABN (mod-89 checksum), Medicare (mod-10 checksum), formatted TFN and IRD numbers (mod-11, separators required), NZ NHI (alpha-prefix regex), and NZ bank account numbers. Bare/unformatted TFN and IRD strings without separators are not detected by value alone — column-name matching remains the safety net for those. Other non-AU/NZ formats rely solely on column-name matching — extend `pii.column_names` or `pii.patterns` for your region.
220220
- **PII already in the model's context** from prior turns, system prompts, file reads, or earlier summarisation. Gate filters what goes *into* the model from configured tools; what's already there stays there.
221221
- **Tool-side network exfiltration.** If a configured tool sends data to an external service directly (rather than returning it via stdout), gate never sees it.
222222
- **Write operations.** `INSERT`, `UPDATE`, `DELETE` are not inspected or blocked.

crates/common/src/patterns.rs

Lines changed: 121 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -95,15 +95,55 @@ pub const BUILTIN_PATTERNS: &[BuiltinPattern] = &[
9595
confidence: 0.85,
9696
},
9797
BuiltinPattern {
98-
// AU mobile: 04XX XXX XXX / +61 4XX XXX XXX
98+
// AU mobile international: +61 4XX XXX XXX / +610 4XX... (stray leading zero after CC)
99+
// +61 is unambiguous — high confidence even without a PII column name.
99100
name: "phone",
100-
regex: r"\b(?:\+61[\s-]?|0)4\d{2}[\s-]?\d{3}[\s-]?\d{3}\b",
101+
regex: r"\+61[\s-]?0?4\d{2}[\s-]?\d{3}[\s-]?\d{3}\b",
102+
confidence: 0.85,
103+
},
104+
BuiltinPattern {
105+
// AU landline international: +61 [2378] XXXX XXXX / +610 [2378]...
106+
// area codes: 2=NSW/ACT, 3=VIC/TAS, 7=QLD, 8=WA/SA/NT.
107+
name: "phone",
108+
regex: r"\+61[\s-]?0?[2378][\s-]?\d{4}[\s-]?\d{4}\b",
109+
confidence: 0.85,
110+
},
111+
BuiltinPattern {
112+
// AU mobile local: 04XX XXX XXX — ambiguous without key context (low confidence).
113+
name: "phone",
114+
regex: r"\b04\d{2}[\s-]?\d{3}[\s-]?\d{3}\b",
115+
confidence: 0.70,
116+
},
117+
BuiltinPattern {
118+
// AU landline local: 0[2378] XXXX XXXX — ambiguous without key context.
119+
name: "phone",
120+
regex: r"\b0[2378][\s-]?\d{4}[\s-]?\d{4}\b",
121+
confidence: 0.70,
122+
},
123+
BuiltinPattern {
124+
// NZ mobile international: +64 2X XXX XXXX(X) / +640 2X... (stray leading zero after CC)
125+
// +64 is unambiguous — high confidence even without a PII column name.
126+
name: "phone",
127+
regex: r"\+64[\s-]?0?2\d[\s-]?\d{3}[\s-]?\d{4,5}\b",
128+
confidence: 0.85,
129+
},
130+
BuiltinPattern {
131+
// NZ landline international: +64 [34679] XXX XXXX / +640 [34679]...
132+
// area codes: 3=South Island, 4=Wellington, 6=Taranaki/Hawke's Bay, 7=Waikato, 9=Auckland.
133+
name: "phone",
134+
regex: r"\+64[\s-]?0?[34679][\s-]?\d{3}[\s-]?\d{4}\b",
135+
confidence: 0.85,
136+
},
137+
BuiltinPattern {
138+
// NZ mobile local: 02X XXX XXXX(X) — ambiguous without key context (low confidence).
139+
name: "phone",
140+
regex: r"\b02\d[\s-]?\d{3}[\s-]?\d{4,5}\b",
101141
confidence: 0.70,
102142
},
103143
BuiltinPattern {
104-
// NZ mobile: 02X XXX XXXX(X) / +64 2X XXX XXXX(X)
144+
// NZ landline local: 0[34679] XXX XXXX — ambiguous without key context.
105145
name: "phone",
106-
regex: r"\b(?:\+64[\s-]?|0)2\d[\s-]?\d{3}[\s-]?\d{4,5}\b",
146+
regex: r"\b0[34679][\s-]?\d{3}[\s-]?\d{4}\b",
107147
confidence: 0.70,
108148
},
109149
BuiltinPattern {
@@ -133,6 +173,9 @@ const TOKEN_SYNONYMS: &[(&str, &str)] = &[
133173
("mobile", "phone"),
134174
("tel", "phone"),
135175
("fax", "phone"),
176+
("phnumber", "phone"), // ph_number / phnumber
177+
("cellnumber", "phone"), // cell_number
178+
("cell", "phone"),
136179
// ── SSN ───────────────────────────────────────────────────────────────────
137180
("ssn", "ssn"),
138181
("socialsecuritynumber", "ssn"), // trigram: social_security_number
@@ -784,7 +827,7 @@ mod tests {
784827
] {
785828
assert!(names.contains(expected), "missing builtin: {}", expected);
786829
}
787-
assert_eq!(patterns.len(), 10);
830+
assert_eq!(patterns.len(), 16);
788831
}
789832

790833
#[test]
@@ -900,6 +943,76 @@ mod tests {
900943
}
901944
}
902945

946+
fn any_phone_matches(s: &str) -> bool {
947+
CompiledPattern::from_builtins()
948+
.into_iter()
949+
.filter(|p| p.name == "phone")
950+
.any(|p| p.regex.is_match(s))
951+
}
952+
953+
#[test]
954+
fn nz_mobile_variants_all_match() {
955+
for num in &[
956+
"0220831619", // local format compact
957+
"022 083 1619", // local format spaced
958+
"+64220831619", // E.164 proper
959+
"+64 22 083 1619", // E.164 spaced
960+
"+640220831619", // stray leading zero after CC (common mistake)
961+
] {
962+
assert!(any_phone_matches(num), "expected NZ phone match: {}", num);
963+
}
964+
}
965+
966+
#[test]
967+
fn au_mobile_variants_all_match() {
968+
for num in &[
969+
"0412345678", // local compact
970+
"0412 345 678", // local spaced
971+
"+61412345678", // E.164 proper
972+
"+61 412 345 678", // E.164 spaced
973+
"+610412345678", // stray leading zero
974+
] {
975+
assert!(any_phone_matches(num), "expected AU phone match: {}", num);
976+
}
977+
}
978+
979+
#[test]
980+
fn au_landline_variants_all_match() {
981+
for num in &[
982+
"+61291711868", // NSW international compact
983+
"+61 2 9171 1868", // NSW international spaced
984+
"0291711868", // NSW local compact
985+
"02 9171 1868", // NSW local spaced
986+
"+61391711868", // VIC international
987+
"+61791711868", // QLD international
988+
"+61891711868", // WA international
989+
] {
990+
assert!(
991+
any_phone_matches(num),
992+
"expected AU landline match: {}",
993+
num
994+
);
995+
}
996+
}
997+
998+
#[test]
999+
fn nz_landline_variants_all_match() {
1000+
for num in &[
1001+
"+6493092677", // Auckland international compact
1002+
"+64 9 309 2677", // Auckland international spaced
1003+
"093092677", // Auckland local compact
1004+
"09 309 2677", // Auckland local spaced
1005+
"+6443092677", // Wellington international
1006+
"+6433092677", // South Island international
1007+
] {
1008+
assert!(
1009+
any_phone_matches(num),
1010+
"expected NZ landline match: {}",
1011+
num
1012+
);
1013+
}
1014+
}
1015+
9031016
// --- Credit card regex ---
9041017

9051018
#[test]
@@ -1014,6 +1127,9 @@ mod tests {
10141127
fn classify_underscore_separated() {
10151128
assert_eq!(classify_column("email_address"), Some("email"));
10161129
assert_eq!(classify_column("phone_number"), Some("phone"));
1130+
assert_eq!(classify_column("ph_number"), Some("phone"));
1131+
assert_eq!(classify_column("phnumber"), Some("phone"));
1132+
assert_eq!(classify_column("cell_number"), Some("phone"));
10171133
assert_eq!(classify_column("first_name"), Some("name"));
10181134
assert_eq!(classify_column("last_name"), Some("name"));
10191135
assert_eq!(classify_column("full_name"), Some("name"));

crates/common/src/redactor.rs

Lines changed: 66 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -571,11 +571,15 @@ fn scan_string(
571571

572572
// 6. Regex pattern scan. column_name_boost is not applied here: any column
573573
// whose name matched the denylist was already handled in steps 1–3.
574+
// Match against a whitespace-stripped copy so that formatting variants like
575+
// "+ 64 022 083 1619" or "+6 40220831619" collapse to a canonical form.
576+
// The original `s` is still used for redaction output and all other steps.
577+
let s_compact = s.replace(|c: char| c.is_ascii_whitespace(), "");
574578
// Short-circuit on the first match whose confidence already clears the threshold —
575579
// no need to continue scanning once we know we'll redact.
576580
let mut best: Option<(&str, f32)> = None;
577581
for p in patterns {
578-
if p.regex.is_match(&s) {
582+
if p.regex.is_match(&s_compact) {
579583
let score = p.confidence;
580584
if score >= config.confidence_threshold {
581585
if vb {
@@ -942,6 +946,67 @@ mod tests {
942946
assert_eq!(out["mail"], "[PII:email]");
943947
}
944948

949+
#[test]
950+
fn phnumber_key_with_nz_phone_is_redacted() {
951+
// "phnumber" is a common abbreviated key; the NZ mobile value alone scores 0.70
952+
// (below the 0.80 threshold), but the key synonym must force-redact it.
953+
let input = json!({"name": "gary", "phnumber": "0220831619"});
954+
let out = redact(input, &plan(), &cfg());
955+
assert_eq!(out["phnumber"], "[PII:phone]");
956+
assert_eq!(out["name"], "gary");
957+
}
958+
959+
#[test]
960+
fn nz_international_prefix_redacted_in_generic_column() {
961+
// +64 is unambiguous (NZ country code) — value-layer confidence 0.85 clears the
962+
// default 0.80 threshold, so redaction must happen even with a non-PII key name.
963+
let input = json!({"name": "gary", "mber": "+64220831619"});
964+
let out = redact(input, &plan(), &cfg());
965+
assert_eq!(out["mber"], "[PII:phone]");
966+
assert_eq!(out["name"], "gary");
967+
}
968+
969+
#[test]
970+
fn au_international_prefix_redacted_in_generic_column() {
971+
let input = json!({"ref": "+61412345678"});
972+
let out = redact(input, &plan(), &cfg());
973+
assert_eq!(out["ref"], "[PII:phone]");
974+
}
975+
976+
#[test]
977+
fn au_landline_international_redacted_in_generic_column() {
978+
// +61 2 9171 1868 is an AU landline (Sydney); +61 is unambiguous so value-layer
979+
// confidence 0.85 clears the threshold even with a non-PII key like "mber".
980+
let input = json!({"name": "gary", "mber": "+61291711868"});
981+
let out = redact(input, &plan(), &cfg());
982+
assert_eq!(out["mber"], "[PII:phone]");
983+
assert_eq!(out["name"], "gary");
984+
}
985+
986+
#[test]
987+
fn nz_landline_international_redacted_in_generic_column() {
988+
// +64 9 309 2677 is a NZ landline (Auckland); +64 is unambiguous.
989+
let input = json!({"name": "gary", "mber": "+6493092677"});
990+
let out = redact(input, &plan(), &cfg());
991+
assert_eq!(out["mber"], "[PII:phone]");
992+
assert_eq!(out["name"], "gary");
993+
}
994+
995+
#[test]
996+
fn spaced_international_prefix_caught_via_compaction() {
997+
// Whitespace is stripped before regex matching, so "+" followed by spaces
998+
// and digits in any position still resolves to the canonical form.
999+
for val in &[
1000+
"+ 640220831619", // space after +
1001+
"+6 40220831619", // space mid-CC
1002+
"+64 0220831619", // space after CC
1003+
] {
1004+
let input = json!({"mber": val});
1005+
let out = redact(input, &plan(), &cfg());
1006+
assert_eq!(out["mber"], "[PII:phone]", "expected redaction for {}", val);
1007+
}
1008+
}
1009+
9451010
#[test]
9461011
fn non_pii_name_token_not_triggered() {
9471012
// "product_name" must not be redacted — bigram "productname" is not in synonyms.

docs/configuration.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ Config lives at `~/.config/gate/config.yaml` (override with `GATE_CONFIG`).
1717
| **Names** | `first_name`, `last_name`, `full_name`, `given_name`, `family_name`, `surname`, `preferred_name`, `middle_name`, `maiden_name`, `salutation`; `<entity>_name` where entity is one of: contact, customer, client, employee, patient, member, owner, recipient, sender, spouse, parent, guardian, manager, sibling, children |
1818
| **Demographics** | `gender`, `sex`, `nationality`, `citizenship` |
1919
| **Government IDs** | `passport`, `license` / `licence` / `driver_license_number` / `driver_licence_number`, `ssn` / `social_security_number`, `national_id`, `tax_number` / `tax_id` / `tax_file_number` / `ird_number` / `tfn`, `abn`, `visa_number`, `resident_id`, `immigration_id` |
20-
| **Contact** | `email` / `email_address` / `mail`, `phone` / `phone_number` / `mobile`, `fax` |
20+
| **Contact** | `email` / `email_address` / `mail`, `phone` / `phone_number` / `mobile` / `cell` / `cell_number` / `ph_number` / `phnumber`, `fax` |
2121
| **Date of birth** | `dob`, `birth`, `birthday`, `date_of_birth`, `birth_date`, `dateOfBirth` |
2222
| **Location of birth** | `birth_country`, `birth_place`, `birth_city`, `country_of_birth`, `place_of_birth`, `city_of_birth`, `state_of_birth` |
2323
| **Address & location** | `address` / `addr`, `street`, `postcode`, `zip`, `latitude`, `longitude`, `gps`, `coordinates` |

0 commit comments

Comments
 (0)