[Bug]: UTF-16 source files extract to nothing, and Windows-1252 names are cut short, with no warning #4145

Pre-flight checks

• I have checked the Troubleshooting section in the README

What happened?

The extractors hand tree-sitter the raw file bytes, and tree-sitter reads them as UTF-8. Two encodings that real Windows projects still contain go wrong, and nothing is printed either way:

• UTF-16 with a BOM (what Visual Studio saves as "Unicode", what Windows PowerShell 5.1 Out-File and > write, what older SSMS scripts are): the file gets its file node and nothing else. No classes, no methods, no calls.
• Windows-1252 (Notepad's old ANSI default): CaféOrder is stored as Caf\xE9Order. \xE9 is not UTF-8, so the identifier is cut at that byte and the class comes out as Order, with node id order_shop_order instead of order_shop_caféorder.

The regex-based readers (Svelte, Vue, Astro, Dart, Razor, ...) read with errors="replace", so UTF-16 turns into replacement characters there too. Three secondary passes (_is_spock_file, _extract_spock_fallback, _augment_cpp_string_tests) call read_text(errors="replace") with no encoding, so they follow the host locale and read the same file differently on Windows and Linux.

Steps to reproduce

python - <<'EOF'
from pathlib import Path
src = "namespace Shop {\n  public class CaféOrder {\n    public void Pay() { Charge(); }\n    public void Charge() { }\n  }\n}\n"
for name, data in {"utf8": src.encode("utf-8"), "cp1252": src.encode("cp1252"), "utf16": src.encode("utf-16")}.items():
    d = Path("enc_" + name); d.mkdir(exist_ok=True); (d / "Order.cs").write_bytes(data)
EOF
graphify update enc_utf8 --force --no-cluster
graphify update enc_cp1252 --force --no-cluster
graphify update enc_utf16 --force --no-cluster
# then compare the node labels in each enc_*/graphify-out/graph.json

Error output or graph output

Same C# / Java / Python file, saved four ways, `graphify update --force --no-cluster` on 0.9.77:

lang  encoding       calls  nodes found
cs    utf-8              1  .Charge(), .Pay(), .Send(), CaféOrder, Invoice, Shop
cs    utf-8 + BOM        1  .Charge(), .Pay(), .Send(), CaféOrder, Invoice, Shop
cs    windows-1252       1  .Charge(), .Pay(), .Send(), Invoice, Order, Shop
cs    utf-16 + BOM       0  (nothing)
java  utf-8              1  .charge(), .pay(), CaféOrder
java  windows-1252       1  .charge(), .pay(), Order
java  utf-16 + BOM       0  (nothing)
py    utf-8              1  .charge(), .pay(), .send(), CaféOrder, Invoice
py    windows-1252       1  .charge(), .pay(), .send(), Invoice, Order
py    utf-16 + BOM       0  (nothing)

No warning is printed in any row.

Graphify version

0.9.77 (5c7b847)

Operating System

Windows

Python Version

3.12

Installation Method

built from source (git clone)

Additional Environment Details

AST-only update, no provider environment variables set. A clean checkout of v8 at 5c7b847 reproduces it.

Additional context

detect._read_ignore_text already solves the same problem for ignore files (UTF-8 first, then UTF-16 by BOM, then a fallback with a one-time warning). Source files never got the same treatment. I have a fix with regression tests ready and will open a PR right after this.