gh-138122: Allow tachyon to write and read binary output

[pablogsal]

---

📚 Documentation preview 📚: https://cpython-previews--142730.org.readthedocs.build/

• Issue: Implement PEP 799 – A dedicated profiling package for organizing Python profiling tools #138122

[pablogsal]

I ran some benchmarks to validate the binary format implementation. Here's what I found.

The test workload ran a bunch of tests from the test suite (test_list, test_dict, test_tokenize, test_exceptions, test_syntax, test_threading) taking approx 28 seconds on Linux with my Intel hybrid CPU clocked at 4.9 GHz, using ZSTD level 5 streaming compression with a 2 MB window.

The binary writer hits 199,175 samples/second in this run, capturing 5.6 million samples. For reference that enough to profile 199 threads at once with 1ms sampling. I ran perf record --call-graph dwarf to see where CPU time actually goes. Here's the breakdown by shared object:

63.1%  python (interpreter running the tests)
31.9%  _remote_debugging (the profiler extension)
 3.9%  libc
 0.19% libzstd

ZSTD compression is 0.19% of total CPU time. The binary format overhead is essentially free.

Within the profiler extension, the hot functions are:

13.2%  _Py_RemoteDebug_PagedReadRemoteMemory (reading target process memory)
 7.4%  _remote_debugging_RemoteUnwinder_get_stack_trace_impl (unwinding stacks)
 3.5%  process_thread_sample
 1.2%  frame_key_compare_func
 1.2%  parse_linetable
 0.7%  string_compare_func
 0.6%  string_hash_func

The binary writing and compression functions don't even show up in the profile: they're below the 0.5% threshold. All the profiler overhead is in reading remote memory and unwinding stacks, not in the output format. Compression gets a 159.6x ratio on profiling data, turning 74.51 bytes per sample into 0.47 bytes. A 1-hour profile at 1000 samples/sec that would normally take 268 MB on disk shrinks to just 1.7 MB. The interning system stores each unique string once and references it an average of 1,658 times. Each unique frame gets referenced 653 times. Without interning, string data alone would be 79 MB. With interning, it's 42.6 KB.

Raw numbers from the run:

Captured 5,659,261 samples in 28.41 seconds
Sample rate: 199,175.43 samples/sec
Error rate: 0.31%

Binary Encoding:
  Records:          111,957
    RLE repeat:     30,314 (27.1%) [5,559,903 samples]
    Full stack:     4,426 (4.0%)
    Suffix match:   22,065 (19.7%)
    Pop-push:       55,152 (49.3%)

Frame Efficiency:
  Frames written:   140,547
  Frames saved:     192,318,495 (99.9%)

The encoding stats show RLE (run-length encoding) is working well as 27% of records are RLE repeats covering 5.5M samples. The frame efficiency shows we're saving 99.9% of frame writes through the encoding schemes.