Optimize jieba hot paths

[messense]

Summary

This PR applies more performance optimizations that measured as wins and leaves out the candidates that either regressed or carried feature-parity risk.

Kept changes:

• Pack HMM emission probabilities into one per-character lookup so Viterbi no longer performs one emission lookup per state.
• Cache Jieba::log_total and refresh it when dictionary totals change.
• Reuse DAG storage in cut_all and replace the route map/max_by chain with an explicit max loop.
• Precompute TextRank keyword candidates once per extraction.
• Add an internal hash lookup table for keyword stop words while preserving the public BTreeSet API.
• Avoid unconditional lowercase allocation during keyword filtering.
• Skip duplicate dictionary checks during the initial bundled default dictionary load.
• Keep clippy cleanups for POS tagging helpers after dropping the scratch-buffer prototype.

Not included:

• The regex-free splitter prototype, because it risked regressing parity with Python jieba splitting behavior.
• The POS tagging scratch/context prototype, because the explicit context version regressed compared with the kept branch.
• The cut_for_search prefix prototype and route-buffer reuse prototype, because they did not improve the measured workloads.

Benchmarks

Measured locally against main at 1e77e50. Criterion values are median estimates from cargo bench -p jieba-rs --features tfidf,textrank; the tag_with_oov row uses the corrected focused rerun after restoring the faster indexed POS loop.

Benchmark	main	this branch	Improvement
jieba/new	55.582 ms	51.975 ms	6.5% faster
cut/no_hmm	646.51 ns	602.35 ns	6.8% faster
cut/with_hmm	1.0369 us	793.84 ns	23.4% faster
cut/cut_all	738.58 ns	620.94 ns	15.9% faster
cut/cut_for_search	1.2418 us	1.0248 us	17.5% faster
tokenize/default_mode	1.0347 us	791.30 ns	23.5% faster
tokenize/search_mode	1.2352 us	1.0230 us	17.2% faster
jieba/tag	1.0989 us	924.55 ns	15.9% faster
jieba/tag_with_oov	2.3739 us	2.1991 us	7.4% faster
keywords/tfidf	2.1164 us	1.4797 us	30.1% faster
keywords/textrank	4.9064 us	3.0443 us	38.0% faster
multithreaded/single_thread	973.91 us	804.46 us	17.4% faster
multithreaded/multi_thread	183.25 us	174.25 us	4.9% faster

Across these Criterion rows, the unweighted average speedup is about 17.3% and the median speedup is 17.2%.

The weicheng release binary improved from a 10-run median of 667.5 ms on main to 593 ms on this branch, about 11.2% faster, using:

cargo build --release -p weicheng
./target/release/weicheng

Validation

cargo fmt --all --check
cargo test -p jieba-rs --all-features
cargo clippy --all-targets --all-features -- -D warnings

[codecov]

Codecov Report

❌ Patch coverage is 95.20000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.37%. Comparing base (1e77e50) to head (21e04f8).

Files with missing lines	Patch %	Lines
jieba/src/lib.rs	95.23%	3 Missing ⚠️
jieba/src/posseg.rs	50.00%	2 Missing ⚠️
jieba/src/hmm.rs	94.11%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #152      +/-   ##
==========================================
+ Coverage   83.10%   83.37%   +0.26%     
==========================================
  Files          10       10              
  Lines        2001     2051      +50     
==========================================
+ Hits         1663     1710      +47     
- Misses        338      341       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[codspeed-hq]

Merging this PR will improve performance by 14.47%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 9 improved benchmarks
✅ 4 untouched benchmarks

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
⚡	single_thread	10.5 ms	8.6 ms	+22.57%
⚡	multi_thread	10.5 ms	8.6 ms	+22.05%
⚡	textrank	67.1 µs	56.1 µs	+19.66%
⚡	tfidf	43.4 µs	38.7 µs	+11.94%
⚡	search_mode	41.5 µs	37.3 µs	+11.43%
⚡	with_hmm	34.4 µs	30.9 µs	+11.14%
⚡	cut_all	23.9 µs	21.5 µs	+11.11%
⚡	default_mode	34.4 µs	31 µs	+11.03%
⚡	tag	37.1 µs	33.6 µs	+10.21%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing codex/perf-optimizations (21e04f8) with main (1e77e50)}