Skip to content

Release v1.5.2: Semantic OPs; Agent Interaction Quality; Cross-Document Dedup; Robustness & Performance

Choose a tag to compare

View all tags
@cmgzn cmgzn released this 29 May 02:10
· 68 commits to main since this release
v1.5.2
1787f47
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Major Updates

📊 Stats: 15 PRs merged, from 11 contributors

📈 Code diff: 217 files changed, with 18,972 insertions and 1,685 deletions

🧠 Semantic OPs MVP shipped: Introduced a new semantic operations framework with extract and condition filter capabilities, laying the groundwork for join/aggregation/top-k in future releases. #948

🤖 Agent interaction quality evaluation: New OPs and recipe for assessing interaction quality, with bad-case HTML report generation, and robust JSONL / HuggingFace meta loading support. #957

📄 Cross-document line-level deduplication: A new DocumentLineDeduplicator enables deduplication at the line level across multiple documents — useful for removing boilerplate or repeated content shared between files. #961

⚡ Lighter default installation: Reduced the size of default dependencies to significantly speed up first-time installation. #959

New OPs

  • DocumentLineDeduplicator: Performs cross-document line-level deduplication — ideal for removing shared boilerplate text, repeated headers/footers, or duplicated content across large corpora. #961
  • semantic_extract_mapper & semantic_condition_filter (MVP): Extract structured fields from unstructured text and filter based on semantic conditions, powered by the new semantic ops framework. #948
  • Interaction quality OPs (via dj-agents): Evaluate the quality of agent interactions with dedicated scoring operators and a recipe that generates HTML bad-case reports for human review. #957

Enhancements

  • O(n²) → O(n) in FrequencySpecifiedFieldSelector: Replaced repeated sum() accumulation with itertools.chain, eliminating quadratic behavior on large batches. #975
  • Exclude work_dir from fingerprint hashing: Cache fingerprints no longer change when work_dir differs, preventing unnecessary recomputation while preserving correct pickling behavior. #967
  • Reduce default dependency size: Trimmed the default install footprint so new users can get started faster without pulling in heavyweight optional packages. #959
  • Replace bs4 stub with beautifulsoup4: Switched to the canonical package name to prevent installation conflicts and deprecation warnings. #977

Fixed Bugs

  • Invalid max_new_tokens injected into API chat completions: The parameter was being incorrectly passed to chat completion API calls that don't support it, causing request failures. Now properly excluded. #983
  • initialize_ray crash without config: Ray initialization failed when no config was provided. Added proper fallback handling. #981
  • exit(1) replaced with raise in dataset processing: Using exit() in a library context kills the entire process. Errors now raise proper exceptions for library-safe error handling. #974
  • PyArrow 20.0.0+ batch JSON reading compatibility: Fixed an issue where open_json batch reading broke under PyArrow 20+. #942
  • text_keys not propagated to ops in service mode: The DJ service was not passing text_keys through to operators via get_init_configs, causing ops to use wrong keys. #960
  • Model utils sampling params & API client initialization: Fixed incorrect sampling parameter handling and optimized API client setup in model_utils.py. #962
  • Aesthetics-predictor normalization check too strict: The substring match for model normalization detection now supports local model paths, not just HuggingFace hub identifiers. #946
  • Temp dir removal failure in RayExecutor: Added fallback handling when temporary directory cleanup fails (e.g., due to permission issues or concurrent access). #943

Acknowledgements

New Contributors

Full Changelog: v1.5.1...v1.5.2

Contributors

ideal, HunterLine, and 4 other contributors
Assets 3
Loading