Skip to content

v2.2.0

Choose a tag to compare

@github-actions github-actions released this 29 Oct 16:55
· 6 commits to main since this release

Added

  • Metadata filtering: New filter_by_metadata() method for efficient key-value filtering using SQLite's JSON_EXTRACT
  • Metadata counting: New count_by_metadata() method to count records matching metadata filters
  • Combined search: New similarity_search_with_filter() method combining vector similarity with metadata filtering
  • Support for nested JSON paths in metadata filters (e.g., {"author.name": "Alice"})
  • Pagination support in filter_by_metadata() with limit and offset parameters
  • New validation function validate_metadata_filters() for secure filter validation
  • New utility function build_metadata_where_clause() for safe SQL generation
  • Comprehensive test coverage for metadata filtering (unit, integration, and security tests)
  • New example advanced_metadata_queries.py demonstrating nested paths and complex queries
  • Updated metadata_filtering.py example with new filtering methods
  • Export/import functionality: New methods for data portability
    • export_to_json() - Export records to JSON Lines format
    • import_from_json() - Import records from JSON Lines format
    • export_to_csv() - Export records to CSV format
    • import_from_csv() - Import records from CSV format
  • Support for filtered exports using metadata filters
  • Batch processing for memory-efficient import/export of large datasets
  • Optional embedding inclusion in exports
  • New io module with import/export utilities
  • Comprehensive tests for import/export functionality
  • New example export_import_example.py demonstrating backup/restore workflows

Security

  • SQL injection prevention in metadata filter keys
  • Validation of JSON paths to prevent malicious queries
  • Parameterized queries for all metadata filtering operations

Improved

  • Data migration workflows now much easier
  • Backup and restore capabilities for production use
  • Interoperability with external systems via CSV/JSON

Documentation

  • Added "Metadata Filtering" section to README with examples
  • Added "Export/Import" section to README with examples
  • Updated examples list in README
  • Added comprehensive docstrings for new methods