Repository navigation
Commit 613ad56
feat(vsjoin): implement VSJoin dual-index architecture (v0.1.3.1) (#101)
* feat(vsjoin): Refine VSJoin plan and add detailed task docs
This commit introduces a comprehensive plan and task breakdown for the new VSJoin implementation.
- **Refine VSJoin Plan:**
- The main design document is updated to unify naming (removing "v2").
- Clarifies the replacement of v1 components with the new architecture (TwoTierWindowState + ConcurrencyManager).
- Adds detailed sections on UID deduplication and the RCU-based load balancing mechanism (AssignmentTable).
- **Add Detailed Task Documents:**
- Creates a new directory.
- Adds markdown files for each implementation step (Task 01 to 09), providing clear instructions for development. These files are force-added as the parent directory is in .gitignore.
- **Cleanup Unused v1 Components:**
- Deletes and as they are no longer needed in the new design.
* feat(test): Add unit test for VSJoin rebuild functionality
This commit adds a new unit test for the VSJoin rebuild process, enhancing the test coverage for the VSJoin implementation. The test file is located at UnitTest/test_vsjoin_rebuild.cpp and is configured to run with a specified timeout of 300 seconds.
* feat(vsjoin): implement VSJoin dual-index architecture (tasks 01-04)
- Task 01: VSJoinMethod basic implementation
- Implement ExecuteEager() with dual-layer query logic (Global + Local)
- Add UID deduplication using local unordered_set
- Implement setGlobalIndexIds/setLocalIndexIds/setWindowStates interfaces
- Task 02: JoinStrategyFactory integration
- Add JoinAlgorithm::VSJOIN enum
- Add vsjoin_* configuration parameters
- Integrate VSJOIN case in factory (fallback to BruteForce for now)
- Task 03: JoinOperator VSJoin special path
- Add vsjoin_local_*_ids_ and vsjoin_global_*_id_ members
- Implement VSJoin-specific updateSideWithState() (insert to local index only)
- Support LSH partitioner for VSJoin
- Task 04: Background rebuild mechanism
- Implement globalIndexRebuildLoop() with periodic rebuild
- Use std::call_once for thread-safe single startup
- Implement local unordered_set deduplication (lock-free)
- Add atomic index replacement via replace_index_by_id()
Integration tests passed:
- bruteforce: 7/7 tests, recall=1.000
- ivf: 7/7 tests, recall=0.999
- hdr_tree: 7/7 tests, recall=1.000
- clustered_join: 3/3 tests, recall=1.000
* fix(vsjoin): fix dangling pointer bug in globalIndexRebuildLoop
The snapshot ownership was released prematurely in the rebuild loop,
causing the pointers stored in unique_left_records and unique_right_records
to become dangling. This led to IVF centroids being initialized with
garbage data (dimension=0), causing 'Vectors must be of the same size'
errors during query_for_join.
Fix: Keep snapshot vectors alive until build_index_from_records completes
by storing them in left_snapshots/right_snapshots containers.
Also includes:
- Improved test assertions for query record dimension
- Code cleanup for VSJoin factory integration
* chore: bump version to 0.1.3.1
* chore: add TODO issue links via todo-to-issue-action
* fix(test): correct LSH window_state_type expectation to PARTITIONED
The inferDefaults() for LSH algorithm returns PARTITIONED, not PARTITIONED_VECTOR.
PARTITIONED_VECTOR is only used for VSJOIN algorithm.
* feat(vsjoin): 完成 Task 06 - VSJoin 集成测试链路打通
sed -n '168,195p' /root/sageFlow/src/operator/utils/join_strategy_factory.cpp
1. 配置验证 ✅
- integration_test_cases.toml 包含 4 个启用的 VSJoin 测试用例
- 配置包含必要参数 (vsjoin_num_hash_functions, vsjoin_boundary_threshold 等)
- num_partitions 参数设置合理
2. 链路打通验证 ✅
- JoinStrategyFactory::create() 正确创建 VSJoinMethod
- TwoTierWindowState 正确初始化
- Global/Local Index 正确创建和管理
- 后台重建线程正常工作
3. 测试执行验证 ✅
- test_join_baseline_integration --gtest_filter='*vsjoin*' 执行成功
- run_integration_test.py --methods vsjoin 执行成功
- 测试报告正确生成
4. 召回率验证 ✅
- vsjoin_baseline: Recall=1.0 (预期>=0.70)
- vsjoin_high_recall: Recall=1.0 (预期>=0.75)
- vsjoin_parallelism_scaling: Recall=1.0 (并行度 1-16)
- vsjoin_low_latency: Recall>=0.60
sed -n '168,195p' /root/sageFlow/src/operator/utils/join_strategy_factory.cpp
- 修复 JoinConfigValidator 允许 LSH + TWO_TIER 组合
- 修复 JoinOperator 中 VSJoin 的 use_index_ 和 index_id 计算
- 更新 integration_test_cases.toml 添加 VSJoin 测试用例
- 更新 task06_integration_test.md 添加集成测试框架链路任务
* feat(vsjoin): implement Task 07-09 - AssignmentTable, LoadMonitor, Logical Partition Routing, and Load Balancing Tests
Task 07: AssignmentTable (RCU) + LoadMonitor
- Implement PartitionAssignment with RCU pattern for lock-free reads
- Implement LoadMonitor for tracking partition load statistics
- Support logical-to-physical partition mapping
- Batch atomic updates for assignment table
Task 08: Logical Partition Routing Integration
- Add VSJoin routing methods in JoinOperator
- Implement routeByLSHBucket() for query routing
- Implement determineTargetPartitions() with multi-partition support
- Integrate with CentroidPartitioner for initial assignment
Task 09: Load Balancing Tests
- Add comprehensive unit tests for LoadMonitor
- Add comprehensive unit tests for PartitionAssignment
- Add integration tests for VSJoin routing
- Add load balancing scenario tests
* test: isolate per-run outputs (reports/logs/charts)
* feat(python): SAGE integration with Python bindings v0.1.3 (#100)
* feat(python): add SAGE integration examples and documentation
- Add test_sageflow_cpp_runtime.py: comprehensive C++ runtime verification tests
- Add sage_sageflow_dual_stream_join.py: dual-stream Join pipeline demo for RAG
- Add SAGEFLOW_SAGE_INTEGRATION_GUIDE.md: integration guide for SAGE + SageFlow
The dual-stream Join demo shows:
- Query Stream + Document Stream architecture
- SageFlow C++ engine for vector similarity join
- RAG context building from join results
* feat(python): complete SAGE integration with Python bindings and examples
Modified files:
- sage_flow/__init__.py: update exports for SAGE integration
- sage_flow/bindings.cpp: enhance Python bindings for dual-stream Join
- test/CMakeLists.txt: add new test targets
New files:
- docs/LLM_INFERENCE_PIPELINE_GUIDE.md: LLM inference pipeline guide
- docs/VSJOIN_DESIGN_REVIEW_REPORT.md: VSJoin design review
- examples/python/llm_inference_service_demo.py: LLM service demo
- examples/python/llm_pipeline_example.py: LLM pipeline example
- examples/python/sage_integrated_pipeline_demo.py: SAGE integration demo
- test/IntegrationTest/test_non_join_operators_pipeline.cpp: non-join ops test
- test/UnitTest/python/: Python unit tests
* chore: bump version to 0.1.3 for PyPI release
This release includes:
- SAGE integration Python bindings
- Dual-stream Join pipeline support
- Comprehensive C++ runtime verification tests
- RAG pipeline examples and documentation
* fix(test): correct LSH window_state_type expectation to PARTITIONED
The inferDefaults() for LSH algorithm returns PARTITIONED, not PARTITIONED_VECTOR.
PARTITIONED_VECTOR is only used for VSJOIN algorithm.
* fix(test): remove index_id checks from LSH test
LSH algorithm doesn't use external index, so left_index_id and right_index_id
can be -1. Removed these checks to match feat/implement_vsjoin branch.
* doc: add sage pipeline markdown file
* fix: unitest failed
* fix: update VSJoin test to expect TWO_TIER window state type
* fix(pybind): fix alpha hardcode bug in NORMALIZED mode and add window_size_ms parameter
- Fix BruteForceBaseline to use configured similarity_alpha instead of hardcoded 0.1 in NORMALIZED mode
- Add 7-parameter join() overload with window_size_ms support in pybind interface
- Add StreamingSource for dynamic streaming input (vs batch-oriented SimpleStreamSource)
- Update JoinStrategyConfig to properly propagate similarity_alpha and window_size_ms
* refactor(pybind): simplify join method bindings and fix test expectations
- Remove unused include of join_strategy_config.h in bindings.cpp
- Simplify join binding: use JoinFunction constructor without time_window param
- Remove redundant SimilarityMode::NORMALIZED and alpha settings
- Fix test expectation: LSH uses PARTITIONED instead of PARTITIONED_VECTOR
- Add note about LSH not depending on external index
* add refactor partitioner doc, stash changes
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>1 parent 85986f9 commit 613ad56
74 files changed
Lines changed: 7241 additions & 2488 deletions
File tree
- cmake
- config
- docs
- refactoring
- tasks/vsjoin
- include
- concurrency
- execution
- operator
- join_operator_methods
- vsjoin_components
- utils
- stream/data_stream_source
- sage_flow
- scripts
- src
- concurrency
- execution
- operator
- join_operator_methods
- vsjoin_components
- utils
- stream
- data_stream_source
- test
- IntegrationTest
- UnitTest
- test_utils
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| 26 | + | |
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| 31 | + | |
30 | 32 | | |
31 | 33 | | |
32 | 34 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
52 | | - | |
| 52 | + | |
53 | 53 | | |
54 | 54 | | |
55 | 55 | | |
| |||
341 | 341 | | |
342 | 342 | | |
343 | 343 | | |
344 | | - | |
| 344 | + | |
345 | 345 | | |
346 | 346 | | |
347 | 347 | | |
| |||
551 | 551 | | |
552 | 552 | | |
553 | 553 | | |
554 | | - | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
555 | 563 | | |
556 | | - | |
557 | | - | |
| 564 | + | |
| 565 | + | |
558 | 566 | | |
559 | 567 | | |
560 | | - | |
| 568 | + | |
561 | 569 | | |
| 570 | + | |
| 571 | + | |
562 | 572 | | |
563 | 573 | | |
564 | | - | |
565 | | - | |
566 | | - | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
567 | 577 | | |
568 | | - | |
| 578 | + | |
| 579 | + | |
569 | 580 | | |
570 | | - | |
| 581 | + | |
571 | 582 | | |
572 | | - | |
| 583 | + | |
573 | 584 | | |
| 585 | + | |
574 | 586 | | |
575 | | - | |
576 | | - | |
| 587 | + | |
| 588 | + | |
577 | 589 | | |
578 | 590 | | |
579 | | - | |
| 591 | + | |
580 | 592 | | |
| 593 | + | |
| 594 | + | |
581 | 595 | | |
582 | | - | |
583 | | - | |
584 | | - | |
585 | | - | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
586 | 600 | | |
587 | | - | |
| 601 | + | |
| 602 | + | |
588 | 603 | | |
589 | | - | |
| 604 | + | |
590 | 605 | | |
591 | | - | |
| 606 | + | |
592 | 607 | | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
593 | 611 | | |
594 | | - | |
595 | | - | |
| 612 | + | |
| 613 | + | |
596 | 614 | | |
597 | 615 | | |
598 | | - | |
| 616 | + | |
599 | 617 | | |
| 618 | + | |
| 619 | + | |
600 | 620 | | |
601 | | - | |
602 | | - | |
603 | | - | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
604 | 641 | | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
605 | 648 | | |
606 | | - | |
| 649 | + | |
| 650 | + | |
607 | 651 | | |
608 | | - | |
609 | | - | |
610 | | - | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
611 | 655 | | |
612 | 656 | | |
613 | 657 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
0 commit comments