enhance: make pchannel level flusher #39275

chyezh · 2025-01-15T02:58:08Z

Add a pchannel level checkpoint for flush processing
Refactor the recovery of flushers of wal
make a shared wal scanner first, then make multi datasyncservice on it

mergify · 2025-01-15T02:58:46Z

Invalid PR Title Format Detected

Your PR submission does not adhere to our required standards. To ensure clarity and consistency, please meet the following criteria:

Title Format: The PR title must begin with one of these prefixes:

feat: for introducing a new feature.
fix: for bug fixes.
enhance: for improvements to existing functionality.
test: for add tests to existing functionality.
doc: for modifying documentation.
auto: for the pull request from bot.

Description Requirement: The PR must include a non-empty description, detailing the changes and their impact.

Required Title Structure:

[Type]: [Description of the PR]

Where Type is one of feat, fix, enhance, test or doc.

Example:

enhance: improve search performance significantly

Please review and update your PR to comply with these guidelines.

codecov · 2025-01-15T04:21:30Z

Codecov Report

Attention: Patch coverage is 79.68338% with 154 lines in your changes missing coverage. Please review.

Project coverage is 80.16%. Comparing base (2d9bef4) to head (b172067).
Report is 12 commits behind head on master.

Files with missing lines	Patch %	Lines
...mingnode/server/flusher/flusherimpl/wal_flusher.go	67.56%	37 Missing and 11 partials ⚠️
...e/server/flusher/flusherimpl/flusher_components.go	74.37%	31 Missing and 10 partials ⚠️
...l/streamingnode/server/flusher/flusherimpl/util.go	68.49%	18 Missing and 5 partials ⚠️
.../server/flusher/flusherimpl/pchannel_checkpoint.go	85.84%	12 Missing and 3 partials ⚠️
internal/metastore/kv/streamingnode/kv_catalog.go	52.17%	10 Missing and 1 partial ⚠️
internal/flushcommon/pipeline/data_sync_service.go	66.66%	2 Missing and 2 partials ⚠️
internal/rootcoord/meta_table.go	0.00%	4 Missing ⚠️
...server/flusher/flusherimpl/data_service_wrapper.go	91.30%	2 Missing ⚠️
pkg/streaming/util/message/message_id.go	60.00%	1 Missing and 1 partial ⚠️
pkg/streaming/util/message/test_case.go	97.05%	1 Missing and 1 partial ⚠️
... and 2 more

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #39275       +/-   ##
===========================================
+ Coverage   69.39%   80.16%   +10.76%     
===========================================
  Files         302     1480     +1178     
  Lines       27077   204774   +177697     
===========================================
+ Hits        18790   164152   +145362     
- Misses       8287    34773    +26486     
- Partials        0     5849     +5849

Components	Coverage Δ
Client	`79.44% <ø> (∅)`
Core	`69.39% <ø> (ø)`
Go	`81.89% <79.46%> (∅)`

Files with missing lines	Coverage Δ
internal/flushcommon/syncmgr/sync_manager.go	`91.13% <100.00%> (ø)`
internal/flushcommon/util/checkpoint_updater.go	`88.57% <100.00%> (ø)`
internal/metastore/catalog.go	`100.00% <ø> (ø)`
...erycoordv2/balance/channel_level_score_balancer.go	`89.24% <ø> (ø)`
internal/streamingnode/server/builder.go	`100.00% <100.00%> (ø)`
.../server/flusher/flusherimpl/vchannel_checkpoint.go	`100.00% <100.00%> (ø)`
internal/streamingnode/server/resource/resource.go	`96.96% <100.00%> (ø)`
internal/streamingnode/server/server.go	`93.10% <ø> (ø)`
...ternal/streamingnode/server/wal/adaptor/builder.go	`61.53% <100.00%> (ø)`
...al/streamingnode/server/wal/adaptor/wal_adaptor.go	`93.45% <100.00%> (ø)`
... and 20 more

... and 1148 files with indirect coverage changes

mergify · 2025-01-23T09:31:45Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2025-01-23T10:44:53Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2025-01-23T11:39:20Z

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2025-02-05T08:17:46Z

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2025-02-05T08:17:46Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2025-02-05T08:29:46Z

/run-cpu-e2e

mergify · 2025-02-05T08:44:13Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2025-02-05T13:22:44Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2025-02-05T13:51:43Z

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2025-02-06T02:58:47Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2025-02-06T03:18:33Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2025-02-07T04:14:35Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

chyezh · 2025-02-07T09:06:01Z

rerun ut

mergify · 2025-02-07T09:10:43Z

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

chyezh · 2025-02-07T09:11:19Z

/run-cpu-e2e

chyezh · 2025-02-07T09:32:36Z

/run-cpu-e2e

mergify · 2025-02-07T10:33:21Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

- Add a pchannel level checkpoint for flush processing - Refactor the recovery of flushers of wal - make a shared wal scanner first, then make multi datasyncservice on it Signed-off-by: chyezh <[email protected]>

mergify · 2025-02-07T13:33:31Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

chyezh · 2025-02-07T15:18:59Z

rerun go-sdk

bigsheeper · 2025-02-08T02:46:04Z

internal/streamingnode/server/flusher/flusherimpl/pchannel_checkpoint.go

+		startMessageID = message.MustUnmarshalMessageID(walName, checkpoint.MessageID.Id)
+		previous = startMessageID
+	} else {
+		startMessageID = vchannelManager.MinimumCheckpoint()


Could vchannelManager.MinimumCheckpoint() possibly be nil?

Yes, if there's no vchannel, the scan operation will consume from the beginning of wal.

bigsheeper · 2025-02-08T03:27:01Z

internal/streamingnode/server/flusher/flusherimpl/wal_flusher.go

+			impl.logger.DPanic("the message type is not CreateCollectionMessage", zap.Error(err))
+			return nil
+		}
+		impl.flusherComponents.WhenCreateCollection(createCollectionMsg)


Whay if the create collection message is produced successfully but the collection creation fails, I think we should add a rollback mechanism?

Got it, I will modify the rootcoord current create collection logic.
if there's a rollback operation, the rootcoord should promise to send a dropcollection message to wal.

Implement it in another PR.

sre-ci-robot · 2025-02-10T08:30:55Z

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: chyezh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jaime0815 · 2025-02-10T08:31:15Z

/lgtm

jaime0815 · 2025-02-10T07:02:20Z

internal/streamingnode/server/flusher/flusherimpl/flusher_components.go

+			dataServices[vchannel] = ds.(*dataSyncServiceWrapper)
+			continue
+		}
+		if firstErr == nil {


return err if any error is not nil directly.

jaime0815 · 2025-02-10T07:56:15Z

internal/streamingnode/server/flusher/flusherimpl/util.go

+			return err
+		}
+		// The channel has been dropped, skip to recover it.
+		if len(resp.GetInfo().GetSeekPosition().GetMsgID()) == 0 && resp.GetInfo().GetSeekPosition().GetTimestamp() == math.MaxUint64 {


Wrapping the isDroppedChannel method clarifies the implicit checks involved with a dropped channel.

sre-ci-robot added area/internal-api area/test sig/testing test/integration integration test size/XXL Denotes a PR that changes 1000+ lines. labels Jan 15, 2025

sre-ci-robot requested review from bigsheeper and congqixia January 15, 2025 02:58

mergify bot added the dco-passed DCO check passed. label Jan 15, 2025

mergify bot added the do-not-merge/invalid-pr-format label Jan 15, 2025

chyezh force-pushed the enhance_make_pchannel_lv_flusher branch from 17e4298 to 44e5e19 Compare January 23, 2025 09:02

chyezh mentioned this pull request Jan 22, 2025

[Enhancement]: Support Growing Search With Querynode Delegator And Streaming Service #38399

Closed

13 tasks

chyezh added this to the 2.6.0 milestone Jan 23, 2025

chyezh force-pushed the enhance_make_pchannel_lv_flusher branch from 44e5e19 to 22ad3ac Compare January 23, 2025 10:03

chyezh force-pushed the enhance_make_pchannel_lv_flusher branch from 22ad3ac to e2f0c28 Compare February 5, 2025 08:14

chyezh force-pushed the enhance_make_pchannel_lv_flusher branch 2 times, most recently from 1c8d96c to 35c53a9 Compare February 5, 2025 12:35

chyezh force-pushed the enhance_make_pchannel_lv_flusher branch from 35c53a9 to ec9d889 Compare February 6, 2025 02:41

chyezh force-pushed the enhance_make_pchannel_lv_flusher branch from 13e31a0 to 86e9c2a Compare February 7, 2025 03:43

mergify bot added ci-passed and removed ci-passed labels Feb 7, 2025

chyezh force-pushed the enhance_make_pchannel_lv_flusher branch from 86e9c2a to c020806 Compare February 7, 2025 07:15

chyezh changed the title ~~enhance: make pchannel level flusher~~ enhance: make pchannel level flusher Feb 7, 2025

mergify bot added kind/enhancement Issues or changes related to enhancement and removed do-not-merge/invalid-pr-format labels Feb 7, 2025

chyezh force-pushed the enhance_make_pchannel_lv_flusher branch from c020806 to 3e8866b Compare February 7, 2025 10:03

enhance: make pchannel level flusher

b172067

- Add a pchannel level checkpoint for flush processing - Refactor the recovery of flushers of wal - make a shared wal scanner first, then make multi datasyncservice on it Signed-off-by: chyezh <[email protected]>

chyezh force-pushed the enhance_make_pchannel_lv_flusher branch from 3e8866b to b172067 Compare February 7, 2025 12:55

mergify bot added the ci-passed label Feb 7, 2025

bigsheeper reviewed Feb 8, 2025

View reviewed changes

jaime0815 added the approved label Feb 10, 2025

sre-ci-robot assigned jaime0815 Feb 10, 2025

sre-ci-robot added the lgtm label Feb 10, 2025

sre-ci-robot merged commit d3e32bb into milvus-io:master Feb 10, 2025
19 of 20 checks passed

chyezh deleted the enhance_make_pchannel_lv_flusher branch February 10, 2025 08:47

jaime0815 reviewed Feb 10, 2025

View reviewed changes

chyezh mentioned this pull request Mar 12, 2025

[Bug]: [streaming] The streaming node was oomkilled when inserting into a collection with partition-key and multi shards #40592

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhance: make pchannel level flusher #39275

enhance: make pchannel level flusher #39275

chyezh commented Jan 15, 2025 •

edited

Loading

mergify bot commented Jan 15, 2025

codecov bot commented Jan 15, 2025 •

edited

Loading

mergify bot commented Jan 23, 2025

mergify bot commented Jan 23, 2025

mergify bot commented Jan 23, 2025

mergify bot commented Feb 5, 2025

mergify bot commented Feb 5, 2025

chyezh commented Feb 5, 2025

mergify bot commented Feb 5, 2025

mergify bot commented Feb 5, 2025

mergify bot commented Feb 5, 2025

mergify bot commented Feb 6, 2025

mergify bot commented Feb 6, 2025

mergify bot commented Feb 7, 2025

chyezh commented Feb 7, 2025

mergify bot commented Feb 7, 2025

chyezh commented Feb 7, 2025

chyezh commented Feb 7, 2025

mergify bot commented Feb 7, 2025

mergify bot commented Feb 7, 2025

chyezh commented Feb 7, 2025

bigsheeper Feb 8, 2025

chyezh Feb 8, 2025

bigsheeper Feb 8, 2025

chyezh Feb 8, 2025

chyezh Feb 8, 2025 •

edited

Loading

sre-ci-robot commented Feb 10, 2025

jaime0815 commented Feb 10, 2025

jaime0815 Feb 10, 2025

jaime0815 Feb 10, 2025

enhance: make pchannel level flusher #39275

enhance: make pchannel level flusher #39275

Conversation

chyezh commented Jan 15, 2025 • edited Loading

mergify bot commented Jan 15, 2025

codecov bot commented Jan 15, 2025 • edited Loading

Codecov Report

mergify bot commented Jan 23, 2025

mergify bot commented Jan 23, 2025

mergify bot commented Jan 23, 2025

mergify bot commented Feb 5, 2025

mergify bot commented Feb 5, 2025

chyezh commented Feb 5, 2025

mergify bot commented Feb 5, 2025

mergify bot commented Feb 5, 2025

mergify bot commented Feb 5, 2025

mergify bot commented Feb 6, 2025

mergify bot commented Feb 6, 2025

mergify bot commented Feb 7, 2025

chyezh commented Feb 7, 2025

mergify bot commented Feb 7, 2025

chyezh commented Feb 7, 2025

chyezh commented Feb 7, 2025

mergify bot commented Feb 7, 2025

mergify bot commented Feb 7, 2025

chyezh commented Feb 7, 2025

bigsheeper Feb 8, 2025

Choose a reason for hiding this comment

chyezh Feb 8, 2025

Choose a reason for hiding this comment

bigsheeper Feb 8, 2025

Choose a reason for hiding this comment

chyezh Feb 8, 2025

Choose a reason for hiding this comment

chyezh Feb 8, 2025 • edited Loading

Choose a reason for hiding this comment

sre-ci-robot commented Feb 10, 2025

jaime0815 commented Feb 10, 2025

jaime0815 Feb 10, 2025

Choose a reason for hiding this comment

jaime0815 Feb 10, 2025

Choose a reason for hiding this comment

chyezh commented Jan 15, 2025 •

edited

Loading

codecov bot commented Jan 15, 2025 •

edited

Loading

chyezh Feb 8, 2025 •

edited

Loading