issue-3: add device timeout notify to rdma partition #3187

vladstepanyuk · 2025-03-14T09:58:14Z

#3
Adding volume notifying if some of requests time outed or rdma endpoint unavailable

github-actions · 2025-03-14T09:58:25Z

Hi! Thank you for contributing!
The tests on this PR will run after a maintainer adds an ok-to-test label to this PR manually. Thank you for your patience!

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor_checksumblocks.cpp

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor_zeroblocks.cpp

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_ut.cpp

github-actions · 2025-03-14T13:07:34Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit be705e7.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
3887	3887	0	0	0	0

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl.cpp

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.cpp

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor_readblocks_local.cpp

github-actions · 2025-03-17T11:09:33Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 3eb347f.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
3888	3888	0	0	0	0

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor_writeblocks.cpp

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_ut.cpp

github-actions · 2025-03-17T13:47:21Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 66289d8.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
3888	3888	0	0	0	0

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.h

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.cpp

komarevtsev-d · 2025-03-17T18:59:33Z

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.cpp

+
+    auto& dCtx = DeviceTimeouted[deviceUUID];
+    dCtx.FirstErrorTs = ctx.Now();
+    dCtx.ParentWasNotified = false;


В interconnect партишионе есть проблема: таймаут запроса может прийти после того как всё раздуплилось. Тут такое может быть?

если пришел таймаут то ответ на реквест уже не получим, если ты спрашивал про это.
Здесь

nbs/cloud/blockstore/libs/rdma/impl/client.cpp

Lines 1529 to 1548 in 9a7b1be

void DropTimedOutRequests()

{

auto endpoints = Endpoints.Get();

for (const auto& endpoint: *endpoints) {

if (!endpoint->CheckState(EEndpointState::Connected)) {

continue;

}

auto requests = endpoint->ActiveRequests.PopTimedOutRequests(

DurationToCyclesSafe(Config->MaxResponseDelay));

for (auto& request: requests) {

endpoint->AbortRequest(

std::move(request),

E_TIMEOUT,

"request timeout");

}

}

}

мы извлекаем таймаученный запрос отвекчаем на него и соответственно уничтожаем

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.h

komarevtsev-d · 2025-03-17T19:10:21Z

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.cpp

+                    requestInfo.Value.ActorId,
+                    std::make_unique<
+                        TEvNonreplPartitionPrivate::TEvCancelRequest>(
+                        EReason::Canceled));


А где обработка TEvCancelRequest ?

Вынесу всю логику с отменой запросов в отдельный пр

komarevtsev-d · 2025-03-17T19:10:39Z

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.cpp

+            {
+                NCloud::Send(
+                    ctx,
+                    requestInfo.Value.ActorId,


Где заполняется поле ActorId?

Вынесу всю логику с отменой запросов в отдельный пр

SvartMetal · 2025-03-17T22:51:19Z

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.h

@@ -67,6 +111,11 @@ class TNonreplicatedPartitionRdmaActor final

    TRequestInfoPtr Poisoner;

+    struct TDeviceTimeoutCtx {
+        TInstant FirstErrorTs;
+        bool ParentWasNotified = false;


А что такое parent?

PartConfig->GetParentActorId()
т.е. волум

переименовал в VolumeWasNotified

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.h

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.cpp

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.h

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.cpp

github-actions · 2025-03-20T12:58:04Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit df05605.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
3888	3888	0	0	0	0

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.cpp

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor_checksumblocks.cpp

SvartMetal · 2025-03-24T11:36:07Z

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.cpp

+            NCloud::Send(
+                ctx,
+                PartConfig->GetParentActorId(),
+                std::make_unique<TEvVolumePrivate::TEvDeviceTimeoutedRequest>(


мы же собираемся переименовать Timeouted в TimedOut?

можно в отдельном ПРе

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.cpp

github-actions · 2025-03-24T12:46:41Z

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 3a73d29.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
3888	3884	0	4	0	0

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.cpp

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.h

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor_checksumblocks.cpp

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor_writeblocks.cpp

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor_readblocks.cpp

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor_readblocks_local.cpp

github-actions · 2025-03-25T11:07:57Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 3903229.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
3888	3888	0	0	0	0

…iceStats

github-actions · 2025-03-25T13:33:18Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 180d548.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
3910	3910	0	0	0	0

vladstepanyuk marked this pull request as draft March 14, 2025 09:58

komarevtsev-d added blockstore Add this label to run only cloud/blockstore build and tests on PR large-tests Launch large tests for PR ok-to-test Label to approve test launch for external members labels Mar 14, 2025

github-actions bot removed the ok-to-test Label to approve test launch for external members label Mar 14, 2025

vladstepanyuk commented Mar 14, 2025

View reviewed changes

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor_checksumblocks.cpp Outdated Show resolved Hide resolved

vladstepanyuk commented Mar 14, 2025

View reviewed changes

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor_zeroblocks.cpp Outdated Show resolved Hide resolved

vladstepanyuk commented Mar 14, 2025

View reviewed changes

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_ut.cpp Outdated Show resolved Hide resolved

vladstepanyuk force-pushed the issue-3/1 branch from d5bf900 to b7debd1 Compare March 14, 2025 10:08

komarevtsev-d added the ok-to-test Label to approve test launch for external members label Mar 14, 2025

vladstepanyuk marked this pull request as ready for review March 14, 2025 10:11

github-actions bot removed the ok-to-test Label to approve test launch for external members label Mar 14, 2025

komarevtsev-d reviewed Mar 14, 2025

View reviewed changes

vladstepanyuk requested a review from komarevtsev-d March 17, 2025 08:38

komarevtsev-d added the ok-to-test Label to approve test launch for external members label Mar 17, 2025

github-actions bot removed the ok-to-test Label to approve test launch for external members label Mar 17, 2025

vladstepanyuk commented Mar 17, 2025

View reviewed changes

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor_writeblocks.cpp Outdated Show resolved Hide resolved

vladstepanyuk commented Mar 17, 2025

View reviewed changes

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_ut.cpp Outdated Show resolved Hide resolved

komarevtsev-d reviewed Mar 17, 2025

View reviewed changes

SvartMetal reviewed Mar 17, 2025

View reviewed changes

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.h Outdated Show resolved Hide resolved

SvartMetal reviewed Mar 17, 2025

View reviewed changes

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.cpp Outdated Show resolved Hide resolved

SvartMetal reviewed Mar 17, 2025

View reviewed changes

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.h Outdated Show resolved Hide resolved

SvartMetal reviewed Mar 17, 2025

View reviewed changes

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.cpp Outdated Show resolved Hide resolved

SvartMetal reviewed Mar 17, 2025

View reviewed changes

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.cpp Outdated Show resolved Hide resolved

SvartMetal reviewed Mar 17, 2025

View reviewed changes

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.cpp Outdated Show resolved Hide resolved

github-actions bot removed the ok-to-test Label to approve test launch for external members label Mar 20, 2025

sharpeye reviewed Mar 24, 2025

View reviewed changes

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.cpp Outdated Show resolved Hide resolved

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor_checksumblocks.cpp Outdated Show resolved Hide resolved

vladstepanyuk requested a review from sharpeye March 24, 2025 11:35

SvartMetal reviewed Mar 24, 2025

View reviewed changes

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.cpp Outdated Show resolved Hide resolved

sharpeye reviewed Mar 24, 2025

View reviewed changes

cloud/blockstore/libs/storage/partition_nonrepl/part_nonrepl_rdma_actor.cpp Outdated Show resolved Hide resolved

sharpeye reviewed Mar 24, 2025

View reviewed changes

vladstepanyuk requested review from sharpeye and SvartMetal March 25, 2025 07:36

vladstepanyuk added 16 commits March 25, 2025 11:52

issue-3: add device timeout notify to rdma partition

a29af18

issue-3: correct issues

e4796e1

issue-3: correct issues

ad0ea91

issue-3: fix timestamp reseting

010e8fc

issue-3: correct small issues

9c14293

issue-3: rmv request cancellation

8befbee

issue-3: correct issues

f16f9a6

issue-3: rename ParentWasNotified to VolumeWasNotified and remove Dev…

fe3eb6c

…iceStats

issue-3: fix build issue

019da42

issue-3:reduce boilerplate

5162774

issue-3: fix test

8bdf10a

issue-3: reduce copy paste code

4856677

issue-3: fix test

702fda3

issue-3: correct issues

05f8d6f

issue-3: fix build

d23769c

issue-3: rename timeouted -> timed out

180d548

vladstepanyuk force-pushed the issue-3/1 branch from 3903229 to 180d548 Compare March 25, 2025 12:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

issue-3: add device timeout notify to rdma partition #3187

issue-3: add device timeout notify to rdma partition #3187

vladstepanyuk commented Mar 14, 2025

github-actions bot commented Mar 14, 2025

github-actions bot commented Mar 14, 2025

github-actions bot commented Mar 17, 2025

github-actions bot commented Mar 17, 2025

komarevtsev-d Mar 17, 2025

vladstepanyuk Mar 18, 2025

komarevtsev-d Mar 17, 2025

vladstepanyuk Mar 18, 2025 •

edited

Loading

komarevtsev-d Mar 17, 2025

vladstepanyuk Mar 18, 2025 •

edited

Loading

SvartMetal Mar 17, 2025

vladstepanyuk Mar 18, 2025

vladstepanyuk Mar 18, 2025

github-actions bot commented Mar 20, 2025

SvartMetal Mar 24, 2025

vladstepanyuk Mar 25, 2025

github-actions bot commented Mar 24, 2025

github-actions bot commented Mar 25, 2025

github-actions bot commented Mar 25, 2025

	void DropTimedOutRequests()
	{
	auto endpoints = Endpoints.Get();

	for (const auto& endpoint: *endpoints) {
	if (!endpoint->CheckState(EEndpointState::Connected)) {
	continue;
	}

	auto requests = endpoint->ActiveRequests.PopTimedOutRequests(
	DurationToCyclesSafe(Config->MaxResponseDelay));

	for (auto& request: requests) {
	endpoint->AbortRequest(
	std::move(request),
	E_TIMEOUT,
	"request timeout");
	}
	}
	}

issue-3: add device timeout notify to rdma partition #3187

Are you sure you want to change the base?

issue-3: add device timeout notify to rdma partition #3187

Conversation

vladstepanyuk commented Mar 14, 2025

github-actions bot commented Mar 14, 2025

github-actions bot commented Mar 14, 2025

github-actions bot commented Mar 17, 2025

github-actions bot commented Mar 17, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladstepanyuk Mar 18, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladstepanyuk Mar 18, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Mar 20, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Mar 24, 2025

github-actions bot commented Mar 25, 2025

github-actions bot commented Mar 25, 2025

vladstepanyuk Mar 18, 2025 •

edited

Loading

vladstepanyuk Mar 18, 2025 •

edited

Loading