issue-1751: [Filestore] Restore and drain WriteBackCache at session recreation #4711

e673 · 2025-11-24T13:21:08Z

Resolve data loss in the following scenario:

WriteBackCache was enabled and used.
ServerWriteBackCacheEnabled flag was disabled in the configuration.
nfs-vhost was restarted.

Actual behavior: nfs-vhost completely ignores the non-flushed requests remaining in the persistent queue.

Expected behavior: nfs-vhost should check for the presence of the persistent queue and initialize WriteBackCache if the persistent queue exists regardless of the value of ServerWriteBackCacheEnabled flag. If ServerWriteBackCacheEnabled is not set, new writes should not be processed until the queue is flushed.

github-actions · 2025-11-24T14:23:55Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 91f6686.

TESTS	PASSED	ERRORS	FAILED	FAILED BUILD	SKIPPED	MUTED^?
3181	3181	0	0	0	0	0

github-actions · 2025-11-25T12:06:13Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit a10526b.

TESTS	PASSED	ERRORS	FAILED	FAILED BUILD	SKIPPED	MUTED^?
3181	3181	0	0	0	0	0

github-actions · 2025-11-26T12:35:04Z

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 3410149.

TESTS	PASSED	ERRORS	FAILED	FAILED BUILD	SKIPPED	MUTED^?
3181	3180	0	1	0	0	0

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 3410149.

TESTS	PASSED	ERRORS	FAILED	FAILED BUILD	SKIPPED	MUTED^?
2	2	0	0	0	0	0

github-actions · 2025-11-27T11:38:16Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 8626f7b.

TESTS	PASSED	ERRORS	FAILED	FAILED BUILD	SKIPPED	MUTED^?
3194	3194	0	0	0	0	0

…e session is destoyed

github-actions · 2025-12-02T13:58:40Z

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 650f286.

TESTS	PASSED	ERRORS	FAILED	FAILED BUILD	SKIPPED	MUTED^?
3196	3194	0	2	0	0	0

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 650f286.

TESTS	PASSED	ERRORS	FAILED	FAILED BUILD	SKIPPED	MUTED^?
63	63	0	0	0	0	0

cloud/filestore/libs/vfs_fuse/fs_impl_data.cpp

cloud/filestore/libs/vfs_fuse/fs_ut.cpp

cloud/filestore/libs/vfs_fuse/loop.cpp

cloud/filestore/libs/vfs_fuse/fs_impl_data.cpp

cloud/filestore/libs/vfs_fuse/fs_ut.cpp

neihar · 2025-12-03T20:18:21Z

cloud/filestore/libs/vfs_fuse/fs_ut.cpp

+
+            // Cache is drained and disabled - new requests go directly
+            // to the session
+            UNIT_ASSERT_VALUES_EQUAL(2, writeDataCalled2.load());


can we check that cache file is empty?

WriteBackCache object is not accessible from this place.
And accessing the cache file directly is not reliable (this is a memory mapped file with no disk synchronization guarantee).
So the answer is: without writing tons of boilerplate code — no.

it is quite easy, you can look here for example

nbs/cloud/filestore/libs/vfs_fuse/fs_ut.cpp

Line 1071 in da489bf

pathToCache = TFsPath(bootstrap.DirectoryHandlesStoragePath) /

I agree that it is a little bit hacky, but allow you to verify file state.

Are you sure that accessing the file while it is being used by WriteBackCache is reliable?

We can suspend and check, but even without that, as I understand it, we can assume the test is deterministic and that nothing (including WBC) touches the file at this point. If that’s not the case, then we probably have a bigger problem.

neihar · 2025-12-03T20:20:02Z

For the future, this PR could be split into two parts: one for draining the cache on shutdown, and another for restoring it after suspension.

neihar · 2025-12-03T20:22:24Z

For the future, this PR could be split into two parts: one for draining the cache on shutdown, and another for restoring it after suspension.
I see, it is already split, just in this pr we have both commits:)

cloud/filestore/libs/vfs_fuse/fs_impl.h

cloud/filestore/libs/vfs_fuse/loop.cpp

neihar · 2025-12-03T20:54:19Z

cloud/filestore/libs/vfs_fuse/fs_impl_data.cpp

+         callContext = std::move(callContext),
+         request = std::move(request)](const auto& f) mutable
+        {
+            f.GetValue();


if (can??) flush fails, we just proceed to write normally

FlushAllData is not expected to throw an exception, we can safely make this call.
I don't like this code and I'd prefer Y_UNUSED(f) but this pattern is commonly used in other places.

cloud/filestore/libs/vfs_fuse/fs_impl_data.cpp

github-actions · 2025-12-04T15:12:56Z

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 20a9d0f.

TESTS	PASSED	ERRORS	FAILED	FAILED BUILD	SKIPPED	MUTED^?
3196	3195	0	1	0	0	0

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 20a9d0f.

TESTS	PASSED	ERRORS	FAILED	FAILED BUILD	SKIPPED	MUTED^?
2	2	0	0	0	0	0

qkrorlqr · 2025-12-04T15:04:50Z

cloud/filestore/libs/vfs_fuse/fs_impl_data.cpp

-            const auto& response = future.GetValue();
-            const auto& error = response.GetError();
-            self->FSyncQueue->Dequeue(reqId, error, TNodeId {ino}, THandle {handle});
+    Y_ABORT_UNLESS(


let's use STORAGE_VERIFY / STORAGE_VERIFY_C macros - they force the developer to provide the entity id (filesystem id / tablet id / client id / etc), otherwise we'll just see that some error happened without knowing anything about the entity that caused the problem (so the debugging process will take more time)

here we can use client id in the STORAGE_VERIFY message

Can be done in a separate PR - there're many other ABORT_UNLESS usages in vfs_fuse which would be nice to replace with STORAGE_VERIFY(clientId)

I would say, that ABORT_UNLESS is the only usage for checks in vfs_fuse and filestore vhost for now. Wasn't able to find a single instance of Verify in those components

e673 changed the title ~~issue-1751: Restore and drain WriteBackCache at session recreation~~ issue-1751: [Filestore] Restore and drain WriteBackCache at session recreation Nov 24, 2025

e673 requested a review from SvartMetal November 24, 2025 13:21

e673 added the filestore Add this label to run only cloud/filestore build and tests on PR label Nov 24, 2025

e673 force-pushed the users/nasonov/issue-4201-create-session branch from 91f6686 to a10526b Compare November 25, 2025 10:48

e673 mentioned this pull request Nov 25, 2025

[Filestore] write-back cache in filestore-vhost #1751

Open

neihar self-requested a review November 26, 2025 08:49

e673 mentioned this pull request Nov 27, 2025

issue-4201 [Filestore] Turning on and off the write-back cache #4578

Closed

e673 force-pushed the users/nasonov/issue-4201-create-session branch from 3410149 to 8626f7b Compare November 27, 2025 10:19

Andrei Nasonov added 2 commits December 2, 2025 14:01

PR-4702: [Filestore] WriteBackCache should flush everything before th…

8b81179

…e session is destoyed

issue-1751: Restore and drain WriteBackCache at session recreation

650f286

e673 force-pushed the users/nasonov/issue-4201-create-session branch from 8626f7b to 650f286 Compare December 2, 2025 13:01

e673 requested a review from qkrorlqr December 3, 2025 19:14

neihar requested changes Dec 3, 2025

View reviewed changes

neihar reviewed Dec 3, 2025

View reviewed changes

cloud/filestore/libs/vfs_fuse/fs_impl.h Outdated Show resolved Hide resolved

neihar reviewed Dec 3, 2025

View reviewed changes

cloud/filestore/libs/vfs_fuse/loop.cpp Show resolved Hide resolved

neihar reviewed Dec 3, 2025

View reviewed changes

cloud/filestore/libs/vfs_fuse/fs_impl_data.cpp Show resolved Hide resolved

e673 force-pushed the users/nasonov/issue-4201-create-session branch 2 times, most recently from f55f953 to 6530b26 Compare December 4, 2025 12:51

e673 requested a review from neihar December 4, 2025 12:55

Comments fix

01d279e

e673 force-pushed the users/nasonov/issue-4201-create-session branch from 6530b26 to 01d279e Compare December 4, 2025 13:07

Refactoring: code deduplication

199581f

Andrei Nasonov added 2 commits December 4, 2025 14:52

Remove micro-optimization

94c0d31

GetServerWriteBackCacheMode -> GetServerWriteBackCacheState

20a9d0f

qkrorlqr approved these changes Dec 4, 2025

View reviewed changes

neihar approved these changes Dec 5, 2025

View reviewed changes

issue-1751: [Filestore] Restore and drain WriteBackCache at session recreation #4711

Are you sure you want to change the base?

issue-1751: [Filestore] Restore and drain WriteBackCache at session recreation #4711

Conversation

e673 commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 24, 2025

Uh oh!

github-actions bot commented Nov 25, 2025

Uh oh!

github-actions bot commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 27, 2025

Uh oh!

github-actions bot commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

neihar commented Dec 3, 2025

Uh oh!

neihar commented Dec 3, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

e673 commented Nov 24, 2025 •

edited

Loading

github-actions bot commented Nov 26, 2025 •

edited

Loading

github-actions bot commented Dec 2, 2025 •

edited

Loading

github-actions bot commented Dec 4, 2025 •

edited

Loading