Skip to content

issue-1751: [Filestore] Restore and drain WriteBackCache at session recreation #4711

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
e673 wants to merge 6 commits into
base: main
Choose a base branch
from
+405 −172
Open

issue-1751: [Filestore] Restore and drain WriteBackCache at session recreation #4711

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8b81179
PR-4702: [Filestore] WriteBackCache should flush everything before th…
Nov 20, 2025
650f286
issue-1751: Restore and drain WriteBackCache at session recreation
Nov 24, 2025
01d279e
Comments fix
Dec 4, 2025
199581f
Refactoring: code deduplication
Dec 4, 2025
94c0d31
Remove micro-optimization
Dec 4, 2025
20a9d0f
GetServerWriteBackCacheMode -> GetServerWriteBackCacheState
Dec 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion cloud/filestore/libs/vfs_fuse/fs_impl.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,20 @@ struct TReleaseRequest

////////////////////////////////////////////////////////////////////////////////

enum class EServerWriteBackCacheMode
{
// The request should go to the session
Disabled,

// The request should go to the WriteBackCache
Enabled,

// The request should wait until the cache is empty then go to the session
Draining
};

////////////////////////////////////////////////////////////////////////////////

class TFileSystem final
: public IFileSystem
, public std::enable_shared_from_this<TFileSystem>
Expand Down Expand Up @@ -395,7 +409,8 @@ class TFileSystem final
fuse_ino_t ino,
uint64_t fh);

bool ShouldUseServerWriteBackCache(const fuse_file_info* fi) const;
EServerWriteBackCacheMode GetServerWriteBackCacheMode(
const fuse_file_info* fi) const;

bool UpdateNodeCache(
const NProto::TNodeAttr& attrs,
Expand Down
181 changes: 116 additions & 65 deletions cloud/filestore/libs/vfs_fuse/fs_impl_data.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -413,45 +413,67 @@ void TFileSystem::Write(
request->SetBuffer(alignedBuffer.TakeBuffer());

const auto size = buffer.size();
const auto wbcMode = GetServerWriteBackCacheMode(fi);

if (ShouldUseServerWriteBackCache(fi)) {
const auto handle = fi->fh;
const auto reqId = callContext->RequestId;

auto callback = [=, ptr = weak_from_this()](const auto& future)
{
auto self = ptr.lock();
if (!self) {
return;
}

const auto& response = future.GetValue();
const auto& error = response.GetError();

if (wbcMode != EServerWriteBackCacheMode::Enabled) {
self->FSyncQueue
->Dequeue(reqId, error, TNodeId{ino}, THandle{handle});
}

if (self->CheckResponse(self, *callContext, req, response)) {
self->ReplyWrite(*callContext, error, req, size);
}
};

if (wbcMode == EServerWriteBackCacheMode::Enabled) {
WriteBackCache.WriteData(callContext, std::move(request))
.Subscribe(
[=,
ptr = weak_from_this()] (const auto& future)
{
auto self = ptr.lock();
if (!self) {
return;
}

const auto& response = future.GetValue();
const auto& error = response.GetError();

if (CheckResponse(self, *callContext, req, response)) {
self->ReplyWrite(*callContext, error, req, size);
}
});
.Subscribe(std::move(callback));
return;
}

const auto handle = fi->fh;
const auto reqId = callContext->RequestId;
FSyncQueue->Enqueue(reqId, TNodeId {ino}, THandle {handle});

Session->WriteData(callContext, std::move(request))
.Subscribe([=, ptr = weak_from_this()] (const auto& future) {
auto self = ptr.lock();
if (!self) {
return;
}
if (wbcMode == EServerWriteBackCacheMode::Disabled) {
Session->WriteData(callContext, std::move(request))
.Subscribe(std::move(callback));
return;
}

const auto& response = future.GetValue();
const auto& error = response.GetError();
self->FSyncQueue->Dequeue(reqId, error, TNodeId {ino}, THandle {handle});
Y_ABORT_UNLESS(
Copy link
Collaborator

@qkrorlqr qkrorlqr Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's use STORAGE_VERIFY / STORAGE_VERIFY_C macros - they force the developer to provide the entity id (filesystem id / tablet id / client id / etc), otherwise we'll just see that some error happened without knowing anything about the entity that caused the problem (so the debugging process will take more time)

here we can use client id in the STORAGE_VERIFY message

Copy link
Collaborator

@qkrorlqr qkrorlqr Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be done in a separate PR - there're many other ABORT_UNLESS usages in vfs_fuse which would be nice to replace with STORAGE_VERIFY(clientId)

Copy link
Collaborator

@neihar neihar Dec 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say, that ABORT_UNLESS is the only usage for checks in vfs_fuse and filestore vhost for now. Wasn't able to find a single instance of Verify in those components

wbcMode == EServerWriteBackCacheMode::Draining,
"Invalid EServerWriteBackCacheMode value = %d",
wbcMode);

if (CheckResponse(self, *callContext, req, response)) {
self->ReplyWrite(*callContext, error, req, size);
auto flushFuture = WriteBackCache.FlushNodeData(request->GetNodeId());
if (flushFuture.HasValue()) {
Session->WriteData(callContext, std::move(request))
.Subscribe(std::move(callback));
return;
}

flushFuture.Subscribe(
[ptr = weak_from_this(),
callback = std::move(callback),
callContext = std::move(callContext),
request = std::move(request)](const auto& f) mutable
{
f.GetValue();
Copy link
Collaborator

@neihar neihar Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if (can??) flush fails, we just proceed to write normally

Copy link
Collaborator Author

@e673 e673 Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FlushAllData is not expected to throw an exception, we can safely make this call.
I don't like this code and I'd prefer Y_UNUSED(f) but this pattern is commonly used in other places.

if (auto self = ptr.lock()) {
self->Session->WriteData(callContext, std::move(request))
.Subscribe(std::move(callback));
}
});
}
Expand Down Expand Up @@ -577,43 +599,66 @@ void TFileSystem::WriteBuf(
request->SetHandle(fi->fh);
request->SetOffset(offset);

if (ShouldUseServerWriteBackCache(fi)) {
const auto wbcMode = GetServerWriteBackCacheMode(fi);
const auto handle = fi->fh;
const auto reqId = callContext->RequestId;

auto callback = [=, ptr = weak_from_this()](const auto& future)
{
auto self = ptr.lock();
if (!self) {
return;
}

const auto& response = future.GetValue();
const auto& error = response.GetError();

if (wbcMode != EServerWriteBackCacheMode::Enabled) {
self->FSyncQueue
->Dequeue(reqId, error, TNodeId{ino}, THandle{handle});
}

if (self->CheckResponse(self, *callContext, req, response)) {
self->ReplyWrite(*callContext, error, req, size);
}
};

if (wbcMode == EServerWriteBackCacheMode::Enabled) {
WriteBackCache.WriteData(callContext, std::move(request))
.Subscribe(
[=, ptr = weak_from_this()] (const auto& future)
{
auto self = ptr.lock();
if (!self) {
return;
}

const auto& response = future.GetValue();
const auto& error = response.GetError();

if (CheckResponse(self, *callContext, req, response)) {
self->ReplyWrite(*callContext, error, req, size);
}
});
.Subscribe(std::move(callback));
return;
}

const auto handle = fi->fh;
const auto reqId = callContext->RequestId;
FSyncQueue->Enqueue(reqId, TNodeId {ino}, THandle {handle});
FSyncQueue->Enqueue(reqId, TNodeId{ino}, THandle{handle});

Session->WriteData(callContext, std::move(request))
.Subscribe([=, ptr = weak_from_this()] (const auto& future) {
auto self = ptr.lock();
if (!self) {
return;
}
if (wbcMode == EServerWriteBackCacheMode::Disabled) {
Session->WriteData(callContext, std::move(request))
.Subscribe(std::move(callback));
return;
}

const auto& response = future.GetValue();
const auto& error = response.GetError();
self->FSyncQueue->Dequeue(reqId, error, TNodeId {ino}, THandle {handle});
Y_ABORT_UNLESS(
wbcMode == EServerWriteBackCacheMode::Draining,
"Invalid EServerWriteBackCacheMode value = %d",
wbcMode);

if (CheckResponse(self, *callContext, req, response)) {
self->ReplyWrite(*callContext, error, req, size);
auto flushFuture = WriteBackCache.FlushNodeData(request->GetNodeId());
if (flushFuture.HasValue()) {
Session->WriteData(callContext, std::move(request))
.Subscribe(std::move(callback));
return;
}

flushFuture.Subscribe(
[ptr = weak_from_this(),
callback = std::move(callback),
callContext = std::move(callContext),
request = std::move(request)](const auto& f) mutable
{
f.GetValue();
if (auto self = ptr.lock()) {
self->Session->WriteData(callContext, std::move(request))
.Subscribe(std::move(callback));
}
});
}
Expand Down Expand Up @@ -960,17 +1005,23 @@ void TFileSystem::FSyncDir(

////////////////////////////////////////////////////////////////////////////////

bool TFileSystem::ShouldUseServerWriteBackCache(const fuse_file_info* fi) const
EServerWriteBackCacheMode TFileSystem::GetServerWriteBackCacheMode(
const fuse_file_info* fi) const
{
if (!WriteBackCache || !Config->GetServerWriteBackCacheEnabled()) {
return false;
if (!WriteBackCache) {
return EServerWriteBackCacheMode::Disabled;
}

if (fi->flags & O_DIRECT) {
return false;
return EServerWriteBackCacheMode::Disabled;
}

return true;
if (Config->GetServerWriteBackCacheEnabled()) {
return EServerWriteBackCacheMode::Enabled;
}

return WriteBackCache.IsEmpty() ? EServerWriteBackCacheMode::Disabled
: EServerWriteBackCacheMode::Draining;
}

} // namespace NCloud::NFileStore::NFuse
Loading
Loading