Skip to content

[Cherry-Pick][BugFix] RDMA: retry ibv_post_send from bad_wr and drain CQ on failure(#8091) #8093

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Sunny-bot1 merged 1 commit into from
Jul 2, 2026
Merged

[Cherry-Pick][BugFix] RDMA: retry ibv_post_send from bad_wr and drain CQ on failure(#8091) #8093

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion fastdeploy/cache_manager/transfer_factory/kvcache_transfer/src/kvcache_rdma.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1390,8 +1390,9 @@ bool RDMACommunicator::post_send_with_retry(struct RdmaContext* ctx,
last_wr->send_flags |= IBV_SEND_SIGNALED;
}

struct ibv_send_wr* cur_wr = wr_list;
do {
ret = ibv_post_send(ctx->qp, wr_list, &bad_wr);
ret = ibv_post_send(ctx->qp, cur_wr, &bad_wr);
if (ret == 0) {
if (need_poll) {
ctx->conn.wc_count = 0;
Expand All @@ -1408,8 +1409,22 @@ bool RDMACommunicator::post_send_with_retry(struct RdmaContext* ctx,
errno,
retries + 1,
max_retries);
// Non-blocking CQ drain to free SQ slots before retrying.
// Do not use poll_cq_with_timeout here: ibv_post_send failure does not
// guarantee a CQE is available (e.g. sync errors, unsignaled WRs), so a
// blocking wait would stall the retry loop for up to
// RDMA_POLL_CQE_TIMEOUT per attempt (up to 210s total).
{
struct ibv_wc wc_array[32];
int n;
while ((n = ibv_poll_cq(ctx->cq, 32, wc_array)) > 0) {

@PaddlePaddle-bot PaddlePaddle-bot Jul 2, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Bug 这里在重试前把 ctx->cq 上的所有 CQE 都直接丢弃了，但这个 CQ 是共享的。

create_qp() 同时把 send_cq/recv_cq 设为 ctx->cq，write_cache 的 pybind 入口会释放 GIL，而且 write_cache() 只在 get_conn() 时短暂加锁，后续 post_send_with_retry()、execute_read_verification()、poll_cq_with_timeout() 都可能在同一个 RdmaContext 上消费这个 CQ。生产高压下如果另一个线程正在等待自己的 signaled write/read completion，这个 drain 可能先把它的 CQE 吞掉，导致对方 30s timeout，或者把非成功 WC 状态静默丢掉。

建议修复方式：将同一 RdmaContext 上的 post/poll/drain 序列纳入连接级互斥，或改成每个并发发送方使用独立 QP/CQ/完成分发器并按 wr_id 归属处理 completion；这个 retry helper 不应裸 drain 共享 CQ。

}
}
usleep(1000);
retries++;
if (bad_wr)
cur_wr =
bad_wr; // resume from the failed WR instead of retrying from head
}
} while (retries < max_retries);

Expand Down
Loading