mlx5: ignore QP max_recv_wr when SRQ is used

[FujiZ]

According to the manual, max_recv_wr and max_recv_sge are ignored by ibv_create_qp if the QP is to be associated with an SRQ. Fix the bug where ibv_create_qp may return EINVAL when attr->cap.max_recv_wr > cap.max_recv_wr even if the QP is to be associated with SRQ.

The code snippet to reproduce this issue:

#include <stdio.h>
#include <stdlib.h>
#include <infiniband/verbs.h>

int main() {
    struct ibv_device **dev_list;
    struct ibv_context *context;
    struct ibv_pd *pd;
    struct ibv_srq_init_attr srq_init_attr = {0};
    struct ibv_srq *srq;
    struct ibv_qp_init_attr qp_init_attr = {0};
    struct ibv_qp *qp;

    dev_list = ibv_get_device_list(NULL);
    if (!dev_list) {
        perror("Failed to get IB devices list");
        return -1;
    }

    context = ibv_open_device(dev_list[0]);
    if (!context) {
        perror("Failed to open IB device");
        return -1;
    }

    pd = ibv_alloc_pd(context);
    if (!pd) {
        perror("Failed to allocate PD");
        return -1;
    }

    srq_init_attr.attr.max_wr  = 1;
    srq_init_attr.attr.max_sge = 1;
    srq = ibv_create_srq(pd, &srq_init_attr);
    if (!srq) {
        perror("Failed to create SRQ");
        return -1;
    }
    printf("SRQ created successfully\n");

    struct ibv_cq* cq = ibv_create_cq(context, 10, NULL, NULL, 0);
    if (!cq) {
        perror("Failed to create CQ");
        return -1;
    }

    qp_init_attr.qp_type    = IBV_QPT_RC;
    qp_init_attr.srq        = srq;
    qp_init_attr.send_cq    = cq;
    qp_init_attr.recv_cq    = cq;
    qp_init_attr.cap.max_send_wr  = 1;
    // fill in random value for max_recv_wr, which should be ignored
    qp_init_attr.cap.max_recv_wr  = UINT32_MAX;
    qp_init_attr.cap.max_send_sge = 1;
    qp_init_attr.cap.max_recv_sge = 0;

    qp = ibv_create_qp(pd, &qp_init_attr);
    if (!qp) {
        perror("Failed to create QP");
        return -1;
    }
    printf("QP created using SRQ successfully\n");
    return 0;
}

[yishaih]

What is the issue that you hit without that change ? not clear from the code and the commit log.

What are the values of the below in your case ?

ctx->max_recv_wr
attr->cap.max_recv_wr

[FujiZ]

In my case attr->cap.max_recv_wr is way greater than ctx->max_recv_wr because it contains random bytes on stack. However, since the manual notes that "The attributes max_recv_wr and max_recv_sge are ignored by ibv_create_qp() if the QP is to be associated with an SRQ" (https://man7.org/linux/man-pages/man3/ibv_create_qp.3.html), I believe that any value of attr->cap.max_recv_wr is acceptable when QP is associated with SRQ.

[FujiZ]

Without this change I will get EINVAL from ibv_create_qp() if attr->cap.max_recv_wr > cap.max_recv_wr even if the QP is to be associated with SRQ.

[yishaih]

I looked around at other drivers, is that case handled at all ?

Why should an application use SRQ and pass attr->cap.max_recv_wr which is different than 0 at all ?

If the issue is really important to handle, we may consider setting attr->cap.max_recv_wr to 0 in the verbs layer to let it work for all drivers around.

Please check and let's get other feedback here.

[FujiZ]

I looked around at other drivers, is that case handled at all ?

I'm not sure if other drivers handle this issue correctly, since I only have mellanox NIC at hand.

Why should an application use SRQ and pass attr->cap.max_recv_wr which is different than 0 at all ?

Because as I mentioned before, the manual said that "The attributes max_recv_wr and max_recv_sge are ignored by ibv_create_qp() if the QP is to be associated with an SRQ" (https://man7.org/linux/man-pages/man3/ibv_create_qp.3.html), which means that assigning different values to max_recv_wr should produce the same result when SRQ is used. Besides, the manual didn't mention that "The attributes max_recv_wr and max_recv_sge should be set to 0 if the QP is to be associated with an SRQ", so it is valid for an application to use SRQ and pass attr->cap.max_recv_wr which is different than 0. The value of attr->cap.max_recv_wr may come from uninitialized stack variables, since the application may construct the ibv_qp_init_attr on stack and not explicitly set the max_recv_wr when using SRQ.

If the issue is really important to handle, we may consider setting attr->cap.max_recv_wr to 0 in the verbs layer to let it work for all drivers around.

Yeah I think that is viable choice, too.

[rleon]

do we have resolution here?

[yishaih]

I would expect another general solution that will fit all drivers and close this one.