libfabric verbs provider not working on Windows 11 (fi_info -p verbs → -120 “no message available” / no RDMA devices found) #11613

ficipo · 2025-11-12T15:22:02Z

ficipo
Nov 12, 2025

I’m trying to use the libfabric verbs provider on Windows 11 with Mellanox ConnectX-6 NIC for RDMA communication, but fi_info -p verbs and the fi_pingpong test do not work as expected. fi_info.exe doesn’t even show the devices which use verbs provider, but with -l argument I can see that verbs is listed (version: 203.10). Running the same version of libfabric on Ubuntu 22.04 with identical Mellanox hardware detects verbs correctly, so this doesn’t appear to be a hardware issue.

I'm getting ret=-120 "no message available".
Here is the full log output with FI_LOG_LEVEL=info set:

fi_info -p verbs

Click to expand full log

C:\Users\Admin\Desktop\Test\libfabric-2.3.1\x64\Release-v142>fi_info.exe -p verbs
libfabric:36824:1762953367::core:core:fi_param_get():373 variable perf_cntr=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable hook=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable hmem=
libfabric:36824:1762953367::core:core:ofi_hmem_init():667 Hmem iface FI_HMEM_CUDA not supported
libfabric:36824:1762953367::core:core:ofi_hmem_init():667 Hmem iface FI_HMEM_ROCR not supported
libfabric:36824:1762953367::core:core:ofi_hmem_init():667 Hmem iface FI_HMEM_ZE not supported
libfabric:36824:1762953367::core:core:ofi_hmem_init():667 Hmem iface FI_HMEM_NEURON not supported
libfabric:36824:1762953367::core:core:ofi_hmem_init():667 Hmem iface FI_HMEM_SYNAPSEAI not supported
libfabric:36824:1762953367::core:core:fi_param_get():373 variable hmem_disable_p2p=
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor uffd
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor memhooks
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor cuda
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor cuda_ipc
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor rocr
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor rocr_ipc
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor xpmem
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor ze
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor ze_ipc
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor import
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor kdreg2
libfabric:36824:1762953367::core:core:fi_param_get():373 variable mr_cache_max_size=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable mr_cache_max_count=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable mr_cache_monitor=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable mr_cuda_cache_monitor_enabled=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable mr_rocr_cache_monitor_enabled=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable mr_ze_cache_monitor_enabled=
libfabric:36824:1762953367::core:mr:ofi_default_cache_size():67 default cache size=1048576
libfabric:36824:1762953367::core:mr:ofi_monitors_init():317 Default memory monitor is: disabled
libfabric:36824:1762953367::core:core:fi_param_get():373 variable provider=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable fork_unsafe=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable universe_size=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable av_remove_cleanup=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable offload_coll_provider=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable enable_passthru=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable buffer_size=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable tx_size=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable rx_size=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable msg_tx_size=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable msg_rx_size=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable cm_progress_interval=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable cq_eq_fairness=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable data_auto_progress=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable use_rndv_write=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable def_wait_obj=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable def_tcp_wait_obj=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable detect_hmem_iface=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable rescan=
libfabric:36824:1762953367::core:core:ofi_register_provider():528 registering provider: ofi_rxm (203.10)
libfabric:36824:1762953367::core:core:ofi_register_provider():528 registering provider: verbs (203.10)
libfabric:36824:1762953367::ofi_rxd:core:fi_param_get():373 variable spin_count=
libfabric:36824:1762953367::ofi_rxd:core:fi_param_get():373 variable retry=
libfabric:36824:1762953367::ofi_rxd:core:fi_param_get():373 variable max_peers=
libfabric:36824:1762953367::ofi_rxd:core:fi_param_get():373 variable max_unacked=
libfabric:36824:1762953367::ofi_rxd:core:fi_param_get():373 variable rescan=
libfabric:36824:1762953367::core:core:ofi_register_provider():528 registering provider: ofi_rxd (203.10)
libfabric:36824:1762953367::core:core:ofi_register_provider():528 registering provider: udp (203.10)
libfabric:36824:1762953367::core:core:ofi_register_provider():528 registering provider: sockets (203.10)
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable prov_name=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable port_high_range=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable port_low_range=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable tx_size=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable rx_size=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable max_inject=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable max_saved=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable max_saved_size=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable max_rx_size=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable nodelay=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable staging_sbuf_size=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable prefetch_rbuf_size=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable zerocopy_size=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable trace_msg=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable disable_auto_progress=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable io_uring=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable firewall_addr=
libfabric:36824:1762953367::core:core:ofi_register_provider():528 registering provider: tcp (203.10)
libfabric:36824:1762953367::core:core:ofi_register_provider():528 registering provider: ofi_hook_perf (203.10)
libfabric:36824:1762953367::core:core:ofi_register_provider():528 registering provider: ofi_hook_noop (203.10)
libfabric:36824:1762953367::core:core:ofi_register_provider():528 registering provider: off_coll (203.10)
libfabric:36824:1762953367::verbs:core:nd_startup():744 nd_startup: starting initialization
libfabric:36824:1762953367::verbs:core:nd_startup():750 nd_startup: WSAStartup complete
libfabric:36824:1762953367::verbs:core:nd_create_factory():436 nd_create_factory: provider 7C4DB90D-6125-43B0-9267-2617E62DAAF2 path: C:\WINDOWS\system32\ch_nd.dll
libfabric:36824:1762953367::verbs:core:nd_create_factory():436 nd_create_factory: provider 134043E9-5D77-47BE-AFA9-5BC722D6F49A path: C:\WINDOWS\system32\mlx5nd.dll
libfabric:36824:1762953367::verbs:core:nd_create_factory():436 nd_create_factory: provider B491F6A6-F964-4111-8DCC-89174580C020 path: C:\WINDOWS\system32\mlx5nd.dll
libfabric:36824:1762953367::verbs:fabric:vrb_init():776 failed in OS specific device initialization
libfabric:36824:1762953367::core:core:fi_getinfo():1386 fi_getinfo: provider verbs returned -120 (no message available)
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable use_srx=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable tx_size=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable rx_size=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable tx_iov_limit=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable rx_iov_limit=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable inline_size=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable min_rnr_timer=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable use_odp=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable prefer_xrc=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable xrcd_filename=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable cqread_bunch_size=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable gid_idx=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable device_name=
libfabric:36824:1762953367:ofi_rxm:verbs:core:vrb_read_params():739 dmabuf support is disabled
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable iface=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable dgram_use_name_server=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable dgram_name_server_port=
libfabric:36824:1762953367:ofi_rxm:verbs:fabric:ibv_get_device_list():50
libfabric:36824:1762953367:ofi_rxm:verbs:fabric:vrb_init_info():1422 no RDMA devices found
libfabric:36824:1762953367:ofi_rxm:core:core:fi_getinfo():1386 fi_getinfo: provider verbs returned -120 (no message available)
libfabric:36824:1762953367:ofi_rxm:core:core:ofi_layering_ok():1265 Provider ofi_rxm is excluded
libfabric:36824:1762953367:ofi_rxm:core:core:ofi_layering_ok():1276 Need core provider, skipping ofi_rxd
libfabric:36824:1762953367::ofi_rxm:core:ofi_check_fabric_attr():470 Requesting provider verbs, skipping tcp
libfabric:36824:1762953367::ofi_rxm:core:ofi_check_fabric_attr():470 Requesting provider verbs, skipping tcp
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable use_srx=
libfabric:36824:1762953367:ofi_rxm:core:core:fi_getinfo():1386 fi_getinfo: provider verbs returned -120 (no message available)
libfabric:36824:1762953367:ofi_rxm:core:core:ofi_layering_ok():1265 Provider ofi_rxm is excluded
libfabric:36824:1762953367:ofi_rxm:core:core:ofi_layering_ok():1276 Need core provider, skipping ofi_rxd
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable use_srx=
libfabric:36824:1762953367:ofi_rxm:core:core:fi_getinfo():1386 fi_getinfo: provider verbs returned -120 (no message available)
libfabric:36824:1762953367:ofi_rxm:core:core:ofi_layering_ok():1265 Provider ofi_rxm is excluded
libfabric:36824:1762953367:ofi_rxm:core:core:ofi_layering_ok():1276 Need core provider, skipping ofi_rxd
libfabric:36824:1762953367::core:core:fi_getinfo():1386 fi_getinfo: provider ofi_rxm returned -120 (no message available)
libfabric:36824:1762953367:ofi_rxd:core:core:fi_getinfo():1386 fi_getinfo: provider verbs returned -120 (no message available)
libfabric:36824:1762953367:ofi_rxd:core:core:ofi_layering_ok():1276 Need core provider, skipping ofi_rxm
libfabric:36824:1762953367:ofi_rxd:core:core:ofi_layering_ok():1265 Provider ofi_rxd is excluded
libfabric:36824:1762953367::core:core:fi_getinfo():1386 fi_getinfo: provider ofi_rxd returned -120 (no message available)
fi_getinfo: -120 (no message available)
libfabric:36824:1762953367::verbs:core:nd_shutdown():764 nd_shutdown: shutdown WSA
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor uffd
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor memhooks
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor cuda
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor cuda_ipc
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor rocr
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor rocr_ipc
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor xpmem
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor ze
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor ze_ipc
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor import
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor kdreg2

Here is the Get-NetAdapterRdma output form Winodws PowerShell:
Name InterfaceDescription Enabled Operational PFC ETS

Ethernet Mellanox ConnectX-6 Dx Adapter #3 True False False False
vEthernet (Default Hyper-V Virtual Ethernet Adapter False False NA NA
vEthernet (WSL (H Hyper-V Virtual Ethernet Adapter #2 False False NA NA
Ethernet 2 Mellanox ConnectX-6 Dx Adapter #4 True False False False

System Configuration:
-OS: Windows 11 Pro(24H2)
-Hardware: Mellanox ConnectX-6 Dx 100 GbE
-Driver: WinOF-2 version 25.7.50000 Windows Client 11, https://network.nvidia.com/products/adapter-software/ethernet/windows/winof-2/
-libfabric version: 2.3.1 (built from source using Visual Studio 2022, Release-v142 x64),extracted header files from downloaded NetworkDirect_DDK.zip:\NetDirect\include\ into include\windows
-RoCE: Enabled in mlxconfig (ROCE_V2=1, ROCE_ENABLE=2)

efschu · 2026-04-09T13:10:15Z

efschu
Apr 9, 2026

I'm using libfabrics with verbs on my linux app. Now trying to port. But I have problem with Windows 11 and verbs too

Bug Report: fi_pingpong fails with verbs provider on Windows (Error -10038 / WSAENOTCONN)

Summary

The fi_pingpong test fails when using the verbs provider on Windows, while it works correctly with the tcp provider. The error occurs during fi_domain() call with error code -10038 (WSAENOTCONN - "Socket not connected").

Environment

OS: Microsoft Windows 10.0.26200.8037 (Windows 11)
Libfabric Version: v2.5.0rc1-135-g72c395f1f (commit 72c395f)
Build Type: Debug-v142 (Visual Studio 2019)
Hardware: Mellanox ConnectX-4 Virtual Adapter
Network Interfaces:
- Ethernet 3: 192.168.40.144 (Mellanox ConnectX-4 Virtual Adapter 2)
- Ethernet 2: 192.168.80.144 (Mellanox ConnectX-4 Virtual Adapter)

Get-NetAdapterRdma

Name                      InterfaceDescription                     Enabled     Operational     PFC        ETS       
----                      --------------------                     -------     -----------     ---        ---       
Ethernet 3                Mellanox ConnectX-4 Virtual Adapter #2   True        True            NA         NA        
Ethernet 2                Mellanox ConnectX-4 Virtual Adapter      True        True            NA         NA

Reproduction Steps

Clone libfabric to C:\Users\someusername\libfabric
include header files from NetworkDirect_DDK.zip and build with Visual Studio (Debug-v142)
Run fi_pingpong test with verbs provider:

cd C:\Users\someusername\libfabric\x64\Debug-v142
fi_pingpong.exe -p verbs -e msg -I 5 -s 192.168.40.144  # Server (background)
fi_pingpong.exe -p verbs -e msg -I 5 -s 192.168.40.144 192.168.40.144  # Client

Expected Behavior

fi_pingpong should complete successfully with verbs provider, similar to tcp provider.

Actual Behavior

TCP Provider (Works)

bytes   #sent   #ack     total       time     MB/sec    usec/xfer   Mxfers/sec
64      5       =5       640         0.00s      0.64     100.00       0.01
256     5       =5       2.5k        0.00s       inf       0.00        inf
1k      5       =5       10k         0.00s       inf       0.00        inf
4k      5       =5       40k         0.00s     40.96     100.00       0.01
64k     5       =5       640k        0.00s    655.36     100.00       0.01
1m      5       =5       10m         0.00s   3495.25     300.00       0.00

Verbs Provider (Fails)

ctrl/read(): C:\Users\someusername\libfabric\util\pingpong.c:566 , ret=-108 (connection reset)

With verbose output:

[debug] C:\Users\someusername\libfabric\util\pingpong.c:671 : Received name
[debug] C:\Users\someusername\libfabric\util\pingpong.c:1402: Opening fabric resources: fabric, eq & domain
fi_domain(): C:\Users\someusername\libfabric\util\pingpong.c:1418, ret=-10038 (Unspecified error)

Server-Side Output (Verbs)

[debug] C:\Users\someusername\libfabric\util\pingpong.c:635 : Sent name

The server successfully sends the name but does not proceed to open fabric resources, indicating the client connection is being reset before the domain can be opened.

Available Providers (fi_info.exe -p verbs)

provider: verbs
    fabric: IB-0xfe80000000000000
    domain: 192.168.40.144
    version: 205.0
    type: FI_EP_MSG
    protocol: FI_PROTO_RDMA_CM_IB_RC

provider: verbs
    fabric: IB-0xfe80000000000000
    domain: 192.168.40.144-dgram
    version: 205.0
    type: FI_EP_DGRAM
    protocol: FI_PROTO_IB_UD

provider: verbs;ofi_rxm
    fabric: IB-0xfe80000000000000
    domain: 192.168.40.144
    version: 205.0
    type: FI_EP_RDM
    protocol: FI_PROTO_RXM

Analysis

Control Socket Works: The TCP control socket connection between client and server is established successfully (client connects, server accepts, name exchange completes).
Domain Opening Fails: The error occurs when calling fi_domain() with the verbs provider. Error code -10038 corresponds to Windows socket error WSAENOTCONN ("Socket not connected").
Endpoint Type: The issue affects FI_EP_MSG endpoint type. FI_EP_RDM requires verbs;ofi_rxm which also times out.
Windows-Specific: The error code -10038 is a Windows-specific socket error being returned directly from the libfabric verbs provider, suggesting an issue with how the verbs provider handles sockets or RDMA connections on Windows.

Potential Root Causes

Windows RDMA CM Integration: The verbs provider on Windows uses the Native Direct (ND) API through verbs_nd_init.c. The integration with Windows RDMA CM may have issues with connection establishment.
Socket Handling: The error suggests a socket-related issue. The verbs provider may be incorrectly handling Windows sockets during domain initialization.
Device Binding: The vrb_open_device_by_name() function in verbs_domain.c may be failing to properly bind to the RDMA device on Windows.
Protection Domain Allocation: The ibv_alloc_pd() call may be failing or returning an invalid pointer on Windows.

Related Code Locations

prov/verbs/src/verbs_domain.c - vrb_domain() function (line ~330-440)
prov/verbs/src/verbs_domain.c - vrb_open_device_by_name() function
prov/verbs/src/windows/verbs_nd_init.c - Windows-specific ND initialization
prov/verbs/include/windows/verbs_osd.h - Windows OS-specific definitions
util/pingpong.c - Test application (line 1418 for fi_domain call)

Workaround

Use the tcp provider instead of verbs:

fi_pingpong.exe -p tcp -e rdm -I 5  # Server
fi_pingpong.exe -p tcp -e rdm -I 5 localhost  # Client

Additional Notes

The issue occurs when testing from the same machine to itself (localhost or same IP)
Multiple domain names were tested (192.168.40.144, 192.168.80.144) with the same result
The verbs provider correctly discovers the Mellanox adapters via fi_info.exe

Request

Please investigate:

Why fi_domain() returns WSAENOTCONN (-10038) on Windows with verbs provider
Whether there are known issues with the verbs provider on Windows
Any required configuration or environment variables for verbs provider on Windows
Potential fixes or workarounds for this issue

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

libfabric verbs provider not working on Windows 11 (fi_info -p verbs → -120 “no message available” / no RDMA devices found) #11613

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

libfabric verbs provider not working on Windows 11 (fi_info -p verbs → -120 “no message available” / no RDMA devices found) #11613

Uh oh!

ficipo Nov 12, 2025

fi_info -p verbs

Replies: 1 comment

Uh oh!

efschu Apr 9, 2026

Bug Report: fi_pingpong fails with verbs provider on Windows (Error -10038 / WSAENOTCONN)

Summary

Environment

Reproduction Steps

Expected Behavior

Actual Behavior

TCP Provider (Works)

Verbs Provider (Fails)

Server-Side Output (Verbs)

Available Providers (fi_info.exe -p verbs)

Analysis

Potential Root Causes

Related Code Locations

Workaround

Additional Notes

Request

ficipo
Nov 12, 2025

efschu
Apr 9, 2026