ENOBUFS write error with many parallel CANables

[GustavoSilvera]

Hi folks,

I currently have a setup with one Ubuntu 24.04 x86 host connected to nine arm64 ECUs with three CANable adapters per ECU, in total there are 27 CANables connected through USB hubs directly to the host. These are all using CANable 0.4 adapters, so no CanFD here.

When I run my CAN program I've noticed that <= 3 active ECUs in parallel works fine, but as soon as I scale up to utilizing all 9 ECUs all the write operations on socketCan fail with ENOBUFS ie. No buffer space available. During this time we can receive messages just fine, only fails on transmission.

For all our CANables, the CAN load does not go above ~70% @ 500kbps, we have a txqueuelength of 10000.

This is what ip -details -statistic shows for a typical interface:

can5: <NOARP,UP,LOWER_UP,ECHO> mtu 16 qdisc pfifo_fast state UP mode DEFAULT group default qlen 10000
    link/can  promiscuity 0 minmtu 0 maxmtu 0 
    can state ERROR-ACTIVE restart-ms 0 
          bitrate 500000 sample-point 0.875 
          tq 125 prop-seg 6 phase-seg1 7 phase-seg2 2 sjw 1
          gs_usb: tseg1 1..16 tseg2 1..8 sjw 1..4 brp 1..1024 brp-inc 1
          clock 48000000 
          re-started bus-errors arbit-lost error-warn error-pass bus-off
          0          0          0          89943      13847      0         numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 parentbus usb parentdev 3-4.3.1:1.0 
    RX:  bytes    packets errors dropped  missed   mcast           
    8240146476 1274823245      0   24402       0       0 
    TX:  bytes    packets errors dropped carrier collsns           
    2198512496  317644290      0      20       0       0 
 

Investigation so far

I was wondering what might cause interference between (supposedly) isolated CANables, so I took a look at the relevant drivers such as gs_usb.

I noticed that the can_send method returns * -ENOBUFS on full driver queue (see net_xmit_errno()) and when I was debugging gs_usb I noticed that these constants looked suspiciously relevant:

/* Only send a max of GS_MAX_TX_URBS frames per channel at a time. */
#define GS_MAX_TX_URBS 10
/* Only launch a max of GS_MAX_RX_URBS usb requests at a time. */
#define GS_MAX_RX_URBS 30
#define GS_NAPI_WEIGHT 32

Which would align with the hypothesis that 9 parallel TX'ers is okay, and 27 parallel RX'ers is okay, which is what we see.

But I'm not seeing where the gs_can::tx_context would be shared across CAN devices, but I can confirm that this atomic counter for active_tx_urbs does surpass GS_MAX_TX_URBS which calls netif_stop_queue and then we see the ENOBUFS errors. I'm wondering if we're hitting an edge case where we are TX'ing faster than we are RX'ing the ack, so gs_can_start_xmit gets invoked a bunch but its corresponding gs_usb_receive_bulk_callback (which frees the held TX context) does not get called as often (or is delayed). This fills up the tx_context buffer and causes the issue.

We've also tried bumping this GS_MAX_TX_URBS to 1000 and even 10000, but ultimately the buffer overflow happens nearly instantly so I suspect there is a bug/race somewhere.

Has anyone else had success running many parallel CANables in this fashion?

Details about our program

My program consumes CAN tx & rx requests in a non-blocking fashion and runs every 1ms. I've noticed the data transmission pattern is somewhat bursty, ie. for transmission we enqueue a bunch of messages and then transmit all at once every 10ms. Our fastest CAN message cycle time is 10ms anyways so I thought this might be okay.

For reference, here is a candump (candump.zip) report of the traffic we were seeing during our test. Replaying it with canplayer -I does seem to reproduce the issue when done on all 27 CANables.

Thank you all for your help!