RoCEv2 OTG Model #405

satyamsinghKey · 2025-02-11T09:33:45Z

Issue: #403

New PR Link: (#415)

Redocly View:
https://redocly.github.io/redoc/?url=https://raw.githubusercontent.com/open-traffic-generator/models/dev_rocev2/artifacts/openapi.yaml&nocors#tag/Configuration/operation/set_config

Objectives:
Add support of RoCEv2 Protocol in OTG model
Added fields under Configuration and Monitor

Sample Code Snippet:

    config = api.config()
    (device,) = config.devices.device(name="device")

    # device config
    eth = device.ethernets.add()
    eth.name = "eth"
    eth.connection.port_name = port.name
    eth.mac = "00:00:00:00:00:11"
    ipv4 = eth.ipv4_addresses.add()
    ipv4.name = "ipv4"
    ipv4.address = "21.1.1.1"
    ipv4.prefix = 24
    ipv4.gateway = "21.1.1.2"

    rocev2v4 = device.rocev2
    rocev2v4_int = rocev2v4.ipv4_interfaces.add()
    rocev2v4_int.ipv4_name = ipv4.name
    rocev2v4_peer = rocev2v4_int.peers.add()
    rocev2v4_peer.name = "RoCEv2 1"
    rocev2v4_peer.num_of_qps = 2
    rocev2v4_peer.remote_end_point_ip_address = "2.2.2.2"
    rocev2v4_peer.ib_mtu = 15
    flow_settings = rocev2v4_peer.flow_settings.add()
    flow_settings.dscp = 25


    # push configuration
    api.set_config(config)

    # start all protocols
    control_state = api.control_state()
    control_state.protocol.all.state = control_state.protocol.all.START
    api.set_control_state(control_state)

    # create a query for rocev2 metrics
    req = api.metrics_request()
    req.rocev2.peer_names = ["RoCEv2 1"]
    results = api.get_metrics(req)

…r/models into dev_rocev2

device/rocev2/flowsettings.yaml

device/rocev2/rocev2.yaml

result/rocev2ipv4.yaml

apratimmukherjee · 2025-03-05T13:56:14Z

config/config.yaml

@@ -51,19 +51,26 @@ components:
          items:
            $ref: '../flow/flow.yaml#/components/schemas/Flow'
          x-field-uid: 6
+        stateful_flows:
+          description: |-
+            stateful flows currently on RoCEv2 present.


This is not for RoCEv2 alone. So probably something like "The stateful flows that will be configured for the traffic generator, for protocols such as RoCEv2 where packet rate and/or contents can change dynamically based on packets received from the connected device "

modified description

@apratimmukherjee , @satyamsinghKey - I have doubt if we will change the packet content. For example for Roce, the content of the packet does not change. What changes dynamically/state-fully is packet rate (due to detection of congestion in the network), sequence of packet (for example the old packet may get send again due to dynamic retransmission) etc. In a prefect scenarios what changes in every packet is SEQ num. iCRC etc. But in the first reading it felt by "content" we are meaning payload, which does not change. So you may think of omitting the word "content".

Maybe instead of 'contents' we can use 'contents of protocol headers' .. I mean flags in IPv4 header or window etc. for TCP etc. which is unlike stateless flows where nothing really changes pkt to pkt .

control/action.yaml

control/protocol.yaml

apratimmukherjee · 2025-03-05T14:27:36Z

flow/rocev2_flow.yaml

+          x-field-uid: 3
+        rate:
+          description: |-
+           Two valid choices are line_rate and burst_mode, if line_rate is selected, per_port_settings will be used to configure.


Some more clarity would be good here.
e.g. when line_rate mode is chosen , it is required that all flows with a tx endpoint on such a port must choose line_rate mode [ unless mixed mode is alowed ???]
In this case, all the rdma flows will transmit data in interleaved manner based on qp.message size as fast as possible but not exceeding the line_rate value configured in port.rate .
If burst_mode is chosen , all flows with Tx endpoint on a port should be configured in burst_mode. Then each rdma flows sends packets as per rflow.message_size configuration for fixed / infinite times with burst_gap time between each iteration.
Some level of clarity on what is expected to go on the wire for each mode is needed.

modified description as suggested

apratimmukherjee · 2025-03-05T14:28:07Z

flow/rocev2_flow.yaml

+           Two valid choices are line_rate and burst_mode, if line_rate is selected, per_port_settings will be used to configure.
+          type: string
+          x-enum:
+            line_rate:


port_line_rate gives a hint to configure the line_rate in the port attributes .

flow/rocev2_flow.yaml

result/rocev2ipv6.yaml

apratimmukherjee · 2025-03-05T14:32:14Z

result/rocev2ipv6.yaml

+          type: integer
+          format: uint64
+          x-field-uid: 5
+        connect_request_tx:


Can't these flap like regular tcp based sessions such as bgp ; any reason not to have a flap count.

apratimmukherjee · 2025-03-05T14:56:28Z

device/rocev2/flowsettings.yaml

+        rocev2_verb:
+          description: >-
+            RoCEv2 Verb, Available options are: none, write, wrtie_with_immediate, send, send_with_immediate and read: The corresponding flow will not take part in traffic.
+          type: string


This could be a choice. Allows immediate_data to be made visible only for _immediate types. Looks like it is not relevant for the other types.

That is correct

apratimmukherjee · 2025-03-05T14:59:22Z

device/rocev2/flowsettings.yaml

+          format: hex
+          default: "00000000"
+          x-field-uid: 7
+        message_size:


Is message_size part of session bring up or is it needed solely for data flows e.g. could a single QP potentially first send a payload of size 1000 bytes and then another of size 10000 bytes ? In which probably should be somewhere in stateful_flow configuration as properties of the data flow , maybe as an array with ixnetwork supporting a single value only for all flows between same qp endpoints. Or is this specification bound that each qp can have single datagram being split and sent on it ?

kunkeysight · 2025-03-06T05:31:30Z

device/rocev2/flowsettings.yaml

+      properties:
+        connection_type:
+          description: >-
+            There are multiple connection types. Valid values are :  Reliable Connection (RC), Reliable Datagram (RD), Extended Reliable Connection (XRC),


At the time of this implementation, please consider only RC. So from description excludes other modes. Or, explicitly mention that supported mode is RC.

This is not applicable for IxN but other test vendors whose configuration will be based on the model

kunkeysight · 2025-03-06T05:34:30Z

device/rocev2/flowsettings.yaml

+          x-field-uid: 7
+        message_size:
+          description: >-
+            The Maximum message size that is allowed to transfer depends on the MTU size and the number of VLANs configured on the interfaces.


Maximum bytes transferred will depend on IB MTU (not Ethernet MTU). So, given message size will be broken into multiple packets viz. WF, WM, WL etc. based on the IB MTU value. We take IB MTU as an user input also.

apratimmukherjee · 2025-03-06T14:41:30Z

config/config.yaml

+          type: array
+          items:
+            $ref: '../flow/stateful_flow.yaml#/components/schemas/StatefulFlow'
+          x-field-uid: 7


You cannot change x-uids of existing attributes. Please add new attributes to existing objects at the end else it will break clients running tests with older snappi versions.

apratimmukherjee · 2025-03-06T14:44:33Z

device/rocev2/rocev2.yaml

+        Configuration for RoCEv2 peer settings.
+      type: object
+      properties:
+        num_of_qps:


Flow settings is an array so number of times user add a new flow setting will define size of the flow settings array not this. Might need discussion.

apratimmukherjee · 2025-03-06T14:46:20Z

device/rocev2/rocev2ipv4.yaml

+      type: object
+      required: [num_of_qps, destination_ip_address, ib_mtu]
+      properties:
+        num_of_qps:


Similar to previous, the times user adds an element into the array will define the size .. should not be needing this .

apratimmukherjee · 2025-03-06T14:49:56Z

device/rocev2/rocev2ipv4.yaml

+      description: >-
+        Configuration for Rocev2 IPv4 peers.
+      type: object
+      required: [num_of_qps, destination_ip_address, ib_mtu]


Name probably should be must , cant ib_mtu be non required with default value ?
num_qps probably can be removed completely.

apratimmukherjee · 2025-03-06T14:51:57Z

device/rocev2/rocev2ipv6.yaml

+      description: >-
+        Configuration for Rocev2 IPv6 peer settings.
+      type: object
+      required: [num_of_qps, destination_ip_address, ib_mtu]


num_of_qps is probably just removable and ib_mtu should not be required since it has a default value.
In general every attribute should either have a default_value or be required but not both at the same time.

apratimmukherjee · 2025-03-06T14:54:29Z

device/rocev2/rocev2.yaml

+          type: integer
+          format: uint32
+          x-field-uid: 1
+        ib_mtu:


I think we can just move the ib_mtu to v4 and v6 peer and just remove this object. Model is easier to review and maintain. Or call it RoceAttributes and just have ib_mtu in that and refer to that . Dont need flow_settings which can be directly in the b4 / v6 peer.

apratimmukherjee · 2025-03-06T15:00:22Z

flow/rocev2_flow.yaml

+          x-field-uid: 1
+        tx_endpoint: 
+          description: |-
+            Tx Endpoint. Empty for Rocev2.


It is not clear what is meant by empty. Is it like RSVP-TE where the src defines the destination and must choose correct egress for specific ingress and no choice is available ?
It is not clear to me how this will help user control which QPs to actually send traffic on if this is empty. If a field is of no value we need to removed it.

apratimmukherjee · 2025-03-06T15:11:13Z

flow/rocev2_flow.yaml

+             on qp.message size as fast as possible but not exceeding the line_rate value configured.
+             If burst_mode is chosen, all flows with Tx endpoint on a port should be configured in burst_mode.
+              Then each rdma flows sends packets as per rocev2_flow_settings.message_size configuration
+               for fixed / infinite times with burst_gap time between each iteration.


This was just a suggestion. The text might need more clarity and accuracy in terms of what the different modes are trying to achieve which needs to come from deeper understanding of rocev2 traffic control needs,
If it is indeed required that certain modes require and mandate all flows on a port to exhibit same type of behaviour, current model would leave a lot of room for error in configuration . We probably need to rethink .
Maybe
rocev2_flows :
array of rocev2_ transmit ports :
port_name
choice
target_line_mode
line rate value desired across all qp values txed from this port
array of flows
src qp endpoint (only qps on this port can be selected)
optional dst port ( for tracking maybe if needed )
other flow parameters
burst_mode ( indivdual control per flow )
array of flows
tx endpoint
optional (expected ) rx endpoint for 2 arm flows and stats
burst_mode
choice
continuous
fixed
burst_count
Not clear whether inter_batch is for line_rate or burst_mode or both -> Needs to fitted in correctly accordingly with proper explanation and expectation from implementation what is the behaviour of that setting,

apratimmukherjee · 2025-03-06T15:12:57Z

flow/rocev2_flow.yaml

+      properties:
+        inter_batch_period_unit:
+          description: |-
+            inter batch period units.


inter -> Inter
Needs more info on that all this means to packet generation by a tool implementing this,

apratimmukherjee · 2025-03-06T15:13:34Z

flow/rocev2_flow.yaml

+        inter_batch_period:
+          description: |-
+            Inter batch Period.
+          type: integer


satyamsinghKey · 2025-03-12T13:38:27Z

A new PR has been created with the latest RoCEv2 model changes. Please refer to that PR for latest updates.
Link to new PR : (#415)

satyamsinghKey and others added 30 commits November 21, 2024 20:49

Initial code checkin for RoCEv2

13da9fe

Update auto generated content

c82ca4f

Correct Description

713ae02

Merge branch 'dev_rocev2' of https://github.com/open-traffic-generato…

4448a94

…r/models into dev_rocev2

Update auto generated content

838d7bc

Adding RoCEv2 in Control Section

6caf646

Merge branch 'dev_rocev2' of https://github.com/open-traffic-generato…

1af5f23

…r/models into dev_rocev2

Update auto generated content

facc567

Adding code for RoCEv2 statistics

2c13e2c

Update auto generated content

172a2c4

Correcting names of RoCEv2

e119120

Merge branch 'dev_rocev2' of https://github.com/open-traffic-generato…

0d02fd4

…r/models into dev_rocev2

Update auto generated content

571fb89

change RoCEv2 peer name in stats section

88875f1

Update auto generated content

94f1f87

Remove RoCEv2 from control_action

424d11c

Update auto generated content

d2af6fb

Changes based on review feedback

fc452b2

Update auto generated content

23a0e8d

Adding description for Flow Settings

b4ed901

Merge branch 'dev_rocev2' of https://github.com/open-traffic-generato…

5d17bb1

…r/models into dev_rocev2

Update auto generated content

4caf28d

Adding Remote End Fields

cc52fcd

Merge branch 'dev_rocev2' of https://github.com/open-traffic-generato…

9d9947f

…r/models into dev_rocev2

Update auto generated content

aa0eecc

default value correction

e650c31

Update auto generated content

1548d44

Correction in default Hex String

ef2d9a1

Update auto generated content

fb8d1e9

Removing commented lines and adding Traffic Parameters in Control Plane

f3a8031

satyamsinghKey changed the title ~~Dev rocev2~~ RoCEv2 OTG Model Feb 11, 2025

satyamsinghKey and others added 2 commits February 11, 2025 16:47

correct traffic burst count uid

eecad6e

Update auto generated content

2d329c5

dkrabort reviewed Feb 17, 2025

View reviewed changes

satyamsinghKey and others added 2 commits February 18, 2025 12:46

incorporate review changes

81cb0a6

Update auto generated content

862526e

satyamsinghKey requested a review from dkrabort February 18, 2025 08:59

satyamsinghKey and others added 4 commits February 25, 2025 19:43

Add RoCEv2 Flow stats

a6d3c25

Update auto generated content

9695955

RoCEv2 Traffic Flow Changes

4c54115

Update auto generated content

503c516

satyamsinghKey requested review from apratimmukherjee, alakendu and PrasenjitAdhikary March 5, 2025 10:19

apratimmukherjee reviewed Mar 5, 2025

View reviewed changes

satyamsinghKey and others added 4 commits March 6, 2025 02:37

correct RoCev2 metric name

7431f37

correct metrix name

2d0df4e

correct metric name

a2d04ec

Update auto generated content

4c9b9d4

apratimmukherjee requested a review from kingshukdev March 6, 2025 05:10

kunkeysight reviewed Mar 6, 2025

View reviewed changes

satyamsinghKey and others added 2 commits March 6, 2025 13:56

Reivew changes

fd09afb

Update auto generated content

397d0e1

satyamsinghKey requested a review from apratimmukherjee March 6, 2025 12:37

apratimmukherjee reviewed Mar 6, 2025

View reviewed changes

apratimmukherjee added the enhancement New feature or request label Mar 7, 2025

satyamsinghKey mentioned this pull request Mar 12, 2025

RoCEv2 OTG model [latest] #415

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RoCEv2 OTG Model #405

RoCEv2 OTG Model #405

satyamsinghKey commented Feb 11, 2025 •

edited

Loading

apratimmukherjee Mar 5, 2025

satyamsinghKey Mar 6, 2025

kingshukdev Mar 6, 2025

apratimmukherjee Mar 7, 2025

apratimmukherjee Mar 5, 2025

satyamsinghKey Mar 6, 2025

apratimmukherjee Mar 5, 2025

apratimmukherjee Mar 5, 2025

dkrabort Mar 6, 2025

apratimmukherjee Mar 5, 2025

dkrabort Mar 6, 2025

apratimmukherjee Mar 5, 2025

kunkeysight Mar 6, 2025

dkrabort Mar 6, 2025

kunkeysight Mar 6, 2025

apratimmukherjee Mar 6, 2025

apratimmukherjee Mar 6, 2025

apratimmukherjee Mar 6, 2025

apratimmukherjee Mar 6, 2025

apratimmukherjee Mar 6, 2025

apratimmukherjee Mar 6, 2025

apratimmukherjee Mar 6, 2025

apratimmukherjee Mar 6, 2025

apratimmukherjee Mar 6, 2025

apratimmukherjee Mar 6, 2025

satyamsinghKey commented Mar 12, 2025

RoCEv2 OTG Model #405

Are you sure you want to change the base?

RoCEv2 OTG Model #405

Conversation

satyamsinghKey commented Feb 11, 2025 • edited Loading

New PR Link: (#415)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

satyamsinghKey commented Mar 12, 2025

satyamsinghKey commented Feb 11, 2025 •

edited

Loading