Merge pull request #92432 from aireilly/manual-cp-TELCODOCS-2211-openshift-418

[Manual CP 4.18] TELCODOCS-2211 - Adding PTP docs for 3-card T-GM

Author: aireilly

_topic_maps/_topic_map.yml

@@ -1516,10 +1516,10 @@ Topics:
 - Name: Using Stream Control Transmission Protocol
   File: using-sctp
   Distros: openshift-enterprise,openshift-origin
-- Name: Using Precision Time Protocol hardware
+- Name: Using PTP hardware
   Dir: ptp
   Topics:
-  - Name: About Precision Time Protocol in OpenShift cluster nodes
+  - Name: About PTP in OpenShift cluster nodes
     File: about-ptp
   - Name: Configuring PTP devices
     File: configuring-ptp

images/561_OpenShift_Using_PTP_network_0124.png

@@ -6,19 +6,20 @@
 [id="configuring-linuxptp-services-as-grandmaster-clock-dual-nic_{context}"]
 = Configuring linuxptp services as a grandmaster clock for dual E810 NICs
 
-You can configure the `linuxptp` services (`ptp4l`, `phc2sys`, `ts2phc`) as a grandmaster clock (T-GM) for dual E810 NICs by creating a `PtpConfig` custom resource (CR) that configures the host NICs.
+You can configure the `linuxptp` services (`ptp4l`, `phc2sys`, `ts2phc`) as a grandmaster clock (T-GM) for 2 E810 NICs by creating a `PtpConfig` custom resource (CR) that configures the NICs.
 
-You can configure the `linuxptp` services as a T-GM for the following dual E810 NICs:
+You can configure the `linuxptp` services as a T-GM for the following E810 NICs:
 
-* Intel E810-XXVDA4T Westport Channel NICs
-* Intel E810-CQDA2T Logan Beach NICs
+* Intel E810-XXVDA4T Westport Channel NIC
+* Intel E810-CQDA2T Logan Beach NIC
 
-For distributed RAN (D-RAN) use cases, you can configure PTP for dual-NICs as follows:
+For distributed RAN (D-RAN) use cases, you can configure PTP for 2 NICs as follows:
 
-* NIC one is synced to the global navigation satellite system (GNSS) time source.
-* NIC two is synced to the 1PPS timing output provided by NIC one. This configuration is provided by the PTP hardware plugin in the `PtpConfig` CR.
+* NIC 1 is synced to the global navigation satellite system (GNSS) time source.
+* NIC 2 is synced to the 1PPS timing output provided by NIC one. This configuration is provided by the PTP hardware plugin in the `PtpConfig` CR.
 
-The dual-NIC PTP T-GM configuration uses a single instance of `ptp4l` and one `ts2phc` process reporting two `ts2phc` instances, one for each NIC.
+The 2-card PTP T-GM configuration uses one instance of `ptp4l` and one instance of `ts2phc`.
+The `ptp4l` and `ts2phc` programs are each configured to operate on two PTP hardware clocks (PHCs), one for each NIC.
 The host system clock is synchronized from the NIC that is connected to the GNSS time source.
 
 [NOTE]

images/holdover_in_t_gm.png

@@ -0,0 +1,106 @@
+// Module included in the following assemblies:
+//
+// * networking/ptp/configuring-ptp.adoc
+
+:_mod-docs-content-type: PROCEDURE
+[id="configuring-linuxptp-services-as-grandmaster-clock-three-nic_{context}"]
+= Configuring linuxptp services as a grandmaster clock for 3 E810 NICs
+
+You can configure the `linuxptp` services (`ptp4l`, `phc2sys`, `ts2phc`) as a grandmaster clock (T-GM) for 3 E810 NICs by creating a `PtpConfig` custom resource (CR) that configures the NICs.
+
+You can configure the `linuxptp` services as a T-GM with 3 NICs for the following E810 NICs:
+
+* Intel E810-XXVDA4T Westport Channel NIC
+* Intel E810-CQDA2T Logan Beach NIC
+
+For distributed RAN (D-RAN) use cases, you can configure PTP for 3 NICs as follows:
+
+* NIC 1 is synced to the Global Navigation Satellite System (GNSS)
+* NICs 2 and 3 are synced to NIC 1 with 1PPS faceplate connections
+
+Use the following example `PtpConfig` CRs as the basis to configure `linuxptp` services as a 3-card Intel E810 T-GM.
+
+.Prerequisites
+
+* For T-GM clocks in production environments, install 3 Intel E810 NICs in the bare-metal cluster host.
+
+* Install the OpenShift CLI (`oc`).
+
+* Log in as a user with `cluster-admin` privileges.
+
+* Install the PTP Operator.
+
+.Procedure
+
+. Create the `PtpConfig` CR. For example:
+
+.. Save the following YAML in the `three-nic-grandmaster-clock-ptp-config.yaml` file:
++
+.PTP grandmaster clock configuration for 3 E810 NICs
+[%collapsible]
+====
+[source,yaml]
+----
+include::snippets/ptp_PtpConfigThreeCardGmWpc.yaml[]
+----
+====
++
+[NOTE]
+====
+Set the value for `ts2phc.nmea_serialport` to `/dev/gnss0`.
+====
+
+.. Create the CR by running the following command:
++
+[source,terminal]
+----
+$ oc create -f three-nic-grandmaster-clock-ptp-config.yaml
+----
+
+.Verification
+
+. Check that the `PtpConfig` profile is applied to the node.
+
+.. Get the list of pods in the `openshift-ptp` namespace by running the following command:
++
+[source,terminal]
+----
+$ oc get pods -n openshift-ptp -o wide
+----
++
+.Example output
+[source,terminal]
+----
+NAME                          READY   STATUS    RESTARTS   AGE     IP             NODE
+linuxptp-daemon-74m3q         3/3     Running   3          4d15h   10.16.230.7    compute-1.example.com
+ptp-operator-5f4f48d7c-x6zkn  1/1     Running   1          4d15h   10.128.1.145   compute-1.example.com
+----
+
+.. Check that the profile is correct. Run the following command, and examine the logs of the `linuxptp` daemon that corresponds to the node you specified in the `PtpConfig` profile:
++
+[source,terminal]
+----
+$ oc logs linuxptp-daemon-74m3q -n openshift-ptp -c linuxptp-daemon-container
+----
++
+.Example output
+[source,terminal]
+----
+ts2phc[2527.586]: [ts2phc.0.config:7] adding tstamp 1742826342.000000000 to clock /dev/ptp11 <1>
+ts2phc[2527.586]: [ts2phc.0.config:7] adding tstamp 1742826342.000000000 to clock /dev/ptp7 <1>
+ts2phc[2527.586]: [ts2phc.0.config:7] adding tstamp 1742826342.000000000 to clock /dev/ptp14 <1>
+ts2phc[2527.586]: [ts2phc.0.config:7] nmea delay: 56308811 ns
+ts2phc[2527.586]: [ts2phc.0.config:6] /dev/ptp14 offset          0 s2 freq      +0 <2>
+ts2phc[2527.587]: [ts2phc.0.config:6] /dev/ptp7 offset          0 s2 freq      +0 <2>
+ts2phc[2527.587]: [ts2phc.0.config:6] /dev/ptp11 offset          0 s2 freq      -0 <2>
+I0324 14:25:05.000439  106907 stats.go:61] state updated for ts2phc =s2
+I0324 14:25:05.000504  106907 event.go:419] dpll State s2, gnss State s2, tsphc state s2, gm state s2,
+I0324 14:25:05.000906  106907 stats.go:61] state updated for ts2phc =s2
+I0324 14:25:05.001059  106907 stats.go:61] state updated for ts2phc =s2
+ts2phc[1742826305]:[ts2phc.0.config] ens4f0 nmea_status 1 offset 0 s2
+GM[1742826305]:[ts2phc.0.config] ens4f0 T-GM-STATUS s2 <3>
+----
+<1> `ts2phc` is updating the PTP hardware clock.
+<2> Estimated PTP device offset between PTP device and the reference clock is 0 nanoseconds.
+The PTP device is in sync with the leader clock.
+<3> T-GM is in a locked state (s2).

images/openshift-ptp-3-card-grandmaster.png

@@ -25,9 +25,6 @@ For example, the PRTC is traceable to a GNSS time source.
 |T-GM clock is in `HOLDOVER` mode, and within holdover specification.
 The clock source might not be traceable to a category 1 frequency source.
 
-|`gm.ClockClass 140`
-|T-GM clock is in `HOLDOVER` mode, is out of holdover specification, but it is still traceable to the category 1 frequency source.
-
 |`gm.ClockClass 248`
 |T-GM clock is in `FREERUN` mode.
 |====

images/openshift-ptp-holdover-tgm-clock-with-gnss.png

@@ -10,7 +10,7 @@ Holdover allows the grandmaster (T-GM) clock to maintain synchronization perform
 
 You can define the holdover behavior by configuring the following holdover parameters in the `PTPConfig` custom resource (CR):
 
-`MaxInSpecOffset`:: Specifies the maximum allowed offset in nanoseconds. If the T-GM clock exceeds the `MaxInSpecOffset` value, it transitions to the `FREERUN` state (clock class state `gm.ClockClass 248`). 
+`MaxInSpecOffset`:: Specifies the maximum allowed offset in nanoseconds. If the T-GM clock exceeds the `MaxInSpecOffset` value, it transitions to the `FREERUN` state (clock class state `gm.ClockClass 248`).
 `LocalHoldoverTimeout`:: Specifies the maximum duration, in seconds, for which the T-GM clock remains in the holdover state before transitioning to the `FREERUN` state.
 `LocalMaxHoldoverOffSet`:: Specifies the maximum offset that the T-GM clock can reach during the holdover state in nanoseconds.
 
@@ -23,7 +23,7 @@ If the `LocalMaxHoldoverOffSet` value is less than the `MaxInSpecOffset` value,
 
 For information about clock class states, see "Grandmaster clock class sync state reference" document.
 
-The T-GM clock uses the holdover parameters `LocalMaxHoldoverOffSet` and `LocalHoldoverTimeout` to calculate the slope. Slope is the rate at which the phase offset changes over time. It is measured in nanoseconds per second, where the set value indicates how much the offset increases over a given time period. 
+The T-GM clock uses the holdover parameters `LocalMaxHoldoverOffSet` and `LocalHoldoverTimeout` to calculate the slope. Slope is the rate at which the phase offset changes over time. It is measured in nanoseconds per second, where the set value indicates how much the offset increases over a given time period.
 
 The T-GM clock uses the slope value to predict and compensate for time drift, so reducing timing disruptions during holdover. The T-GM clock uses the following formula to calculate the slope:
 
@@ -43,9 +43,9 @@ The phase offset is converted from picoseconds to nanoseconds. As a result, the
 The following figure illustrates the holdover behavior in a T-GM clock with GNSS as the source:
 
 .Holdover in a T-GM clock with GNSS as the source
-image::holdover_in_t_gm.png[Holdover in a T-GM clock with GNSS as the source]
+image::openshift-ptp-holdover-tgm-clock-with-gnss.png[Holdover in a T-GM clock with GNSS as the source]
 
-image:darkcircle-1.png[20,20] The GNSS signal is lost, causing the T-GM clock to enter the `HOLDOVER` mode. The T-GM clock maintains time accuracy using its internal clock.
+image:darkcircle-1.png[20,20] The GNSS signal is lost, causing the T-GM clock to enter the `HOLDOVER` mode. The T-GM clock maintains time accuracy by using its internal clock.
 
 image:darkcircle-2.png[20,20] The GNSS signal is restored and the T-GM clock re-enters the `LOCKED` mode. When the GNSS signal is restored, the T-GM clock re-enters the `LOCKED` mode only after all dependent components in the synchronization chain, such as `ts2phc` offset, digital phase-locked loop (DPLL) phase offset, and GNSS offset, reach a stable `LOCKED` mode.
 
@@ -55,4 +55,4 @@ image:darkcircle-4.png[20,20] The time error exceeds the `MaxInSpecOffset` thres
 
 image:darkcircle-5.png[20,20] The GNSS signal is restored, and the T-GM clock resumes synchronization. The time error starts to decrease.
 
-image:darkcircle-6.png[20,20] The time error decreases and falls back within the `MaxInSpecOffset` threshold.
+image:darkcircle-6.png[20,20] The time error decreases and falls back within the `MaxInSpecOffset` threshold.

images/openshift-ptp-using-dual-nic-ptp.png

@@ -0,0 +1,41 @@
+// Module included in the following assemblies:
+//
+// * networking/ptp/configuring-ptp.adoc
+
+:_mod-docs-content-type: REFERENCE
+[id="nw-ptp-three-nic-hardware-config-reference_{context}"]
+= 3-card E810 NIC configuration reference
+
+Use this information to understand how to configure 3 E810 NICs as PTP grandmaster clock (T-GM).
+
+Before you configure the 3-card cluster host, you must connect the 3 NICs by using the 1PPS faceplate connections.
+The primary NIC `1PPS_out` outputs feed the other 2 NICs.
+
+When you configure a 3-card T-GM, you need to compensate for the 1PPS signal delay that occurs when you connect the NICs by using the SMA1 connection ports.
+Various factors such as cable length, ambient temperature, and component and manufacturing tolerances can affect the signal delay.
+To compensate for the delay, you must calculate the specific value that you use to offset the signal delay.
+
+.3-card E810 T-GM PtpConfig CR reference
+[cols="1,2" width="90%", options="header"]
+|====
+|PtpConfig field
+|Description
+
+|`spec.profile.plugins.e810.pins`
+a|Configure the E810 hardware pins with the PTP Operator E810 hardware plugin.
+
+* `$iface_timeTx1.SMA1` enables the `1PPS OUT` connection for `SMA1` on NIC 1.
+* `$iface_timeTx1.SMA2` enables the `1PPS OUT` connection for `SMA2` on NIC 1.
+* `$iface_timeTx2.SMA1` and `$iface_timeTx3.SMA1` enables the `1PPS IN` connection for `SMA1` on NIC 2 and NIC 3.
+* `$iface_timeTx2.SMA2` and `$iface_timeTx3.SMA2` disables the `SMA2` connection on NIC 2 and NIC 3.
+
+|`spec.profile.ts2phcConf`
+|Use the `ts2phcConf` field to configure parameters for the NICs.
+Set `ts2phc.master 0` for NIC 2 and NIC 3.
+This configures the timing source for NIC 2 and NIC 3 from the 1PPS input, not GNSS.
+Configure the `ts2phc.extts_correction` value for NIC 2 and NIC 3 to compensate for the delay that is incurred for the specific SMA cable and cable length that you use.
+The value that you configure depends on your specific measurements and SMA1 cable length.
+
+|`spec.profile.ptp4lConf`
+|Set the value of `boundary_clock_jbod` to 1 to enable support for multiple NICs.
+|====

modules/nw-ptp-configuring-linuxptp-services-as-grandmaster-clock-dual-nic.adoc

@@ -4,40 +4,49 @@
 
 :_mod-docs-content-type: CONCEPT
 [id="ptp-dual-nics_{context}"]
-= Using dual-NIC Intel E810 hardware with PTP
+= 2-card E810 NIC configuration reference
 
-{product-title} supports single and dual-NIC Intel E810 hardware for precision PTP timing in grandmaster clocks (T-GM) and boundary clocks (T-BC).
+{product-title} supports single and dual-NIC Intel E810 hardware for PTP timing in grandmaster clocks (T-GM) and boundary clocks (T-BC).
 
 Dual NIC grandmaster clock::
++
+--
 You can use a cluster host that has dual-NIC hardware as PTP grandmaster clock.
 One NIC receives timing information from the global navigation satellite system (GNSS).
 The second NIC receives the timing information from the first using the SMA1 Tx/Rx connections on the E810 NIC faceplate.
 The system clock on the cluster host is synchronized from the NIC that is connected to the GNSS satellite.
-+
+
 Dual NIC grandmaster clocks are a feature of distributed RAN (D-RAN) configurations where the Remote Radio Unit (RRU) and Baseband Unit (BBU) are located at the same radio cell site.
 D-RAN distributes radio functions across multiple sites, with backhaul connections linking them to the core network.
-+
+
 .Dual NIC grandmaster clock
-image::561_OpenShift_Using_PTP_network_0124.png[Dual NIC PTP grandmaster clock connected to GNSS timing source and downstream PTP boundary and ordinary clocks]
-+
+image::openshift-ptp-using-dual-nic-ptp.png[Dual NIC PTP grandmaster clock connected to GNSS timing source and downstream PTP boundary and ordinary clocks]
+
 [NOTE]
 ====
-In a dual-NIC T-GM configuration, a single `ts2phc` process reports as two `ts2phc` instances in the system.
+In a dual-NIC T-GM configuration, a single `ts2phc` program operate on two PTP hardware clocks (PHCs), one for each NIC.
 ====
+--
 
 Dual NIC boundary clock::
-For 5G telco networks that deliver mid-band spectrum coverage, each virtual distributed unit (vDU) requires connections to 6 radio units (RUs). To make these connections, each vDU host requires 2 NICs configured as boundary clocks.
 +
+--
+For 5G telco networks that deliver mid-band spectrum coverage, each virtual distributed unit (vDU) requires connections to 6 radio units (RUs). To make these connections, each vDU host requires 2 NICs configured as boundary clocks.
+
 Dual NIC hardware allows you to connect each NIC to the same upstream leader clock with separate `ptp4l` instances for each NIC feeding the downstream clocks.
+--
 
 Highly available system clock with dual-NIC boundary clocks::
-You can configure Intel E810-XXVDA4 Salem channel dual-NIC hardware as dual PTP boundary clocks that provide timing for a highly available system clock.
-This is useful when you have multiple time sources on different NICs.
-High availability ensures that the node does not lose timing synchronisation if one of the two timing sources is lost or disconnected.
 +
+--
+You can configure Intel E810-XXVDA4 Salem channel dual-NIC hardware as dual PTP boundary clocks that provide timing for a highly available system clock.
+This configuration is useful when you have multiple time sources on different NICs.
+High availability ensures that the node does not lose timing synchronization if one of the two timing sources is lost or disconnected.
+
 Each NIC is connected to the same upstream leader clock.
 Highly available boundary clocks use multiple PTP domains to synchronize with the target system clock.
 When a T-BC is highly available, the host system clock can maintain the correct offset even if one or more `ptp4l` instances syncing the NIC PHC clock fails.
 If any single SFP port or cable failure occurs, the boundary clock stays in sync with the leader clock.
-+
+
 Boundary clock leader source selection is done using the A-BMCA algorithm. For more information, see link:https://www.itu.int/rec/T-REC-G.8275.1/en[ITU-T recommendation G.8275.1].
+--

modules/nw-ptp-configuring-linuxptp-services-as-grandmaster-clock-three-nic.adoc

@@ -0,0 +1,29 @@
+// Module included in the following assemblies:
+//
+// * networking/ptp/about-ptp.adoc
+
+:_mod-docs-content-type: CONCEPT
+[id="ptp-three-card-grandmaster_{context}"]
+= 3-card Intel E810 PTP grandmaster clock
+
+{product-title} supports cluster hosts with 3 Intel E810 NICs as PTP grandmaster clocks (T-GM).
+
+3-card grandmaster clock::
++
+--
+You can use a cluster host that has 3 NICs as PTP grandmaster clock.
+One NIC receives timing information from the global navigation satellite system (GNSS).
+The second and third NICs receive the timing information from the first by using the SMA1 Tx/Rx connections on the E810 NIC faceplate.
+The system clock on the cluster host is synchronized from the NIC that is connected to the GNSS satellite.
+
+3-card NIC grandmaster clocks can be used for distributed RAN (D-RAN) configurations where the Radio Unit (RU) is connected directly to the distributed unit (DU) without a front haul switch, for example, if the RU and DU are located in the same radio cell site.
+D-RAN distributes radio functions across multiple sites, with backhaul connections linking them to the core network.
+
+.3-card Intel E810 PTP grandmaster clock
+image::openshift-ptp-3-card-grandmaster.png[3-card PTP grandmaster clock connected to GNSS timing source and downstream PTP boundary and ordinary clocks]
+
+[NOTE]
+====
+In a 3-card T-GM configuration, a single `ts2phc` process reports as 3 `ts2phc` instances in the system.
+====
+--

modules/nw-ptp-grandmaster-clock-class-reference.adoc

@@ -30,10 +30,12 @@ include::modules/nw-ptp-introduction.adoc[leveloffset=+1]
 Before enabling PTP, ensure that NTP is disabled for the required nodes. You can disable the chrony time service (`chronyd`) using a `MachineConfig` custom resource. For more information, see xref:../../machine_configuration/machine-configs-configure.adoc#cnf-disable-chronyd_machine-configs-configure[Disabling chrony time service].
 ====
 
-include::modules/ptp-dual-nics.adoc[leveloffset=+1]
-
 include::modules/ptp-linuxptp-introduction.adoc[leveloffset=+1]
 
 include::modules/ptp-overview-of-gnss-grandmaster-clock.adoc[leveloffset=+1]
 
 include::modules/cnf-about-ptp-and-clock-synchronization.adoc[leveloffset=+1]
+
+include::modules/ptp-dual-nics.adoc[leveloffset=+1]
+
+include::modules/ptp-three-card-grandmaster.adoc[leveloffset=+1]

modules/nw-ptp-holdover-in-a-grandmaster-clock.adoc

@@ -25,21 +25,25 @@ include::modules/nw-ptp-device-discovery.adoc[leveloffset=+1]
 
 include::modules/nw-ptp-configuring-linuxptp-services-as-grandmaster-clock.adoc[leveloffset=+1]
 
-include::modules/nw-ptp-configuring-linuxptp-services-as-grandmaster-clock-dual-nic.adoc[leveloffset=+1]
+include::modules/nw-ptp-configuring-linuxptp-services-as-grandmaster-clock-dual-nic.adoc[leveloffset=+2]
+
+include::modules/nw-ptp-configuring-linuxptp-services-as-grandmaster-clock-three-nic.adoc[leveloffset=+2]
 
 [role="_additional-resources"]
 .Additional resources
 
 * xref:../../networking/ptp/ptp-cloud-events-consumer-dev-reference.adoc#cnf-configuring-the-ptp-fast-event-publisher-v2_ptp-consumer[Configuring the PTP fast event notifications publisher]
 
-include::modules/nw-ptp-grandmaster-clock-configuration-reference.adoc[leveloffset=+2]
+include::modules/nw-ptp-grandmaster-clock-configuration-reference.adoc[leveloffset=+1]
 
 include::modules/nw-ptp-grandmaster-clock-class-reference.adoc[leveloffset=+2]
 
 include::modules/nw-ptp-e810-hardware-configuration-reference.adoc[leveloffset=+2]
 
 include::modules/nw-ptp-dual-wpc-hardware-config-reference.adoc[leveloffset=+2]
 
+include::modules/nw-ptp-three-nic-hardware-config-reference.adoc[leveloffset=+2]
+
 include::modules/nw-ptp-holdover-in-a-grandmaster-clock.adoc[leveloffset=+1]
 
 [role="_additional-resources"]