IBM z16 “Time Direct to the CEC”

The IBM z16 became generally available (GA) in May 2022. It introduced yet another significant change to IBM Z time synchronization via the new “time direct to the CEC” functionality. This article will briefly review the z15 time synchronization change that served as the first step in the transformative process and then covers in more detail the “time direct to the CEC” feature.

Since the introduction of IBM Server Time Protocol (STP) for IBM Z in 2005/2006, the Network Time Protocol (NTP) was supported as an external time reference source for STP. When the accuracy capability of NTP alone was not good enough to comply with applicable industry regulations, IBM Z clients could supplement NTP with pulse per second (PPS) capability via coaxial cable connection from the same time server providing the NTP connection. Government regulations mandating minimum levels of accuracy to Coordinated Universal Time (UTC) for end users doing business in trading financial instruments (stocks, bonds, etc.) became significantly more stringent in the mid-2010s. IEEE 1588 Precision Time Protocol (PTP) has been a standard since 2002. But its adoption was limited primarily to power utilities or telecommunications end users. The new, more stringent financial industry regulations on UTC accuracy made PTP an attractive option for the financial/banking industry because the new regulations could be complied with via PTP without the need for the additional coaxial PPS connection to supplement NTP. Recognizing a client need for PTP, IBM Z became active in the PTP standards bodies and communities and in May 2020, IBM Z introduced support for PTP as an additional option for the external time reference source for STP.

While introducing support for PTP was an important step to improving the accuracy of UTC, it was only the first step for IBM Z. With the z16, we introduced the second, more important step and that was bringing the connection to the external time reference (PTP and NTP) directly to the card on the z16 hosting the oscillators. Hence, the term “time direct to CEC”. For IBM z15 and prior machines, the external time reference was connected to the HMC/SE and then would be passed via an additional connection, known as the service word interface, from the SE to the CEC. The service word interface was not exclusively for the time data traffic and, therefore, subject to “jitter” which could limit overall accuracy to the external reference source, and in turn, to UTC. The advertised accuracy for z15 and earlier to the external source was 100 milliseconds or less. This meant that starting in 2018 when the new aforementioned UTC accuracy regulations went into effect, IBM Z clients subject to those regulations would need to implement PPS to ensure their compliance with the new regulations. Due to the limitations on coaxial cable length, this was not ideal for these customers due to 1) having to purchase the cables and 2) the length of the cable restricting how data center floor planning could be accomplished.

So, what is it that makes the z16 and “time direct to the CEC” different? First, the reduction in connection steps from the IBM Z to the time server providing the external reference simplifies the overall connectivity. Secondly, the network connection providing the external time reference to the IBM z16 is exclusively for the time protocol traffic. Third, the card hosting the

oscillators where the external reference connection comes into has a new network adapter that supports PTP and NTP and also provides hardware timestamping capability. What does that mean? Simply stated, with hardware timestamping, the network adapter is capable of generating a time stamp for supporting the time protocol accuracy determination. The ideal timestamp for accuracy purposes would be on the actual PHY itself, followed by somewhere on the NIC, followed by the OS, and then the application. Time stamps generated by the OS and applications are referred to as software timestamping. With the z16’s new capability, we can make use of the NIC’s timestamping feature. How much better than the 100 milliseconds tested and advertised for the z15 and earlier methodology? Our preliminary testing has shown much better results for both PTP and NTP. At the time of writing and submitting this article, the full performance testing and verification have not been completed. We will publish those results in an IBM Techdocs white paper, articles, and via other methods once we have completed the testing and verified the results.

The final thing to mention on the z16 “time direct to CEC” paradigm that is a significant enhancement is that we now run the external time services (ETS) for NTP and PTP in a firmware management partition rather than on the HMC/SE. Moving forward, this allows us to more easily support/introduce new enhancements for the PTP (and NTP) protocols as the standards development organizations support them. It also will allow us to introduce more resiliency enhancements to STP and parallel sysplex.

In summation, the new time synch enhancements introduced on IBM z16 will provide clients with improved accuracy performance to better comply with governmental regulations on UTC accuracy. These enhancements also set the stage for continued evolutionary advancements in time synchronization technology for IBM Z and the benefit of our clients.