Timing, timekeeping, time synchronization, more specifically accurate synchronization, is a key requirement for modern information technology systems. This is especially true for industries involved in transaction processing such as the financial industry. As such, the IBM Z sysplex needs highly accurate timing/timekeeping and synchronization technology to ensure data integrity, and to also provide the ability to reconstruct a database based upon logs. There have been 3 phases in the evolution of time synchronization on IBM Z.
Phase 1: The 9037 External Time Reference (ETR) aka the Sysplex Timer was introduced with Parallel Sysplex. An ETR network was comprised on one or two 9037 devices, which synchronized time among themselves. The ETRs provided time services for the zSeries CECs. Support for the 9037 ended with the z10 mainframes.
Phase 2 began with Server Time Protocol (STP). STP is a message based protocol, similar to the industry standard Network Time Protocol (NTP). STP allowed a collection of IBM mainframes to maintain time synchronization with each other using a time value known as Coordinated Server Time (CST). The network of servers is known as a Coordinated Timing Network (CTN). STP improved time synchronization compared to the ETR, and it also scaled much better over longer distances of up to 200km. STP could co-exist with an ETR network (known as a mixed CTN), or it could be an STP only CTN. The mainframe’s Hardware Management Console (HMC) plays a critical role with STP CTNs: The HMC can initialize CST manually or initialize CST to an external time source. The HMC also sets the time zone, daylight savings time, and leap seconds offsets. It also performs the time adjustments when needed.
STP currently has two external time source options. One is an external NTP server, which provides up to 100 milliseconds (ms) accuracy. The second external time source option, which is what I like to term phase 3, is the use of the NTP server with pulse per second (PPS). PPS provides 10 microseconds (μs) of accuracy, and requires a connection directly to the oscillator.
Recent changes in U.S. and European regulations have brought increasing attention to time synchronization accuracy. The US based Financial Industry Regulatory Authority (FINRA) is a not for profit entity that is responsible for overseeing US brokerage firms and works closely with the SEC. In August 2016, the Securities and Exchange Commission (SEC) approved a rule change to require synchronization to within 50 ms of Universal Coordinated Time (UTC), as well as requiring audit log capability to prove compliance. At the same time, the European Securities and Markets Authority (ESMA) came out with revised directive known as the Markets in Financial Instruments Directive II (MiFID II). MiFID II applies to any organization dealing in European stocks, commodities, and bonds. MiFID II requires time synchronization to within 100us of UTC.
These regulations have spurred a renewed interest in another time synchronization protocol, initially introduced in 2002. IEEE 1588 Precision Time Protocol (PTP) enables heterogeneous systems that include clocks of various inherent precision, resolution, and stability to synchronize to a grandmaster clock. PTP is a message-based time transfer protocol that enables synchronization accuracy and precision in the sub microsecond range for packet-based networked systems. The standard was first released in 2002 as Version 1 and then revised in 2008 as Version 2. The IEEE Standards Association is currently leading an initiative to update the standard again (Version 3). The protocol supports synchronization in the sub-microsecond range with minimal network bandwidth and local clock computing resources. PTP has played an important role in power transmission, telecommunications, and laboratories. PTP is starting to find its way into computer networking and is seeing increasing support in hardware such as switches. Will PTP become increasingly adopted in the financial sector as a way to better comply with the new regulations?
Only time will tell (pun intended).
Steve Guendert, aka “Dr. Steve”, is in an IBM Z senior engineering role based in Poughkeepsie where he is on the IBM Z Hardware architecture team. He works on all aspects of IBM Z I/O and timing architecture, including the Server Time Protocol (STP), FICON and zHPF architectures. He is a member of the Mainframe Hall of Fame and the IBM Academy of Technology.