Streaming z/OS IT Operational Data with IBM Z Common Data Provider

In today’s business world, the pressure to deliver faster to be “in the business”, the importance to reduce outages and the time to solve any issues to have business applications available 24×7, increase exponentially. Mobile and internet penetration in the daily activities require business applications to be always available, scalable and timely updated to react to business or users’ requests. All of this agility and reliability is achievable having in the backend a team that works together on problem determination beyond the specific subsystem point of view. The team needs to have a complete end-to-end view of the applications components, and access to the log data from different subsystems of the final applications. The fast and comprehensive access to the IT Operational Data is crucial to achieve the goal given from current business expectations.

IT Operational data is data that is generated by the z/OS system as it runs. This data describes the health of the system and the actions that are taking place on the system. The analysis of operational data by analytics platforms and cognitive agents can produce insights and recommended actions for making the system work more efficiently and for resolving or preventing problems.

IBM Z Common Data Provider (ZCDP) is a product that is able to collect all the z/OS IT Operational Data, including System Management Facilities (SMF) data and z/OS log data, such as SYSLOG (IBM z/OS System Log and USS Syslogd), JOBLOGs and application logs (IBM CICS Transaction Server logs and IBM WebSphere Application Server logs),as a single source provider for both structured and unstructured data in near real-time and on demand. All these IT operational data are filtered and streamed by ZCDP to the configured analytics platforms like Elastic Stack, Splunk and IBM Z Operations Analytics, in a consumable format for health monitoring and problem determination.

As SMF data is an important type of z/OS operational data. The following section will focus on the ways to configure ZCDP to stream SMF data.

Configuring ZCDP to stream SMF records

There are three methods to configure in ZCDP to stream SMF records in near real-time to analytics platforms. Each one has its advantages and considerations. See the following for more information.

1. SMF in log stream mode with SMF in-memory buffer

For customers who run SMF in log stream mode, it is best for ZCDP to collect SMF from the SMF in-memory buffer using the SMF real-time interface. This SMF in-memory buffer is not automatically configured with the log stream configuration in SMF and needs additional configuration. You can see the SMF documentation for details. ZCDP must be configured to pull data from the SMF in-memory buffer using the SMF real-time interface.

If SMF is configured for log stream mode but the SMF in-memory buffer is not configured, configuring ZCDP to pull SMF records directly from log stream has a side consequence that when ZCDP pulls data from the log stream (at a configurable interval, which is 1 minute by default), it causes a log stream switch. This switch results in a file being generated with the SMF records since the last switch from which ZCDP can read the latest SMF records.The undesirable side effect is that you end up with a lot of smaller files (potentially 1per minute if you leave the default interval of 1 minute), this can be an annoyance to clean up or potentially cause a concern if you drive automation on the log stream switch. It is for this reason that it is strongly suggested that when SMF is running in log stream mode, you also have the SMF in-memory buffer configured and configure ZCDP to pull from the SMF in-memory buffer.

2. SMF in data set recording mode (using SYS1.MANx, not log stream mode)

For customers who run SMF in data set recording mode (SYS1.MANx) (and aren’t moving SMF configuration to log stream mode), they can use the capability to collect and stream SMF data through a set of SMF user exits. Note that the SMF records are not pulled from the SYS1.MANx data set, the records are processed and streamed in near-real time through the new SMF user exits provided by ZCDP.

When ZCDP is configured for collecting SMF records in data set recording mode, the HBOSMFEX user exit is configured at installation exit points IEFU83, IEFU84 and IEFU85 to process SMF records for ZCDP.As SMF records are processed in SMF and the appropriate ZCDP user exit is called, the ZCDP user exit will copy the record to an above-the-bar 64-bit shared storage buffer. The System Data Engine component of ZCDP will periodically read the SMF records from the shared storage buffer and process them, performing the configured transforms and passing them off to the Data Streamer component of ZCDP to send them to the configured subscribers (e.g. Splunk, Elastic Stack, IZOA, etc.).You can configure the interval at which the System Data Engine reads records from the shared storage buffer, and the default value is 1 minute.

It is quite simple to install and configure ZCDP to collect SMF records via the user exits. Add the required module to the system link pack area (LPA), install the user exits by updating SYS1.PARMLIB and update SMFRPMxx to call the user exits. There are no other dependencies.

3. Batch

As an alternative to ZCDP streaming SMF data in near-real time as the SMF records are submitted, the System Data Engine component of ZCDP can be run stand-alone in batch mode to read SMF data from a data set and writing it to a file. You can create the System Data Engine batch job for writing SMF data to data sets and create the System Data Engine batch job for sending SMF data to the Data Streamer by using the sample job HBOJBCOL in the hlq.SHBOSAMP library and the sample job HBOJBCO2 in the hlq.SHBOCNTL library respectively, and updating the copies.

Summary

For all or specific operational data, collecting and streaming them for further analysis are vital for the healthy and sustainable operation of the z/OS system. IBM Z Common Data Provider enables your business to stream these data in a consumable format to multiple analytics platforms to monitor the system and quickly find where problems lie. For more information about ZCDP, see https://www.ibm.com/us-en/marketplace/common-data-provider-for-z-systems

[starbox id=68]

[starbox id=69]