Mainframe Penetration Testing: Why It Matters More Than Ever

The goal is not disruption but discovery.

For decades, mainframes have been at the heart of critical industries such as finance, healthcare, government, and retail. Since IBM introduced the System/360 in 1964, these systems have earned a reputation for being reliable, secure, and almost “unhackable”. 

But reputations can be dangerous. In an era of persistent cyber threats, ransomware, and insider risks, assuming that the mainframe is inherently safe is no longer acceptable. We know that Swedish bank Nordea was compromised in 2012, while others rarely make headlines.

How do companies protect themselves from modern mainframe vulnerabilities?

What is penetration testing

The answer is Penetration Testing (pen testing), which simulates the actions of a malicious attacker to uncover vulnerabilities before they are exploited.

It has become a vital tool in the mainframe security arsenal. Yet many organizations still struggle with the concept. “Why test a system that has passed every audit for the last five years?” is a common refrain.

The uncomfortable reality is that even in a supposedly clean environment, penetration testers (pen testers) can and often do elevate their privileges, gain inappropriate access, and exfiltrate sensitive data. 

How penetration testing works 

According to the standard definition, a penetration test is a security exercise, conducted with permission, using the same tools and techniques as real attackers. 

The goal is not disruption but discovery: to identify cracks in a system before someone else does. On the mainframe, this typically unfolds in two stages:

1. Footprinting (data gathering)

Step one involves collecting as much information as possible about the target system, infrastructure, and networks. In practice, this might involve nothing more than READ or BROWSE access to z/OS logs, IPL information, system configuration files, or dataset catalogs. Tools such as SDSF, SYSVIEW, eJes, and TASID are invaluable here.

2. Exploitation (penetration)

In step two, pen testers use a low-level, non-privileged ID to explore whether the information gathered can be leveraged to escalate privileges, access restricted datasets, or alter the running system.

This isn’t a script-driven process. Each test adapts to what the environment reveals: sometimes weak APF dataset protection, sometimes poorly defined SURROGAT profiles, and sometimes overly permissive system command protection.

In every engagement my team at Vertali has conducted so far, testers have succeeded in demonstrating privilege escalation or data exfiltration.

Why mainframe weaknesses persist 

A unique challenge in the mainframe world is the long lifecycle of configurations and security databases. It’s not unusual for RACF, ACF2 or TSS databases to have been migrated from version to version for decades.

Customization choices made in 2000 – or even earlier – can persist unnoticed into today’s z/OS environments. Parameters copied forward without review become latent vulnerabilities, ready to be exploited.

Testing a real-world scenario 

Recently, a client asked me a chilling but practical question: What would happen if a ransomware attack hit the mainframe?

Using a disaster recovery (DR) environment, testers built custom REXX and assembler tools, and created dummy datasets to test the code and explore the question.

The testers then launched the ransomware attack on the DR system. The results were sobering: within 90 minutes, six batch jobs encrypted the system datasets of the running system. The takeaway? Ransomware on the mainframe isn’t science fiction; it’s a here-and-now risk.

Mainframe penetration testing tools and techniques 

Penetration testing on the mainframe requires both creativity and a toolkit. Ample resources can be found by performing a simple Google search for “Mainframe Hacking” or “Mainframe Hacking Tools”. 

Most pen testers have their own toolkit, created and configured for the way they tend to work. The point isn’t the tools themselves, but how easily standard, legitimate functions can be chained together by someone with standard access to expose privilege gaps and misconfigurations.

How to strengthen mainframe security 

Penetration testing shouldn’t be seen as an optional extra. It’s a critical exercise in risk management.

• Uncover misconfigured or outdated security controls.
• Identify residual vulnerabilities inherited from older z/OS releases, which are carried forward when systems are upgraded.
• Validate the effectiveness of current RACF/ACF2/TSS and z/OS controls.
• Build resilience against modern threats such as ransomware.

The next generation of mainframe security talent 

A final takeaway: penetration testing isn’t just about tools but also skills. Coding skills are essential. 

The breadth of available languages makes it easier for young and old to participate. I prefer Assembler and REXX, but many others prefer C++ and Python. 

Choose your language of choice and get going. Mainframe security professionals must blend decades of platform knowledge with modern offensive security techniques. For those entering the field, this is both a challenge and an opportunity.

Mainframes remain indispensable, but they are not invulnerable. Penetration testing exposes environments that may otherwise appear secure through an honest, sometimes uncomfortable lens. 

By investing in regular pen testing exercises, organizations can protect their data and modernize their approach to security while respecting the platform’s history and future.

▶️Catch more about mainframe security in this interview with Mark Wilson.