Why Galasa is a ‘big deal’ for mainframe testing

The news that IBM’s Galasa software testing tool has been welcomed by the open mainframe project is extremely positive for the mainframe software testing community. As Macro 4 is an early adopter of Galasa, I’m able to shed light on the experience of using it, the advantages it brings to us and its significance for software testing within the IBM mainframe customer base.

Testing is an important part of today’s DevOps software development philosophy. One of the core principles of which is that testing should be automated and performed as early as possible in the development process – and repeated throughout the cycle. The aim is to identify and fix defects faster, reduce rework, and ensure the quality and functionality of the software before it reaches production.

The Galasa story

This is where Galasa comes in. While it started life as an internal IBM tool for software testing within IBM z/OS teams, Galasa has since been released externally for use by customers. Importantly, it is designed to replace the (still) very widely used manual and semi-automated software testing processes which soak up time, increase complexity and are liable to human error. Galasa is an open source framework enabling end-to-end testing of applications running across multiple platforms, including hybrid cloud environments – something that is increasingly important for mainframe customers today.

Macro 4’s testing regime

At Macro 4 we had written our own bespoke software tool many years ago to automate the continuous testing we do on all our commercially available mainframe software products. 

Called Retest, this internal software runs directly on the mainframe, taking input data stored in a file at one end and capturing the resulting output screen results at the other. Each time we replay a test, Retest compares the output produced under test against our expected ‘gold’ results to ensure all functionalities are working as they should. If any discrepancies are flagged, we know recent changes or updates made by our developers are having an unintended impact.

Why consider changing?

Retest has continued to serve us extremely well for over 30 years. However, there are two main reasons why we are interested in evolving our testing regime with an open source product like Galasa.

Firstly, Retest is a bespoke software tool built using the Assembler programming language. As older mainframe technicians retire, Assembler skills are becoming scarce, making it harder to support the product or fix any issues that might arise.

Secondly, Retest runs directly on the mainframe which also requires specialist mainframe skills to run the testing process. By contrast, Galasa runs remotely via a terminal emulator which is more in keeping with how the new generation of developers prefer to work now. Many developers today do much of their work off the mainframe developing their code on the likes of VS Code accessed from GIT or Subversion.

Working off the mainframe means developers have access to a growing ecosystem of open source tools and development environments which have sprung up to make mainframe development more intuitive and less reliant on specialist z/OS expertise.   

For these reasons, we made the decision to run an initial project to see if Galasa would be a good fit for our testing processes. This project involved trialling Galasa in the continuous testing we do for a single Macro 4 product. The product we chose was InSync, a file and data management tool we supply for native z/OS, DB2, IMS, IBM MQ.

Early reservations

It’s fair to say we had some reservations at the outset. For example, we understood that running ‘out of the box’, Galasa is primarily suited to unit testing i.e. running discrete tests on individual pieces of code after each code change or update – with each test lasting only a few minutes at most. So how would Galasa cope with the full-scale regression testing that is part of the testing regime for InSync? 

Part of the concern related to the speed and efficiency of executing the testing we needed to do. Using Retest, our regression testing consists of 60,000 tests running unattended overnight on a z15 mainframe. Retest is written in Assembler which is classified as a ‘lower level’ language making it super-efficient – but even then, it takes 16 hours to complete the full testing process. 

Galasa drives tests remotely via a terminal emulator, so there is an additional path which can introduce delays. So we were concerned that it might not be able to reliably complete our InSync testing in good time.

There was also a question mark related to whether Galasa would allow us to use the existing input data and output results that are part of our testing regime. Our impression was that Galasa out-of-the-box would require input data to be individually hard coded for each test, which would not work for us. 

Our original input and output screen recordings were meticulously generated by human programmers many years ago. It would take thousands of man-hours to recreate them from scratch.

Galasa passes test with flying colours

The good news is that despite our initial concerns, we were able to use Galasa to do everything we needed. 

While we had to make some refinements to the way we were using it, we’re now confident of using Galasa to complete our full overnight regression test for InSync in a time frame that fits our requirements. We introduced some efficiencies in the way the tests were being executed to help achieve the target time. 

For example, our original recordings were by humans, who had examined the data in real-time before proceeding to the next part of the test. Those tests ran within our acceptable time frame when running natively under z/OS so timing was never a worry. With automated testing under Galasa, those human timings were introducing delays, so we were able to create efficiencies by speeding up those sequences.

We were also able to successfully execute the test with Galasa using the same input data file and output screen results that we had been using for many years with Retest. This minimized the effort on our part. Instead of hard coding the input data, we abstracted the process using an input data file written in the Rexx scripting language. Rexx generates a script to manage what needs doing at each stage of the testing process.

IBM’s Galasa team was very interested in what we were trying to do in our initial trial. We met with them and explained what we were aiming to accomplish. Importantly they were very supportive and responsive to any questions we had while running the project. 

The future

With Galasa now part of the open mainframe project, I am sure there will be increasing interest from other mainframe shops who will be keen to explore how they can integrate it into their testing regime. At Macro 4, we plan to extend our own use of Galasa to the testing regime for other products where we are currently using Retest for regression testing. We are also hoping to replace the manual IVP (Installation Verification Procedure) that we provide with our products with an automated Galasa process. We are delighted to be part of the steering committee for Galasa, to support its continued growth.