Central Bank Case Study
Learn how the solution powered by Axe and our partner Intelligent Delivery Services delivered a 94% reduction in the monthly cost of regression testing for a Central Bank in the United Kingdom.
If you require more information on Intelligent Delivery Services visit their website.
The Challenge: Developing a leaner, more cost-effective testing process
The bank is a central player in the UK banking industry and are undergoing internal changes and increased control around quality assurance practices. This, coupled with a strong desire to reduce costs associated with testing, means they are keen to deliver an efficient execution capable of satisfying its internal customers.
The Test & Quality & Release Management and Domain Lead Test Automation Manager at the bank, recognised the importance of completing regression testing of the application quickly. However, the traditional manual approach they adopted was becoming a costly bottleneck.
The requirement from the client was to build an automation process that would scrape data from an Excel exported XBRL file into a 2D table. Then it would use a 2D table setup and pivot tables in Excel, which connects to Cube (Microsoft SQL Analysis Server), to compare the two tables and report any failures.
On top of the strain that manual testing creating was the growing complexity of their testing coverage. The team had an overhead of 15 man days to complete for each monthly deployment of the application. Coupling this timescale with heavily involved data related validation of results exposed the manual testing approach to human error which could be catastrophic for the organisation.
Faced with the increasing cost and complexity of their legacy testing approach, the bank has requested the support of our partner organisation to help deliver a more cost effective process through automated testing and continuous delivery.
The Solution: Automated Regression Testing
Counting the cost of manual testing
Working closely with Test management and business unit, the first task was to establish the cost of their UK based regression testing:
Day rate cost of a single test analyst |
£400 |
Total days to manually execute regression |
15 |
Cost of regression testing |
£6,000 |
Frequency of regression testing (monthly) |
1 |
Total spend (monthly) |
£6,000 |
Ideally, one should execute regression tests each time you build and deploy an application. For a typical software application with active development procedures this should be held on a monthly basis.
In the case of the bank, the equivalent costs of their traditional manual testing become even higher:
Cost of regression testing |
£6,000 |
12 iterations of manual regression testing |
£72,000 |
As this testing had previously been completed manually, it would take up to 3 weeks to complete. With this solution, it has been calculated that the entire test process could be completed in just 1 day and without the need to transfer knowledge to new onboarded staff.
If the client had chosen to execute the same level of automation using manual techniques the cost would be £72,000. In a like for like comparison they are making an immediate yearly saving of £67,200 using the solution.
With a clear understanding of the costs and return on investment opportunities available, we began the process of establishing an automated solution for the execution and re-execution of tests that will contribute to the overall application lifecycle management (ALM) process.
There had been limited success with automation previously, but with our background in automated testing it was felt that using specialist help to define and implement a standardised approach would prove decisive.
The Outcome: Eliminating the testing bottleneck, cutting costs and increasing quality
- The solution was to create a function format using C# which is built together in sequence using Axe.
- The report is shown in Excel and a summary of it is available on the Axe report.
- This was a fully bespoke implementation, and outside of the normal use of Axe, we also provided a IE dialog which showed the actual progress of the run in a progress bar format.
- The bank now has the benefit of a fully functioning, repeatable regression test pack working as part of a continuous delivery model.
- Delivery of the solution uses Odin’s Axe Automation tool using the embedded UFT coding functionality across the internal banking system.
- Enabling the bank’s development teams to build-deploy-test as often as required, resulting in a much more robust build, better deploy practices, and increased confidence in the application being deployed.
Having achieved ALM, the bank have increased the volume of testing as well as improvements in:
- Build & deployment reliability
- Overall quality
- Better reporting metrics
- Automated reports
Through the specialist implementation of a test automation solution powered by Axe, the bank now benefits from a considerable saving in staff utilisation, freeing up skilled resources to focus on manual and exploratory testing or other areas of concern.
The quality of testing is also greatly improved with the bank having greater confidence in the application under test conditions as the possibility of human oversight is eliminated.
