Have your software failures resulted in lost time, fraud, or injury? This is far too common a problem. The good news is that Software Test Engineering helps improve your ability to find and fix these issues before they’re experienced by your customers.
So, what are the first three steps to conquer this problem?
Develop a Software Test Strategy
Test Plan a Project with Configurations, F/F Tree, & Data
Measure the key attributes of your test organization and use the metrics to drive change. Adjust your Software Test Strategy and Test Plans based on metrics and feedback.
This may seem like a simple list, but it is the foundation for change. Contact firstname.lastname@example.org to get the full STE white paper.
Review of The broken promise of test automation? by Wayne Arioli appeared in 16th June, 2017 issue of SD Times.
Reports 20-30% of tests being automated. I believe that percentage but don’t believe tools are the issue – several tools out there that are sufficient for data-driven software test automation. Agree you must plan for maintaining tests, but not that this maintenance isn’t worth the cost. Automated scripts are like other software, they must be designed and organized in a manner that minimize impact from planned or unplanned product changes. Recovery scenarios and logging are necessary in automated test design.
“Testing these technologies requires either a high degree of technical expertise/specialization or a high level of business abstraction…” – Amen – test automation is a specialty skill set and the ability to quickly understand business process and requirements – all keys to successful automation.
Bottom Line Analysis – software test automation isn’t for every tester but good automation is worth the effort. Start with these software test automation steps:
- Define your test strategy including scaling up your automation >50%
- Have a dedicated test team responsible for improving FPY and % automation
- Organize and scale your automation including data driven features and scenarios
- Continuously test and continuously improve your software test automation!
Finding the most critical defects early is an important objective of all test plans, along with finding all critical and major defects before release. To achieve early critical defect prevention, prioritize test procedures into high, medium, and low by how critical defects found in those procedures are likely to be. In the first X hours (we had resources to achieve the objective in the first 2 hours), execute the most critical procedures (auto and manual) and cover a majority of test configurations. We reported out defects and procedures executed in the first 2 hours (with some show stoppers), and proceeded on to other tests while those initial issues were being sorted out (with test assistance as needed). After the initial report, test updates are given daily. Learn from each test cycle and improve your next test plan!
Recently working on a new Coded UI software test automation project being deployed on an MSTest “server” without Visual Studio debugging capability, I have revisited First Pass Yield (FPY) metrics for these automated tests. FPY in this context is not the yield of software under test but the percent of times your automated tests execute and return correct Pass/Fail results. In my example, getting the correct Pass/Fail results was not an issue, but getting all data rows to execute for each test procedure and running all tests was the problem. My initial expectations and goals were to be >90% FPY. In the first month, I was <75% FPY. Breaking down the issues, they were grouped into file handling exceptions, timing/control differences between development machine and deployment server, data errors / differences on server. Know your FPY, pareto your failure modes, and fix them before moving on to more automation – then continue to monitor!
1. Define and adapt your test strategy and goals.
2. Plan data driven tests using customer configurations. Give teams time to establish baseline automation & manual tests.
3. Measure your test effectiveness and efficiency, and adjust your strategy and plans.
Find out more – email@example.com