Cost Benefits Analysis of Test Automation
Cost Benefits Analysis of Test Automation
Douglas Hoffman, BACS, MSEE, MBA, CSQE Software Quality Methods, LLC. 24646 Heather Heights Place Saratoga, California 95070-9710 Phone 408-741-4830 Fax 408-867-4550 doug.hoffman@acm.org
Keywords: Automated Testing, Automation Tools, Cost of Testing, Intangible Costs, Return on Investment, Tangible Costs
Introduction
Many managers today expect software test automation to be a silver bullet; killing the problems of test scheduling, the costs of testing, defect reporting, and more. Automating testing can have positive impacts in many areas, and there are many success stories to provide hope that test automation will save money and solve some testing problems. Unfortunately, there are many more horror stories, disappointments, and bad feelings, even in cases where automation has been beneficial. I have been brought into more than one situation where previous attempts at automating software testing have failed; where large investments have been made in shelfware, and many years of effort creating automated tests abandoned. The purpose of this paper is to provide some practical guidance for understanding and computing cost and benefits from test automation. It describes some financial, organizational, and test effectiveness impacts observed when software test automation is installed. The paper also advises about areas that are difficult or impossible to factor into the financial equations and addresses some common misconceptions management holds about test automation. There are many factors to consider when planning for software test automation. Automation changes the complexion of testing and the test organization from design through implementation and test execution. It usually has broad impacts on the organization in such things as the tasks performed, test approaches, and even product features. There are tangible and intangible elements and widely held myths about benefits and capabilities of test automation. It is
Douglas Hoffman, BACS, MSEE, MBA, CSQE Software Quality Methods, LLC. 24646 Heather Heights Place Saratoga, California 95070-9710 Phone 408-741-4830 Fax 408-867-4550 doug.hoffman@acm.org
Keywords: Automated Testing, Automation Tools, Cost of Testing, Intangible Costs, Return on Investment, Tangible Costs
Introduction
Many managers today expect software test automation to be a silver bullet; killing the problems of test scheduling, the costs of testing, defect reporting, and more. Automating testing can have positive impacts in many areas, and there are many success stories to provide hope that test automation will save money and solve some testing problems. Unfortunately, there are many more horror stories, disappointments, and bad feelings, even in cases where automation has been beneficial. I have been brought into more than one situation where previous attempts at automating software testing have failed; where large investments have been made in shelfware, and many years of effort creating automated tests abandoned. The purpose of this paper is to provide some practical guidance for understanding and computing cost and benefits from test automation. It describes some financial, organizational, and test effectiveness impacts observed when software test automation is installed. The paper also advises about areas that are difficult or impossible to factor into the financial equations and addresses some common misconceptions management holds about test automation. There are many factors to consider when planning for software test automation. Automation changes the complexion of testing and the test organization from design through implementation and test execution. It usually has broad impacts on the organization in such things as the tasks performed, test approaches, and even product features. There are tangible and intangible elements and widely held myths about benefits and capabilities of test automation. It is