Estimating effort for test automation

Once we identify what tests need to be automated (read previous blog on choosing the right tests for automation), the first thing we usually do before initiating automation is the estimation of effort for automation.

Automation is time consuming initiative, sometimes running for several months & years. It is important to use the right estimation approach to ensure that the automation deliverables are inline with product development roadmap. We don't wish to promise delivery of scripts in one month which actually needs two team months. And, at the same time, we don't wish to allocate four months for a two months activity, making automation even more expensive.

Let us learn about the steps involved in arriving at optimal effort estimation for automation.

1. Identify the size & complexity of tests

2. Categorize tests by size & complexity

3. Sampling of tests for individual effort

4. Average automation effort with resource experience levels

5. Effort estimation with team composition

This estimation method is proposed assuming the framework is already in place. Please read our blog on choosing the right automation framework, if not done already. I will cover the Framework Development effort estimation in next blog.

1.      Identify the size & complexity of tests

The test cases shortlisted for automation need to be categorized in to Simple, Medium & Complex based on their size and complexity.

When it comes to size, the tests can be categorized based on the number of test steps in the test case. Tests with less than 10 steps can be considered as simple, tests with 10-20 steps can be categorized as medium and above 20 steps can be marked as Complex. Additional categories can be added based on the application nature and type of tests performed.

Alternate way to categorize the tests is by complexity. Tests with plain web/screen interactions navigating through 2-3 pages can be marked as Simple. Tests with navigations of 5-6 cases and simple querying of database can be considered as Medium. If the tests include navigation of more than 6 pages or have multiple database validations can be considered as Complex. Tests that have multiple systems involved, like switching between two or more applications as part of the same test, or tests that include web and web-service transactions can also be considered as complex.

2.      Categorize tests by size & complexity

Once we mark each test case as Simple, Medium or Complex based on their size and complexity, identify the total test cases under each category for each application / product.

Let us assume we arrive at the following number after the study from above step.

Feature	Total Cases	Simple	Medium	Complex
Application 1	200	60	100	40
Application 2	200	50	120	30
Application 3	200	30	110	60

3.      Sampling of tests for individual effort

After categorizing the test cases, we need to understand what would be the effort to automate a simple test, medium and a complex test for the given application. There are few industry standard test case level estimates like 1-hour for Simple, 2-hours for Medium and 4 hours for Complex test. This effort might vary from product to product. The sampling of tests helps in coming up with more accurate estimates.

In the sampling, we pick up few cases (usually 5 cases or 2% whichever is higher) from each category for the given application and try to automate them using the framework to understand the average effort.

4.      Average automation effort with resource experience levels

The productivity of automation resources with different experience levels varies. Estimating effort of automating a test with resources of different experience levels provides more accurate estimates as the automation teams consist both junior (<=3 years exp) and senior (>3 years exp) resources.

Once the sample cases are identified in above step, we need to have these cases automated with a junior and a senior resource in the team / organization. The average effort for each category of tests needs to be documented. If a junior resource is assigned to automate 10 simple cases and it took 15 hours to automate them, the time is 1.5 hours per test. A senior resource might automate the same 10 cases in 10 hours with an average of 1 hour per test. We need to identify the average effort for Medium and Complex test cases as well and mark them in a simple table like the one below.

Resource	Average Effort (Hrs) – Simple Test	Average Effort (Hrs)  – Medium Test	Average Effort (Hrs)  – Complex Test
Junior Engineer	1.5	3	7
Senior Engineer	1	2	4

5.      Effort estimation with team composition

In the above step we have calculated the effort of automating tests with different complexity by a junior and a senior level resource. But the automation team might consist of multiple resources at each experience level. The average effort per test varies based on the team composition. The average effort of a test with a team of 1 junior & 2 senior resources would be different from a team of 2 junior & 1 senior resources though the team size is same. We need to calculate the average effort per test with given team composition.

If the team consists of X junior and Y senior resource, the average team effort needed to complete 1 test would be:

(X * 1/Jr. Effort) + (Y * 1/Sr. Effort) = 1/Avg. Team Effort

Avg. Team Effort = (Jr. Effort * Sr. Effort) / (Y * Jr. Effort + X * Sr. Effort)

From above table, the team effort for different category of tests with 1 junior and 1 senior resource would be:

Avg. Team Effort for Simple test = (1.5 * 1) / (1.5 + 1) => 1.5 / 2.5 => 0.6 hours

Avg. Team Effort for Medium test = (3 * 2) / (3 + 2) => 6 / 5 => 1.2 hours

Avg. Team Effort for Complex test = (7 * 4) / (7 + 4) => 28 / 11 => 2.55 hours

Feature	Average Effort (Hrs) – Simple Test	Average Effort  (Hrs) – Medium Test	Average Effort  (Hrs) – Complex Test
Avg. Team Effort (1 Jr + 1 Sr)	0.6	1.2	2.55

Let us assume our automation team size would be 3 junior and 2 senior resources. Here is the effort estimation for given team composition:

Avg. Team Effort for Simple test = (1.5 * 1) / (2 * 1.5 + 3 * 1) => 1.5 / 9 => 0.17 hours

Avg. Team Effort for Medium test = (3 * 2) / (2 * 3 + 3 * 2) => 6 / 12 => 0.5 hours

Avg. Team Effort for Complex test = (7 * 4) / (2 * 7 + 3 * 4) => 28 / 26 => 1.08 hours

Resources	Average Effort (Hrs) – Simple Case	Average Effort (Hrs) – Medium Case	Average Effort (Hrs) – Complex Case
Avg. Team Effort (3 Jr + 2 Sr)	0.17	0.5	1.08

This is the effort for one single test from different complexities. In Section 2, we have listed the test case count across different categories for 3 applications. The total team effort to automate these test cases with a team of 3 junior and 2 senior resources would be:

Feature	Effort (Hrs)  – Simple Cases	Effort (Hrs)  – Medium Cases	Effort (Hrs)  – Complex Cases
Application 1	60 * 0.17	100 * 0.5	40 * 1.08
Application 2	50 * 0.17	120 * 0.5	30 * 1.08
Application 3	30 * 0.17	110 * 0.5	60 * 1.08

Which is:

Feature	Effort (Hrs) – Simple Cases	Effort (Hrs) – Medium Cases	Effort (Hrs) – Complex Cases	Total Effort  (Team hours)
Application 1	10.2	50	43.2	103.4
Application 2	8.5	60	32.4	100.9
Application 3	5.1	55	64.8	124.9

Here is the effort in Team Days (with 8 hours per day):

Feature	Total Effort  (Team Days)
Application 1	13
Application 2	11.2
Application 3	15.6

So, the total effort to automate Application 1 with 3 junior and 2 senior resources would be 13 days. Application 2 will need 11.2 days and Application 3 will take 15.6 days.

There would be large applications with thousands of tests and automation takes several months. The same approach would work for application of any size. The sampling of tests needs to be increased based on the size of the application.

More reusable steps in automation can save some automation time as the steps can be copied between tests. This is possible when a standard keyword-driven approach is followed. We will discuss more about this in upcoming blogs.

In next blog, we will understand the process of estimating effort for Automation Framework Development, Script execution and Script maintenance.

By
Automation Mentor
www.automationmentor.in

We provide hands-on training on automation tools and frameworks