Do we find bugs with automation?

When we automate a test for an application we know one thing up front. With this certain input, we expect this output. This is what I call a known known. We know the input(s), we know the output. This test can be automated and can be added to a regression test suite to check the functionality for every delivery to production.

After a while, out of the blue, the output of this automated test gives a different value/behaviour as we expected. Do we have a bug? I don’t know. Is something changed in the application that has broken this specific functionality? Probably. We have found a regression issue as it reveals itself due to an unknown situation in the application.

When we write some code to automate a test, we work with expectations. With the expected output observed, we make a judgement call and say the feature of the application works. When the output differs with what we expected, we have a failed automated test. Now we have to determine what happened. At this moment we do not talk about a bug, but about a problem (as I said, a failed automated test).

With automation we signal that there is a problem when we encounter an unexpected output. A situation that we did not expect. At this moment, we start to look why this is a problem. The automation has stopped and a person (most of the time a tester) takes over. This person will investigate what happened. Communication to a specific server is lost. Login credentials to a specific environment have been changed. An interface has been changed. A parameter not sent within the message over this interface, because it is no longer mandatory.

So, the answer to my question is, with automation we will not find bugs. We only find problems, a result we did not expect. It is up to a person to determine, after thorough investigation, what the cause of this problem is. The outcome can be that it is a bug.

But automation is not a waste. Automation reveals information. Automation provides signals for a person to react upon. It helps us to shift focus to the areas where we think there might be a problem. At that point automation ends and a person takes over.

2 Comments Add yours

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s