A Complete Guide to Exploratory Testing
June 14, 2016
In order to understand anything, we need to explore- this is always true whenever we visit new places to wander/explore around. The same holds true in case of any application also. In order to understand the application potential and limitation, testers need to navigate through the application thoroughly. In Software testing life cycle sometimes we use terms like adhoc testing, monkey testing and exploratory testing interchangeably. But adhoc and monkey testing are different than exploratory testing, just like wandering is different from exploring. We will be lost while wandering if we do not have precise direction or a road map.
It does not matter how many test cases we add, how many we execute or how many times we repeat our scripted tests, testers are always encountered by several unknown issues when they do random scenarios which were not documented in test suits. In other words, it is difficult to cover all real time scenarios while designing a test plan. Since there may be variations in system configuration, time zones, user interactions and other factors, crazy things can happen in live environment which deviate live server data from test server. So it is important for testers to emphasize on good exploratory testing practices.
When we say Exploratory testing, some questions may arise like:
- When should testers carry out it ?
- What are the pre-requisites ?
- What should be the tester's skill set to perform it?
- When to stop it ?
- What should be the process to follow it ?
- Its pros and cons ?
When should testers carry out exploratory testing ?
- Initial product learning: In agile environment, there may be no design documents available like in traditional method, or there may be no time for product training, in such a scenario team can go with exploratory testing. Here main focus is given on product understanding rather than actual testing. This gives complete knowledge of product ecosystem and its dependencies.
- Initial writing of test cases: Testers can perform exploratory testing while writing tests cases for new product or for new features. This fits in agile environment where testers are not provided with design document.
- Rapid feedback: Some time testers may not have time to create test document for new features or for new product, in such cases a quick feed back can be provided by doing exploratory testing.
- Investigating issues: Testers need to do exploratory testing while reproducing issues.
- To complete the testing coverage: There are some risk areas where testers need to update their test document. Risk areas can be investigated through explorative testing and test document can be updated accordingly.
- Integrate with agile test process: High quality work can be deliver, if we integrate exploratory testing in agile test cycle along with integration testing and system testing.
- Regression testing: In STLC, regression testing is crucial one, when developers fix maximum number of defects, this can be further enhanced by exploring defect areas by experienced tester.
What are the pre-requisites?
Pre-requisites for exploratory testing may vary in different scenarios. I have listed some of the general pre-requisites:
- Product basic functionality a.k.a. product working knowledge
- How our product is different from competitor's product
- What are the recent changes incur in our product
- In case of mobile application we need to study the nature of application users and their usage
- Previously identified risk areas
- Critical issues which occurred in previous releases
Before starting exploratory testing, testers need to know - what is the best test scenario that can be executed instead of following instructed pattern to get surprise defects.
Exploratory testing consists of test design and test execution. It emphasizes on use case scenarios around the feature/area to be explore. Random test data is generated, in order to break the application or to understand how application can handle such situations.
To execute exploratory testing, testers need to think diversely based on their past experience and should have a keen eye for detail . It has gain its popularity majorly because of rigorous involvement of listening, reading, thinking out of the box, reporting skill etc.
As a summary, exploratory tester should have following skills:
- Good listening skills
- Fast grasping skills
- Keen observation skills
- Diverge thinking skills (out of box thinking)
- Fast and accurate reporting skills
- Good communication skills
When to stop exploratory testing ?
It is a known fact that since there is no pre-designed test document in exploratory testing it is difficult to estimate the timeline. This leads to obvious question - when one should stop it? I have summarized general completion criteria based on our past experience:
- In product learning we stop exploratory testing once tester acquires enough product knowledge or scripted around 80% test cases
- In case of issue investigation, we stop testing once issue is reproduced successfully or once we get the pre-production scenario
- In case of regression testing, we stop it once product become stable with random test inputs
- If we incorporate exploratory testing in agile process (along with integration and system testing) , we stop it, once tester met the product acceptance criteria.
If we talk about mobile application, testers should plan for periodic exploratory testing with live server. In such cases, it should stop with allotted time frame, if the issues are increasing at abnormal rate then testers need to continue till it reaches some stability.
Exploratory testing process
There is no defined process for exploratory testing yet we have tried to formulate a structured process (based upon our prior experience)
Exploratory Testing Pros and Cons
The main advantage of exploratory testing is that the issues are identified quickly, since there is no test document required to include before execution. Based on previous test report analysis, testers can create random test data which will help them to decide what to test and where to focus on & to skip script testing in case of small update release. Effective use of this testing with skilled testers will undoubtedly accelerate issue detection and in turn enhance the application quality.
Disadvantage is that, it's difficult to reproduce defect scenario. If any severe defect occurs, it may take longer duration to reproduce the same. But tester attitude here should be "Problem persist till we know its root cause". It's difficult to automate some test scenarios since exploratory tests cannot assure same set of data that will give same set of result every time. Hence some time tester need to execute same scenario repeatedly. But where ever possible testers need to automate the test scenario and push it to regression suit.
Keeping the advantage and disadvantage of the exploratory testing in mind, it is good if we incorporate it in test life cycle. When it comes to mobile testing, it is essential to practice it every now and then with live application to ensure live server data is intact and less prone to break the app.