Timeboxed exploratory testing

Whether you build or maintain a solution, or a set of it, whatever your business sector is, you need to test the new functionalities and the ones you have largely modified. In such cases, you cannot not rely on your existing tests: regression testing is not an option.

You have different cumulative options to verify the quality and validate the new or modified functionalities:

• Automated tests
• Scripted manual tests
• Timeboxed exploratory testing

Automated testing

This is the most recommended practice.
It requires you to build repeatable tests set that you can use in various campaigns: smoke testing, regression testing, and reliability testing to name a few.

Nevertheless, anyone in charge of automating tests knows that doing it at the same time as you build or modify functionality is hard and sometimes even inefficient or too expensive. It requires clear and stable requirements and design. For that reason, teams/squads tend to defer the automation (at least of some of the tests) after the demonstration (end of sprint or iteration).

Using a hypothetic automated test described below, the total time would be 7h57, including implementation, execution and analysis.

Scripted manual tests

Scripted manual tests are faster and cheaper to create than automated tests.
Most of the time, it is the first step before automating tests.

It requires the same level of implication and information as when automating tests. However, it is clearly easier to get it done during a short-term development cycle.
With scripted manual testing, the same Hypothetic test would take only 2h22, including creation, execution and analysis.

Timeboxed Exploratory Testing

With Exploratory Testing the same Hypothetic scripted test would take 1h53 to create, execute and analyse.


One should not consider it as an alternative practice to scripted manual tests or automated tests, but rather as a complementary practice - which is more often use in Agile 2 contexts though.
There is an extensive set of authoritative documentations addressing exploratory testing on the web; hence, we will not describe the practice.

We will rather explain:

• How XStudio makes it easy to use exploratory.
• How it slashes the time to create Scripted Manual Test using XQual PDNL.
• How it supports Risk Based Testing.

When do you do timeboxed exploratory testing?

There is no good or bad time about doing exploratory testing. However, most teams/squads generally engage in exploratory testing when there is no existing test (either manual or automated), and not much resources and time to verify or validate new features or functionalities.

Often, they don't benefit from reviewed specifications neither have direct access to the end-users that expressed the needs.

In such cases, you will assign specific aspects to be verified by some available actors - testers, business analysts, power users, developers ... any stakeholders having an interest in the feature/functionalities.

When saying "timeboxed", we mean the duration to test must be limited (1h to 3h).

When saying "exploratory testing" we expect a charter (What? Why? to test), an assignee and the effort to execute the test to be defined upfront.

Still using our hypothetic scripted manual test, the total time would be 2h05, including creation, execution and analysis.

How do you do it with xstudio?

XStudio manages exploratory testing as specific test campaign-sessions.
You create an exploratory session, define its charter, assign team members and select the specific System Under Test (SUT) against which the tests will run. A good practice is to indicate in the charter the type of risks or failure you fear to encounter, and wish to verify against. This guides the "tester" without imposing any tests practice or path.

When doing exploratory testing, it is critical to take "smart notes" and it is efficient to declare potential "defects" as you go.

"Smart notes" are critical because they allow developers to refer to the session, to analyze and eventually to reproduce potential defects. The cleaner and the more structured are the notes, the easier will be the work for the developers. To the contrary, imposing a stringent format and language to takes the note will impair the ability of the "tester" to focus on actual testing.

For that reason, XQual provides, but does not impose, the Procedure Definition in Natural Language (PDNL).

PDNL is simple to learn and use. It is a descriptive notation of the tasks sequence used by a tester to write the test:

• testcase
• do
• check
• and

PDNL is not only usable when you take notes but also when documenting the "steps to reproduce" a defect.

Keyword	Meaning
testcase	Indicate a new test case. A name for the test case can be provided next to this keyword.
do	Define some steps/actions in the test case procedure.
check	Define checking points in the test case procedure.
and	can be used either after do or check keywords to repeat the keyword.

Example:

How do you turn exploratory tests into scripted testing?

Once you have done a new exploratory test, you will often want to debrief and decide if this test should be just a one-off test or if it makes sense to turn it into a test that can be reused - for regression testing mainly.

There you get the benefit of XStudio allowing you to translate the test notes written in PDNL to automatically generate tests along with their detailed procedures. This is done in no time.

Transforming exploratory test notes into test cases is so handy that most customers tend to use exploratory testing when designing new manual scripted tests.
Doing so, they build and try their tests, making sure the tests meet the test objectives, and then generate the scripted tests in one click.


From exploratory test to scripted tests using XStudio: 1h25 from start to first results

This brings an important time and effort saving.
we measured it to be at a minimum of 35% saving over traditional manual scripted test authoring.