Agile Testing Chronicles: From Exploration to Explanation

Test cases: Confidence Led to Overconfidence

Early in our project, we relied heavily on structured test cases. Initially, this approach appeared successful—test coverage was solid, and we uncovered a significant number of defects. However, User Acceptance Testing (UAT) exposed critical gaps. Business users discovered defects that our test cases hadn’t covered, forcing us to reevaluate our strategy.

A team retrospective revealed key shortcomings:

• Testers focused only on executing test case steps, rarely exploring beyond them.
• Knowledge about system behavior was siloed, meaning testers were only familiar with areas they had personally tested.

These findings made it clear: structure alone wasn’t enough. We needed a balance between discipline and Agile adaptability.

Blending Structured Test Cases with Exploratory Testing

Our new strategy combined structured test cases with exploratory testing. Exploratory testing is led more by the tester’s instincts rather than following a strict script. In this form of testing, the tester’s mind isn’t occupied by the constant questions of “What else do I need to include?” or “What steps am I missing?”. Instead, the tester is free to explore the system more spontaneously, allowing unexpected scenarios to arise.

Similar challenges in automated testing, especially dealing with dynamic element IDs, have been explored in detail in our blog post on ExtJs and test automation, which discusses practical solutions for improving test automation reliability.

This strategy allowed testers to:

• Dynamically investigate unexpected behaviors.
• Share knowledge about system quirks in real-time.
• Expand test coverage beyond scripted scenarios.

After implementing this dual approach across four sprints, the results were striking:

• Found defects now included subtle, business-critical issues.
• Testers gained deeper system knowledge, improving their confidence and satisfaction.
• Collaboration increased as testers discussed findings in real-time.

Strategic Test Charters in Exploratory Testing

To make exploratory testing even more effective, we introduced test charters—guiding testers toward high-risk or complex areas while still allowing flexibility. By focusing exploration on:

• Critical defect-prone functionalities (high-level exploration with test charters).
• New, complex features (medium-level exploration with test charters).

We saw:

• Higher defect capture rates during exploratory sessions.
• Better knowledge sharing as testers collaborated on uncovering system behaviors.

Sharing Exploratory Testing Insights Through Conferences and Publications

We believed this experience was valuable to share, and thanks to conference organizers, we got to present it at both SEETEST (in-person) and A4Q Summit (online). The two experiences were quite different – at SEETEST we enjoyed direct interaction with attendees after our presentation and could discuss real-world testing experiences, while A4Q’s virtual format made audience engagement more challenging but still valuable. Both events exposed us to exciting topics and speakers, helping us expand our exploratory testing knowledge through conversations with other exploratory testers. We met many professionals equally passionate about the topic and exchanged insights that enriched our approaches. These conferences truly showed us how much we can learn from being part of the testing community.

Along with sharing our insights on conferences, another opportunity for sharing was writing an article for Quality Matters magazine. Writing the article felt like placing another puzzle piece into a picture we didn’t even realize we were assembling. First came the project. Then the presentation. Then the questions, reflections, and hallway conversations. And finally, this article—just one more piece that helped us see the shape of the whole a bit more clearly.

You can read the full article here https://quality-matters.org/index.php?page=qm-issues-archive&issue=19.

Agile Testing Strategy: Balancing Structure and Exploration

Testing is like navigating uncharted territory—too much rigidity can limit discovery, while too little can lead to inefficiency. Our journey taught us:

• Structured test cases provide a necessary foundation.
• Exploratory testing brings flexibility and creativity.

When reflecting on our Agile journey—full of structure, spontaneity, and a few missteps—Sinatra’s timeless words come to mind:

Our “Agile way” means never settling—learning, iterating, and continuously refining our strategies. As we move forward, we remain open to new testing innovations, ensuring we deliver the highest-quality products with confidence.