Lisa Crispin, Janet Gregory - Agile Testing A Practical Guide for Testers and Agile Teams [2009, PDF, ENG]

Chapter 1 What Is Agile Testing, Anyway? 3
Agile Values 3
What Do We Mean by “Agile Testing”? 4
A Little Context for Roles and Activities on an Agile Team 7
Customer Team 7
Developer Team 7
Interaction between Customer and Developer Teams 8
How Is Agile Testing Different? 9
Working on Traditional Teams 9
Working on Agile Teams 10
Traditional vs. Agile Testing 12
Whole-Team Approach 15
Summary 17
Chapter 2 Ten Principles for Agile Testers 19
What’s an Agile Tester? 19
The Agile Testing Mind-Set 20
x CONTENTS
Applying Agile Principles and Values 21
Provide Continuous Feedback 22
Deliver Value to the Customer 22
Enable Face-to-Face Communication 23
Have Courage 25
Keep It Simple 26
Practice Continuous Improvement 27
Respond to Change 28
Self-Organize 29
Focus on People 30
Enjoy 31
Adding Value 31
Summary 33
Part II Organizational Challenges 35
Chapter 3 Cultural Challenges 37
Organizational Culture 37
Quality Philosophy 38
Sustainable Pace 40
Customer Relationships 41
Organization Size 42
Empower Your Team 44
Barriers to Successful Agile Adoption by Test/QA Teams 44
Loss of Identity 44
Additional Roles 45
Lack of Training 45
Not Understanding Agile Concepts 45
Past Experience/Attitude 48
Cultural Differences among Roles 48
Introducing Change 49
Talk about Fears 49
Give Team Ownership 50
Celebrate Success 50
Management Expectations 52
Cultural Changes for Managers 52
Speaking the Manager’s Language 55
Change Doesn’t Come Easy 56
Be Patient 56
Let Them Feel Pain 56
CONTENTS xi
Build Your Credibility 57
Work On Your Own Professional Development 57
Beware the Quality Police Mentality 57
Vote with Your Feet 57
Summary 58
Chapter 4 Team Logistics 59
Team Structure 59
Independent QA Teams 60
Integration of Testers into an Agile Project 61
Agile Project Teams 64
Physical Logistics 65
Resources 66
Tester-Developer Ratio 66
Hiring an Agile Tester 67
Building a Team 69
Self-Organizing Team 69
Involving Other Teams 69
Every Team Member Has Equal Value 70
Performance and Rewards 70
What Can You Do? 71
Summary 71
Chapter 5 Transitioning Typical Processes 73
Seeking Lightweight Processes 73
Metrics 74
Lean Measurements 74
Why We Need Metrics 75
What Not to Do with Metrics 77
Communicating Metrics 77
Metrics ROI 78
Defect Tracking 79
Why Should We Use a Defect Tracking System (DTS)? 80
Why Shouldn’t We Use a DTS? 82
Defect Tracking Tools 83
Keep Your Focus 85
Test Planning 86
Test Strategy vs. Test Planning 86
Traceability 88
xii CONTENTS
Existing Processes and Models 88
Audits 89
Frameworks, Models, and Standards 90
Summary 93
Part III The Agile Testing Quadrants 95
Chapter 6 The Purpose of Testing 97
The Agile Testing Quadrants 97
Tests that Support the Team 98
Tests that Critique the Product 101
Knowing When a Story Is Done 104
Shared Responsibility 105
Managing Technical Debt 106
Testing in Context 106
Summary 108
Chapter 7 Technology-Facing Tests that Support the Team 109
An Agile Testing Foundation 109
The Purpose of Quadrant 1 Tests 110
Supporting Infrastructure 111
Why Write and Execute These Tests? 112
Lets Us Go Faster and Do More 112
Making Testers’ Jobs Easier 114
Designing with Testing in Mind 115
Timely Feedback 118
Where Do Technology-Facing Tests Stop? 119
What If the Team Doesn’t Do These Tests? 121
What Can Testers Do? 121
What Can Managers Do? 122
It’s a Team Problem 123
Toolkit 123
Source Code Control 123
IDEs 124
Build Tools 126
Build Automation Tools 126
Unit Test Tools 126
Summary 127
CONTENTS xiii
Chapter 8 Business-Facing Tests that Support the Team 129
Driving Development with Business-Facing Tests 129
The Requirements Quandary 132
Common Language 134
Eliciting Requirements 135
Advance Clarity 140
Conditions of Satisfaction 142
Ripple Effects 143
Thin Slices, Small Chunks 144
How Do We Know We’re Done? 146
Tests Mitigate Risk 147
Testability and Automation 149
Summary 150
Chapter 9 Toolkit for Business-Facing Tests that
Support the Team 153
Business-Facing Test Tool Strategy 153
Tools to Elicit Examples and Requirements 155
Checklists 156
Mind Maps 156
Spreadsheets 159
Mock-Ups 160
Flow Diagrams 160
Software-Based Tools 163
Tools for Automating Tests Based on Examples 164
Tools to Test below the GUI and API Level 165
Tools for Testing through the GUI 170
Strategies for Writing Tests 177
Build Tests Incrementally 178
Keep the Tests Passing 179
Use Appropriate Test Design Patterns 179
Keyword and Data-Driven Tests 182
Testability 183
Code Design and Test Design 184
Automated vs. Manual Quadrant 2 Tests 185
Test Management 186
Summary 186
Chapter 10 Business-Facing Tests that Critique the Product 189
Introduction to Quadrant 3 190
Demonstrations 191
xiv CONTENTS
Scenario Testing 192
Exploratory Testing 195
Session-Based Testing 200
Automation and Exploratory Testing 201
An Exploratory Tester 201
Usability Testing 202
User Needs and Persona Testing 202
Navigation 204
Check Out the Competition 204
Behind the GUI 204
API Testing 205
Web Services 207
Testing Documents and Documentation 207
User Documentation 207
Reports 208
Tools to Assist with Exploratory Testing 210
Test Setup 211
Test Data Generation 212
Monitoring Tools 212
Simulators 213
Emulators 213
Summary 214
Chapter 11 Critiquing the Product Using Technology-
Facing Tests 217
Introduction to Quadrant 4 217
Who Does It? 220
When Do You Do It? 222
“ility” Testing 223
Security 223
Maintainability 227
Interoperability 228
Compatibility 229
Reliability 230
Installability 231
“ility” Summary 232
Performance, Load, Stress, and Scalability Testing 233
Scalability 233
Performance and Load Testing 234
Performance and Load-Testing Tools 234
Baseline 235
CONTENTS xv
Test Environments 237
Memory Management 237
Summary 238
Chapter 12 Summary of Testing Quadrants 241
Review of the Testing Quadrants 241
A System Test Example 242
The Application 242
The Team and the Process 243
Tests Driving Development 244
Unit Tests 244
Acceptance Tests 245
Automation 245
The Automated Functional Test Structure 245
Web Services 247
Embedded Testing 248
Critiquing the Product with Business-Facing Tests 248
Exploratory Testing 248
Testing Data Feeds 249
The End-to-End Tests 249
User Acceptance Testing 250
Reliability 250
Documentation 251
Documenting the Test Code 251
Reporting the Test Results 251
Using the Agile Testing Quadrants 252
Summary 253
Part IV Automation 255
Chapter 13 Why We Want to Automate Tests and What
Holds Us Back 257
Why Automate? 258
Manual Testing Takes Too Long 258
Manual Processes Are Error Prone 259
Automation Frees People to Do Their Best Work 259
Automated Regression Tests Provide a Safety Net 261
Automated Tests Give Feedback, Early and Often 262
Tests and Examples that Drive Coding Can Do More 262
xvi CONTENTS
Tests Are Great Documentation 263
ROI and Payback 264
Barriers to Automation—Things that Get in the Way 264
Bret’s List 264
Our List 265
Programmers’ Attitude—“Why Automate?” 265
The “Hump of Pain” (The Learning Curve) 266
Initial Investment 267
Code that’s Always in Flux 269
Legacy Code 269
Fear 269
Old Habits 270
Can We Overcome These Barriers? 270
Summary 271
Chapter 14 An Agile Test Automation Strategy 273
An Agile Approach to Test Automation 274
Automation Test Categories 274
Test Automation Pyramid 276
What Can We Automate? 279
Continuous Integration, Builds, and Deploys 280
Unit and Component Tests 282
API or Web Services Testing 282
Testing behind the GUI 282
Testing the GUI 282
Load Tests 283
Comparisons 283
Repetitive Tasks 284
Data Creation or Setup 284
What Shouldn’t We Automate? 285
Usability Testing 285
Exploratory Testing 286
Tests that Will Never Fail 286
One-Off Tests 286
What Might Be Hard to Automate? 287
Developing an Automation Strategy—Where Do We Start? 288
Where Does It Hurt the Most? 289
Multi-Layered Approach 290
Think about Test Design and Maintenance 292
Choosing the Right Tools 294
CONTENTS xvii
Applying Agile Principles to Test Automation 298
Keep It Simple 298
Iterative Feedback 299
Whole-Team Approach 300
Taking the Time to Do It Right 301
Learn by Doing 303
Apply Agile Coding Practices to Tests 303
Supplying Data for Tests 304
Data Generation Tools 304
Avoid Database Access 306
When Database Access Is Unavoidable or Even Desirable 307
Understand Your Needs 310
Evaluating Automation Tools 311
Identifying Requirements for Your Automation Tool 311
One Tool at a Time 312
Choosing Tools 313
Agile-Friendly Tools 316
Implementing Automation 316
Managing Automated Tests 319
Organizing Tests 319
Organizing Test Results 322
Go Get Started 324
Summary 324
Part V An Iteration in the Life of a Tester 327
Chapter 15 Tester Activities in Release or Theme Planning 329
The Purpose of Release Planning 330
Sizing 332
How to Size Stories 332
The Tester’s Role in Sizing Stories 333
An Example of Sizing Stories 334
Prioritizing 338
Why We Prioritize Stories 338
Testing Considerations While Prioritizing 339
What’s in Scope? 340
Deadlines and Timelines 340
Focus on Value 341
xviii CONTENTS
System-Wide Impact 342
Third-Party Involvement 342
Test Planning 345
Where to Start 345
Why Write a Test Plan? 345
Types of Testing 346
Infrastructure 346
Test Environments 347
Test Data 348
Test Results 349
Test Plan Alternatives 350
Lightweight Test Plans 350
Using a Test Matrix 350
Test Spreadsheet 353
A Whiteboard 353
Automated Test List 354
Preparing for Visibility 354
Tracking Test Tasks and Status 354
Communicating Test Results 357
Release Metrics 358
Summary 366
Chapter 16 Hit the Ground Running 369
Be Proactive 369
Benefits 370
Do You Really Need This? 372
Potential Downsides to Advance Preparation 373
Advance Clarity 373
Customers Speak with One Voice 373
Story Size 375
Geographically Dispersed Teams 376
Examples 378
Test Strategies 380
Prioritize Defects 381
Resources 381
Summary 382
Chapter 17 Iteration Kickoff 383
Iteration Planning 383
Learning the Details 384
Considering All Viewpoints 385
CONTENTS xix
Writing Task Cards 389
Deciding on Workload 393
Testable Stories 393
Collaborate with Customers 396
High-Level Tests and Examples 397
Reviewing with Customers 400
Reviewing with Programmers 400
Test Cases as Documentation 402
Summary 403
Chapter 18 Coding and Testing 405
Driving Development 406
Start Simple 406
Add Complexity 407
Assess Risk 407
Coding and Testing Progress Together 409
Identify Variations 410
Power of Three 411
Focus on One Story 411
Tests that Critique the Product 412
Collaborate with Programmers 413
Pair Testing 413
“Show Me” 413
Talk to Customers 414
Show Customers 414
Understand the Business 415
Completing Testing Tasks 415
Dealing with Bugs 416
Is It a Defect or Is It a Feature? 417
Technical Debt 418
Zero Bug Tolerance 418
It’s All about Choices 419
Decide Which Bugs to Log 420
Choose When to Fix Your Bugs 421
Choose the Media You Should Use to Log a Bug 423
Alternatives and Suggestions for Dealing with Bugs 424
Start Simple 428
Facilitate Communication 429
Testers Facilitate Communication 429
Distributed Teams 431
xx CONTENTS
Regression Tests 432
Keep the Build “Green” 433
Keep the Build Quick 433
Building a Regression Suite 434
Checking the “Big Picture” 434
Resources 434
Iteration Metrics 435
Measuring Progress 435
Defect Metrics 437
Summary 440
Chapter 19 Wrap Up the Iteration 443
Iteration Demo 443
Retrospectives 444
Start, Stop, and Continue 445
Ideas for Improvements 447
Celebrate Successes 449
Summary 451
Chapter 20 Successful Delivery 453
What Makes a Product? 453
Planning Enough Time for Testing 455
The End Game 456
Testing the Release Candidate 458
Test on a Staging Environment 458
Final Nonfunctional Testing 458
Integration with External Applications 459
Data Conversion and Database Updates 459
Installation Testing 461
Communication 462
What If It’s Not Ready? 463
Customer Testing 464
UAT 464
Alpha/Beta Testing 466
Post-Development Testing Cycles 467
Deliverables 468
Releasing the Product 470
Release Acceptance Criteria 470
Release Management 474
Packaging 474
CONTENTS xxi
Customer Expectations 475
Production Support 475
Understand Impact to Business 475
Summary 476
Part VI Summary 479
Chapter 21 Key Success Factors 481
Success Factor 1: Use the Whole-Team Approach 482
Success Factor 2: Adopt an Agile Testing Mind-Set 482
Success Factor 3: Automate Regression Testing 484
Success Factor 4: Provide and Obtain Feedback 484
Success Factor 5: Build a Foundation of Core Practices 486
Continuous Integration 486
Test Environments 487
Manage Technical Debt 487
Working Incrementally 488
Coding and Testing Are Part of One Process 488
Synergy between Practices 489
Success Factor 6: Collaborate with Customers 489
Success Factor 7: Look at the Big Picture 490
Summary 491
Glossary 493
Bibliography 501
Index 509