The Feature Flag
We tested the feature flag both on and off. Production used new_checkout_enabled. The test environment used new_checkout_enable. One missing letter had its own release strategy.
Article archive
We tested the feature flag both on and off. Production used new_checkout_enabled. The test environment used new_checkout_enable. One missing letter had its own release strategy.
I reported that checkout failed on Safari. The developer said he could not reproduce it because he did not own an iPhone. Our customers remained strangely unwilling to switch phones for him.
The checkout test failed eleven times out of twenty. The developer marked it as flaky automation. I placed twenty orders manually. Eleven failed. Production was apparently flaky too.
The change affected customers with ten years of transaction history. The test data team gave us one customer. His name was Dave. Dave had joined that morning and had never bought anything.
I raised a Severity 1 because every user could view the payroll screen. The delivery manager lowered it to Severity 3 because the screen was restricted to administrators. The defect was that every user was an administrator.
The Jira ticket said: QA required: No The release note said: QA approved QA had never seen the ticket.
Project manager: “Can you just test the happy path?” Tester: “Which path is that?” Project manager: “The one where everything works.”
Development finished three weeks late. The release candidate reached QA on Friday afternoon. Testing found a blocker on Monday morning. The steering group slide said: Go-live delayed by testing: one day.
“We don’t need testers. Our developers will get it right.” The first production defect arrived six hours later. The incident opened with: “How did testing miss this?”
The release went live at 4:00 pm. At 4:07 pm, customers could no longer log in. At 4:12 pm, the project manager asked why testing had missed it. The tester opened the test evidence. Login had passed every day for two weeks. The defect appeared only after someone changed the authentication settings during deployment. Nobody […]