AWS 2017: the typo was not the outage [BYpGNhN5Dmm]
28 February 2017. An engineer at Amazon runs a command from an established playbook to take a few billing servers out of S3. One input is typed wrong. Slack goes down. Medium, Quora, Docker, Coursera, Expedia. Nest cameras stop recording. A quarter of the visible internet is dark for four hours. Everyone tells this story as "a typo took down the internet". That is the boring half. The command removed more servers than intended, and the extra ones were not billing servers. They were serving the index subsystem, which holds the metadata for every object in the region, and the placement subsystem, which decides where new objects go. A library with no catalogue: every book still on its shelf, nothing lost, nothing findable. Then the part that should stop you. For two hours AWS could not update its own status page, because the Service Health Dashboard console depends on S3. Their words. The tool that tells you the system is down was inside the system that was down. They used Twitter. And the sentence everyone skips, from AWS's own post-mortem: they had not completely restarted the index or placement subsystems in their larger regions for many years. S3 had grown enormously in that time. So the typo was four seconds. The outage was a restart nobody had ever timed at that size. Timeline, from AWS: 9:37 AM it starts. Reads return 12:26. Placement whole 1:54. Four hours seventeen. You almost certainly have one. A database you have never failed over since it tripled. A cluster you have never cold started since launch. A backup that restores fine at 10 GB and has never been tried at 400. Find the thing you have never restarted at today's size, and restart it on purpose, while you are watching. AWS's own fix was not smarter engineers: they made the tool remove capacity more slowly, and refuse to take any subsystem below its minimum. Boring guardrails on the untested path. Source: AWS, Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region. Full lesson on testing the path you never test: 766 lessons, built exactly like this. One payment, lifetime, not a subscription. Rs 499 India / $7.99 everywhere else, systemdesign.academy #Shorts