This site had minor difficulties over the weekend, due mostly to (S)FTP failure: no connections on port 21, or on any of the usual alternates. Nothing to get hung about, as Lennon used to say: it didn’t affect external operations, only my ability to send up files. (WordPress has its own media handler, which may or may not use FTP or its brethren, and which I don’t use anyway.) Other customers of this host were not so fortunate; there was wailing and gnashing of teeth, prompting the Head Honcho to explain what had happened:
We run Debian OS and have used autoupdates to ensure security packages are installed as soon as they are available. We’ve had some breakage in the past from this approach, but nothing major. However last night’s autoupdate went badly wrong, removing essential packages from dedicated, VPS and some shared servers. Our monitoring and support team flagged the issue fast, and we scrambled our admin, dev and NOC teams to reinstall the packages that had been removed by autoupdate, reboot servers, fix package dependencies, and test that individual services were live. Given the number of services affected, this took a long time to complete. Rest assured we had all hands working on the issue, but I know it was still a frustrating experience for customers.
To mitigate the risk of anything like this happening again, we’re immediately switching off autoupdates, and moving to a manual process where we’ll only push out Debian updates after significant testing. There’s always a balance to be struck between speed, efficiency, security and issue prevention, but this event has shown us that we need to take a different approach.
Debian 6.0.4 was released Saturday; it was Sunday when Things Happened.
This has, I assume, nothing to do with the DDOS attack on Monday.