Upstart said it had outgrown the cloud – now five out of five restore tools have failed
1 Feb 2017 at 02:02, Simon Sharwood Source-code hub GitLab.com is in meltdown after experiencing data loss as a result of what it has suddenly discovered are ineffectual backups.
On Tuesday evening, Pacific Time, the startup issued a sobering series of tweets we've listed below. Behind the scenes, a tired sysadmin, working late at night in the Netherlands, had accidentally deleted a directory on the wrong server during a frustrating database replication process: he wiped a folder containing 300GB of live production data that was due to be replicated.
Just 4.5GB remained by the time he canceled the rm -rf command. The last potentially viable backup was taken six hours beforehand.
Full Article