💾 World Backup Day - Backup Horror Stories: Share, Learn, and Win! 😱

When you have an ESXi cluster, running from a SAN, it is NEVER a good idea to have the same volumes mounted to the Management Server that host your backups as well. The scenario is like this, customer got the volumes mounted via iSCSI on the Management Server, as well as an iSCSI volume from the QNap where the onsite backups is stored. 2 drive failures on the QNap, and the volume is gone. Client replace the 2 drives, and rebuild the raid. When remounting the iSCSI volume, the client brought the “wrong” iSCSI volume online, delete the partition, and repartitioned and formatted it. BOOM, all Vm’s offline, end entire business down…… but wait, there is more. The last successful backup was more than 2 weeks old. A massive amount of data loss for this customer. THIS is my “horror world backup day” story in stead of an Honor World Backup Day. Countless and countless hours worth of work and recovery time here before the customer was up and running again. Needless to say, now the customer have not only onsite backups with daily offsite backups that get rotated daily, BUT also got full daily backups going to the cloud. 3-2-1-1 backup solution is a have to have, it like the “religion” of data protection.

We have a small server that we use here at are business to run a Virtual Machine for QuickBooks, and we thought we had everything setup correctly, and we were running it for several years until one day it just died and we discovered that the backups had never been setup correctly and we lost all the data, none of our recovery tools were able to get it back, we did have a backup from a external device that was about 6 weeks old so we were able to restore from that, and rebuild everything since the backup data. Lesson learned to always test your backups and make sure they are working correctly, and to never be to busy to check on your own equipment!

Always backup your backup. The amount of clients who think one backup is enough as they now have 2 copies of their data (the live content they access, plus the backup)

lockdown proved a point for this with our clients that had on-premise servers and backups and vpn issues and on-site power cuts caused lack of access to both live data and backups.

covid was/is such an awful global tragedy but I’ve sold a lot of cloud data hosting and cloud backups since covid

@Martin.1 @russell.harris @Rodney18 Y’all win gift cards

Extending this another day to try and get some more comments!

Awesome thanks so much, I would have thought there would be more discussion about this topic, but I guess that's a good thing if there isn't to many horror stories :)

Thanks @TylerM 

Yes, Surprised it’s been so quiet. Maybe everyone’s dealing with current backup horrors!

@TylerM  thank you Sir. Appreciated as always. Sharing our stories (even if Freddie will not even enter the room due to the content) hopefully can help someone else prevent the same mistakes. 

@Rodney18  I am sure that there are more. I think some people may have missed it due to it not being “advertised” on the Luminaries side, only in the community side.