In between my music and photography work, I have a lot of hard drives. But even so, I've never had one fail, until just two days ago. It went without warning: no crunching sounds, no notifications, just a red light.
The Story
It's been a rough few weeks for me computer-wise. I was editing client photos earlier this weekend, and all was fine. Then, when I turned on my computer Sunday, I received the following message: "Your device has a RAID configuration issue." I figured it was something a restart might fix, but I peered at my enclosure (it's hidden behind my monitor) and saw solid red lights (never a good sign). I shut down my computer, then restarted it and watched the enclosure as the soft white light blinked on, thinking all was fine, only to watch it suddenly switch to solid red again. I opened the drive utilities app and saw the dreaded status telling me one of the drives had failed and needed to be replaced.
Thankfully, I use two 8 TB hard drives in a single enclosure in RAID 1 for my photos. RAID isn't a backup solution, but it did allow me to immediately switch over to the redundant drive and continue working. Everything is backed up with Backblaze (at $5 a month, I really can't recommend them enough) and on another drive in my apartment, but even so, the thought of losing literal terabytes of not just data but hard work, creativity, and memories was sobering.
The 3-2-1 Strategy
If you're never heard of the 3-2-1 strategy, it's the best way to back up your data. It goes like this: at the minimum, you have three copies of your data, two of which are local and one of which is offsite. So, for example, my current setup is:
- Main external hard drive: My photos live on an 8 TB external hard drive in RAID 1 configuration with an identical drive. I do not count the RAID configuration as a backup; it simply makes it very easy to get back up and running if a drive fails.
- NAS: I have a second external hard drive attached to my router. My computer automatically backs up the photos drive to that drive. I prefer it this way because my router is in another room on another circuit, so I do get at least some isolation between the two local drives (imagine a burst pipe in the ceiling pouring on both if they were on the same desk).
- Backblaze: Every night, my computer syncs to Backblaze, so my offsite backup is always up to date. The initial backup took about 40 days, but if that's too long for you, you can send them a hard drive of your data.
With this method, you can easily get up and running again if your local drive fails, and if something catastrophic happens, you have the offsite backup. And I can't stress the importance of an offsite backup enough. It doesn't matter if you have 500 copies; if they're all in the same place and a fire/flood/theft occurs, you're done for. If your Internet connection is slow, another option is buying a backup drive that you bring home every few weeks or so, then store elsewhere, such as an office or relative's house. At $5 a month and with the benefit of real-time backups, Backblaze is a no-brainer for me, but if your connection speed precludes its use, bringing home an extra external drive every few weeks isn't a bad alternative.
Yes, the extra hard drives and subscription services add cost to the equation. Nonetheless, the thought of losing my work is terrifying enough that I'll gladly pay that extra cost, and I definitely recommend you do too.
Lead image by Pixabay user 422737, used under Creative Commons.
I have everything stored on one external travel-sized drive. It's called the 50-50 strategy as every day it feel like I have a 50% chance of making it without losing it all.
You like to live your life dangerously... (Seriously Ryan please follow Alex's advice)
livin on ze edge
As a film shooter and self-developer I live on the edge anyway so I am pretty much in the same boat when it comes to backup! I have my negatives, but man would that be a bitch to re-scan!
Perfect article Alex, I quote every single word! Especially "RAID isn't a backup solution" I work in IT and sadly most customers understand how important backup is, only after a major failure.
I could add another reason to have an offsite backup and is computer virus such as Cryptolocker. Usually, photo/videographers PCs/Macs are really fast and this makes the Cryptolocker work easier, so having an offsite backup that is not connected to our LAN is vital.
I agree that most understand how important this is only after a major failure. Everyone in my family has experienced something along these lines, including myself when I spilled a glass of wine on my laptop. I've made them all install Google Drive and pay the $2 per month to have their important documents backed up. And it's those people that need a solution like Backblaze or Google Drive that does this automatically otherwise they get lazy and forget why it was so important in the first place.
Yes backup has to be automatic otherwise no one will do it
I shoot them --> images go from the card to to my laptop --> laptop gets backed up to two different loca drives --> Edited session with both RAW and PSD files gets achived to two different mirrored drives --> final web jpg and full res TIFFs go to the cloud.
Also if I shoot client work tethered I backup all RAW files in 15min intervals to a external SSD.
Requires some manual work and has a point of failure in that all files are locally stored for a while but it is a cheap and pretty robust solution for me at the moment.
Good job with the 3-2-1, but your NAS component has a major downside. Here's your test case: turn off your router, unplug the USB3 drive from it and plug it into a computer. Can you read it? Most likely not, as the drive is formatted in EXT4, which isn't natively supported on Windows and Mac computers. If you're lucky you'll spend an hour or two setting up a virtual machine with a Linux LiveCD and hope you can share the drive back to yourself.
Yes, home routers 'can' share out a drive like a NAS, but it's not a full featured device.
Had a VERY similar experience earlier this week with my RAID drives. It was a sobering reminder that they're not a backup solution (and luckily I wasn't treating them like one!).
Which NAS brand do you use? I'm considering adding one to my backup strategy as well.
QNAP and Synology are very good brand regarding NAS, just make sure to fill them with NAS-specific Hard Drives such as Western Digital RED or RED PRO series
I use WD as I like their drives, but I'll defer to Andrea Re Depaolini's expertise here; he's the IT guy. :)
«…but if your connection speed precludes its use….»
Just to make it clear to those who may be unaware, the Internet speed here in question is your ‘upload’ speed, not your download speed.
Your Internet “connection speed” has two values; download and upload. Most ISPs will quote your download speed to you. If you are not on a fiber connection, your connection is probably asymmetrical, meaning that your download speed is NOT the same as your upload speed.
Most non-fibre telco connections give a relatively low upload speeds. Most residential non-fibre cable connections, although much better than telcos, usually are no higher than 10-15Mbps upload. Even with a 1Gbps download speed, some cable providers limit uploads to no more than 35Mbps (which isn't too shabby, but expensive).
Fibre connections tend to be symmetrical. This means that if your advertised download speed is 100Mbps, then your upload speed is probably 100Mbps, also.
The point is that many purchase a 250 Mbps download connection for ¤299.95 per month (plus modem rental, fees, taxes, for upward of ¤330), then wonder why their backup is taking so long. The answer is that their upload speed may still be as slow as 10Mbps.
Exactly why I pay for a 60/6 connection. I'd give anything for symmetrical d/u speeds.
From the beginning, I've stored everything on matching externals. As drive space has gotten cheaper I buy larger drives, 250GB, 500GB and now 1TB drives. Like CF and SD cards, I never have all the images on one big card but write to two in camera. If one fails, I simply replace and backup from the matching drive or card. I believe in redundancy and so far it hasn't failed me. Luckily, I've had one drive fail and it's since been replaced.
Question about Backblaze. Is it actually unlimited, or is it one of those unlimited where the fine print actually indicates X gigabytes Is what they mean by "unlimited?" I already back everything up in quadruplicate, including off site, but would be nice to have a cloud backup as well. But I've got roughly 20TB and counting, so not sure if that would still be $5/month?
Truly unlimited! I've got somewhere around 6 TB with them and pay $5 a month.
I'd like to add that whilst a NAS is a good way to add resilience, thought is required if you want to do it well.
I strongly recommend only selecting a NAS that supports either BTRFS or ZFS. This will provide features like CoW and snapshots, which will give you a better chance of getting your file back if you accidentally overwrite it. Don't forget to ensure that the system regularly does a scrub. These terms may sound intimidating at first, but they're easy enough to understand with a few minutes reading.
Bit rot. A quick read on Wikipedia is enough to give you nightmares! Spinning rust (HDD), SSDs and CF cards are all vulnerable to some extent. Suffice to say that sometimes a drive can't read the data without error, even though modern drives have heroic abilities to correct data reads.
Today, most consumer drives are rated 10 to the 14th power regarding their Unrecoverable Read Error Rate. That means that there is a _chance_ that you'll get a read error that can't be corrected every 12.5 TB of data. So with a 4 TB drive, if you read the entire drive more than 3x, there is a _chance_ you'll get a read error that the drive can't automatically correct.
What this means in practical terms is that if you have a really large array, there's a very real possibility that it will fail to rebuild when a drive fails, simply due to unrecoverable read errors. To be fair though, this is not usually an issue unless you have many TB of storage.
On single drives, that's not to say that if some of your data was corrupted with a non-recoverable read error you'd be left with a useless file - If it's just one bit flipped, you probably won't notice the corruption.
Nowadays I have several backups. I have my NAS with two hd.
I have a backup on a second harddisk. I have a backup on an external HD and I have a copy in the cloud.
In the past when I had to make backup on RW DVD's, I lost an entire year of pictures (all backups were corrupt) and I swore to do a better job in the future. Lately I erased all my pictures on my HD (tired, the flu, and a glass of red wine too many) because I delete the Lightroom catalogue (yeah daft, I know) and I was very glad to have a uptodate backup.
I'm going to start the Router/External outline you mention above. Been a Backblaze user since last month. 20 days or so for initial backup, but sleeping better for....$5 bucks a month! Seriously!
My backup system is a little easier than Alex uses.
My files are on my computer. Once a month they get backed up to an onsite external drive. The next day I bring my offsite drive to the office, back it up overnight, then take it back to its offsite location. If I put anything on my computer between the backups, I back it up onto a flash drive until my monthly backup.
My way is a little less secure than Alex uses, but it's worked well for the past 12 years.
Have Fun,
Jeff
PS On Facebook last week a photographer and artist posted that he had everything on a RAID drive, but a power surge ran through his house and fried both drives. He's now raising $2,000 to have a recovery company pull his data from the drives.
The other drive will die shortly