If you’re relying solely on cloud-based storage alone such as Dropbox, Box, Google Drive, etc., stop! This is a disaster if you’re a professional or a semi-pro photographer. Allow me to explain why.
This week in my technical support for creatives business, Tech for Togs, I had a call from a friend who is a portrait and family photographer. He called to ask my opinion on a cloud-based service, who shall remain nameless. My experience using this piece of software had not been a good one. The interface was cluttered and confusing, and there was no mobile app to check your files had uploaded to your phone. Bi-directional syncing was the only option for updates on your computer, meaning if you deleted a file off your computer, it would also delete the identical copy of the file from your backup. Also, with this software, no backup would occur if your computer was asleep or switched off. Technical support was only available through the British photographic association that provided the offer, so any problems with the service, app, or pricing would go back to the association, not the company who made it. This was unbelievable to me: who better to support an application than the company that created it?
After discussing this with my friend, there was a long pause, then “I wish I’d know that sooner, as I’ve been using them for the past three years and I’ve run into a bit of a problem.” The problem was that he’d not only lost the file he was looking for, but it looked like he’d lost the whole lot. At the time of writing, this issue is still ongoing and I’m not confident that the missing files will be successfully retrieved.
So, outlined below are my reasons for not relying on cloud storage alone and what you can do to minimize risk when you do decide to use them.
Test Lots of Different Backup Solutions
There’s lots of them available, right? Google Drive, Dropbox, Box, Backblaze, Carbonite to name a few. I personally use the Backblaze B2 option, as while they aren’t the cheapest, they offer a good app that allows you to download and check the single files you need. It also integrates with my Synology DS1821+ NAS (network attached storage) perfectly, giving me a hands-off backup solution that runs every night backing up incrementally to my cloud storage provider. All I need to do is run a check every few months that files are still backing up as they should. I deliberated for a long time and tested a lot of options before finally settling on this company, and my experience with them has been brilliant. The B2 version backs up your network storage device but they do a cheaper single computer option also.
Spread Your Risk
Putting all your eggs in one basket isn’t a good idea when it comes to backup. Single hard drives reach the end of their life or might get stolen, dropped, or damaged. NAS devices can suffer a similar fate and can be a little complicated to set up or be a bit too expensive when you’re just starting out. Cloud storage is only as good as the internet network connection you’re using to download or upload files. The reality of it is that a good backup solution relies on spreading the risk across the “Magic Triangle” solution: three copies of your data.
- Onsite (hard drive/NAS)
- Offsite (drives elsewhere or synchronized backups in another location, not your studio or house)
- Cloud (online available via app, webpage, computer, etc.)
Look at Backing up to the Cloud From Your NAS Rather Than Your Computer
Backing up from your computer and storing your pictures on the computer’s internal hard drive you’re editing on is a risky idea. The computer is handling thousands of files and calculations every second, and if the computer fails due to the many reasons computers can fail (malware, viruses, corrupted updates, drivers, faulty internal hard drives, etc.), all your images on there are at risk. At worst, this would critically impact your business. At best, it would be time-consuming and stressful, but recoverable. Cloud backup alone, depending on the provider, relies on us as humans to save, sync, or drag and drop, and we are fallible, we are busy, we have other things going on in our lives and we forget. Automating this process makes it much more reliable, not to mention less stressful!
Have a Local Backup
The second part of the “Magic Triangle” solution is your onsite backup. This might be a controversial viewpoint, but single drives are just not the way to go to do this. There’s too much pressure on this single piece of equipment being the single point of failure. As reliable as companies make out their equipment is, there is an accepted failure rate with all drives. SSDs are much more reliable, but it is worth noting that the cost per GB is much higher.
Backblaze tested its SSDs performing the same tasks it previously assigned to hard drives after a full company transition in 2018. This includes serving as boot drives, primary storage, temporary SMART storage, et cetera. After four years of aggregated data, the SSDs from a variety of manufacturers are showing a failure rate of 1.05%, well below the 1.83% failure rate of hard drives over the previous four-year period. Dramatically, the SSDs showed a 0.00% failure rate in their first year of service compared to .66% for hard drives. Even the most pessimistic projections show the SSDs outperforming hard drives for reliability by a considerable margin. Source
This really means looking at a RAID-based solution locally spreading the risk across drives. Seagate makes a great product, the LaCie 16TB 2big 2-Bay USB 3.1 Type-C RAID Array. This allows for a RAID 1 mirrored replication locally and is fast enough to edit from directly when plugged into a USB-C port. While not technically a backup if it’s the only copy you’re working from, the RAID 1 array allows a copy of the data stored on it to be stored on the other internal drive in the array. Keep in mind, if you buy this as a 16 TB array, it’s actually going to be 2 x 8 TB drives, and the Raid-1 configuration means you’ll actually only have 8 TB of useable space. When buying, always choose double the useable space you’d actually want to have available if you’re considering setting it up in a RAID 1 configuration. Also keep in mind the RAID is not a backup. If one drive corrupts, there's a good chance the issue will be copied to the other.
Don’t Just Go for the Cheapest
I don’t know about you, but if I was ever under the knife for surgery, I’m certainly not going to say: “who’s the cheapest surgeon who can do this?” I’m going to evaluate all the factors. Of course, your budget is an important consideration, but don’t let it be the main deciding factor.
Sort Out What Files You Actually Want to Back Up and What You Don’t To Save You Space, Time, and Money
This is a crucial point. These days, you can drill down into the folder structure of almost any online backup and choose what needs to be backed up and what doesn’t. Do you really want your Aunt Mary’s 69th birthday photos being backed up online where it’s costing you money? Back up only what’s important.
Hopefully, these points have given you some food for thought on how to create a backup solution that is effective, reliable, and stress-free. Considering your options and spreading the risk is key. You can always reach out to me for more support or leave a comment for me to get back to you on.
I totally agree that it is important to have 3 copies of all your files. Also, it really needs to be automated because you'll forget to do it manually. I personally use the computer internal SSD drives only to run apps.(when I set up my computer I put in 2 separate hard drives one to back up the other) I put all of my created files on an external hard drive. I then back up that hard drive to another external hard drive. I also back up all of my files, (including emails) to the cloud via Carbonite. I use external hard drives that last me about 2 years (2-3 Gigs each). I use them as they are easy to set up and very inexpensive compared to other types of RAID systems. Carbonite costs me about $100 per year. I can download any file directly or they will deliver a new hard drive to me within 24 hours if I have a total loss. A very simple and cost effective system.
Good solution Dennis Hill
Nicely summarised, Peter Morgan. I've never been able to explain it succinctly as you have.
My understanding from this article is: Work on internal drives which are backed up to the cloud, and run regular backups of this same data to a local backup drive / NAS which is also backed up to the cloud... Finally, keep offsite backups which are taken at regular intervals from your local backup.
Could you explain a little further about why a dual RAID 1 mirror makes sense if corruption could possibly copy itself onto the mirror? I've never used a NAS, or RAID.
A RAID 1 mirror will only handle corruption that happens at the drive level, e.g., a drive experiences a bad sector and ECC deems the read error uncorrectable, a RAID 1 array will simply be able to gather the error-free data from the other drive.
Errors that happens upstream, e.g., RAM stick goes bad and causes corrupt data to be written, then RAID will not do anything about it.
These days, many people are preferring file systems that have some RAID like benefits rather than setting up a true hardware RAID. The main reason is when you do something like build a PC and install 8 hard drives and set up a RAID 5 array with them, while it may work fine, and handle drive failures just fine. A failure of the RAID controller can make recovery a nightmare, because often you can't install the drives into a different PC and just have it use that RAID array from the old system.
Thus recovery becomes very annoying.
On the other hand, ZFS based systems can often be transferred to a different system without issue, best of all, in many cases the drive order doesn't even matter when physically transferring drives.
If starting fresh, I recommend building a system, especially since money can be saved with older gen Ryzen 3000 CPUs that will also support ECC RAM.
A full sized ATX board will also often mean getting 6-8 onboard SATA ports, and the PCIe slots can be used for multiple 16 port SATA expansion cards if you need to add more hard drives.
Then installing a NAS focused OS such as TrueNAS Core will make for an easy ZFS based NAS.
It is also far cheaper to build your own NAS than it is to buy a premade one, especially if you need a larger number of drives. You save even more if you need features such as 10GbE+ networking, since premade NAS makers charge a huge premium for 10GbE support, while for a desktop PC, it is a simple $70-90 PCIe card.
Furthermore, if your storage needs increase massively, you can install an old cheap used enterprise HBA card to the NAS build, then buy a cheap used JBOD DAS enclosure that a company may be getting rid of. Often many 24-60 drive enclosures can be had for cheap on the used market.
corruption comes in 2 forms really. Drive corruption caused by hardware failure and corruption caused by files saved incorrectly - hardware failure on one drive (i.e. bad sectors) wouldn't necessarily copy to a second drive as the copy is instantaneous and if corruption happened on one drive then it wouldn't necessarily mirror across to the other drive (it wouldn't damage the sectors on the second drive to mirror the first). I'd have to look into it more but thats my best guess.
Thanks Naruto Uzumaki and Peter Morgan for your input.
@Naruto, I've heard about RAID cards failing, and causing catastrophic data loss when the same model is not available as a replacement... Isn't that a cause for concern when building your own NAS?
If using a hardware RAID that relies on the unique implementations that many companies do (everyone has their own trade secret ways of managing drives while reducing CPU overhead), then a controller failure will make recovery of the array difficult.
Many pre-made NAS devices will use many unique hardware functions as shifting the drive management to a dedicated ASIC, allows them to maintain good performance while using a much lower end CPU.
With a JBOD implementation where the file system handles the redundancy and fault tolerance, a failure of a RAID card, SAS controller, or motherboard will be unlikely to impact the drives since it is not relying on the unique functions of the hardware. The downside is systems like ZFS require more RAM and CPU time to perform well.
Many modern CPUs have hardware acceleration for most of the computation done for these file systems, then many people are able to easily saturate connections well over 10GbE with entry level 4 to 6 core mainstream desktop CPUs.
This provides a good rundown of why many people like some of the DIY solutions. The plug in issues have since been fixed, and it is always best to install the OS to an internal SSD and not a USB flash drive.
https://youtu.be/P6O1mYgf0eo
Many thanks for that run-down, Naruto Uzumaki.
First of all you write a large introduction about a cloud system that clearly is neither Google, nor Microsoft or other consecrated cloud systems with storage solutions. Dear all, when choosing the cloud you don't only save money by not buying expensive equipment, keeping it maintained and becoming sort of IT admins with a lot of skills required, but you also need to always think about capacity, electric shocks that could destroy your hardware and lose all your data in a matter of seconds. This happened to me. One power fluctuation burned all 4 disks in RAID at the same time. If you're choosing a cloud option just because it's new or free, that's going to remain an option you have to consider: free is not for professionals. Google cloud for photos, which was unlimited for a long time had always the 12 mb limitation that meant a compression applied by Google over all uploaded pictures. Don't understand it wrong though: for many pictures this is a fantastic option as long as we don't need a high resolution with 0 compression rate on top of what jpg already does. But storing there is not possible for anything than jpegs. However storing versus a paid version of it, really is worth it's price. Microsoft Office personal package comes with 1TB storage for absolutely anything you want and their options about maintaining data while cleaning it up from you devices is really easy (there's an option called "keep on device / free space on device -- keep on cloud" that does the trick. Using delete button will delete the file from device and cloud too). Saving files in onedrive permits absolutely anything, including pictures in original parameters. There are more cloud options out there like Dropbox for example, but always keep in mind: free is not for professionals.
The primary reason that we should not use cloud storage is that it is very detrimental to the environment. We should also be selective about the photos that we keep so as to need fewer storage devices. Small is beautiful. Buy less.
If using cloud is detrimental, using crypto is a disaster.
Totally agree!
If you have any close family or friends in the same country, a good backup solution to consider is having a good NAS setting in both homes. Share the space in 2 separate encrypted pools. Each person is each others off site backup. While there is a higher upfront cost, it is far cheaper in the long run.
In addition to that, for the most important data, also have cold backups. Get a USB to SATA dock, and then when there are good sales, puck up an extra hard drive. In the past few years, you could get 8TB WD red CMR drives for around $130-$150 on sale.
Now you can often find 12TB drives for around $160-170 on sale. For your most important data, add an additional backup copy to an encrypted volume on the drive, and then stick the drive in a fire/ water resistant safe. A cold backup is also great for if if for whatever reason the hot backups encounter a ransomware attack (which can also impact automated cloud backups as well).
Aside from that, for your home desktop PC, it is good to have ample internal storage. Most modern motherboards offer 6-8 SATA ports, in addition to 2-3 m.2 slots. Gradually over time when you get extra funds and a decent sale is going on, work your way up to 6-8 8TB+ drives in your PC.
For your NAS build, SATA cards are cheap, thus you can easily add an 8 SATA port card to your PC when you run out of onboard SATA ports, and things like the ZFS file system will work well across all of them and provide good redundancy.
Here is a simple way to make sure you always have a reliable backup, after every photo session, transfer what files you want to work on to your PC, then make the SD card read only (the little sliding button on the side flick it upwards) date and mark the SD card with what camera it was used in, and store in safe or I recommend a safety deposit box, you will then have an uncorrupted backup of your files, once that card is made read only, nothing can be added to it or deleted from it, until you make it read/write capable by flicking the switch on it down again.
SD cards are relatively cheap, compared to taking the chance of your cloud service being corrupted, or your personal backup array being corrupted.
Remember this if you have a virus/malware on your computer, when you back it up, it is also going to be backed up to whatever system you use, no antivirus or anti-malware program is 100% capable of discovering every virus or malware program.
Please remember this,
any photos you upload to Facebook, gives them a license to use or sell it, it is in their TOS.
Also using Microsoft Onedrive, Google Drive , etc.... Gives them permission to access and read, or use.