RAID is often times one of the most misunderstood topics that photographers don’t take advantage of. So often I hear horror stories of photographers losing images or irreplaceable data from crashed hard drives. I’ve made a video explaining a few popular RAID options and a strategy to ensure your files never get lost.
Photography and video have very different requirements when it comes to data storage and backup. In this article I will be talking about mixed usages for both jobs we are shooting photography and video and some strategies that we’ve found useful to ensure the content is backed up. I will take you through our entire workflow on how data moves through our studio.
Workflow Backup
Mobile Bus Powered Backup
As mentioned in the video we use Carbon Copy Cloner to ensure files are copied to an external device every 15 minutes. This ensure that no matter what happens to my computer, the files are in multiple locations. We just began using the new All Terrain G-DRIVE ev ATC drives from G-Technology because they appear to be the most rugged and only water proof drives on the market. Not to mention they are the fastest drives we’ve ever used that are 7200 HDD. In a perfect world we would have all SSD mobile drives but due to size/cost constraints this is the next best thing. Unfortunately they aren't yet offered in Thunderbolt 2, which is twice as fast as Thunderbolt 1 but they are still blazing fast. The last thing we want to do is wait for something to backup on set, or interfere with the job at hand.
RAID 0
If we are in a location indoors, we will often times also bring a smaller and faster RAID 0 solution as well, especially if we are shooting both photography and video. We are sometimes limited to the size of the 1TB drives so we always travel with two additional 6TB Thunderbolt 2 drives that act as mirrors of each other, both set to RAID 0. If we are shooting video, we tend to backup data every 30 minutes and copy cards to one of these RAID 0 systems, which is then cloned using Carbon Copy Cloner to the second RAID 0 system. These are great solutions for speedy backup but provide no redundancy so we have to travel with 2 to ensure redundancy. It's never worth risking having your data in just 1 location. Again for us, the G-RAID Studio drives have offered us the faster read/write speeds and also have enterprise class drives installed in the enclosure. When so much money and time goes into creating a digital file, make sure you have a rock solid foundation to back you up no matter what.
If you are unfamiliar with enterprise-class drives it means that they are rated at 2.0 million hours mean time between failure (MTBF), and receive additional manufacturing and testing at extended temperature ranges to ensure a target of less than one non-recoverable bit error per 1015 bits read (i.e. no dropped frames). It's a better build quality and better manufacturing. We've yet to ever have an enterprise drive fail (knock on wood).
RAID 5
Once we get back to the studio from shooting on-location we then copy everything from our mobile drives and RAID 0 systems to our 24TB RAID 5 G-SPEED Studio system. All live media that we are currently working on for both photo and video are kept on a RAID 5 system until the project is published, delivered, and ready for archive. In the video we demonstrate that a RAID 5 system is the perfect combination for both speed and also redundancy/security. As a rule of thumb you typically get 75% of the storage with RAID 5 systems but this depends on the number of drives the enclosure has. For live editing we typically use the G-SPEED Studio. Again, they are rated with Enterprise class drives and for us, have shown the fastest read/write speeds available. We use a 24TB drive for each workstation (18TB available) for video and a smaller 12 TB option (9TB available) for photography.
RAID 5 Archive
Once we finish and deliver a product then the library and working files are ready for backup. Once this happens we dump everything onto a monster 64TB G-SPEED Studio XL that has a total of 56 TB available. We have this drive hooked up to a Mac Mini server that everyone can access through the network. One thing to note is that when you have a RAID 5 drive with 8 drives inside, you are able to exceed the typical 75% storage capacity limitations. In this case we are able to have access to 88% of the total available storage in the drive itself.
As files in medium format get bigger and video files/quality become monstrous, we’ve run into a major cost consideration with Archive storage. We did quite a bit of research for thunderbolt 2 systems (Mac Only) and found that G-Technology was actually the most affordable option for storage of that capacity.
You really have to treat your storage and archive solution as a major line item when doing your purchasing for your studio so always project your size needs each year based on worst case scenarios.
Off-Site Storage RAID 5
At the end of every job archive we also have to take into consideration fire, theft, on-site destruction. Because of this we have an additional layer of security for off-site video storage. For this, we use another RAID 5 system from Other World Computing. The drives aren’t as fast as G-SPEED Studio drives, but for the price they are a better bargain as an off-site backup. With large capacity options of 24 or 32 TB, these are a great solution for off-site storage. You can also custom build these on your own with any drives you want, as long as they are compatible to the RAID requirements. These drives remain off and don't get a lot of use, so for us it's not important to have an enterprise class solution.
If you compare the OWC cost for a 24 TB RAID 5 Thunderbolt 2 drive for $1,959 against the $2999.95 price tag of the G-SPEED Studio, its a matter of what works with your budget. For us we love the speeds of the G-Technology system and the security of the enterprise drives included. In our initial tests we found the G-Speed Studio drives to be about 20% faster than the OWC drives for live video editing read/write in Adobe Premier. Even copying files makes a big difference in how much waiting you have to do if you are constantly moving libraries between team members.
If you are doing Photography ONLY and are on a tight budget, I think you could implement some of the cheaper OWC options into your fleet of backup devices. If you are doing video anything I would strongly recommend spending the few extra bucks for the G-Technology. When it comes to video capture, it’s essential that you have the proper tools to protect your data and that fastest options to increase your workflow.
In addition to this workflow I’ve outlined in the article, I strongly recommend you follow the workflow in the above video if you are just doing photography. Adopt the 3-2-1 backup plan and sleep better at night knowing your work is backed up no matter what.
RAID is ok, but the best is to use ZFS. RAID doesn't protect you from bit rot, whereas ZFS (and possible BTRFS if you can't get zfsonlinux) will.
ZFS and BTRFS are the file system that you put ON TOP of the RAID volumes. It's like saying you shouldn't use tires because your engine is the fastest.
It's worth to note that RAID 5 is painfully slow when you have many small files. Don't backup small preview files or thumbnails :)
In general, you are correct, but this can be mitigated with proper block-size and a hard-ware raid implementation with decent cache.
Also as a note, RAID 0 is not a data protection scheme within the RAID setup, if you lose a disk in RAID 0, the rest of the data is cooked. RAID 0 is great for performance and when you want really large volumes, but not so great for data protection. I think based on the setup you are describing you are talking about RAID 1 where you are writing the same data to two or more different disk at the same time. RAID 0+1 gives the best of those worlds, but is a bummer on the number of disk you need to implement.
We have 2 x RAID 0 enclosures setup to mirror one to the other.
using Carbon Copy is not RAID. It just copying files, not real time redundancy, which is what RAID is. Even having CC performing copies at a set schedule is just backing files up. Should not be using it as a RAID solution.
I'm not using it as a RAID solution. I'm using it to backup my RAID 0 drive to another RAID 0 drive. I'm also using it to backup my internal drive on the machine to an external drive while I'm shooting. Just for redundancy. I actually use another piece of software that I didn't mention in the video called Hazel (Mac only) that watches the tethered folder . . . anytime a new file hits that folder, it copies it to the external drive immediately. But I still use CCC just-in-case.
"...we always travel with two additional 6TB Thunderbolt 2 drives that act as mirrors of each other, both set to RAID 0. If we are shooting video, we tend to backup data every 30 minutes and copy cards to one of these RAID 0 systems, which is then cloned using Carbon Copy Cloner to the second RAID 0 system."
You're wording is a bit opaque. What I think you're saying is that you have two portable enclosures with thunderbolt 2 interfaces. Each enclosure has two 3TB drives configured for RAID-0, giving you the performance you need for video. You are using two of these enclosures as "near line" mirrors of each other.
Every 30 minutes you sync the data between the two enclosures so that,at worst, you'll lose the last 30 minutes of changes if you suffer a failure in one of the enclosures.
Do I understand the setup you described above?
RAID isn't a form of backup, it's there for redundancy. If a file gets deleted or corrupted on a RAID then the file is lost forever. It doesn't matter how many drives you have. Carbon Copy is the backup element and can protect against this. A RAID backing up a RAID isn't an issue.
RAID 0 = disk striping, 2 x 1TB in RAID 0 gives you a 2TB drive, benefit speed in write/reads
RAID 1 = disk mirroring, 2 x 1TB in RAID 1 gives you two 1TB drives with same data, possible read increase, no write increase
RAID 5 = You loose one drive's worth of capacity, not a percentage. 3 x 1TB = 2TB space, 4 x 1TB = 3TB, etc. Benefit is can sustain a single disk failure and still rebuild the array and some read increase.
http://en.wikipedia.org/wiki/Standard_RAID_levels
For PCs, I would say check out SyncBack for file syncing between drives, free version is pretty handy:
http://www.2brightsparks.com/syncback/syncback-hub.html
That's all correct James, I think that's what I said? However as you add more disks to your RAID 5 array, the relative percentage of space goes up as compared to a typical 4 disk RAID 5 array. But yes, the end result is that you lose capacity equal to 1 drive. It's just a different way to say the same thing.
There is an extremely important side to all of this you did not cover, especially with respect to raid-5. Recovery time in the event of a disk failure. The larger the drives, the longer the recovery. The longer the recovery, the higher the chances of a 2nd drive failure, or unrecoverable data error, during recovery. In either of those two events, you've pretty much lost the entire array.
I agree with your multi-layered approach to backups. I use something similar in my own studio but include Amazon's S3 storage and glacier storage as an additional 2 layers of protection,especially for the "deep archives' of stuff that's more than a year old.
We use Glacier via Arc as well (for deep storage) in addition to Backblaze. This article was mainly geared toward RAID systems. But you're absolutely correct, a cloud storage option is important.
RAID != Backup, it's a uptime storage solution. If your data is corrupted you have no fall back. You really need a multi-point redundancy solution, and for large amounts of data Ultrium Solutions are still the most cost effective on the market despite their high upfront cost.
The other main reason for off-line back ups is to prevent Ransomware/Malware attacks and data corruption from spreading as they can spread via network shares.
So your primary back up should be on-site, speed is irrelevant as back ups should occur at the end of the day (or real time if it's necessary), with your generational back ups should be offsite. If you're looking at real time back ups storage you need to consider what's hot/warm/cold storage and how it impacts your day-to-day operations.
If you're going for speed you're far better to use a LAG connection, most NAS solutions support this and most managed switches do as well. QNAP and Synology are two of the best NAS solutions in the SMB market with Dell and HP having excellent SAN array options for Enterprise.
One other thing, as we approach larger storage arrays we face the increasing risk of Bit Rot corruption, this is why most Server OS systems are moving to new file systems (ReFS/ZFS and the like).
So can we please stop spreading false information about RAID being a back up solution, it's in the name what it does: Redundant Array of Inexpensive Disks. It's a big misconception and gives a false sense of security about your data. The only data that's safe is offline and offsite, properly archived.
This is probably the best advice here. Honestly, im a little perplexed why 2 raid 0 solutions are necessary for when they are out of the office. To me that just screams twice the amount of data loss. What happens if the drives get knocked around while traveling? If they are SSD drives then there wont be an issue of mechanical failure(aside from something smashing it). It just seems overly complicated for the initial copying of images. And like you mentioned with the long term storage, I would even throw in tape as a cold storage option.
If youre really worried about speed and backups while traveling, invest in 2 large SSDs. Get the Intel 730 or the dc3700 series drives. Battery backup for data loss prevention when not used. Also, look into raid 1+0(10) for your in house storage array. You loose more space but you get striped performance with mirrored backup.
Sorry people are being critical as well as I am but I find it weird that you have to raid 0 arrays that travel with you. It screams failure to me.
We're not talking about a day out of the studio here. This is for extended (sometimes week long productions) that include both stills and video, that use a lot of storage. Redundancy is the point. If one drive fails, since they're in a RAID 0 array, there is no redundancy within the enclosure itself, therefore we use a second RAID 0. It's possible, but highly unlikely that both RAID 0 systems will go down at the same time. It also allows us to have two different people travel with the units, keeping them separate until we get back to the studio.
Have you ever lost a drive while your RAID is rebuilding from a previous disk failure? If not, you will and it will HURT. RAID 6 is the way to go. You will take a hit on writes and you will lose more of that precious space for another parity block but it is worth the hit.
Lose any significant amount of data and you will have plenty of time to reflect when you are researching expensive data recovery options.
Software / hardware RAID, backups, redundancy and disaster recovery are all technologies you NEED to know and know well if you are using a camera to make a living. If you don’t you are playing with fire.
What drive becomes your 'working' drive. Do you work off of the Raid or do you work off of the small external drive?
See RAID5 section above for detailed explanation of our working drive. "For live editing we typically use the G-SPEED Studio."
Thanks for the awesome article, Tony! Is there anyway you can visually map out the road that files take from camera to final drives/cloud storage? I am understanding most of it, but seem to be having trouble visualizing all of this. I am currently building my storage workflow and could use some help. I have two mac pro towers that I can use as RAID backups, but I am not really an IT guy. Thanks!
I'm going to try and put something together for you. Give me a it of time. Thx