Storage management is a common issue shared by many photographers due to the constant increase of the megapixel counts in our cameras. Here is a detailed guide about the storage options and how to make the right choice based on your photo-video needs.
The classic 3.5” hard drives are bulky, fragile, and slow but they still find their way into our computers because they offer the best value in terms of price per bit of data. Because of the mechanical architecture (spinning disks inside), these hard drives are painstakingly slow, both in terms of transfer and access speed. Therefore, only use them to store cold files and archives but don’t install your operating system and hot files on this type of drive as it will create a massive bottleneck that will slow down your entire system. A common practice is to install two large hard drives in RAID 1 (mirror) in order to create a temporary backup system before moving your files to the cloud or an external storage solution.
Selection Criteria
Storage Capacity and Price
Current hard drives based on the PMR architecture are reaching a storage limit. The largest models can store up to 14 TB but Seagate and Western Digital, the two main players in the industry are going to introduce new technologies next year that will allow to manufacture higher capacity hard drives. I will cover the HAMR and MAMR drives soon in another article.
One key element to pick a hard drive is to select the model with the best storage value. There is a capacity versus cost sweet spot ratio for any given hard drive family. Small and large capacity hard drive tends to be more expensive per TB of data than the middle range models. At the moment, the 4 to 12 TB hard drives are within the sweet spot.
Physical Size
They come in two size 3.5” and 2.5”. Assuming you are looking for space, only pick the 3.5” as they can store more data at cheaper price. The 2.5” format used to be installed in laptop computer but they are now being replaced by entry level SSD. However the 2.5” hard drive remains a common format for external USB powered hard drive (3.5” external drives are bigger and must be plugged to a power source).
Spinning Speed
As the name indicated, spinning hard drive contain spinning disks inside. At equivalent capacity, the faster they rotate, the quicker they can read data. The standard spinning speeds are 7,200 RPM and 5,400 RPM. As a rule of thumb, faster disk tends to be a bit nosier, pricier and less durable than the slower one but there are many exceptions depending of the category of product (consumer or professional grade). Usually, 5,400 RPM drives are enough for cold storage and archives but 7,200 RPM is faster.
Cache Memory
This is the buffer of the disk. Higher cache doesn’t necessarily translate into faster transfer speed but it may help for multi-tasking operations. This is a minor feature for our cold storage purpose. Usually, manufacturers integrate an appropriate amount of cache memory based on the specifications and capacity of the hard drive.
Hybrids SSHD
Some hard drives combine the traditional spinning disks with a small SSD to boost the performances. Typically, 8GB of NAND flash memory is added to the disk in order to create a large buffer and improve the access speed. Hybrid can be a good solution in some computers but they don’t substitute for a proper SSD and don’t impact the transfer performance of large files. Hybrids SSHD are not very relevant for cold storage purpose.
Helium Filled Drive
A few high capacity hard drives are now filled with Helium instead of air to reduce the internal friction of the moving parts and allow for higher density models.
Green Model
Green is a label used to define entry level model with low power consumption and performance, typically spinning at 5,400 RPM. Western Digital did some rebranding in the past and put the Green denomination under the umbrella of the Blue line. Current 5,400 RPM Blue hard drives correspond to the former Green offering.
MTBF
MTBF stand for mean time between failures. It’s a measure of how reliable a product or component is and indicates the predicted elapsed time between inherent failures of a system. This value is not a guarantee but a relative indication of durability (kind of like the shutter rating on cameras). The higher the number, the more reliable the product is. Entry level hard drives come with low MTBF while the NAS and professional versions can reach up to 2.5 million hours of MTBF.
Versions and Performance
For convenience and because Seagate and Western Digital are the two main hard drive manufacturers, I’m going to focus on their offering. As you can see in the chart below, they offer similar products at similar prices. The main difference between the consumer and professional grade models is related to durability and higher MTBF but professional models are usually more expensive. Here are the main categories of hard drives:
- Entry level: the WD Blue and Seagate Barracuda drives are cheap but offer low performance (5,400 RPM).
- Intermediate: the WD Black and Seagate Barracuda Pro are the standard “Jack of all trade” consumer hard drive. They are faster (7200 RPM) than the entry level and comes with a higher warranty period (5 years instead of 2 years).
- NAS: the NAS drives are designed to be installed in NAS and server enclosures. The emphasis is placed on durability and continuous operating with a much higher MTBF compared to the regular non-NAS hard drive. Seagate proposes the IronWolf version while Western Digital call them Red. Each brands also comes with a “Pro” version with higher MTBF and/or warrantee.
- Video surveillance: the WD Purple and Seagate SkyHawk are optimized to withstand the particular condition of video surveillance application where the drive spends the vast majority of its time writing.
- Ultimate: when absolute performance is required, both in terms of speed and reliability. These hard drives like the WD Gold standard or Seagate Exos don’t come cheap.
That being said, which hard drive should you pick? Is it possible to install a NAS drive in a desktop computer or a regular hard drive in a NAS?
Simply put, always choose the right type of drive for the given application or use-case. Even though a NAS drive can be installed in a regular desktop computer and vice versa, the best practice is the follow the label of the product. Getting the most out of the drive depends on having the right model for that environment.
Technically NAS and regular desktop hard drives are very similar but there are major differences in the firmware (the internal software that controls the drive). A NAS drive is designed to be up and running 24/7, so its priorities include constantly being ready to go at a moment's notice, like in a cloud storage environment where you might have a number of users who want to access any information held on the drive at any given time, and not have to wait a long time for access. NAS drives, as mentioned, are rated for continuous use and also designed to push bigger workloads year round. They are also designed to handle more vibration than a standard desktop drives because server environment is rough. Imagine eight huge 8TB or 10TB drives all stored right next to each-other in an enclosed server rack. Therefore, the firmware on NAS drives is designed to account for, protect against issues with, and manage these vibration considerations. However, NAS drives can be quit noisy and they wouldn't be best to fit in a desktop environment. For instance, NAS drives don’t aggressively fix some errors, figuring they can just pass them on to the next drive in the RAID array for faster performance. On the contrary, a desktop drive aggressively corrects these errors because it isn't intended to be used in the same way.
Long story short, here is what I suggest:
- Install a NAS hard drive for RAID and NAS setup. The WD Red and Seagate IronWolf drives are good for this type of use. If the prices are similar, go for the Pro version as they offer a longer warrantee period.
- Install a regular version for desktop use like the WD Blue or Seagate BarraCuda. If budget permits, get the faster WD Black or Seagate BarraCuda Pro version. They also comes with five years warrantee against two for the basic model.
Reliability
Manufacturers don’t publish the failure rate of their products but BlackBlaze, one of the largest cloud storage service regularly publishes the failure rates observed in its server farms. Despite what people say based on their personal experience, there isn’t much reliably difference between manufacturers. Some models were notoriously bad in the past like the 4TB from Seagate but actual statistics show similar results and both brands can be equally trusted.
In any case remember to always save your data on separate locations such as different hard drives or to the cloud. Hard drive fail and the probability of failure increases over time.
With pretty much all new computers coming with smaller SSD drives for speed, what kind of drive do you recommend attaching to your computer as an external to store say your Lightroom library, or your video files for iMovie, etc. Is a 5400 RPM drive sufficient for LR, or do you need more speed? All of my photos and video will currently fit on a 2 TB drive, so I don't need a huge amount, but I'm wondering how to handle things once I transition from my current computer to one that only has a 256 gb SSD.
Hi Justin, a few remarks:
- 256GB SSD is very small, if budget permits, get a least a 512GB
- A 2TB external hard drive doesn't fall in the "sweet spot" in terms of price per data (see my chart above). But if you only need 2TB it's fine.
- To answer your question: 5400 RPM drive are enough for LR and basic video files. The vast majority of USB powered drives (WD My passport, Seagate Backup) are 2.5" format at 5400 rpm. Some external 3.5" disks are 7200 rpm and they'll be a little bit faster to transfer large files. But they are also much bigger in size and you need to connect them to a socket. So it depends what you need.
I use 1tb SSD drives in all my machines and 5tb drives as secondary. I would not use a non SSD drive anymore. Too slow.
SSDs drive are a must for active files and operating system but they are very expensive. For cold storage and backup, regular HD are good for the money.
Hi Dan. Out of curiosity, why do you use 2TB drive to store 6TB of files per year? If I understand correctly you buy 3x 2TB (6TB) for A copy and 3x 2TB (6TB) for B?
It would be cheaper to buy 4TB drives instead of 2TB. You do this for spread the risk of disk failure?
Got it, tks
I'd argue a"Virtual Nas" or Nas as a service is better than buying HDDs.
This only works for those with fast connections. 10mbps with 1mbps upload is out of the question.
Paying $5/TB/month Vs buying and maintaining disks.
Using a 500gb SSD for £100 (Samsung 860 Evo) is very low cost for speed.
As for backup there are companies offering cloud storage backup solutions which is a good idea.
As a long time I.T. professional, also disparagingly referred to as a techy, I should point out that the "Professional Grade" tag on the chart is slightly misleading. All these drives are very much consumer grade. We do not typically use SATA drives at all. Typically we are using SAS drives fro spinning disks with the slowest ones running at 7,200 RPM, then 10,000 RPM, and finally the 15,000 RPM drives. As more drives are added to a SAN or local attached storage you get more performance as well. The last server I built for storage has 20 SAS drives running in a RAID 6 configuration. Even though they are spinning disks there are enough drives that they easily can handle the 1 Gb/s network speed the server was designed for. Storage gets very complicated very fast in an enterprise environment.
What is probably the most relevant thing about spinning disks is that for consumers the spindle speed has been mostly stuck at 7,200 RPM for over twenty years now. Performance has been essentially stagnant and only capacity has truly increased. So pay attention to the MTBF rating if one is published, do not pay a premium for something labeled professional grade, and make sure your data storage can handle at least one disk failing without losing data.
Thanks for the feedback. Obviously the "professional grade" are just a marketing thing. Real pro like you use more specific hardware. But for consumer the technology is relatively stagnant. I remember having my first SCSI drive in the 90s and the the raptor at 10,000 rpm. Now 7200 rpm are already "fancy". Only the plater density has increased.
And we agree, actually I didn't recommend any of the "pro" drives in this article. Personally I think that the "consumer" one are enough, especially since the data should be saved over several disks. Now, if you can get a "pro" model with double the MTBF for a couple of $ more, why not.
Out of curiosity, did you receive some of the early HAMR or MAMR drives? WD and Seagate said they released a few thousands to selected customers. Perhaps you don't even use these two brands.
Agreed that we agree. I was quite happy to see that you were not recommending the most expensive drives. We did not get an opportunity to play with any HAMR or MAMR drives at work. Honestly, I have been moving more and more out of managing storage day to day. We have petabyte storage arrays, but that just is not that exciting to me and backup strategies are important as well as sleep inducing.
If you want to check out something really exciting in storage that I recently saw a presentation on from VMWare at VMWorld, I am really finding Intel's Optane DC to be one of the most exciting changes in computer technology in some time. This is different than Optane. It is a DIMM form factor using the memory controller, but with Intel/Micron 3D-Xpoint persistent memory. It is persistent memory with more than 100 times less latency than NVMe; nearly RAM speeds.
https://newsroom.intel.com/editorials/re-architecting-data-center-memory...
(We don't normally specify a brand of drive in servers. All of the Dell's I have managed over the years have shipped with Seagate drives. I like Western Digital products for my home/photography use.)
Thanks Mark for this great feedback and the link. I'm going to cover SSD, NVMe and Optane in another article.
Okay. So sounds like I can use internal SSD for operating system and apps, and keep my Lightroom library on an external drive, with a 5400 rpm drive being fine. I will import new RAW files to Lightroom which will put them in the library on the external drive and edit them from the external drive. The 5400 rpm drive will not affect performance. Am i understanding that correctly?
The 5400 rpm will affect the performance. Self powered 2.5" external drives are usually slow 5400 units. That's being said, it's enough for LR use and it won't impact your workflow in negative way except during the import process or if you really have high resolution files (eg. 5Ds, D850). But again, I personally have a lot of stuff on these HD and this is not a deal breaker for LR. It's more important for high resolution video editing though as you may create an access bottleneck. But I'm talking about 4k+ files encoded in high bitrate (ProRes, etc.).
You will be fine for your type of use with LR.