When it comes to storing data, photographers have it easy. What if you’re a filmmaker shooting on multiple cameras across multiple projects for multiple clients? Resolution is increasing by the year, and fortunately, hard drives are also growing in size, but what do you do if you have 200 terabytes of data to look after?
Gene Nagata (a.k.a. Potato Jet) runs you through his situation and explains his reasoning, immediately rejecting a rather expensive option: the Lumaforge Jellyfish Tower. I’ll let Nagata reveal the price for you, but be assured: it’s not cheap.
The insights into all of the different gear in this video are incredibly useful, as is the breakdown of convenience, versus storage, versus cost per terabyte. All of that said, perhaps the most important takeaway from this video is the sheer amount of time and effort invested into making sure that the data is not simply secure but also carefully organized.
If like me you suddenly feel the urge to splash out on a label printer, even the digital ones aren’t too expensive. For example, this Brother PT-M95 is just $24 from B&H Photo.
What’s your system for storing vast amounts of data on a super tight budget? Leave your suggestions in the comments below.
That is why IMO:
4k is not always advisable, more than that is just flexing and not worth it.
Tape storage for cold storage is still as relevant today as it was a couple decades ago.
I dunno man, tape is still kind of unproven. I save all my work on punchcards and haven't lost a single bit of data yet!
He's one of the few consistent YouTubers who puts out quality work.
the fun part of being a photographer or videographer is capturing the moments.. the un-fun part is the business side of things.. and the technical side of things. You need to factor in "the cost of doing business" that means storage, backup, and insurance (if something goes wrong)..
if you're doing 200tb worth of business.. you need to be charging enough to store 200tb worth of business.. and back it up.
"cheap solution" - synology 12 bay NAS = 1500.00
16tb drive x 12 = (499.99) x 12 = 6000.00
That'll get you to 176 usable terabytes.
if you really need more you can get a 12 bay expansion chassis for 1200.00
add drives and expand it as you need it.
its a big investment to start out with.. but it is expandable and redundant storage. (not a backup.. but can survive a drive failure)
if you are doing 200tb worth of work.. you should be able to factor in this kind of purchase every 3-5 yrs along with drive upgrades.. again its just the cost of doing business..
you will have a drive fail.. its just a matter of time.. and the cost of reshooting.. or drive recovery .. or the lawsuit.. could easily be 2x to 3x this.
You also need a workflow that allows you to keep track of your data... probably more than a label maker
I've not yet seen the video, so I could be putting my foot into my mouth but... Tape. This is the perfect application for tape.
Have a NAS with enough space for current and recent work, the rest goes to tape. LTO-8 holds 12TB uncompressed per cartridge and is borderline affordable for a small business - An internal drive is only $3450US and tapes are only $160US and those are New Zealand prices converted to USD, so it's probably even cheaper in the US. Being tape, it's also small, fairly rugged and very reliable.
Tapes need proper storage conditions, otherwise it's just another (or a different) disaster waiting to happen. There are commercial storage space providers also for tapes, the monthly costs should fit even the ills of a small scale photographic business.
i've been burned by tapes way way too many times to ever trust them again. I've worked in the technology industry for 20+ yrs. if you factor in the price of the tape drive and tapes.. i'd rather just buy spinning disks. Faster, more reliable, and i dont need a special drive to access them.
To put it simply, this is terrible advice and that's me being nice about it.
Even asking around on reddit.com/r/datahoarder would have been a better place to start than settling on just the $5k Jellyfish.
The moment you cross the 50TB threshold, you need to consider a few racks of storage servers as it will offer redundancy, speed, and reliability.
The 12-bay Synology Rackstation is the best place to start especially when one has expansion in mind.
I repeat, this is a disaster in the making, this guy's advice is terrible, absolutely terrible.
Source: I work in the data storage industry and this is NOT how to store 200TB of footage, reliably, securely, or redundantly.
couldnt agree more
First of all, there is a difference between availability (think RAID) and backup (think what if my house burns down or if I delete a file and then discover some time later that I need it). An intriguing choice is something like Amazon S3 in something of a staged approach. Stuff you are working on/likely to need local, stuff on S3 for like 6 months that gets migrated to Glacier after that. It's off-site with 11 9's of reliability (that's a 99.999999999% chance it'll still be there if you need it). It takes a while to get there but it's somewhere else in case disaster strikes. They even offer something called Snowball where they'll ship device(s) to you to get started if you have a lot of data. And, like another poster suggested, if you have 200TB worth of data, then this must be factored into the cost of doing business which created that 200TB.
Agreed, off to the cloud with these loads. The monthly costs are cheap - up to the point when one wants to get the data back. But if cloud is just the last stage of a proper data storage concept then it should work well. Clients who need data from donkey years back will also pay the bill for data transfer back from the cloud storage.
Read the fine print on cloud services like amazons glacier. and S3.. they charge you for the upload, the storage, and the download.. it adds up fast.
glacer charges you .01 per access per gigabyte.. sounds good.. but if i understand it.. thats upload, store, access, download, or delete.. so some quick math..
1024 gb in a terabyte = ~10.50 per terabyte per month.. so thats ~2100.00 just to upload the 200tb.. and 2100.00 a month to store it. (i think) and then they charge you to access it.. you cant edit it on their system.. glacier is meant for backup and old data. like cold storage of data.
Things like backblaze and carbonite say "unlimited backup." i dare you to try to backup 200tb.
thats not even opening the discussion on the transfer times.. yes you can use amazons import/export service with drives.. but they still charge you for the data transfer rates.
If you cant afford a 1 time 10-12k charge, i assure you you cant afford amazons pricing.
https://wasabi.com/ is one of the cheaper more aggressive cloud storage companies right now.. they charge 5.99 per month per terabyte.. so.. $6.00*200= 1200 a month *12 = $14,400 a year.. and thats IF you can get the data there.. i would think you'd want to edit this stuff.. so you'd be sending it back and forth.. so you'd need a stupid fast internet connection.. (faster than a gigabit)
I typically like his content and I am subscribed but he is admiralty ignorant about what he is doing and in this case no one should be following his advice on this subject nor should it be published here. Without an underlying filesystem protection like ZFS and redundancy in an array of discs you can literally put a file on a drive and spin it up a year later and it doesn't exist or is corrupt, this is especially true for video files which can break easily.
My photo storage is nearly the same price as his but is way more redundant. a NAS server which holds 8x 12TB drives in a Z2 array for 96TB of storage, about $2k. I use Lightroom, all photos are stored as converted DNG files which include a built in checksum. The underlying OS and Filesystem on the NAS handle any issues with bit rot or random file corruption between disks as it is constantly looking for inconsistencies. Files are written to the drive once, in one place and stay there forever. Lightroom has a DNG verify function where it looks at each file and individually confirms the actual file is still valid. Now I know my local files are valid and not corrupt in any way. All of these are backed up to the cloud as an offsite and backup solution on a daily/weekly/monthly basis and I am also taking "snapshots" of my local copies. This means that if I get a ransomware virus that starts encrypting my files I can easily and quickly roll back to a known good state locally without needing to even pull cloud backups. I can also locally or remotely recover deleted files. I know I can trust my local storage, my backups, and the photos themselves. Anyone can buy or build one of these systems in a weekend, with a following like his I'm surprised someone hasen't... or maybe thats the next video?
Brett, thanks for your informative post. I've recently implemented a NAS and cloud backup for my video and Lightroom photos. I was unaware of the DNG built in checksum - definitely going to check that out as I am now backing up camera raw.
You talk about storage - but where are the backups?