I can remember when I first got my 36 MP Nikon D800 a few years ago. I actually bought three of them, and I took them out with two assistants to shoot a 10-hour wedding... in raw. We came home with around 3,000 images. That worked out to 180 GB of files I had to transfer, edit, and then save forever. It was a time-consuming process to say the least.
As technology advances, our need for data storage has continued to grow. Our camera's files are bigger than ever, and our video cameras shoot more frames per second at a higher resolution than the most expensive Hollywood cameras could just 10 or 15 years ago.
Storage has certainly gotten cheaper and more convenient. We use a Synology NAS box in our office as a server for all of our computers to connect to. We currently have 27 TB of storage on, it but that isn't necessarily the problem. The speed at which we can transfer data hasn't kept up with the sheer volume of content we are able to create and need to access.
Yes, solid state drives are significantly faster than standard hard drives, but their storage capacity and cost haven't made them practical for standard data storage. Loading 200 gigs of raw files into Lightroom is slow. Editing 4K or 120 fps video footage isn't possible from our NAS box without rendering the footage first. Transferring footage from a server to a local SSD to edit and then transferring it back to the server is also extremely time-consuming.
I just ran across a funny and informative video about the exponential growth of data that humans create as technology advances. We take for granted that our computers today can easily hold a few terabytes of data. It's impossible to understand how much data that really is.
I have no doubts that humans will find a better way to store the incredible amount of data that we will create in the future; I just hope they can come up with something soon. These spinning hard drives simply are not fast enough to keep up with what my current gear is able to produce.
I feel your pain. This is not evolving fast enough compared to the rest. I did a bullet-time project and within one week, I created 4TB of data by shooting RAW + stop-motion with 32 dslrs... and of course I do backups. I'm starting to get lost under a pile of hard drives.
WTF is that point of that video exactly? It kinda annoys me when people throw out the phrase "big data" randomly. It's like no shit we're getting more data and the data is big. The data has always been big. Back in the days when we used to use backup tapes, the tapes were so big they took up all the space in the datacenter. Now we have magnetic and sometimes solid-state drives in the datacenter, but it's the same thing: most of that is data.
To quote your last sentence, "These spinning hard drives simply are not fast enough to keep up with what my current gear is able to produce." <~~~ we're not using any spinning device to store the media that your gear produces, at least not directly anyway. SD/CF/XQD are all flash media; there's no spinning part. As for storing/archiving your photos and videos, hard drive manufacturers are coming up with these 8TB drives and more, but they're only spinning at 7200rpm. If you want them to spin at 15k rpm, or you want there to not any spinning at all, they do make server-grade SSDs that have very high fault tolerance. They are of single-level-cell SSDs, as opposed to multiple-level-cell SSDs that consumers normally buy. The question is, can you afford them?
Of course working on your photos/videos isn't possible from your Synology, it's not what it was designed to do. You're essentially working off data that's transferred over 1 Gbps wires to an array of drives that write your data across the drives plus a few redundant ones. If you want to keep the same NAS-like workflow setup, get some server grade stuff like NetApp, install some fiber optic cables in your setup, and get some SSDs in that NetApp. If not, edit your photos/videos off of a local drive and then move the project to the Synology once you're done. That's what Synology/Drobo and any other consumer NAS device is designed to do: archive.
FYI I haven't had coffee yet.
Btw this video was made 6 years ago, and Chase had already had server-grade equipment in his office, and he wasn't complaining. https://www.youtube.com/watch?v=Y-6EQo6it7Y
He also wasn't editing 4K video... ;)
actually.. yes he was. they show them using the RED camera which was 4K and beyond.
Both Synology, QNAP and others have expanded into the enterprise segment years ago (their rack-based systems are very capable indeed), but you briefly touched the actual problem: Connectivity, not disk speed.
Even with 10Gbit/s ethernet, real-time editing probably isn't really possible for 4K video unless you run two NIC's between your workstation and the storage, and having multiple users working simultaneously on the same storage will most likely be out of the question - which kinda defeats the purpose of a NAS.
I think I would look into having all raw media on a huge-ass SAN running off 10K Raptor SATA disks with 10Gbit/s network to all workstations for fast copying, and do the editing on their local SSD RAID (like you suggested). Editing over the network is simple not feasible.
I love that the keyboard isn't plugged in. Took all the realness out of this video... :)
Apple keyboard is bluetooth dude.....
Ha. It was supposed to be a joke on the video, nothing more. I'm not sure if people knew exactly what I was talking about in the video. Added a pic...
Hey Floppy Disk!
(well.... 3 1/2 hard disk....)
I was running into a pretty critical storage crisis this winter. I work on an iMac, and buying another external HDD was becoming stupid. (I always backed things up, but my system was kind of a nightmare that i dont even want to get into here.)
My dad snagged me a few 1st-gen Intel Mac Pro towers from his office, which was a huge help. I networked them and put 12TB in each, one with faster drives than the other to save a few bucks. Then I moved everything from before 2014 down to the faster machine from my external HDD "array" and then backed those files up to the 2nd tower.
My next step is to get a fast Thunderbolt RAID as a working disk, use the older firewire drives as back-ups for that year's files and expand each tower to 24TB. The towers aren't practical for work, but I can send batches of raws to Amazon storage from them each night. I'm eventually going to have year-specific LR libraries and map them to the shared drives.
Of course, I only shoot stills. You video guys are screwed.
I've been in the middle of researching the perfect NAS system for several months, now. And while I thought Synology was the answer, I quickly realized it would not work for the reason in the article. However, I've found the answer.
Currently, there is only one system in existence that can handle networked connections (for when you're on the road) simultaneously with Thunderbolt connections. It's an 8-bay NAS/DAS box that QNAP came out with at the end of last year called the TVS-871T (the last 'T' is the key, here). https://www.qnap.com/i/useng/product/model.php?II=198
I'm pretty sure that's the answer to all of our nightmares as of late. Took long enough...man! And it isn't cheap. But for what it does and considering the alternatives (none, really), it's not expensive either.
Yes, I like the Synology system a bit better. But I think I could easily manage to enjoy the QNAP software, especially considering I wouldn't personally use it too often, since I'd do most of my stuff over FTP most likely while I'm gone or treat it as DAS when in the studio.
First, I'd say that a 36MP camera for a wedding (let alone THREE) in general is just overkill. For the hero shot portraits of the bride and groom? Yeah, sure, maybe. For EVERYTHING ELSE. Noap (that's no). 12 to 16MP is more that enough for most clients who aren't going to print larger than 20x30". 180GBs of data for ONE wedding make my stomach hurt.
Second, I'd say for weddings, family portraits, senior portraits, heck even product/commercial stuff, you don't need to save every freakin' file FOREVER. I keep whole session files on disk for ONE YEAR. Then I delete/dump everything except the absolute hero shots that I'd maybe want featured in an article or exhibition about me when I'm old and famous. The only files I would absolutely keep forever forever forever are my personal photographs of my family and even THAT gets heavily culled and edited down like it's a pro job. I think part of the problem is that photographs think that they have to keep EVERYTHING FOREVER and there's simply no need for that if you are editing stuff down. Even back in the film days, photographers only kept negatives and contact sheets for their really big historical jobs.....everything else they got rid of or they'd risk being buried alive under a mountain of silver and plastic. It's ok to delete files guys. You. Will. Not. Die.
It's pretty simple.
Long-term storage ( "safehouse" ) = HDDs ( with redundancy or 2 backups in case of paid work that needs to be always available ) [ have them uploaded to an online cloud service to be 100% safe ]
Short-term storage ( pre-editing, editing phase ) = SSD array.
If you can afford $10.000+ on camera gear, surely you can afford 2.5k for an Areca/LSI/Adaptec controller + multiple high-grade SLC SSDs ( MLCs will suffice too ).
"The glass storage discs can hold a whopping 360 terabytes each, are stable at temperatures up to 1,000°C (1,832°F), and are expected to keep the data intact for 13.8 billion years at room temperature (anything up to 190°C, or 374°F)."
This is awesome for 16K video. ;-)
http://petapixel.com/2016/02/16/glass-disc-can-store-360-tb-photos-13-8-...
Lee,
My answer to your question in the title is "I don't know."
However, in the article, you lament about the speed to transfer data. I offer these recommendations:
I think the technology in your office needs to be upgraded. You need to install fiber-optic for support of a gigabit network and there are also upgrades to the computers and NAS to support gigabit networking.
Because of the huge volume of data involved for photos and movies, you also need to install a "rendering farm", Pixar uses rendering farms.
However, based on my experience in the technology industry from college in 1971 and in the workforce since 1976, developing computer applications, these things have happened and will happen again:
1) File storage changes. Gone are the 8 inch, 5.25 inch, and 3.5 inch floppy drives. When personal computers first got hard drives, the storage capacity was in MB, now it's GB and TB. Optical drives are the CD and DVD. What's next in optical storage?
2) File formats change. There may become a time when the JPEG organization decides that prior standards of JPEG are not compatible with the new version.
3) CPUs change and die. Apple once powered their computers with the IBM Power PC, now they use the same chips that powers Windows computers.
4) Operating systems change or die. Remember DOS? Unix has been around the longest since the 70's and Linux is a clean slate of Unix. Windows is 1990.
5) Companies change or die. I have seen a number of companies bought out or die. Yes, Adobe is going gang-busters; but what if 50 or 100 years from now, Adobe is no longer a company. Who will support their software?
I don't have a "crystal ball" to predict the future, otherwise I would short those companies in decline and invest in the new IBMs.
The 5d disc!
It can store 360 terabytes and has the size of a coin.
http://www.engadget.com/2016/02/16/5d-discs-can-store-data-until-well-af...
Would love for you guys to do a writeup on the your NAS setup and suggestions for accessing data from multiple computers. Since getting A7Rii bodies last year and dipping a bit into 4K I'm feeling the pain now. Have 4 Thunderbolt drives chained as is, but I'm starting to wonder if there's a better solution for expanding both, the working and the backup drives.
If you daisy chain a bunch of them and one dies, does the whole thing dies, e.g. none of your data is then inaccessible?
No, no RAID 0 or similar setup, just chained in a physical sense, besides, each one has a redundant drive on it's own. Plus I do a monthly backup off-site (when I'm not too lazy, admittedly). But still, as I'm looking at my LR and Catalyst Prepare libraries grow I'm resorting to external storage despite having relatively new iMac and MB Pro. The latter of which is basically used to just initially ingest and sort data because it's a faster SSD and I'm constantly finding myself transferring data and being a whole of a lot more picky during culling... which, actually, is a good thing for me.
I've been shooting a lot of 4K to learn motion work and it's just a PITA to deal with lot's of footage that I know I'll throw away after tinkering with it... I just get annoyed at having to do it right away because of lack of space. Maybe I'm just too much of an amateur with my total external storage of just 8TBs not counting the off-site backup.