In my journey to separate my family life from my work life, as detailed in my last post, a change in my work environment has been key. Namely, my wife was tired of seeing my hard drive sitting on the kitchen island and I was handily kicked down to the basement. I took this opportunity to switch up my workflow from using a local external hard drive to a NAS (network-attached storage). Here are some interesting things I've discovered along the way.
First and foremost, I have to acknowledge: I've been pretty dumb about storage.
I had no real backup. All of my files were hanging out on a 6 TB LaCie Thunderbolt drive in RAID 0. Dumb. If something were to happen to that drive I would have been royally screwed. This is totally not a knock on the drive itself. It was a beast. It had super fast read and write and I had no complaints. I also had no redundancy with my data.
When I moved my work area to the basement, I didn't want to bring the same bad habits with me. I decided to do some research into NAS drives. Now, I've built many computer systems in my day, but when it comes to networking I'm a rank amateur. As such, I wanted a drive that would simply work. I didn't want to have to fool around with too many settings and I wanted backup of my shoot files to be automatic. I ended up purchasing a Synology DS215j NAS.
I settled on it because of a few factors: Price, reviews, and — uh — price. Did I mention price? I'm cheap. I also picked up two Western Digital 5 TB Red Drives that I would be setting up in the drive. I've never had WD drives fail on me so I went with what was familiar. I'm sure there are options out there that are just as good.
After popping the top on the Synology, inserting the 2 drives, and closing it up (roughly about 10 minutes of work), it was time to plug this bad boy in. I was pleasantly surprised at how simple the setup was. I hooked the NAS up to a gigabit Ethernet port on my router, went to the web address printed in the manual, followed the instructions, and I was formatted, up and running in about 20 minutes.
Now for the fun part. I had roughly 3 TB of shoots to put onto the drive. Easy! I figured I'd just mount the NAS to my Mac (no problems there), move the files over in Finder (check), go to bed (I'm good at that!) and all would be done when I woke up in the morning.
Nope.
Apparently my transfer had gotten about halfway through during the night, then the NAS reset to apply some updates without my permission. Oy. Alright, I changed the permissions on the NAS to ask before updating and away I went. Thirteen hours later, I'm in network nirvana! I can open my files in with read times only slightly slower than my local LaCie drive was giving me.
Next step: I'll just put my Lightroom catalog on the network and fire that bad boy up to check how Lightroom performs on the network.
Nope.
Apparently, unless you hack your own computer, the Lightroom catalog has to be local to the machine. No network drives allowed. You can store the files themselves anywhere you'd like, but the catalog has to be local.
I've got a flash drive coming in the mail so that I can take my Lightroom catalog with me from machine to machine with no issues. I'll update when I receive it.
In the meantime, I wanted to check the import times I'd be getting. Long story short: Lightroom is slow. Really, really slow. At first I thought maybe I had done something wrong with my install of the NAS, so I set up a little experiment.
I downloaded Capture One and did imports with 27 files from an SD card with both Lightroom and Capture One.
Capture One imported the files in half the time. Since I was using a crappy USB 2.0 reader, I thought maybe that was the issue. My shiny Lexar USB 3.0 reader is still in transit. To make sure the slow reader wasn't the issue, I copied the 27 files directly to a desktop folder and did the import to NAS from there. Again, Capture One finished in half the time.
Zooming in to the photos in Capture One was also an order of magnitude faster than Lightroom. It was taking forever for the images to render fully at 100 percent. I have a feeling Capture One will be getting a lot more love from me in the near future.
Some perks I didn't count on with having this NAS were the ease of having access to my files from anywhere. With free apps, I can pull up any files I want to my cell phone or tablet at any time. Amazing! I'm sure this is old hat for all the tech-heads, but I might have to cancel my Dropbox subscription. I could use an extra $10 a month in my pocket. Also, I can set up folders for my family photos and files and give access to the wife for certain folders.
I still have tons more to learn about working with the NAS, but I'm really enjoying the safety and simplicity of it for now. Do any of you have tips for dealing with network storage for newbies? Are there any apps for the NAS that can make my life easier? Is Lightroom this slow for everyone else or am I missing something? Let me know!
Oh god, please no. Just no. This article is a cry for help with a bunch of ‘what not to do’ points. Please take this down before you steer people onto the wrong path. Clearly you don’t know what you’re doing (which is fine, every starts somewhere) but please don’t write articles like this making it seem like these are good approaches. Fstoppers why in the hell would you publish something like this? This is seriously detrimental rather than helpful.
UPDATE:
All of the down voters and commenters are right. I should have approached this with advice rather than criticism, but I only caught this article moments before I had to fly out the door and was so agitated (and still am) that it was published on this type of site that I ended up venting to try and get a warning up in the absence of time for a proper response.
To clarify I have absolutely zero issues with someone undertaking this kind of project and blogging about it, writing an article about it, etc. Everyone has to start somewhere and as the author wrote in one of this comments sharing the experience is a great way for people to learn. My frustration with the article is this… Fstoppers has caught flack for fluff articles before, but worse than fluff this article could put readers photos and maybe even business and livelihood seriously at risk. Readers come here for advice, often from industry professionals. Those who don’t know better could easily be inclined to start following this advice and may or may not read the comments for clarifications and corrections. This, I wholeheartedly believe, is providing misinformation and a disservice to the reader base.
Now that we have that out of the way, here are some of the issues with your setup.
1) “I had no real backup. … If something were to happen to that drive I would have been royally screwed.”
Sorry to say that you’re not much better off now. I must stress something that was stressed to me many years ago. RAID IS NOT BACKUP. RAID is only designed to reduce/prevent downtime. You still have a single point of failure in several regards. The only thing that the NAS could protect you from above and beyond what your external drive did is a hardware failure of a single drive within the NAS. It will not protect you any better from file corruption, accidental deletion, user error, fire, theft, or even a total device failure, etc, etc.
2) “… the Lightroom catalog has to be local to the machine”.
Correct, and rightfully so. The read/write from the catalog has some of the biggest impacts on overall light room performance. As you edit Lightroom stores the data inside the catalog, in real-time. This includes data intensive adjustments such as the Local Adjustment Brush and others. Storing your catalog on anything other than a fast internal disk is just a bad idea from a performance perspective which is likely one of the reasons (maybe the main reason) Adobe attempts to prevent you from storing the catalog on, say, a network drive. Even if you get this working (which you can without too much trouble) you will (under most circumstances) take a performance hit.
3) “I've got a flash drive coming in the mail…”
When I first read this I assumed you wanted to store your catalog on the USB flash drive all the time, especially since you were talking about storing the catalog on a network location. Re-reading it I’m not so sure. Regardless this is also a bad idea. See point 1.
4) “Capture One imported the files in half the time.” / “Is Lightroom this slow for everyone else…”
That’s great, but it has nothing to do with the NAS. You even somewhat proved it with your additional test. This feels like an irrelevant distraction and probably doesn’t need to be in the article at all. Again an open call for help in something that should be in a forum in the first place.
Honestly it just sounds much more like you’re reaching out for help in general with this article and doing through a larger platform like Fstoppers because you have access too it rather than doing it in a user forum where it should be done or just reading the many, many, articles and discussions that already exist on the topic.
lolwut did someone piss in your breakfast?
You should probably pick a couple of the bad points in the article and tell why you think they are wrong so you don't seem like as much like a stupid person that doesn't know what he is talking about.
Yes please post some of your ideas instead, keep it positive :)
Sometimes, the best way for people to learn is to show their own process and the things they learn along the way. I could have spent time doing all of this in anonymity, but I believe that the best way to benefit from a community like Fstoppers is showing that process to the public, warts and all. When I get solutions to my questions (as evidenced by the many comments below) everyone benefits! I think that's far more productive for the community as a whole.
This is exactly what discussion forums are for. Not articles on sites that typically provide readers with details guides, how-tos, professional advice, and alike.
The biggest problem is that there is absolutely no backups in this system. Sure it's great to be able to access files over a network, but if something happens to that Synology, you're still out of luck just like you were with the Lacie drive. For a beginner trusting Fstoppers to point them in a good direction on workflow and storage, this article needs to include notes that you should ALWAYS have backups.
An easy addition would be to buy another duplicate Synology and have them sync over internet or such which is easy to due via their system if I remember correctly (I don't use Synology but it's a feature on several of these NAS I believe.)
The synology is in raid 1, so there is a backup. Thats half the point of the article. That being said, I definitely need to get a third backup off-site
This is WRONG, so very wrong. RAID is NOT a backup. It is for more convenient / speedy recovery of hardware failure, nothing more. It does not protect against corruption, accidental deletion, etc, etc, etc.
Dude. Define backup.
A backup is a totally separate version of your files.
There are different types of raid. Striping, Mirroring, Striping + Mirroring. With & without parity or distribution.
You didn't specify which raid setup you were using. With only two drives, your choices are either striping (raid 0 for speed) or mirroring (raid 1 for redundancy)
Mirroring does not give you a "backup". If you're writing to that disk and files become corrupted during the write, you're not safe.
I have a computer engineering background and worked with storage arrays in the 1990s and early 2000s.
I can tell you from experience that the purpose of having drives in a raid 1 array is to reduce the probability of downtime. It is for scenarios where any downtime for the storage system could be bad news but not catastrophic.
Raid 1 is two drives operating as one but the drives are mirrored for redundancy. If one of the drives experiences a failure; you can replace the drive with no downtime to that storage segment. The new drive will then be written to with the data from the remaining drive.
So this is basically a single drive.
A backup would be an entirely separate drive (whether that drive is two drives in a raid array or a single drive doesn't matter as much as the fact that it's a separate drive)
I personally keep a backup on site, a backup off site and have a cloud backup...
Welcome to the often confusing arena of "backup". First you need to define your needs for storage.
1. Fast read/write access for day to day productivity.
2. Secure local data backup.
3. A place to put data for multiple computers on your network.
4. Off site data backup.
There may be more but these are the basics.
Each of these four require different solutions and are not compatible with each other.
1. From your article I see that you use Lightroom and Capture One. I assume Photoshop as well. For day to day productivity the best solution is storage, either internal or directly attached using USB 3 or equivalent. There is no NAS solution that can match directly attached storage.
2. Backup is just that, backup. You should never combine productivity and backup on the same device. The main reason for this is to maintain the integrity of your backup. This can be either directly attached or a NAS solution.
3. This is the place where you put what I call overflow data such as documents, family photos, etc. It is a NAS solution that is available to all of the computers on your network. This should also be backed up to your NAS backup system.
4. Off site means just that. I use a cloud based solution called Crashplan but there are many good ones out there. After the initial upload of my data the program runs in the background daily and backs up anything new to the cloud. This is a very slow solution and recovery of data can literally take days but if my house burns down I can still get everything restored.
Raid 1 is just fine for your NAS. You are not running the kind of production environment that requires a more exotic raid solution.
I hope this helps. Good luck and keep it secure.
I commented back in March about RAID not being a back up solution. A back up solution protects you from data corruption or loss, RAID doesn't do both.
https://fstoppers.com/education/six-photography-mistakes-you-may-not-rea...
Most of you are right, I probably should have approached this in a more tactful way and provided some suggestions instead of just criticism, that is my typical approach. BUT, when there are so many whole I just don't know where to start. On top of that readers of this site look here for advice and in most cases the articles are written by professionals who know what they are doing. How many readers will be lead astray by this and put their photos at risk? I find it irresponsible on the part of Fstoppers. When I can find some spare moments I will write up a few notes, if they haven't already been pointed out by others in the comments.
At no point did I claim to be a professional when it comes to computer networking. The entire article is set up (and very transparently so) as a "follow along while I try to figure this out" experience. I'm sorry you don't find the article useful, but I bet there are many others out there who have learned something by the ensuing discussion - which is the entire point.
Yes this was a great article in this regard. Writing an article that meets Oxygen’s criteria that no one will read is easy. I’ve done it myself it’s so easy. The right people will read your article for the reasons you say, you bring them to people like me and Oxygen.
Not piling on! Peace to all!
Michael
Again, Im sorry that to you it sounds like Im reaching out for help. But the thing is, you're wrong. Its a learning post. I, for one, feel encouraged when i know that others out there struggle with the same questions and issues as myself. You have some really great info in your rebuttal. I wish we didnt have to wade through all the negativity to get to it.
If you could write this up in a post format and send it to either chelsey@fstoppers.com or david@fstoppers.com, I think we could make it a post on the front page.
I'm pretty disappointed with this article too but I have to disagree on one point - RAID 1 most definitely is a form of backup. I think you even unintentionally contradict yourself by giving the definition of RAID 1 - it's two drives containing identical information, which by definition is a form of backup.
Is the poor author completely clueless? Yes.
Does he enlist help from an expert/friend/colleague/redditor? No.
Does he need lessons in computer? Definitely.
Should we bash him? No... maybe a tiny bit, but at least not in a mean way.
I have artist friends who take paths down the road to computering properly and some reach their destination but the point here is to make the difficult explanation easily digestible which unfortunately cannot be done if you want to actually do it properly. For example Tricking a program or OS into using remote storage (whether it's connected to your router or half way around the world in the cloud) is a pretty simple procedure in itself (especially on a *nix like OSX) but can't be explained without technical details and shouldn't be dismissed as "hacking"
I love the first line though "oh God no. please no. just no."
Assuming those drives you have in there are 7200 rpm, on average, they max out at around 90-120 MB/s best case scenario (megabytes per sec); that's around 960 Mbps (megabits per sec). Your average ethernet wire is 1 Gbps (gigabits per sec). I don't see how the speed of your ethernet could ever be the bottleneck. If you get LaCie or G-Tech shit, they all pack with Seagate or WD drives with the same speed anyway. You're pretty much paying for the brand.
Btw, I wouldn't call "mounting a network drive on your local computer" hacking it.
FYI:
1 byte = 8 bits
1 mega = 1,000
1 giga = 1,000 mega = 1,000,000
Gigabit ethernet is 125MB/s and that's not counting overhead for the packets so you never end up with a pure 125MB/s of the original data.
If he set the NAS up in RAID 0 like his old USB drive was configured, I could see it easily maxing out a gigabit connection.
Right, but he's not setting it up as RAID 0. I mean, if performance is your #1 concern and you still want some redundancy, RAID 10 is a safer bet. At which point, yes, I agree, a NAS is no longer adequate, unless you have a commercial switch to hook it up using CAT-7 cables.
FYI:
1 byte = 8 bits
1 kilobyte = 1,000 byte
1 megabyte = 1,000 kilobyte = 1,000,000
1 gigabyte = 1,000 megabyte = 1,000,000 kilobyte = 1,000,000,000 byte
or
kilo = 10³ byte
mega = 10^6 byte
giga = 10^9 byte
No, 1 kilobyte = 1,024 bytes!!! 😂😉👍😎
I do not sweat these details but if you make this suggested adjustment to these numbers the math will absolutely add up!! I am thrilled to see someone else focus on the fundamentals. Great job.
The Synology systems can easily saturate a single link, and even a dual link Gb/e. I'm using a dual link DS415 and can pull anything from 200-220mb/s across the LAN but that's on a managed switch and with the network fabric to support it.
WD Reds are more than fast enough for this solution and the WD Reds have proven to me to be the most reliable out-with the Hitachi He drives...
I've been dealing with the NAS/lightroom workflow problem at work for a few weeks now.
A little background:
For the past few years I've been able to use lightroom over Iscsi. iscsi connection is a way of mounting a network drive as local, but it requires a software controller and a separate network connection (can't be part of a home or business larger network). Works great for video, audio, and photo workflows. Problem is we are trying to set up a easily accessible photos drive for multiple departments to us (print, web, communications, etc) and decided to go with a drobo (cause you know cheap compared to an actual server). Turns out, as this author discovered, lightroom literally will not work over a NAS connection (we are using for the drobo because iscsi would require a lot more infrastructure cost and user training).
Solutions I've tried:
1) Keeping my lightroom catalogues local and synced to other users via dropbox while keeping the stills on the drobo
--outcome: works great, except dozens of photos have already corrupted through this method...so no go
2) Playing with CaptureOne as an alternative.
Outcome: So far so good--and it actually may work better for our office structures and institutional needs, but I still need to do much more extensive testing before rolling it out.
I would love to hear thoughts on this NAS problem since multiple persons on campus are still using contact sheets as a means of culling and organizing photos from a digital library--this makes collaboration a real chore!
What about keeping the catalog in a mac disk image. Put the disk image on the server, double click it to open on the local desktop. Never tried this but would love to know if it works. I use catalogue sync via dropbox with image files on a NAS. So far no issues, but you have to be meticulous in making sure that all syncing has completed before opening the catalog on another machine
If you shall work in a real multiuser enviroment you need a seperate DAM software or a RAW Editor thats multiuser enabled, because you need to check oout the files when editing them ...
There are a couple of multiuser editors out there, but the DAM/RAW Editor solution will be a better way of doing this. If you use something like Daminion Server or Fotoware Fotostation in combination with Lightroom/C1 pro you will get a system that not only are multiuser, but where you also can configure Access rights for det mediafiles ...
The files itself can be stored where ever you like when using a DAM solution.
Thanks Jaran! That is the direction we are heading for our video workflow, but I hadn't thought to apply it to the photo workflow. We are still a year or two away from implementing any kind of DAM on the video side, but I'll be interested to see how it works for photos. For now I've settled on Chronosync and Adobe Bridge as a workaround workflow. I was trying to push our cataloguers (don't edit, just collect for print and web) and Aperture editors towards LR for both cataloguing and editing, but Bridge will have to do for cataloguing, and I can't really do anything about the Aperture hold-outs at this point. With Chronosync I end up with two copies of every file so at least its a form of backup since I'm storing photos locally now. It also lets me continue to use LR.
What will happen if a fire breaks in the basement?... hmmm..
I'd be screwed! Off-site storage is definitely next on my list to secure my data. Any good suggestions for myself and the rest of our readers?
Check out Amazon Cloud Drive. For 60$/year you get unlimited storage and the Synology NAS have an "app" that can sync your files and folders automatically for you.
Quentin, how is Amazon Cloud with large amounts of .RAW files and .PSD's?
As the copy is made in the background and on the NAS, I must admit it's hard to tell if it's fast or not to copy…
I tried to recover a couple of files – to see if it works before copying everything – and so far so good. The user interface is far from pretty, but it does the job.
A great addition to Amazon Cloud Drive for Mac is https://www.syncovery.com/
You can even encrypt your files that goes to Amazon.
I have 5.7 TB stored in an unlimited Amazon Cloud Drive account. 60 bucks a year plus 10 bucks a month for odrive pro (syncing software due to Amazon's lack of a dropox-style desktop client) . www.odrive.com - this software with Amazon Cloud Drive works flawlessly. A bit of a learning curve but waaaay cheaper Dropbox for Business. It syncs very quickly. It does have a file size limit that is very large, but a limit nonetheless. I can easily upload 10-20gb files though.
Yes, it's very good. I have 2 TB and no problem.
The web and iOS app for Amazon Cloud Drive are simple, but does its job.
I hope it gets better.
I was a QNAP (recently also asustor) user for many years and I never found a cloud backup solution on either NAS that was stable enough. The dropbox and crashplan clients I tried always stop working for some reason or another. I realised that the only software that was reliable was the client software for the PC/Macs themselves. I sold the NAS and purchased a cheap Mac Mini to use as a NAS and never looked back. Cloud backup and time machine work faultlessly also.
.
If you have access to your own studio or a family residence you can easily set up another Synolgoy box to sync it's data. I do this with my home and studio over a VPN link and leave it to run overnight. This is on top of the Samsung S3 drive I use to carry the data for offline storage. when I get home which typically goes to a WD Passport drive.
Remember also to take regular backups of you catalog ... and I will advice you to enable the writing to XMP sidecar files in Lightroom for additional safety around Your edits, if you use sidecars and you edit a image, only the small xmp textfile will be copied, you do not need to copy/backup Your catalog every time you have done a small adjustment.
There are three good reasons for doing so, speed when you set up a off site backup, cloud or a second NAS.
You get another point of safety for your adjustments, and the adjustments can be read directly from ACR/PS if you for some reason should be without the Lightroom catalog.
And its the keywording/metadata, that also will be stored with the files in the xmp sidecar.
My studio uses a Synology NAS for all of our lightroom catalogs. RAW's are stored in a shared folder on the drive and I use Cloudstation to sync the lightroom catalogs themselves along with my lightroom presets and settings across my computers. Works similarly to the dropbox idea someone's already mentioned, but keeps everything technically on my NAS. That way I know it's redundant and local.
You might want to activate two-factor authentication to your NAS (Synology and Qnap make this easy) since it's hanging off your internet connection etc. Also, it's a good start - but you need a 3-2-1 system (three copies, on two media, with one set offsite). And agree, LR is much too slow under these situations.
Thanks! Ill look into that. Yeah, its shame since I enjoy the program
Regarding Capture One, if you use catalogs, try to keep the one you're working on on the local drive and the original files on the NAS. It'll go even faster as C1 works on the previews found in the catalog and not on the raw files directly. If you work with sessions, work on them locally, with backup on your NAS and then move the final files to your NAS once you're done editing ;)
Oh… and regarding the NAS and its apps, look into Plex. Not photography related, but you'll be able to watch your movies anywhere (TV, phone, tablet, etc.).
+1 on plex
This is ok advise if youre locked into the Apple ecosystem.
I use Plex, my family uses my Plex, my phones, TV, , Apple TVs, computers, Tablets, consoles as well as remote versions of all of the above use them. Plex is infinitely more capable than airplay and doesnt require you be at the host computer to do any of it like airplay might.
Plex itself is a whole different beast andd i would recommend it over staying only in the apple ecosystem.
* No one is "locked into the Apple ecosystem," and as it relates to say watching videos their devices support the most commonly used video file format. What more would one need?
- If you are using Airplay, then you are locked into the Apple Ecosystem. There are devices that can use Airplay to accept incoming streams but its from an Apple device only, thus, you are locked into having an Apple device for part of the system to work.
* How is Plex "infinitely more capable than AirPlay," especially as it relates to most people's needs?
- Lets say i set up a Plex server. To clarify Plex server, i dont
mean a separate computer that runs plex but a computer that has the plex server software on it. Then i load on my media including TV shows, Movies, Music and Pictures. So now, i have one central device that has all of my media on it which is accessable by all of my Apple devices, not only in my home but also outside my home. I can play any movie, from any device and pick up where I left off, on any device and it comes over. I can start watching a TV show or series and pick up where I left off. It also pulls down all media for all TV shows, including artwork, trailers, theme songs, subtitles and meta info. Itunes eco system only lets you integrate with iTunes which gives 0 control over all of this and doesnt pull down half the info/features. Not to mention you need to be in different apps on the iphone just to access each of these or a different program on the mac to do the same thing. The interface for Plex is way ahead of itunes as well.
* Plex "doesn't require you be at the host computer to do any of it like AirPlay might"? You're declaring something to be true and then follow that up with "might"? Clarify.
- Lets say you want to play a movie on your Apple TV that you have on your Mac, you can do that relatively easily but its only easy from the Apple TV. Lets say you want to play a movie on your iphone thats located on your Mac....well thats impossible UNLESS you sync it first with your iPhone. Or lets say you want to Airplay a movie from your Mac to your phone....you cant, you have to sync first. Want to play music thats on your Mac on your iphone? You cant unless you sync first. You can use the remote app(another app you need to fire up) on your iphone and do this but it also requires you to be on the same network and connected to wifi. Plex on the other hand lets you play all of this on ANY device that has a plex app and not only play local to that device but also airplay to devices on the same network you are currently on even if its not the same network the Mac is on.
There are SO many more possibilities with Plex for consuming media, not only in the same network but also while youre not at home.
I should also add that only media bought thru iTunes is able to be played on other apple decives that isnt local. Bought a CD and imported it? Gotta play it on the device its imported to or synced which requires, you guessed it, to be at the Mac to do it.
I always liked the rendering of Trans X files on C1 more than Lightroom anyway, so maybe this is just the reason I need to make the switch back.
Good idea for a quick write up!