"Trevor, I have been editing this wedding for the last few days and now my Lightroom catalog says it is corrupted! When I opened a new catalog and reimported the photos from my hard disk I can't find any of my previous edits I worked so hard on. I am so frustrated! What did I do wrong?" Keep this from happening to you by having Lightroom automatically save your changes, but this setting is off by default. I'll show you where to turn it on.
If this hasn't already happened to you, knock on wood because surely it will. Seems like every week I see another photographer post desperately in a Facebook group about having lost all their recent edits because of a corrupted or lost Lightroom catalog. Often this means that hours of work, maybe even days or weeks is lost.
However, this should never be the case if they just turn on one simple setting in Lightroom. Sadly, this setting is off by default. The reason the team at Adobe has it off by default is because at times, depending on your system and how many files you are working on at one time, it can slow things down a smidge. But these days our computers keep getting more and more powerful and often we will not even notice any difference in speed.
You will find the setting in the Catalog Settings of Lightroom.
MAC > Lightroom > Catalog Settings > Metadata
PC > Edit > Catalog Settings > Metadata
The important setting that I would recommend having checked is "Automatically write changes to XMP."
By having this button checked you are telling Lightroom to constantly be "saving" the changes you make to your images. If you are working on proprietary RAW files such as .CR2 or .NEF files you will see a .XMP sidecar file created alongside your image file in its folder. That XMP file contains all your changes you have made to the file in Lightroom. Which means if your Lightroom catalog gets corrupted, lost or for whatever reason is not working right all you will need to do is reimport the images into a new catalog and Lightroom will automatically read those XMP files in the same folder and apply those changes immediately to the files. Then you can wipe those tears away knowing all your hard work wasn't lost.
If you are working with DNG, JPEG, TIFF, or PNG files the XMP information will be stored directly inside your files so you will not see the accompanying XMP file in the folder, but if you were to send those files to someone and they opened them in Lightroom they would see the edited version of the image. So again, if you had to reimport the DNG files into a new Lightroom catalog because the old one got corrupted - no worries, no tears, your work is all there and up to date.
Keep in mind though that the history of your edits is not stored in the XMP file which is one of the benefits of the Lightroom catalog.
So if you don't already have that box checked in the Lightroom Catalog Settings, stop what you are doing and go turn it on now. Do some editing and see if you notice it slowing your computer down. Hopefully it does not. If so, the other option you have is keep the setting off but every few hours when you want to take a break from editing hit the keyboard shortcut, Ctrl+S to save the metadata to files. That will do the same thing, it just won't be autosaving as you work.
If this was helpful please share so we can save the world of angry and depressed photographers who have worked for days only to find out a Lightroom corrupted catalog has lost all their work. Cheers!
UPDATE: For those more interested in learning a good technique on how to back up Lightroom Catalogs, check out this article and accompanying video on Fstoppers, "Good Technique for Backing Up a Lightroom Catalog."
Thanks for the tip!
You bet. Thanks Bjorn.
This is (arguably, obviously) bad advice. Enabling this option can have a major performance impact, both on day-to-day Lightroom use and on your backup solution. One should fully understand the ramifications before deciding whether it's appropriate for their workflow, but before even considering this option, one should have a solid catalog backup solution in place.
The thing is, once you have a solid catalog backup solution in place, the need for ongoing XMP writes is greatly reduced.
It might be different advice from your workflow, but this is far from "bad advice." Tone it down there Jeffrey.
It's bad advice to just say "turn it on" without fully discussing the ramifications, and even worse to misrepresent the ramifications, as he did. Auto-XMP makes perfect sense for some folks, but not most.
Care to elaborate a bit? What negative ramifications are you talking about?
The ones I mentioned in my initial comment: a substantial performance impact on Lightroom and on your backups. It's a YYMV thing as to how much of an impact, but I've heard horror stories as well as success stories like tdayley's. My initial point is that it's not a no-brainer to just turn on blindly.
I did not get that feeling from this article. I think people realize it is their choice, and they were clearly informed that performance may take a small hit.
I can't imagine how updating a small plain text XMP file could have a "substantial" performance impact. This operation requires so little processing power and memory compared to editing the graphics, that I doubt it will be noticeable at all on any machine that is reasonably fast for Lightroom.
I understand how one might think that at first, and for some folks that may well always be the case, but I can tell you that in my 7½ years working with Adobe on Lightroom (and for some of that time as an employee working for Adobe on Lightroom) that horror stories have popped up about it bogging down systems in strange and unexpected ways. Sometimes the "unexpected" was because you didn't think about it deeply enough, and sometimes it remained a mystery.
Remember, for example, that every XMP update to all but proprietary raw files results in a full read-and-rewrite of the entire master image file. Work with DNG and just loaded a 10-gig shoot to work on? You're going to read and rewrite that 10 gigs over and over and over again as you update keywords, ratings, and other metadata, then dive into the develop phase.
Have any large panos that you update a single keyword on and didn't budget the time to wait for 100GB to be read and rewritten with every minor move, then be prepared to wait. For example.
Ripple effects including trashing your OS disk cache and having to re-backup gigabytes of stuff after some seemingly innocuous catalog maintenance.
Smarter backup solutions may help mitigate that last issue, but you're still reading and writing your master image files; many folks prefer to keep them read only.
(Some folks have asked for sidecar files for all image formats, which would mitigate many of these issues, but would raise its own set of issues).
I'll repeat: the idea of auto-write is not bad, but turning it on blindly is a bad idea.
That problem with the 100GB file, in your example, isn't an inherent problem with the automatic saving of metadata. It's an inherent problem with how LR deals with large files. In short, it doesn't. Or at least didn't. In earlier versions, if you were working on large files LR would choke whether the auto-save feature was on or off. Recent versions are significantly better but LR is still not the optimal solution for large image files.
You're still only offering broad 'have a good backup solution' comments without specifics. Are you suggesting the CTRL/CMD+S idea or are you suggesting no saving of the metadata?
DNG is not a panacea. Nor is it a 'one-size-fits-all' solution. Plenty of thought needs to be given to it before switching from a RAW workflow to DNG. Are you going to outline the considerations that need to be made there?
Rather than simply criticise, do something. Offer a valid alternative with specifics. Ask the Fstoppers folks if you can write a guest post to outline your ideal LR workflow.
I was just struggling with sluggish performance of Lightroom. Turning OFF this setting resolved my issue. For what it's worth, here is what I have been doing:
I have some old folders of images saved on external hard drive (USB 3 connected). I am addding this folders to the LR catalog and assigning keywords to all images. With the setting turned on, it has to create or update an XMP file for every file I touch. Since I'm doing them in batch, that is a LOT of files. Furthermore, some files are updated multiple times as I add more than one keyword.
I realized turnning off this setting is what I needed to do to improve performance because when I quit LR, I got a message stating that it was not finished updating XMP files. Given my current workflow, it makes more sense to finish adding folders of images and keyword them and update XMP files only when I am finished.
I've had this setting turned both on and off and have never seen a performance difference between the two. That's on four computers; two desktops/two laptops, with vastly different hardware configurations.
I think the tone of your message is, perhaps, drowning out the actual information of the message.
The article wasn't well written. A better approach would have been to make sure the same information that is includedin the video is also included in the text. That is true. But the pissing contest going on now doesn't really serve any purpose either. You're all big swinging dicks in the photography world. OK. Egos stroked?
What does "substantial performance impact on Lightroom and your backups" mean? That's not overly specific either. What kind of "horror stories"? What are the reasons for the performance impacts? How does it affect backups? What is your alternative recommendation? Having a "solid backup solution in place" is pretty vague. You want to actually elevate the discussion or just criticise?
If your computer can't handle even lightroom and you are worried about performance impacts, maybe you should be rethinking your career choice... since its blatantly obvious you aren't making enough to even cover basic gear.
why wouldn't it be a no brainer? why wouldn't I want to conveniently and discretely have my computer automatically back up all my hard work at regular intervals? what kind of horror stories have you heard from people who so hatefully had their work backed up?
please elaborate on your 7.5 years experience working with the now not-even-8-years-old lightroom.
to this point, you still have not backed up any of your blatant claims. i'd like to see some evidence and not just some cry for attention through back-handed words and obvious condescension .
He has written LR plugins, which are widely used.
See my previous post about adding thousands of images in batch and adding keywords to them. Since the images are on an external hard drive, the USB 3 connection is likely the slow step in the process. In any case, turning the setting off changed LR from being very sluggish to being very responsive.
Don't discount an argument just because you can't think of the scenario where it's relevant. My scenario is definitely relevant to the discussion.
It will increase the number of write requests which can shorten SSD lifespan, it lowers performance because by default lightroom saves your edits in memory for a while and updates the catalog all at once so it writes less often. Some people report more crashes with this method as well.
The "ramifications" are mentioned in paragraph 3, and discussed in more detail along with an alternative in the video. But the great thing about LR is that there are a lot of ways things can be done. That said, this works fantastic for me and my system and hopefully helps many others.
Note that [CTRL/CMD][S] only works if the images currently selected are the images which changed. If you don't have the edited images selected, they aren't being written to. Test this: Hit [CTRL/CMD][D] to deselect everything. Then hit [CTRL/CMD][S]. Nothing happens - no dialog. Now select a couple of files: [CTRL/CMD][S] See the dialog?
Anyone reading this would think [CTRL/CMD][S] is a panacea that will save them. Without first selecting the changed files, it does nothing. The incomplete advice offered here is potentially dangerous.
There are somethings contained within the LR Catalog that are not included in XMP. Had all these disaster-ridden photographers followed a reasonable backup regiment, or at least backed up their catalogs frequently (and properly) there would have been less gnashing of teeth and wailing in the darkness and far more information would have been saved that Autowrite or [CTRL/CMD][S] could have offered.
The bigger fear is that people will abandon good backup strategies thinking Autowrite or [CTRL/CMD][S] has their back.
@Rick - I am pretty certain it says in the video that if you are going to choose not to have it auto-write the XMP data and instead use the Ctrl+S or CMD+S it says to select the files first. I believe it also mentions the fact that not everything is contained in the XMP files (for instance the LR history.) If you watch the video it is there. But I do understand not every one has 4 minutes to watch it so I appreciate you spending the time to comment here and clear that up. I think there also might be a misunderstanding that this article is saying to abandon all other LR workflow systems - which it is not. This is definitely not a detailed explanation of an entire LR data backup workflow - just covers one aspect that is quite useful for many people and often ignored. Hopefully it is helpful for those photographers. ;)
BTW my name is spelled Rikk... There are many who never look at videos and only read the text. I never click on a CNN story if it has a video icon-why? Because I want to read-not watch. The question you need to ask yourself is: Do many of my viewers/readers do the same? And, if they do, do I dare put anything in text that will misinform them if they fail to watch the video.
I want happy Lightroom users-that is my end goal and that means informing them properly.
Thank you for the opportunity to respond.
@Rikk - I absolutely agree that not everyone will stop to watch the video. I don't always do it myself. But I would say that if someone is going to stop to comment and say that the "incomplete advice offered here is potentially dangerous" then I would hope that at least they took the time to consume the content before disagreeing with it. Also there are much better ways of adding value to an article in the comments versus trying to discredit it. Like you, I want happy Lightroom users as well - hence the reason I have put out numerous videos giving them tips and information to help them out. You can find more of them here on my YouTube page. http://www.youtube.com/user/tdayleyphotography/videos
Rikk - And the word you were looking for is 'regimen', not regiment. So maybe a little less pedantry on the spelling would be a good idea. Glass houses and all that.
I would like to see if I can add light onto what Jeffery is saying. I learned this the hard way and wish someone would have explained the pluses and minuses before I committed my workflow to converting my RAW images to DNG.
I too wanted my images to be "self-contained", no XMP files needed. I turned on the auto update flag, there wasn't any day to day performance hit that I noticed. I have a decent PC, no problem.
The trouble for me happened in my backup routine. All those writes to the DNG's trigger the backup attribute for this file (and any others) to be rewritten both to my local copy and my cloud backup. This was causing me NOT to be backed up, they weren't able to finish over night. The simple act of trying to organize my collection with keywords killed any productivity. It took me a lot of investigating of why all these images were being backed up again, I didn't edit them, but it finally hit me, I had triggered a update by adding flags, keywords etc, this was happening to all my old scanned images as well. Some of those are bigger TIFF's then the DNG's. As Jeffery was saying, it's not the XMP files that are the issue, its when you have chosen to convert to DNG (or any non-RAW format) that auto update has impact, as least for me it did.
The solution for me was to turn off the auto update, life was good again.
With due respect, I disagree. This is not bad advice, this is more information. Information is never a bad thing. People can use this if they desire redundancy in their workflow as a means to reduce the chance of losing completed work. This is wonderful advice.
Also, next time, maybe add to the knowledge by clearly stating the issues with Lightroom performance when this option is selected (even though he clearly states the issues in the article and the video).
if youre working with sidecar xmp files, they are really tiny and quick to flush to the disk, so the speed drawback is negligible
Put a hundred brush strokes on an image, compare its XMP file size to a non-brushed image's and watch the speed drop...
Trying to remember the last time I put 100 brush strokes on an image. That is probably why I never experienced the "speed drop" you speak of.
You obviously have a sheltered workflow then. There are many without a pixel-based editor who do just this. If you spend some time helping out on support boards, you will realize that the Adjustment Brush gets used heavily by many users. Users, who just might take your advice.
Personally, I don't recommend it either but then I don't poopoo those who work this way.
A hundred brush strokes? Isn't that what Photoshop is for? Brushing in LR is such a pain (quite slow, not very precise, not as customizable as in PS), why would you even try brushing that much a file in LR? :/
@Rikk just for giggles I grabbed a full size CR2 file, changed every single slider in the Lightroom Develop module including Lens Correction. I then did 46 spot removals, 52 different adjustment brushes (before I got tired of doing any more), 3 graduated filters, 1 radial filter, red-eye reduction - basically everything I could think of to try and push the size of the XMP file. After all that I got it to 154KB as seen in this screen shot. Even with all those edits (which I would never do in LR) the size still seems quite small to me and one that should be pretty manageable for a computer these days.
The size of the file that needs to be updated raises substantially to 10~20~30+ megabytes if you're working with DNGs or TIFFs or PSDs, which perhaps becomes a bigger issue both for the incessant updates, but also for your backups. It has no effect on an all-CR2 workflow, but perhaps some of your readers work with DNG, PSD, JPEG, etc.
Once again, what is your alternative solution?
My alternate solution to a blanket "this should be on by default" suggestion is to point out that there are important considerations (well beyond "slows things down a smidge") that each person should be aware of before making their own decision. I don't understand why "yo, hold on there, there's more to it than that" garners so much hate.
The original article presents it as a no-brainer option that everyone should turn on, and for reasons I stated I think that's bad advice. If you disagree, then we disagree.
I don't think there's one data-integrity solution for everyone, and so I don't have one data-integrity solution to present, sorry. Personally, I keep good catalog backups and I don't use any kind of XMP write, but that's merely what works for me; YYMV.
By the way, Robert, I've gotten email notifications from Discuss about a couple of longer (reasonable, well-presented) comments from you, but when I come to the web site to address them, I can't find them.
Hate? Come on, that's more than mildly hyperbolic. I agree, as noted earlier, that it isn't a well-written article. But rather than simply criticise, we could, perhaps, use this space for better purposes.
I'm simply trying to get past the 'that's a bad idea' type of comment that several seem to have left and get to a real discussion of alternatives. As I said, I've tried both ways on several computers and not found a siginificant performance disadvantage. That's my experience. But then, I've only had a corrupted catalogue once and that was back with LR2. And I was able to recover it. I've had more problems with total disk failures, but that's a different story. And I don't think the XMP issue should automatically be conflated with the issues around the handling of large files.
When you say you use a no-XMP workflow, are you saving the metadata directly into the files then? If you use a DNG rather than RAW workflow then I understand. Or are you not saving out any of the metadata?
WRT why other of my replies can't be found, I have no explanation for that. Sorry.
My initial comment had exactly two words of criticism ("bad advice"), followed by information that would start a reader to question a blind following of the advice, and perhaps start the discussion. I did not feel the need to address every detail nor provide a full solution in place of the advice I thought was bad. Books are written to do that. I was just intending to provide a speedbump to the blind following the advice.
I didn't mean to be melodramatic with "hate"; in these internet-forum contexts, it's often used to mean "unjustified antagonism". My two words of criticism don't fall under that heading, but the response to them sometimes did.
I, too, have never had a truly bad experience with the auto-write option. I used to use it many years ago. But I also understand that my personal experience does not apply to everyone else, and I know at least some of the problems that lurk when you use it, which is perhaps one reason I could avoid a truly bad experience with it; I knew what I was dealing with. I don't know all the problems. I'm not an expert, but my not being an expert doesn't make the problems I mention any less impactful, nor my mentioning them any less relevant.
To answer your questions, XMP is the metadata format for associating data with the files, whether written directly into the file (JPG, DNG, etc.) or as a standalone *.xmp sidecar to a proprietary raw file (NEF, CR2, etc.). I don't use any of it because of the drawbacks I mentioned don't eclipse the benefits for me and my workflow. It's only a partial backup of the image data; the full data is kept in the catalog, so one needs a good catalog backup to be covered, and if you've got that, the benefit to me for writing to the individual files is negligible. Others may well feel differently, but they can't balance the tradeoffs if they aren't aware of them.
I've never said that auto-write XMP was bad. It's a tool. I said advising it to be turned on without advising on the ramifications is bad.
"Unjustified antagonism"? Since when did asking questions, asking for clarification, prompting for more information and the like become "antagonism" let alone it being unjustified? Bloody hell.
I wasn't suggesting you were an expert. Nor am I. I simply asked questions to try and get more information and generate discussion. I apologise. Not for nothin', but when you say you worked for Adobe on Lightroom, it kind of paints a picture of something approaching expertise.
"... and I know at least some of the problems that lurk when you use it, which is perhaps one reason I could avoid a truly bad experience with it; I knew what I was dealing with." This is an example of what I was talking about though. It's vague. For those who don't knownwhat those lurking problems are, it's not helpful. Why not provide examples so there's context around the caution and readers can be more informed. I'm not trying to be antagonistic. I'm trying to learn and to help others learn as well.
XMP is *one* format for associating data with files. There are others (e.g., EXIF and IPTC).
i don't see any way a professional workflow would include editing jpeg/dng/psd/tiff on lightroom. please enlighten me.
well you obviously have no clue.
where do you do all the keywording etc. for JPG and TIFF if not in LR?
only the metadata part is writen to the image file.
not the whole 30MB file.
you don´t need to rewrite the whole file when you only wnat to write new metadata.
correct me if im wrong. but i think im not.
For non-propitiatory image formats, the data is added *into* the (presumably large) image file. Yes, you don't want to rewrite the whole file just to update some metadata, but you have no choice unless the size of the metadata block has not increased (in which case Lightroom could well optimize to an in-place overwrite, but I don't know whether Lightroom takes that optimization).
i doubt that LR re-writes the whole file.
What about disk fragmentation? Having 30,000 10KB files isn't something that would definitely speed that process? I think this is good advice for those who don't use LR as their primary editing tool, if you have let's say 20 images you need to adress to, ACR can be a bit painfull to work at for more than a few files. But other than that, I guess saving your LR catalog and doing weekly backups (or more depending on your workflow) is more than enough, and a lot less stressfull for your disk I'm guessing.
there is no disk fragentation issues with todays harddisk. we have 2014 not 1993.
it´s an urban myth that won´t die.. get some information on that topic.. actuall information from today.
things have changed.
Really? So writing many files into a disk over and over doesn't shorten it's life? What makes disks die? I'm asking because I have a 1TB disk that I use mainly for backup but would love to use it as a primary disk.
Agreed!
Agreed. Why anyone wouldn't have a backup of their catalog should be the issue here. I keep 3 copies of my catalog (on my PC, on my Local Network, and on a USB).
No one has suggested, as far as I can tell, that users should use the auto-saving of metadata as a proxy for backing up their catalogues.
well.. simple solution.... get a faster system!!!
i have it enabled "write XMP" and i don´t notice any slowdown.
a bad advice is to lose your work because of not enough failsafe measures.
and a catalog dose not help when you also use OTHER programs who have no access to the lightroom catalog.
what does all the keyword etc. metadata help when it is only in the LR catalog?
you really only work with LR?... i am not.
Trevor you are not any more PC guy ;)
I bat for both teams. Mac laptop and PC desktop. Though I got to say if I can get my hands on a new Mac Pro I think I will be won over for good. :)