In a perfect world, raw files would be as light as JPEGs. Even better yet, every camera manufacturer would use a similar and open format such as DNG. Unfortunately, as we all know, we don't live in a perfect world. However, one company named Dotphoton is about to change it all!
Changing it all may sound presumptuous, especially as there hasn't been much evolution in terms of raw file format since digital cameras have been around. Most raw formats are proprietary, aren't well optimized, and aren't designed with perpetuity in mind. But Dotphoton is genuinely about to break most of the issues raw files present while retaining all the benefits.
Enter the World of True Lossless Raw Compression
You've probably already seen or heard of lossless raw files. Many companies offer this option, Nikon being one. However, even with this option, my D810 files are still quite large and make archiving a pain as well as a costly business expenditure! I've even come to ask myself multiple times if keeping raw files was making any sense and if I shouldn't just keep the final file.
But a few weeks ago, I was introduced to Dotphoton Raw. What it offers with its solution is a way to compress raw files while retaining all useful data. So you may think this is too good to be true, and that compression is just like what's already available in camera when enabling the lossless compression. Well, the result is actually very far from what you've ever seen. Files can be compressed to a third or a fifth of their original size!
On the left is the original raw file from my D810 (NEF file), it takes over 40MB on my hard drive. On the right is the compressed raw produced by Dotphoton Raw and it's only around 12MB. This is one example, but in my testing over the past two to three weeks, I haven't had a file that wouldn't get compressed by less than three times.
Both files look similar and the metadata are recognized by Capture One.
Is It Truly Lossless?
Now, let's answer the question that you should be asking: is there truly no loss of quality with this compression system? Well, it seems like it! I couldn't say yes or no for sure, but it definitely feels like there is no difference. What I can tell you about are the different experiments I've done so far.
When Dotphoton gave me access to its app, I was extremely excited and compressed files from my last shoots, then opened them up in Capture One. At first, I thought it was too good to be true. So I looked closer at the files in Capture One and started looking for differences. After pixel peeping for a bit, I noticed a slight difference in certain colors and how they were rendered. The shift is very minimal, but still present.
Click the image to better see the values.
On the left is the same original file as before, with L.A.B. values sampled on different areas. On the right, the compressed file with the same areas being sampled – and yes, these are precisely the same points on the images as they are placed automatically by Capture One. As you can notice, there is a difference, but it's minor, and I doubt most people would even see it without color readouts. From what I understand, the problem lies in how Capture One interprets DNG files – yes, the compressed files are DNG, but more on that later. The difference in Capture One probably won't bother +95% of photographers, and Dotphoton has told me that its team is working to improve the color rendering with Capture One. So hopefully, we should see an improvement on this side with the upcoming updates.
So I brought my files into Photoshop, opened them up with Camera RAW, no adjustment, selected ProPhoto RGB and 16 bits to be sure to get a fair comparison. I then placed the compressed file as a layer on top of the original and changed the blending mode of the compressed file layer to difference so that I could see if there was any difference. All sample images gave me the same result, a fully blacked out image, meaning there is absolutely no difference between the two layers. And that was the case even if I tweaked the images in Camera Raw with similar settings.
From Proprietary to Perpetuity and Interoperability
There is a lot of debate on the web as to whether to use DNG or not. I used to rely on DNG when I was using Lightroom as it saved me a bit of space on my hard drives. But when I moved to Capture One, I stopped bothering with it as the saving wasn't all that great.
However, there are other advantages to DNG format, and they are often overlooked. It's an open format, designed to help preserve digital images, and offer a file format that can integrate into different workflows. For companies and professional photographers who want to archive files so that they can be preserved for decades to come, it's easy to understand that DNG is the superior choice compared to camera manufacturer proprietary formats. If you look at Canon, since 1997, they've moved from CRW to CR2 and are now transitioning to CR3 while DNG is still DNG since 2004.
Isn't It Too Perfect to Be Real?
That's what I thought right away. How can it be possible to save space, get a better file format, not lose any data, and no one is talking about it? If it's not better known, it's probably because Dotphoton is a young startup. It was created two years ago, and Dotphoton Raw is being released this week.
It also comes with a few limitations. First, it's not compatible with all cameras out there as the magic behind it is apparently different for each camera. So far, the most popular cameras are supported:
Supported cameras at launch
The company is also working hard to add any other relevant cameras, and you can even vote for the ones you'd like to see being added first. In the upcoming weeks, we should also see the Fujifilm GFX 50S/R, Nikon D750, and Canon 1DX mark II added to the list of supported cameras.
For professional working with Phase One or Hasselblad medium format cameras, unfortunately, it's not supported yet. However, I've been told there would be another dedicated product tailored explicitly for MFD systems, but no estimated release date was given to me. Hopefully, we can see it being integrated right into digital backs! It'd be an absolute dream to see such a solution being available directly into a camera, especially with digital backs now offering a resolution of 100 or even 150 megapixels.
Another development we should see are plugins for better integration with the different photo editing solutions on the market. One example I hope we'll see improved – and from what I understood should be improved – is the integration with Capture One and Lightroom. The workflow with Dotphoton Raw isn't bad but isn't the best when shooting tethered. The problem is, the compression app is a standalone solution. Meaning when shooting tethered, at the end of the session, you have to take all your camera raw files and put them through the Dotphoton Raw solution to get the compressed DNG. Then import these back into Capture One or Lightroom and copy/paste the settings you had previously applied. For wedding photographers or anyone who doesn't shoot tethered, the workflow won't change much though. It's just a matter of letting the app convert your files, which require a bit of time, but it's quite reasonable, and then import them into your photo editing software of choice.
Conversion process sped up
Price
At the time of writing the article, the Dotphoton Raw solution isn't available for everyone. It will be released the week of 27th May for $49, tax not included, per year.
What I Liked
- Ease of use.
- Storage space saving.
- Choice of DNG format.
- Pricing is very attractive when compared to the money you would otherwise spend on hard drives and storage solutions.
What Could Be Improved
- Lack of camera support – should be improved within the next couple of months.
- Lack of Windows support – should be supported once MacOS growth is stable.
- Integration with Capture One or Lightroom when shooting tethered is not the best – future plugins should help.
Conclusion
Dotphoton Raw is so far the best thing I've discovered this year. I haven't been excited about an app, solution, or even product in general for quite a while. It's not perfect yet, but for a first release, it's very stable and showing great opportunities for the future. I genuinely can't wait to see how it's going to be evolving with future plugins and updates.
The only difference most will ever see from RAW vs JPEG is the consumption of disk space
Sounds like someone’s been reading Ken Rockwell.
...or Toni Notorious.
True. However, I have on many occasions gone back to old raw files and been very thankful that I can now reprocess them using all the available dynamic range, even though I didn't have the know-how to post process the image properly when I shot it.
this.
plus, as i change and my tastes change, some photos are truly timeless and can benefit from being reprocessed to match my changing tastes. sometimes it can better reflect the moment when looking back or improve upon the processing as i learn new skills.
This +1
I am currently reprocessing images from 2014 for a new project. That processing requires making extreme changes of the images WB (both globally and locally) that wouldn't be possible in JPG.
Noting your use of the word "most"; I think you may have ruffled a few feathers. I suppose using RAW files makes us special.
$49 or ~3.75TB worth of HDD space. Pass.
Tried it, just amazing. $49 is nothing compared to the price of TB on my Mac. And the quality is just raw...
The third account with only one comment that is just in awe of this app. Gosh. So genuine!
And doesn't even bother to get a profile picture to make it look more real :-))
Oh it just did! At least we know it’s a human writing fake comments and not a bot.
Sock-puppeting is an astonishingly effective way of losing credibility for your product. Please don't do it.
Hi Peter,
We don't. This comment comes from the beta testers of the app. I hope he has a right to say his opinion too and I am thankful for his support. :)
Cheers,
Eugenia from Dotphoton
People who create accounts on a website only to leave glowing praise is a pretty common and sleazy tactic. One, it says your company has questionable ethics. Two, it shows a lack of respect for users.
Sorry but this simply doesn't ring true.
1. Real people who like a product go to communities they are already members of (and where they are known/know people) to extol the virtues of whatever it is they like. They don't go around signing up to new communities to convince people they have never met (unless they are being paid/encouraged to do so).
2. The chance of a real person deciding that they must sign up to some new community just to post such a message in tiny. The chance of three of four doing it at the same time is simply not real. Especially when.....
3. The comments are clearly "boiler plate" enthusiasm. Real people who this new products are good write real comments about the product that will help the other members (and they do it in communities they are already part of/care about.
Sorry but big negative marks.
Hi Dan, Well, I think if you look form the human point of view it will look different. 1. I am a professional photographer and yes, I have registered here only to answer the questions about this company. The registration process takes literally one minute. Not a rocket science really. I have been reading Fstoppers for years without registration. 2. People commenting are beta users who spent months with us testing, debugging and became kind of friends. The comments are super enthusiastic because we are all very happy that there is an article about the company right the first day of the release and a very good article. We are very proud about it and surely have shared the link on all our social networks. So, it is a thumb up message for the team as well or their feelings that finally they have a non beta version of the app they helped us to build. 3.I am sure, Dan, if let's say, one of your photos wins an award and is published on a certain website your friends will comment online and say " Amazing photo, Dan!" even if they are not experts in photography. There is a chance that you have three such friends who care enough. I would understand if you were making so much fuss about hundreds of comments, but 3! Come on, there is quantum physics, DNGs good or bad, will Lightroom disappear in 5 years from now and TONS of other interesting and relevant topics which could be discussed here, rather than 3(!) comments of people who simply wanted to support us.
Cheers,
Eugenia
Possibly licensing the tech to manufacturers would be the best avenue to get it widely adopted
Storage is cheap and the whole argument about DNG never made any sense to me. 99.999% of people will process their files within a year. Raw format obsolescence is mostly an imaginary problem. What you really want to do is losslessly archive your final result. For that we have TIFF. TIFF will be around as long as there’s ones and zeros.
Wow, so good! Finally happening!
Another account with only one comment that LOVES this product. What are the odds?!
Oh my... Gonna save so much time and money. Can't believe it's true :-D
Wow. This enthusiastic comment is the only ever made on your very recently created account on your account. Seems legit.
Sounds so good, and yet, the proliferation might be a problem. Lets look at the archaic JPEG, how enduring its presence is. Every now and than someone publishes a new format, beating JPEG in almost every aspect (remember J2000?... or PNG?) and yet nobody bites. Most recently Apple embraced HEIF, and at first I thought that with their record of success in establishing new technology finally JPEG will be replaced by something better, having more than 8bit per color. Not so, even them cannot go through the inertia.
The less it will happen with the zoo of the raw formats. However, if they are really so good, maybe some of the camera makers would license their encoding algorithms for yet another "proprietary" raw format.
hmm… PNG is a very useful and very used format…
Anyone who chooses to use a rental model to unlock their own photo files has rocks in their head!
Um, DNG is an open, unlocked format. Where did you learn that a subscription is necessary to unlock the files, especially given that their website goes to pains to point out the exact opposite? It's fine to be skeptical, but not when it crosses over into baseless accusations.
...another subscription model? ...Really? Have you heard of Adobe? Not if my life depended it on it. Thanks...I'll pass.
Given that storage is dirt cheap, I wonder what the actual time + subscription vs. storage cost is.
I'm inclined to suggest it's not economic.
Hi, dev here. Thanks for all the comments. I am Also amateur photographer. For me, the main advantage of having the files compressed is that they are way easier/faster to handle. And that I can hope to fit them all in one place.
For the same price as a lage 5TB external drive, I bought tiny 1TB SSD. Backing it up takes less than one hour vs more than a day.
Also, if you use any cloud or NAS storage, the 80% file size reduction really helps.
Personally, I am not a massive fan of the subscription model either, but it seems increasingly hard to make the "classic" model sustainable for newcomer companies.
Sounds a lot like Lossy DNG (from adobe) which I love.
Hi Adam, not exactly: the only thing in common is that a lossy DNG and Dotphoton compressed images would be the same size. But Dotphoton compressed images have significantly higher quality (well, raw quality) and fidelity of a raw file, i.e. they are fully editable and great for archiving.
Lossy DNG also has "raw quality" (i.e. you can adjust all parameters in post).
These products sound very alike, so addressing the differences would be a good idea for Dotphoton/this article to do.
We definitely will make such article in the nearest future, thanks for a good idea!
A real shame there is no windows support,
I shoot weddings with a 42mp camera and recently Ive been downsampling to DNGs in LR, but this introduces visible compression. Clients will never notice this, but I do, and id happily pay for an alternative.
It does look great though. Fingers crossed for windows sometimes soon.
Hi Matthew, I am a wedding photographer too and I totally know the pain :)
I would recommend you to subscribe to Dotphoton's newsletter. We won't spam you weekly, but we will let our subscribers know when there are new cameras added or when (and hopefully very soon ) Windows will be supported. Cheers, Eugenia
Do I understand that correctly: I should convert 12Tb of NEF Files to DNG. Then import the DNG to LR or CO and then I should copy the development settings from the old raw to the new imported DNGs? And then I delete all my original RAW? Sounds like the embedded JPG is compressed and then copied to a DNG. I would jump on it if I just could compress my RAW and that's it. Nothing to change or to import and copying development values.
Hi Bert, ideally you should compress your NEF files first thing after your transmit images from the camera, and then you can go with LR or CO workflow using compressed files.
I guess they use JpegMini Pro to compress the embedded JPEG and then generate a DNG from it.
Not at all. Dotphoton uses its proprietary compression technology which comes from quantum physics research. It has nothing to do with JPEG's lossy compression.
I don’t want to rain on your parade and to each his own. But seriously guys? Even on my phone with a blind eye you immediately see a difference for example in the saturation of her hair color!
Also if you have thousands of dollars invested in professional camera equipment why cheap out on storage and in stead turn to lossy compression effectively making your said investment in camera gear more or less meaningless.
Apparently this company has learned nothing from Adobe's recent troubles. Why is this software $49 PER YEAR? Just sell it to me and be done with it. What happens when I stop paying the rental fee?
Hey, Phillip, please don’t think we don’t know what you are talking about. We weren’t very happy with the latest pricing events in the photography/graphic industry either.
Problem is, subscription model is the only one currently known to humanity allowing business planning in software development for indie startups and at the same time keeping customers happy with regular updates and keeping prices low.
As to non-renewal of your subscription — it’s damn easy: NOTHING will basically happen to you or your images. Your images will be yours forever where you keep em since they are open format DNGs, but you will not be able to compress more images and will have to spend 5 times more on storage as you used to before Dotphoton Raw.
You could learn more about subscription-based models in any article by any startup founder on Medium, I really encourage you to 🖤
This is the ONLY business model that allows you to produce software?!? How did developers get anything done before the subscription model was available?
It was there for ages, you just thought you were paying for a “version 2”. But, let’s be honest, your second option was using outdated software with zero bug-fixing quickly losing interoperability and depriving you from innovations and this is where you also start losing both your money and time.
And yeah, this is the only model which allows our customers to have unlimited access to the permanently updated camera support database.
There always will be people unhappy about subscription or payed updates, but our market research showed photographers already pay loads and it would be easier for them to have that pricing split over time.
Well this false mantra "archive files so that they can be preserved for decades to come" about DNG will probably never die. It is essentially not true. I learned hard way. Adobe did everything to detour any thoughts against their new format, to make it popular. It seems camera manufacturers did not take care of DNG (only Panasonic and Pentax uses it natively), so they started to advertize space saving and compatibility preserving feature. The last one is fake.
The truth is DNG is just container is it is PEF, RAF, NEF, ... and inside this container are uncompressed or lightly compressed 16-bit TIFF files that represents all original captured data. Lightroom or CaptureOne or any other raw developer reads this file and debayer it before further editing. CameraRAW does that on technology of DCRAW, as meny (if not all open source, free or cheap alternatives do). CaptureOne uses their own technology which they tuned for each camera specifically. There were numerous updates latelly just for FUJI cameras. They want to represent files the best. Have the best profiles among all. Lr is sloppy and often does not have separate profiles for each version of cameras but are more generic. Since they rely on DCRAW, they open all legacy cameras in existence. While C1 opens only recent ones since they started to build up database.
DNG is NOT future proof of any way better than proprietary format. What DNG converter does is extract original TIFF image from raw container and repacks it in DNG. Nothing more. Since they open everything, they will open everything forever until they wipe legacy support out or when they stopy use DCRAW for extracting.
Real example: In C1 there is no support for older FujiFilm bridge cameras that had raw option (S5600, S9500 PRO). I used those two for years, so not being able to open them was deal breaker. I quickly converted them into DNG and images was still unrecognized by C1!! Of course. If I use DCRAW to extract original bayer TIFF image out from RAW, both images were bit-to-bit the same, regardless if I pulled it from RAF or "converted" DNG (repackaged). Why is fine is DNG does not alter the image whatsoever. Since it is software process with plenti if time/cpu, compression is better. So converting into DNG might be beneficial. But it might some metadata be lost. Some (not so important). I think most users do not convert images to DNG since they gain nothing, but 10% of space.
If camera is supported in C1 (or anything), it will open as original raw or DNG. If camera is not supported, than rewrapping to DNG will do nothing. It will work only if it is converted into lossy DNG (original image is debayered by DNG converter than it is saved using JPEG compression). While I can say it retains 90% of fidelity and for most cases it is almost like original raw, so quite usable and good solution for not so important files, this is not a solution of one would want to avoid debayering being done by Adobe. C1 conversion is slightly better at raw level. Slightly. In fact raw pixels can not be "develop" better than they are. All raw developers make exactly the same file IF noise reduction and any effects are off. Differences comes with noise reduction and representing curves. Since the most true to original translation is DCRAW itself, I use it when I want maximum control over file. Old Fuji files are best when extracted to 16-bit TIFF and I edit that further in C1.
Hi Mihael,
dev here, and I must say that DNG does help a lot in interoperability, as it is a well documented format. It's important to distinguish the actual raw image data and the metadata. Indeed, DNG copies the raw image data unmodified, but where it helps is that it translates the metadata from each proprietary file format to the DNG "standard", so that an application only needs to be able to read DNG to open and interpret the raw data. Profiles, color matrices, and a lot of other data which is important in the development pipeline is embedded in DNG, and applications can make use of that in their development pipeline. So, DNG does help in the long-term preservation of an archive of RAW files. Indeed, with non-Bayer cameras (e.g. Sigma, some Fuji) DNG will only work if you embed the RGB tiff or JPEG.
C1, and other software have each their own debayering/denoising/processing pipelines these are advanced and give (at least subjectively) better results that DCRAW. These can work with bayer DNG regardless of whether they know the camera or not. But how do you get the raw data extracted with DCRAW into C1 or other?
My point is that since DNG does not change actual raw data, it does not make uncompatible camera compatible. So it does not warrant future compatibility for that camera. Yes, DNG itself is future proof, but if raw image inside is from obsolete camera, future applications will not benefit from being saved in DNG container. When new cameras come to market, all application vendor compete who will add that camera sooner (or better). So for all future cameras we are safe they will be compatible, regardless. Also, there is little to none chance or reason, why established application will wipe out older cameras once they already made pipelines for them. It's just database. They all want to support "XYZ" cameras and compete with this number as well. So the only case when older cameras will not be supported in future applications is with newcomers. They will ad just most popular legacy cameras, and just focus on new ones, that will come. C1 is like that. They did not bother with obscure cameras like Fuji S5600. And for those cameras DNG could not make a difference in those applications that does not support those cameras. That was my point.
(Fuji S5600 still have Bayer matrix, but it is rotated 45 degrees.)
I pixelpeeped a lot of applications, including compare that to DCRAW. What I can say, C1 is no better than Lr in "resolving" pixel data, since it can not be. Image stored there is the same, none of those application is creating non-existing data. Also de-Bayer process is same, for particular camera. The most obvious difference is to normalize RGB pixel data, to match certain sensor, so that white is white in the end. That can be analyzed using RawDigger or DCRAW. Both can extract raw image or de-Bayered one.
The only difference between developing applications is how they handle clipped highlights and noise. Especially color noise. And after that how they fake sharpness. Mostly in area how to mask halos around edges, to cover up contrast sharpening. At (color) noise handling and sharpening C1 is in lead against Lr. But Lr is meant to be used with Ps, where C1 advantage vanish. After using Lr since 1st version and C1 since V10, I will say that both can the same end IQ for 99% of images (few extreme cases give lead to one or another. Where C1 outclasses Lr is convenience and speed. Due to raw speed of application and better workflow (in most parts, some are still better in Lr).
I use DCRAW to extract and de-Bayer original raw data into 16-bit TIFF (either linear or not) - exactly what all DCRAW based application do. Than I use denoiser like Neat Image and than edit that 16-bit TIFF in chosen way.
Benefits: DCRAW is the best at recreating (smart faking) clipped areas. Also it leaves image as is, without processing. Other application always add something to image even if everything is zeroed. But if image is imported as TIFF, they handle it as is and only developing part of pipeline applies. I have more control that way.
Deficiency: In everyday life, that workflow is too cumbersome to edit 1000's of images. For old RAF files not supported in C1, I can still convert them into JPEG-DNG using DNG converter and for most uses it is good enough. If not, I can edit just those images long way.
PS: I loved DNG + Lr for a while because image developing could be saved into files, and that was read in WindowsExplorer (if DNG codec was installed) and even thumbnails reflected the editing. Also if database in Lr crashed, I just read metadata from files. What I did not know is that metadata changes file (size). So when I had hard drive crash I had hard time to recover all from backup, since files were not matching so MD5 was useless. I had a lot of manual comparing. How could I know if files does not match since corruption or just different at few bits due to metadata saved in. Now I use sidecar files. But I also does not save edits into them, since I rather save finished image. If I need image after many years I will most likely re-edit it with new workflow (and my better skill).
Hi Mihael,
Interesting tech discussion. My understanding is that in order to make the DNG, the DNG converter or other raw library (like DCRAW) does need to have the original camera in the database. This is because it must have a way to read and understand the data and metadata in the original raw. Then, the data is copied as-is to the DNG file, and the metadata is split into two: 1) "standard" metadata, which is required to correctly develop the DNG, this contains stuff like color matrices, black/white levels, bit depth, ISO, exposure time etc. This metadata is mapped to the respective fields in the DNG file.
2) "non-standard" metadata which is typically used only by the manufacturer's own software. This is copied as-is, and not used by lightroom, C1 or other, besides the manufacturer's own SW.
To read a DNG, you do not need to have the specific camera in the database, and DNG is truly future-proof in this sense.
There might be some exceptions to this rule, as DNG only supports Bayer data, so that some cameras (Fujifilm and Sigma) that have non-bayer sensors embed an RGB image into DNG. However, if this image is RAW, there are currently no software that I know of that can actually dovelop it.
About the debayering, of course this is a matter of how much you pixel-peep, but during testing of Dotphoton Raw we opened many images in many raw converters (~10), and I found that there is quite a difference in how debayering is done in each, although which is better is a matter of taste. Also, I believe that debayering, denoising and sharpening need to be well-matched algorithms to get the best results out.
You are right that after many years, if you open the dng or raw, it will most likely look different because of the new workflows and development pipelines in the various software, also because the color space, and resolution of both print and screen has changed a lot.