Content-Aware Fill in its concept is nothing entirely new. But fresh off the back of their photo-merging software, NVIDIA has released fresh technology that can reconstruct photos like never before. Check out the photo examples to see how extraordinary the technology has become.
The company released the below image to demonstrate the full effects of the technology. Removing chunks of the portrait, the technique is challenged to fill in the blanks. We’re told the difference between NVIDIA’s new tool and Photoshop’s existing capabilities is that the former allegedly “understands” what the original photo should look like.
Current Content-Aware Fill systems take information from pixels surrounding the blank area; NVIDIA’s, on the other hand, knows what is missing and tries to replace it. Were an eye removed from the face, it wouldn’t attempt to fill the gap with the eyebrow, nose or cheek, for example – but would know to instead attempt to recreate an eye.
Photoshop's attempt
NVIDIA's result
It’s not perfect and likely never will be, but it’s at least a realistic effort. In a smaller surface area (and with an area less detailed than an eye), the technology could likely fill in a blank that would appear normal to the average viewer.
The name of the new technology was announced by NVIDIA researchers as “Image Inpainting for Irregular Holes Using Partial Convolutions.” They explained:
“Our model can robustly handle holes of any shape, size location, or distance from the image borders,” the researchers write. “Previous deep learning approaches have focused on rectangular regions located around the center of the image, and often rely on expensive post-processing. Further, our model gracefully handles holes of increasing size.”
It was created using 55,116 random masks of holes and streaks which were trialed on a wide range of example photos. In cross-referencing the original, unedited photos with those that researchers had deliberately created holes in, the system “learned” how to guess what should be in the gaps.
More images – which include much greater detailed landscapes – are below to demonstrate the capabilities of this new tech.
I wonder how it would do in real world use. Such as reparing worn photos or torn ones, etc.
I think that will be one of it's better uses when it's released. I think they even marketed it as such.
I see possibilities of it upping low resolution photos to higher resolution. Very cool!
Can't wait to see that working in photoshop nativeley / by plugin. Sounds great even if not perfect would save me lot of time trying to fix things uhh...
Photoshop?! This will be native in your system. More work taken away form retouchers.
I'm unsure what you mean in my system - in camera? :)
Well, I meant your computer running on an nVidia card.
Those are researchers from nVidia, but technology itself has to be used itself on a photo that a user select (not everything on computer). For that you need any kind of software. Best would be to use a industry standard for those manipulations, so Photoshop would be perfect.
I'm not sure how you expect this to work on your card? It can be a set of instruction on a card for sure, but in the end a program would need to call those instruction to make them work.
I meant any app that can access the processes in the nVidia card would be able to do this, it wouldn't need to be an app like Photoshop. There could well be an applet supplied with the drivers that could do it.
Holy shit