Content-Aware Fill in its concept is nothing entirely new. But fresh off the back of their photo-merging software, NVIDIA has released fresh technology that can reconstruct photos like never before. Check out the photo examples to see how extraordinary the technology has become.
The company released the below image to demonstrate the full effects of the technology. Removing chunks of the portrait, the technique is challenged to fill in the blanks. We’re told the difference between NVIDIA’s new tool and Photoshop’s existing capabilities is that the former allegedly “understands” what the original photo should look like.
Current Content-Aware Fill systems take information from pixels surrounding the blank area; NVIDIA’s, on the other hand, knows what is missing and tries to replace it. Were an eye removed from the face, it wouldn’t attempt to fill the gap with the eyebrow, nose or cheek, for example – but would know to instead attempt to recreate an eye.
It’s not perfect and likely never will be, but it’s at least a realistic effort. In a smaller surface area (and with an area less detailed than an eye), the technology could likely fill in a blank that would appear normal to the average viewer.
The name of the new technology was announced by NVIDIA researchers as “Image Inpainting for Irregular Holes Using Partial Convolutions.” They explained:
“Our model can robustly handle holes of any shape, size location, or distance from the image borders,” the researchers write. “Previous deep learning approaches have focused on rectangular regions located around the center of the image, and often rely on expensive post-processing. Further, our model gracefully handles holes of increasing size.”
It was created using 55,116 random masks of holes and streaks which were trialed on a wide range of example photos. In cross-referencing the original, unedited photos with those that researchers had deliberately created holes in, the system “learned” how to guess what should be in the gaps.
More images – which include much greater detailed landscapes – are below to demonstrate the capabilities of this new tech.