If you're one of the hundreds of millions of people who uses Snapchat every day, you've probably played with its lenses that transform your face into all sorts of humorous, interesting, and horrifying things (trust me, I face-swapped with a clock once, and I looked like a Dali disaster). Making all that magic happen takes a lot of math and computing power, however, and the process is fascinating.
Making a filter that recognizes and modifies faces in real-time is no small task, especially when you consider that a computer doesn't "see" a face, it merely sees RGB values for each pixel. So, how does a computer detect a face? It uses something called the "Viola-James Algorithm." The idea is quite simple: light and dark areas of a face generally occur in a predictable way, so if we look for those abstract patterns of contrast, we might suspect the presence of a face. If enough of these patterns occur in the same area, we'll conclude that indeed, a face is present.
Once a face is detected, a "mesh" that divides the face's structure is applied, then customized to fit the target face, again using areas of contrast as the guide. Once the mesh is in place, applying transformations, rotations, scaling, or most anything else you can think of is the next step. I've worked with meshes when solving partial differential equations before, and after leaving my code to run overnight to solve one problem, it amazes me that these can be generated and manipulated in real-time on a phone. While it's all for fun with Snapchat, it's a sign of how remarkable our phones have become. Check out the video above for more.
Cool! Thanks a lot for sharing this, it's incredible to see how good this kind of technology has become and most of all I'm amazed to see how powerful smartphones are nowadays.
My smartphone has four time the RAM of my first PC :D