Real-Time In-Camera VFX: Next Generation of Filmmaking

Visual effects (VFX) are a great way of achieving spectacular results when the budget doesn't allow filming for real or it's too risky for the actors. Up till now, these were achieved mostly by filming on green or blue screens. This new way of filmmaking presents a way for shooting scenes with VFX directly in camera.

For the last several years I've seen real-time VFX compositing, but the actors areĀ  still in a green or blue screen environment. In this presentation the guys from Unreal Engine have managed to utilize LED screens for both lighting and projecting visuals. Together with the help of some real-time camera tracking, the filmmakers can create impressive scenes that are naturally blending with the actors and the set. The realism now depends not on the way the characters are keyed-out from a green screen and composited into the computer-generated scene, but from the realism of the projected imagery on the screen and additional on-set lighting. If you have been working with 3D, you will know about using HDRI backgrounds to light a scene which is basically what is being done with these LED walls. The colors of the projected screen reflect perfectly over the actors and the set to create realistic lighting conditions.

While this set up is very handy for directors and DPs, it will be interesting to hear how actors feel like being filmed for VFX in comparison to the green or blue screens.

Tihomir Lazarov's picture

Tihomir Lazarov is a commercial portrait photographer and filmmaker based in Sofia, Bulgaria. He is the best photographer and filmmaker in his house, and thinks the best tool of a visual artist is not in their gear bag but between their ears.

Log in or register to post comments

This is insane! This is basically the start of the holodeck.

Really cool. This seems to be the next evolution of what they did when filming Oblivion. For the skytower sequences they used projection mapping to not only eliminate blue/green screen keying, but to also light the scene itself...

Exactly what I thought when I first saw the technology. It's not something unseen before, but it was nicely explained in the video.

I think the big difference here is that the scene is computer generated, whereas in Oblivion it was recorded video.

Yes, I don't remember they did real-time camera tracking on Oblivion.

So I know you guys are owned by Sony and Apple but any change the brand new awesome Blackmagic pocket 6k will get any attention or are you waiting to do the comparison video where you can say it's not as good as Sony or an iphone?

How does your comment have anything to do with this article?

Article talks about 3d modeling but no articles about new camera. Which one would match this site better? This is awesome but so are new cameras and the site is called fstoppers so wondered why it wasn't mentioned.

The word "camera" was mentioned three times in the article. I hope that counts.

The word "Sony" can be found in the trending articles on the right. I hope that counts.

The word "Apple" is in the related articles. I hope that also counts.

But for those who are into filmmaking and care about something else than just cameras there are two hundred fourty one words somewhere above the comments talking about that.

That is a question you should send to the Fstoppers staff...and they have done articles on the blackmagic. Oh...WAIT...I just scrolled down and theres an article comparing the Blackmagic to several other cameras. And it took me 30 seconds to find it.

6k just came out a week ago. Did you find that article?

That was really awesome

Ok, that's bonkers.
Time to look for another job.

I don't think you should. That's like saying the invention of the crane (or "Technocrane", as people usually refer to it) will make Steadicam operators to be out of business. It's just a different technology. If you want to have a big set with real people and assets around, it will be quite expensive to light it with huge LED screens.

Learn to operate that rig. New job. :-)

I have a robot to do that. ;)

The Terminator wasn't wearing a helmet when he was doing his motorcycle riding either.

The close-ups with the gun are usually filmed with the bike on a platform so that it's safe for the actor. Also, back then kids were riding their bicycles without helmets and people weren't that overly-anxious. I've never used a helmet with my bicycle either.