Real-Time In-Camera VFX: Next Generation of Filmmaking

Visual effects (VFX) are a great way of achieving spectacular results when the budget doesn't allow filming for real or it's too risky for the actors. Up till now, these were achieved mostly by filming on green or blue screens. This new way of filmmaking presents a way for shooting scenes with VFX directly in camera.

For the last several years I've seen real-time VFX compositing, but the actors areĀ  still in a green or blue screen environment. In this presentation the guys from Unreal Engine have managed to utilize LED screens for both lighting and projecting visuals. Together with the help of some real-time camera tracking, the filmmakers can create impressive scenes that are naturally blending with the actors and the set. The realism now depends not on the way the characters are keyed-out from a green screen and composited into the computer-generated scene, but from the realism of the projected imagery on the screen and additional on-set lighting. If you have been working with 3D, you will know about using HDRI backgrounds to light a scene which is basically what is being done with these LED walls. The colors of the projected screen reflect perfectly over the actors and the set to create realistic lighting conditions.

While this set up is very handy for directors and DPs, it will be interesting to hear how actors feel like being filmed for VFX in comparison to the green or blue screens.

Log in or register to post comments

18 Comments

Martin Van Londen's picture

This is insane! This is basically the start of the holodeck.

Raymond Craig's picture

Really cool. This seems to be the next evolution of what they did when filming Oblivion. For the skytower sequences they used projection mapping to not only eliminate blue/green screen keying, but to also light the scene itself...
https://youtu.be/DzqdzBZRM9k

Tihomir Lazarov's picture

Exactly what I thought when I first saw the technology. It's not something unseen before, but it was nicely explained in the video.

Chris Cameron's picture

I think the big difference here is that the scene is computer generated, whereas in Oblivion it was recorded video.

Tihomir Lazarov's picture

Yes, I don't remember they did real-time camera tracking on Oblivion.

David Love's picture

So I know you guys are owned by Sony and Apple but any change the brand new awesome Blackmagic pocket 6k will get any attention or are you waiting to do the comparison video where you can say it's not as good as Sony or an iphone?

Steven Magner's picture

How does your comment have anything to do with this article?

David Love's picture

Article talks about 3d modeling but no articles about new camera. Which one would match this site better? This is awesome but so are new cameras and the site is called fstoppers so wondered why it wasn't mentioned.

Tihomir Lazarov's picture

The word "camera" was mentioned three times in the article. I hope that counts.

The word "Sony" can be found in the trending articles on the right. I hope that counts.

The word "Apple" is in the related articles. I hope that also counts.

But for those who are into filmmaking and care about something else than just cameras there are two hundred fourty one words somewhere above the comments talking about that.

That is a question you should send to the Fstoppers staff...and they have done articles on the blackmagic. Oh...WAIT...I just scrolled down and theres an article comparing the Blackmagic to several other cameras. And it took me 30 seconds to find it.

David Love's picture

6k just came out a week ago. Did you find that article?

Steven Magner's picture

That was really awesome

Rob Mitchell's picture

Ok, that's bonkers.
Time to look for another job.

Tihomir Lazarov's picture

I don't think you should. That's like saying the invention of the crane (or "Technocrane", as people usually refer to it) will make Steadicam operators to be out of business. It's just a different technology. If you want to have a big set with real people and assets around, it will be quite expensive to light it with huge LED screens.

Spy Black's picture

Learn to operate that rig. New job. :-)

Rob Mitchell's picture

I have a robot to do that. ;)

The Terminator wasn't wearing a helmet when he was doing his motorcycle riding either.

https://youtu.be/uGVZbqEIteo

Tihomir Lazarov's picture

The close-ups with the gun are usually filmed with the bike on a platform so that it's safe for the actor. Also, back then kids were riding their bicycles without helmets and people weren't that overly-anxious. I've never used a helmet with my bicycle either.