5 Ways To Blend The Seam Between The Real & Virtual World in Virtual Production

On the set of a virtual film production, the cast and crew know what's real and what's virtual. But, for the audience, the two should be seamless.

So how do filmmakers decide what should be real versus virtual?

Your first instinct might be to make as much virtual as possible. This line of thinking has apparent advantages. With virtual production, you don't have to worry about sourcing physical set dressing. Virtual production also makes modifying digital assets—like changing the color or size of digital props—a breeze.

But it doesn't matter how easy your production is if the end result looks fake. And that's bound to happen if 100% of your set is virtual—or if you blend the physical and virtual worlds improperly.

Most TV and film productions aim for the highest level of realism. Directors usually don't want their movies coming out looking like video games or cartoons.

Here are 5 tried-and-true ways to blend the real and unreal, and thereby make virtual production look more realistic:

1. Use Layers Of Physical Set Decoration to Add Depth

Adding physical objects in the foreground and mid-ground adds dimension to your shots ups the realism.

If you look at this video from Resolution Studios, you'll see that they placed a cooler, tree, and fire pit in the foreground and a chair and camping stove in the mid-ground. This positioning makes the shot look real because it adds depth that you wouldn't get if it was just the actor and the LED wall.

Experienced virtual production filmmakers will give this rule of thumb: the more layers of fore- and mid-ground you add in front of your virtual background, the more your audience's eyes will be tricked by it.

2. Strategically Use Bokeh & Shallow Depth Of Field

Another reason to add physical set dressing is due to an occasional technology constraint: The processing power required to run big scenes on LED walls can sometimes be so high that your environments may not look photorealistic up close.

When this happens, the camera operator can get around this is by adjusting aperture, or otherwise creating bokeh or a shallow depth of field.

This blurs out the background and mid-ground to a varying degree. It works best when you have actors in the foreground and props or set decoration in the mid-ground. Here's an example of a close up shot where you'd never guess the background was virtual:

3. Use Camera Movement To "Sell" The Illusion

Camera movement is fundamental to film production — whether it's a dolly shot, panning, handheld, or Steadicam. In virtual production, you can add a tracker to the camera and achieve parallax in moving shots, which helps make the shot look and feel natural.

In general, when the camera is moving, the audience's eyes are more easily tricked by the virtual art in the background—if things are moving with the right mathematics (which camera tracking accomplishes automatically).

In the picture below, from the set of the TV show Resignation, you can see one type of camera tracking system.

In the still shot here, you can tell that the background wall is fake. But when you see the shot in-camera with the camera tracking active, it looks like this:

Movement, especially handheld shots, can mask aspects of an environment that doesn't look as real as you want.

4. Use Art To Literally Cover The Seam Between The Floor And Wall

There is usually a literal seam in virtual production between the floor and the LED wall. If you look at it directly, it will look fake. But you can design your scene around anything that naturally would be a seam if this were a real environment.

For example, SMASH Virtual recently showed us a demo of a temple complex where they carted in a bunch of sand and rocks from the abandoned lot across the street from their studio. The image below shows a little wall they built that covers the break between the floor and the wall. Everything below the wall is dirt and debris, whereas everything above is the LED wall.

5. The Big One: Match Your Physical and Virtual Lighting

The #1 thing that can make virtual production look fake is when the lighting in the physical world does not sync with the lighting in the virtual world.

Conversely, the #1 thing you can do to blend your physical and virtual into one world is to get the lighting to match.

The position of the virtual lights should correspond with the placement of the real-world lights. If this does not occur, it could appear strange to viewers, as the illumination would be coming from various directions ,and the shadows would not be positioned correctly. Even if you don't consciously think about this, your brain will register it as phony.

In a traditional green screen shoot, the visual effects department must match the lights in post-production. But with virtual production, you do the matching while you're shooting. The ability to tweak the physical and virtual lights simultaneously benefits the production process. It saves time (and therefore money) because you can see the final outcome in-camera. And it eliminates the headaches caused by guessing what the final VFX will look like based on previsualization.

On the other hand, with a green screen, you'd have to light the physical scene first as best you can and then, maybe months later, do the final VFX lighting. In that scenario, if you really screwed up on the green screen lighting, you'd have to do reshoots.

Below is an image, again from the set of Resignation. This scene has large windows behind the actor at the desk. The windows have sunlight shining through them which are virtual lights. You can see the physical lights above and behind the actor, illuminating him and matching the virtual and physical worlds.

And the result is what we call movie magic: