Well, even with that, at a macro level filmmaking is still a Waterfall Process. You can’t alter the order of the big steps:

1. Pre-Production, which has to happen before:

2. Production, which has to wrap up before:

3. Post-Production.

Or… can you?

For the Resignation pilot, our post-production partners at Prysm Chicago volunteered to do something unique. We had a couple of their VFX artists actually create some of the virtual environments for the pilot’s 22 virtual locations.

That meant that the post-production team had an excuse to join in our pre-production planning meetings, production design meetings, and shoot days themselves. And that turned out to be game changing. We iterated effectively on art in a way that got us a higher quality outcome, and we wasted no time in the communication or handoff for what needed to be fixed in post. Post was there for it all.

We don’t think this is a one-off type of situation. If your post-production house has good visual effects chops, we’d dare say you SHOULD get them involved in creating your virtual environments. Or at least get them in the room where Virtual Art design decisions are made.

Engaging your Post team earlier can have positive second-order effects for the whole production. For us, it paid off in the form of a more agile feedback loop in pre-production, and a more efficient post-production timeline.

Watch Prysm Studios talk about their experience on Resignation in our final episode below. (And by the way, if you’re looking for great VAD and post services in Chicago, you should hire them.)

Thanks for joining us as we shared the lessons from this journey. If any of this resonates with you—or if you have any strong counterarguments from your own virtual production experience, we’d love to hear your perspectives.

We’ll share occasional updates on Resignation as it makes its way to screens, but from here on this Substack will go back to our usual One Post Every Week Or So schedule.

Stay tuned for our upcoming State of the Soundstage Report and some chewy content on virtual production cost comparisons.

Til then, we hope The Making of Resignation inspires you to change the way you work. We know we’re never going back to the way we used to do film, that’s for sure!

—Team SHOWRUNNER

1. Locations team preps the shooting location

2. Everyone else waits

3. Art department decorates the set

4. Everyone else waits

5. Grip & Electric department lights the scene

6. Everyone else waits

7. Camera team sets up the upcoming shot(s)

8. Everyone else waits

9. Actors get called to set, and Director and Camera team shoot the scene

10. Everyone else watches

This process is not arduous on purpose. G&E literally can’t get a set lit if it’s not decorated yet. Camera can’t really dial in a shot if the set isn’t lit yet. And all these people can’t fit in the same space all at the same time.

Virtual production, however, gives us a chance to hack the filmmaking waterfall.

When we planned out the pilot shoot for Resignation, scheduling constraints dictated that we had to squeeze a star actor's scenes all into two shoot days. Normally in this situation, you’d rewrite the script, or go drum up more money from your financiers to add extra shoot days, or find something to compromise.

In our case, we purposely took on the challenge of shooting everything with our star in two shoot days. And that meant a day with 8 locations.

But you can’t do 8 locations using the above work-and-wait-and-work-and-wait waterfall process.

We literally had no way to accomplish this without rethinking the way things are done.

Luckily, we were shooting in virtual production—which meant locations could be transported to us in the click of a button. And using our SHOWRUNNER technology, we could save lighting and virtual location settings ahead of time and instantly load them up.

Perhaps most importantly for this challenge, on these days we were shooting at Resolution Studios in Chicago, where we could configure two LED walls inside the main soundstage.

So we designed a process where the crew could bounce back and forth between the two walls in what you might call a Double Waterfall methodology:

1. World operator brings up Location A on LED wall A, and Location B on LED wall B

2. Art department decorates Set A

3. This first set is the only time everyone else waits

4. G&E quickly lights Set A, (Gaffer loads up pre-light settings from the day before using SHOWRUNNER), while Art department decorates Set B.

5. Camera team sets up the shots on Set A, while G&E lights Set B.

6. Actors get called to set, and Director and Camera team shoot Set A.

7. Actors, Director, and Camera pop over to shoot on Set B, while Art and G&E turn Set A into a new location.

8. Actors, Director, and Camera pop over to shoot at the new Set A, while Art and G&E turn Set B into a new location.

9. Repeat the process, bouncing between the two sets.

This might sound exhausting for the crew (especially Art), and it was. But we wrapped early on our 8-location Do-Si-Do day.

This day took careful planning—and truly played out like a giant choreographed dance.

And afterwards, we decided we’ll do as many Do-Si-Do days as we can in our future productions. It was exhilarating and productive and collaborative—and our crew got home in time for dinner.

The moral we learned from this story is this:

Virtual production can allow a few people do more planning up front, so more people can waste less time later.

Enjoy Episode 7 of The Making of Resignation:

See you next week with our final episode in this series.

—Team SHOWRUNNER

P.S. Big shout-outs to Big Works (our intrepid Art department), 2nd Decade Productions (our Camera team) and Miller Creative (our VAD and World Operator team for the Do-Si-Do). If you’re ever shooting VP in Chicago, you should hire them.

“What do actors think of it?”

It makes sense that this is top of mind for producers considering whether the learning curve is worth it.

Even if costs can go down and creativity can go up (which if you’ve been watching this Making of Resignation series, you know we’re convinced are both the case), you can’t fault a producer for saying:

“Yeah, but what if Tom Cruise doesn’t want to act in front of an LED wall?”

When we shot the pilot for Resignation, we sought out actors who would be willing to be experimental with us. Not just from a technology and workflow standpoint, but also in doing an indie TV show. That’s not your usual gig. We briefed them endlessly (it probably felt like to them) about what they were walking into.

And then when they walked in and saw the LED volumes for the first time, they were awestruck. As performers, every single one of them saw the creative possibilities. Every single one realized they were about to get the chance to try more takes, do more pages, and wait around less each day.

And whereas every one of them is a pro, we’re glad we gave them the heads up on virtual production before they walked in the door.

Our rule: Give your actors the virtual production tour ahead of time.

(The reason is not about persuasion…)

We haven’t found an actor who isn’t willing to perform in front of a virtual background. Turns out, actors don’t need to be convinced to do virtual production. After all, Tom Cruise has been acting in front of virtual backgrounds since at least the ‘90s.

However, actors want to be mentally prepared for the environment they’re going to perform in. We recommend giving your cast a tour of your LED studio before their first shoot day. (And if they can’t swing that, schedule-wise, at least show them that famous Youtube video on the making of the Mandalorian).

Most actors are curious about the way it all works, so give them the inside look before they walk in. The ulterior motive is to help them accurately visualize their performance in the space they’ll be in, so they can focus on acting on the day.

Sure, a pro can walk onto any set and knock it out of the park (and one of our players did just that). But the magic of virtual production is so in-your-face that letting your cast process the crazy environment before the day they have to show up to work their magic is a pro move on the filmmaker’s end.

With that preamble, we’re excited to share Episode 6 with you, and introduce you to the soon-to-be-award-winning cast of Resignation!

Stay tuned for Episode 7 tomorrow! It’s the doozy where we shot 8 locations in a single day.

— Team SHOWRUNNER

And it turns out that some configurations of LED volumes can be quite echoey—and the computers that run virtual production can be loud (or require lots of fans).

In the past, our team had always shot on flat or gently curved LED volumes in large soundstages with good acoustics. But we came across an auditory land mine early on in our pre-production process for our 100% virtual production TV pilot Resignation, when we started prepping inside of a brand new J-shaped volume with an awesome full-resolution ceiling. The new studio was efficient, gorgeous… and had sound issues in unexpected places.

It turns out, though, that there was a solution to our sound issues that was both elegant and ironic:

When shooting in an LED volume, plan on recording your audio like you would outdoors. It solves any virtual production audio issues 99% of the time.

Watch today’s episode for the breakdown:

See you tomorrow for Episode 6—

—Team SHOWRUNNER

But if you’re a filmmaker, you know that actually shooting in an LED studio is not as straightforward as a typical studio setup. In this episode, we show how our director and cinematographer solved problems and inspired a film crew that was entirely new to virtual production.

Shooting A Movie Ain’t Easy. Shooting It A Better Way For The First Time Ain’t Easy Either. But It’s Worth It:

Stay tuned for Episode 5 tomorrow!

—Team SHOWRUNNER

Last week, we posted the first » two» episodes of our behind-the-scenes series about making a TV pilot entirely in virtual production.

In today’s episode, you’ll see how our Production Designer and his intrepid team of artists at Big Works Industries (Chicago) integrated physical set decoration with Unreal Engine and video plate backgrounds to make our virtual locations actually look real.

Check it out:

Stay tuned for Episode 4 tomorrow—

—Team SHOWRUNNER

Today, we’re posting Episode 2, which is about creating virtual locations—and how challenging ourselves to do EVERY location with virtual backgrounds forced us to think outside the box and learn lessons that any virtual production filmmaker can use.

The moral of the story:

It’s possible to shoot a lot more—and a lot more types of locations—in virtual production, than the common wisdom says you can:

If you missed Episode 1, check it out here »

We’ll be back next week with Episode 3!

– Team SHOWRUNNER

1. We decided to make an independent TV pilot, 100% in virtual production. 

2. The show is not sci-fi; it’s a sitcom. “They” say virtual production doesn’t make sense for sitcoms with normal locations: offices, classrooms, bathrooms, bars, coffee shops, and the streets of Chicago. So we challenged ourselves to do all of those.

3. There were 16 locations in this 30-min episode. “They” said that’s too many for an indie budget, especially in virtual production. So we upped it to 22.

4. We made a list of camera shots “they” say you can’t get in virtual production, and wrote them all into the storyboards: walk-and-talks, whip-pans, ceiling shots, etc.

5. And then, as you’ll see in Episode 7, there was one day where we shot 8 locations in a single day. “They” didn’t have to tell us that was absurd; we knew it was absurd.

6. We aimed to do all of the above within a budget that’s a fraction of what “they” say you can do virtual production for. If Mandalorian is 8x cheaper than a Star Wars movie; our show would be a level of magnitude cheaper than that.

7. And we did it all with a rule: No overtime for the crew; and all crew get paid a real living wage.

Does this all sound insane? It was.

Over the next few days, we’re going to share the story of what happened, along with specific innovation and teamwork lessons from them.

So without further ado, here is:

The Making of Resignation:

Episode 1: So You Wanna Make A TV Show?

[click the video thumbnail to watch]

***

We believe that the most exciting innovation in Hollywood right now—virtual production—can be accessible to more than just the big shots.

Big-budget TV series like The Mandalorian, House of the Dragon, and 1899 use newfangled “virtual production” technology to do something magical: They can put actors and props inside of a room made out of video walls, and make it look like they’re on a desert planet in the morning, an icy castle at lunch, and a jungle in the afternoon.

This technology is incredible. Whereas the latest 2-hour Star Wars movie cost well over $200 million, virtual production has allowed Disney to make 8 hours of the Star Wars series The Mandalorian for a bit more than $100 million. That’s at least 8x cheaper.

But… it still cost $15 million per episode.

So whereas virtual production technology is exciting, conventional wisdom has it that you can only use it for big, splashy, fancy, expensive content. Think: Super Bowl commercials, Formula 1 intro sequences, and The Batman.

Filmmakers who have dug into virtual production are convinced that the costs are going down. (We recently conducted a state-of-the-industry survey that showed a ton of growth and optimism in the virtual production space.)

But even so, you need special expertise; you need expensive computer servers and technicians to operate it all; and, well, “they” say you shouldn’t use it for anything with crazy camera shots, or for grounded content that isn’t big-budget sci-fi.

And that’s the part that got our team going.

The “you can’t” part. 

About a year ago, we launched our technology company to help film studios run like smart homes. Our SHOWRUNNER platform is designed to help filmmakers who don’t have $100 million budgets do what Star Wars does.

And since the parsnippety rumors about what “can’t be done” in virtual production persist, we decided to do something brash: make a whole TV show using this tech to prove that virtual production can actually democratize filmmaking.

Stay tuned for Episode 2 tomorrow.

Thanks for watching!

— Shane, and the SHOWRUNNER Team

When new technology suddenly foists itself upon an industry, leaders can choose from a limited set of approaches. I like to chart them like so:

The most common reaction, I'd dare say, is the middle one—in part because most people don't know much about new technologies as they're emerging. But when you're the leader, doing nothing is a choice itself.

We see this spectrum of leadership reactions on full display today with AI (with some leaders fully embracing it, some fighting it tooth and nail, and some saying "let's wait and see"). But I have my eye on another technology trend in my own industry (filmmaking, where I run a company called SHOWRUNNER).

This trend is called virtual production, where film studios—including those used for commercial advertising—use technology to bring virtual locations into their studios instead of sending crews out to real locations. This technology was especially appealing to filmmakers during the pandemic, which was also when it started becoming available to more of the industry. But the question now that pandemic restrictions have lightened, is how should the technology be embraced in the long term?

This year, my colleagues and I conducted a set of surveys of 800 professionals in the film industry to gauge their opinions on virtual production. The result was a 25-page report called The State of Virtual Production 2023. And the results reveal that the real choice leaders have in front of them has fewer sensible options than the chart above would lead one to believe.

Insight 1: Skeptics, Enthusiasts, and Holdouts Are Split... Unless They Have Experience With The New Technology

The first set of stats that jumps out of the State of Virtual Production report is about the industry's level of enthusiasm. Among the general population of filmmakers surveyed, about half are "excited" about the rise of this new technology, with 22.3% "skeptical."

But when we filter for just those professionals who have a year of experience using the new technology (about one-sixth of the survey population), the enthusiasm jumps, and the percentage of skeptics is cut in half.

The real kicker, though, is when we filter for those professionals with three or more years of experience with the new technology. At this point, the number of respondents who fit the criteria was in the low double digits—so we'll need more data for statistical significance—but rarely do you see a pie chart that looks like this:

When the more experience folks have with a new technology correlates with increased excitement about the technology, it's a pretty clear indicator that the tech is here to stay.

Insight 2: The Amount of Work Being Accomplished With This Technology Is Increasing

When asked, more than 75 percent of filmmakers said they anticipated doing at least some work using virtual production technology this year.

And more than three-quarters of studio owners and operators indicated that they would be doing more virtual production work this year than last. Fifty-seven percent said they anticipated doing "a lot more."

These charts perhaps explain the data on the survey respondents' excitement. If you anticipate getting more work because of a new technology, chances are you'll be excited. But this second set of charts indicates that the excitement is likely merited.

Insight 3: The Industry Is At The Beginning Of Its Learning Curve

The surveys asked a variety of questions about specific tools and techniques in virtual production, but two high-level stats jumped out at me, as they indicate what work needs to be done before every leader in the film industry might fully embrace the new tech.

Those stats are about just how unsure folks are about the quality and technical difficulty of virtual production.

This says to me that some filmmakers are going to still play the "wait and see" game until other, more bullish early adopters figure out these answers definitively.

It's important to remember: that disruptive innovation, as it was originally defined, begins with lower-end solutions that are more cost-effective—and which then increase in quality until there is no excuse to do things the old way. In the virtual production space, slightly fewer people think that the new technology is not cheaper. As the cost of computing goes down, virtual production will likely get cheaper.

But if one is placing bets on the virtual production technology, data indicates that the choice to fight against it is a losing one.

And for those leaders reading this who aren't in the film industry, that is the general lesson of this post:

The choice to adopt any new technology is really a choice between Embrace vs Ignore, not Embrace vs Fight.

In purely pragmatic terms (leaving moral dilemmas aside), the chart at the beginning of this post really only has good options on the left-hand side. That's because, if a new technology ultimately fails to improve things in your industry, spending effort fighting that technology is wasted effort.

It's better to have ignored the new tech, not over-invested or over-stressed about it, and continue focusing on your goals.

However, if the new technology ends up being useful, it's highly advantageous to have embraced it early on. The rising tide will create outsize rewards for your early outsize efforts.

This is why many sensible leaders choose the middle path when it comes to new technology. Waiting for others to place bets, and only diving in when things are proven, is a luxury that organizations with resources can make. But with truly disruptive technology, the catch-up effort will be expensive. (Just look at Kodak.)

The worst choice, it appears, is to deplete your resources by fighting the new technology, so when it becomes inevitable you don't have the time to adopt it. (Just look at Blockbuster.)

It's 2023, and virtual production is a hot topic of discussion on the lips of filmmakers worldwide. The cutting-edge process is making waves across the film, TV, and commercial production landscape, according to data from two industry surveys conducted this January.

Virtual production (VP) is the use of technology to create high-quality locations in studio settings, often using LED walls to make it appear that actors are in faraway places. Think Disney Plus' Mandalorian, HBO's House of the Dragon, and Netflix's 1899. All of these shows were set in detailed, fantastical locations, and shot using VP technology.

www.starwars.com

Some VP studios use green screens, a standard in traditional production, while most major studios utilize LED walls and video game world-building technology to create massive, fantastical, and realistic sets with technology alone.

Since virtual production is still an emerging industry, an initial investment in the technology can be high. However, studios recoup their investment over time through cost savings. For example, VP allows studios to reduce or eliminate expenses related to building sets and flying to remote locations, while still relying on added visual effects (VFX). These benefits make VP an exciting, appealing, and cost-effective production method for many studios. As VFX Supervisor Rob Legato (Titanic, Avatar) told Forbes, “Productions can now do for $25 million what used to cost $100 million."

The same practices that bring costs down also help reduce studios' carbon footprint. Commercial producer Chris Chaundler told Variety, “Our industry hasn't got a good track record when it comes to its environmental responsibilities. Huge crews are traveling all over the world with energy-intensive generators, equipment, and large one-off-disposal sets."

Expensive and environmentally detrimental scenes, like car commercials, chase scenes, and those with explosives, can now be produced at a fraction of the cost with significantly less carbon output.

The Virtual Sky Is The Limit

The growth we've seen in VP recently is just the tip of the iceberg that will be 2023 and beyond. We at SHOWRUNNER wanted to understand how virtual production is affecting the film landscape, how filmmakers and studio owners are responding to it, and what big needs and questions filmmakers have about it. We surveyed hundreds of filmmakers and industry experts to compile our 2023 State of Virtual Production Report. We found the results exciting and enlightening, and hope they help illuminate more of this groundbreaking era of filmmaking.

Download a copy of the full report here.

1. A survey of 771 film industry professionals in Los Angeles; and

2. A survey of 72 virtual production studio owners and operators across North America and the Europe.

While virtual production (VP) is still an emerging industry, the results point to a general excitement and openness to its adoption. However, plenty of filmmakers are also wary of high initial costs for technology that leaves some things to be desired. Below are some of the most striking takeaways gleaned from the survey.

Overall Excitement In The Promise of Virtual Production

When asked how they felt about the rise of VP in the film industry, 53.4% of the professionals surveyed reported feeling excited about it. Overall, virtual production is being embraced by those in the industry as well as viewers.

And the excitement just increases the more experience filmmakers seem to have in virtual production.

Only 23% of those surveyed said they anticipated having no virtual production work in 2023. Even those who aren't exclusively working in VP are very likely to end up involved in VP work in some capacity this year.

The Two Biggest Keywords About The Specifics Of Virtual Production: "Not Sure"

35.1% of respondents believe VP is technically difficult, but about 43.8% are not sure if it's difficult.

We've heard a recurring experience from filmmakers, which is that they're by and large excited about VP, but have no idea what to do once they're confronted with the technology and equipment.

Is VP faster? Higher quality? Less expensive? The most common response to these questions is a resounding, "Not sure."

Shooting with Rear Projection has been a practice for a while, which makes shooting VP seem intuitive to many at first. However, there are plenty of unintuitive snags that filmmakers run into in order to master VP—the sort of technical knowledge that is impossible to guess or troubleshoot without some program-specific training and understanding.

The film industry as a whole may be charging forth into VP with a fair amount of naiveté—which makes VP tech solutions that are intuitive and easy to use all the more valuable.

What The Producers Are Saying About Virtual Production

As with any technological advancement, the expectation is that some producers naturally find it thrilling and liberating, while others are more hesitant to jump on board. Among excited adopters, we heard about the revolutionary capabilities that VP will provide.

The skeptics are naturally concerned about the cost and newness of it all. While the initial VP spend is prohibitive to some, other filmmakers say that the overall return on investment makes VP cost-efficient in the long run.

Tech That Leaves Some To Be Desired

While some VP tech systems like outside-in tracking are favored among filmmaking professionals with a year or more of VP experience, some of the current technologies in place aren't up to snuff. Most of survey participants, in fact, are currently unsatisfied with the state of real-time rendering technology, and most participants are unsatisfied with the current state of virtual production workflow tools.

As an increasing number of filmmakers turn to VP capabilities, tech developers are going to be tasked with meeting the demand and improving upon existing solutions.

Upcoming Virtual Production Projects

When asked what sort of VP work production facility owners and operators anticipate doing during the upcoming years, nearly one-third said car commercials. When combined, film and TV make up the largest category at just under 40 percent.

While it isn't entirely likely that there will be a similar amount of auto commercial and narrative film or TV work, it's also interesting to gauge these survey results as hopeful projections based on filmmakers' interests. The results point to increased interest in more VP film and TV, and hope that producers will rise to meet the demand.

Similarly, when we asked this group how much virtual production work they anticipated doing this year, 57.6 percent said: "a lot more." Whether they actually plan to do a lot more VP work or just hope this will be the case, the interest in a higher concentration of VP projects remains.

But strikingly, more than three-fourths of respondents said they would likely do at least a little more VP work this year than last.

When asked if they could "wave a magic wand and change one thing about the virtual production industry," results were fairly expected for an industry as nascent as VP. Respondents reported wanting more buy-in from producers, increased adoption, and lower hardware costs. Overall, the survey pointed to a general enthusiasm for VP from filmmakers, coupled with some pauses and hesitations about cost and the current available tech systems.

Want to see the results and our additional conclusions? Download a copy of the full report here.

• Insights from a survey of 771 film professionals in Los Angeles

• Filmmaker attitudes toward virtual production

• What producers are saying about virtual production

• Predictions and opinions from virtual production professionals who have been actively working in the field

• Insights from 72 virtual production studio owners and operators

• 25 page PDF

Download or view online here »

Read our additional coverage and analysis here:

• Leadership Insights from The State of VP 2023 Report

• Attitudes Toward Virtual Production

• 2023 Is The Year Virtual Production Goes Mainstream

Have you ever watched a movie and wondered how something this bad could possibly have gotten made? If you saw that movie in the theater, chances are it cost tens of millions of dollars, and hundreds of people worked on it. Somebody had to have known this was a bad movie. Right?

If you've had that thought process, you're not alone. And you're right: a lot of somebodies did know. But by the time it was too late, it was too late.

The problem doesn't lie in talent or taste. The hundreds of cognitively diverse people that typically make up a film crew come together with different skills to create something bigger than them all. It's an incredible example of the kind of teamwork that most businesses aspire to.

When a high-potential creative process turns out disastrously bad, the problem usually isn't talent. It's often the waterfall process that leaves each skilled team member powerless to improve the final product.

As a friend of mine (film producer and former President of Amazon Films Ted Hope) put it in his excellent Substack recently, "Everyone wishes all it [took] to have a movie is that 'good idea' and the 'good filmmaker,' but it really is about good processes. And we currently don't have them."

This problem isn't peculiar to Hollywood. The underlying issue is the same reason corporations filled with talented people often see mediocre results despite massive effort.

Visual effects artist Felix Jorge had this in mind when he started his own digital studio, Happy Mushroom. Jorge was frustrated, having seen great artistic work like his—which can take place very early in the filmmaking process—often get thrown away.

It comes down to alignment issues that don't get resolved early on, Jorge says. “It has to do with the fact that the guy at the beginning thought they were doing something. The guy that came after did something else. The guy after that, something else. And by the time you get to the end, it's all being remade."

Lack Of Alignment Creates A Mess Teams Often Only See Too Late

It's sort of like if you were cooking a complicated meal. Your job is to prep ingredients for the chef to use.

Say the chef took the ingredients and decided, in the moment, that she wanted to do something a little different with the recipe. Something that promises to be better than the original plan. She combines the ingredients, and so far so good. Adaptation! Innovation! Huzzah!

But then—let's say it's someone else's job to actually cook this thing the chef prepared. The cook looks at the original recipe, checks what the chef has handed them, and decides, in the moment, the best way to cook this thing. There is no true recipe to follow anymore, so the cook does their best.

Then let's say someone else's job is to take what got cooked and assemble it into a meal. They have the original recipe, but what they've pulled out of the pot is something else. And at this point, it's looking like a bit of a mess.

After all this, the chef looks at the result and panics. They either freak out a la Gordon Ramsay, or ask the assembler to make changes. The cook stops by and asks for changes, too.

At some point, the person paying for this meal might call an emergency meeting. So, the chef, the cook, the assembler, and the ingredient prepper all get in the same room to stop the back-and-forth. No one leaves until they agree on what to do next. But, at this point, the food is cooked with little opportunity to salvage it.

When a film goes well (or any big team project in the workplace, for that matter), it's a function of good alignment and communication across all the steps in the process, even when changes to the original plan get introduced along the way. When the outcome is terrible, it's often because of a variation in this cooking analogy.

“It makes sense," Jorge says. “At the beginning, you have the script. The script turns into a picture. The picture turns into a 3D picture. Then you shoot it, edit it, et cetera. Everything was in order. However, all of a sudden, now you have work that is, frankly, mediocre."

In the cooking analogy, artists like Jorge are the ones prepping the ingredients, using the script (recipe) as their guide. A production designer will take those ingredients and create a film set for a director to shoot in. The director then has their cinematographer capture the action. All this “cooking" gets taken out of the oven by an editor, who figures out how to make it a meal.

The obvious solution to this problem is to hash out the disagreements and get on the same page before the cooking work begins. If you can get all of those parties to engage in the process early together—even before it's time to do one's specific job—a group can hash out changes to the recipe and get alignment before the chopping and mixing and cooking starts.

The problem is it's hard to convince busy people to take time to engage so far ahead of the time to do their job.

“If I can get the intention of the director, the DP [director of photography], and the production designer [up front], it's way harder to tell me that my project or the thing that I'm doing is a throwaway," Jorge says.

But directors and DPs and production designers are busy. If your job is to cook meals all day, it's hard to tear you away to talk about ingredient prep—or to interest you in it.

Besides, this is the way everyone cooks. It's hard to rip people out of best practices that an entire industry has used for years.

Jorge's big epiphany at Happy Mushroom has been that you can use new tools and technology to “trick" people into engaging early.

For example, in the emerging field of virtual production in filmmaking (in which my current company, SHOWRUNNER, works), exciting new tools are expanding the possibilities for how movies can be made. Instead of sending a film crew out to a far-flung location, virtual production allows filmmakers to put gorgeous locations up on giant LED walls, making it look like you're in, say, the jungle, when you're just inside a studio.

Directors and production designers and DPs who are excited about this new technology are having to learn new ways to work anyway—and that's the perfect time to influence positive changes in an established process.

To do that, Jorge says it's helpful to use tools that interest his downstream collaborators—production designers, in his case—in engaging in the creative disagreements earlier on.

A visualization artist like Jorge (in the cooking analogy, our ingredient prepper) isn't the party with the most power in the filmmaking process, but they can “trick" (in Jorge's words) those with influence in the process downstream into weighing in earlier. People are naturally curious about new technology, so Jorge advocates for using it as a hook to get downstream collaborators to pay attention to earlier work.

I recently put this advice into practice with a film project of my own. We got our editing and post-production contractors (Prysm Chicago) involved in our planning process in the beginning of the project months before they would normally get involved. They were specifically interested in seeing the newfangled tools we were planning to use, and that created a natural segue into collaborative creative discussions far earlier than usual.

“If [all collaborators] see things as quickly as possible, they're going to arrive at a stronger product at the end," Jorge says. “And their creative vision can carry all the way through. That's what I want."

The details of how you can do this will vary greatly by industry and company, but the principle is this: seeking to unearth disagreement early on helps a team to get to alignment early on—and that prevents the team from painting itself in a corner later.

“I'm really excited for the future," Jorge says. “I'm excited for people to see examples of projects that are engaging in creative disagreements early [and then succeeding], and for people to start thinking about the future process" with these new technologies that can help filmmakers avoid the cooking problems of years past.

Lucas's companies' technological breakthroughs, particularly those from Industrial Light and Magic (ILM) and Skywalker Sound, have influenced engineers and creatives worldwide, including in industries like aeronautics.

In film production, Lucas's innovations have opened up at least 3 major paradigm shifts that the world of cinema has followed.

When A New Hope premiered in 1977, it changed the film industry's fundamental relationship with visual effects and sound design. And Star Wars continues to do so nearly five decades later. Not bad for a film that no one initially wanted to finance. Here are just some examples of how the Star Wars franchise has helped new technologies go mainstream:

Star Wars Paradigm Shift #1: From Film to Digital Cameras

WWW.STARWARS.COM

Digital photography rose in the 1990s, but no major film had been willing to use digital for motion photography. The cameras weren't good enough or reliable enough for the demands of film directors and their audiences

But George Lucas foresaw the benefits of making digital cameras good enough for the film. For 2002's film Attack of the Clones, the Lucasfilm team asked Sony to develop a digital camera that was as good as or better than a standard 35mm film. Sony, along with Panavision, delivered six prototype cameras capable of recording HD images at 24 frames per second.

With this, the team could replay footage instantly and could make quick decisions on set.

Attack of the Clones became the first studio film primarily shot digitally. It was like suddenly being able to do the Kessel Run in fewer parsecs. No wonder digital cameras were in massive demand after that.

WWW.STARWARS.COM

Star Wars Paradigm Shift #2: CGI Characters

Did you know Jar Jar Binks was the first fully computer-generated supporting character in a live-action film with motion capture? Before George Lucas' 1999 feature, The Phantom Menace, no one had ever created a fake character with dialogue, character, plot, and a considerable amount of screen time.

This was a significant breakthrough as it allowed for the development of characters with physical features and facial expressions, paving the way for the next generation of movies.

WWW.STARWARS.COM

ILM's efforts to create this fantastic character demonstrated something movie studios weren't willing to take a chance on before: CGI could be economical and realistic. The plan was for ILM to construct Jar Jar as a prosthetic costume on set with a digital head added later.

Fortunately, a comparison test examining the benefits of full CGI for the film laid the groundwork for the future of filmmaking.

Not A Paradigm Shift Per Se, But Still Awesome: 3D Holograms

Although George Lucas didn't invent 3D holograms, A New Hope educated viewers about what holograms were and the possibilities they carried. The enduring image of a 3D Princess Leia is iconic to this day. Holograms would become a recurring tool for Star Wars filmmakers, appearing regularly.

3D holograms are seen in many sci-fi movies, such as Iron Man, when Tony Stark interacts with the suit or his office's surroundings, or Back to the Future 2 when Marty McFly encounters a gigantic hologram of a shark promoting Jaws 19. On Blade Runner 2049, for example, we can see holographic versions of famous singers in an abandoned hotel.

ELVIS PRESLEY IN A SCENE FROM BLADE RUNNER 2049

We at SHOWRUNNER Studios recently used a Star Wars-style hologram ourselves in an all-virtual production television pilot, blending practical effects with the latest in-camera virtual effects.

Speaking of which...

Star Wars Paradigm Shift #3: From Greenscreen to LED Virtual Production

We know The Mandalorian is a great show—it remains the most-watched Star Wars streaming series as of this writing. For Star Wars fans, the series meant extending the world of their beloved film franchise. But for the media, the tech behind it established a fresh benchmark. That tech is called virtual production.

Movies can now be filmed anywhere by displaying a live picture with LED panels behind the performers, a departure from the rear-screen projection that has been used in films for decades.

WWW.STARWARS.COM

ILM built a massive 75-foot-diameter circular LED soundstage to make the Star Wars series The Mandalorian. This vast space is utilized for motion capture and compositing and provides a full-rounded solution for virtual production; and has since been used to make Kenobi, The Book of Boba Fett, and Andor. The LED walls make the image look realistic, and the camera movements and settings affect the display.

And though virtual production had been an option for tech-forward filmmakers for some time before Star Wars, the making of The Mandalorian became a turning point in the film industry—an obvious example of the power of a new tool and a new way of making television.

We're in the middle of this third paradigm shift right now, and so far, filmmakers are finding that well-planned virtual production saves money and it's better for actors that don't like working in front of green screens.

It's one more example of how the Star Wars universe continues to break paradigms and inspires creators to embrace the future of filmmaking.

Who knows what paradigm Star Wars will shift next? But it's a fair bet that the film world will take cues from it again.

So how do filmmakers decide what should be real versus virtual?

Your first instinct might be to make as much virtual as possible. This line of thinking has apparent advantages. With virtual production, you don't have to worry about sourcing physical set dressing. Virtual production also makes modifying digital assets—like changing the color or size of digital props—a breeze.

But it doesn't matter how easy your production is if the end result looks fake. And that's bound to happen if 100% of your set is virtual—or if you blend the physical and virtual worlds improperly.

Most TV and film productions aim for the highest level of realism. Directors usually don't want their movies coming out looking like video games or cartoons.

Here are 5 tried-and-true ways to blend the real and unreal, and thereby make virtual production look more realistic:

1. Use Layers Of Physical Set Decoration to Add Depth

Adding physical objects in the foreground and mid-ground adds dimension to your shots ups the realism.

If you look at this video from Resolution Studios, you'll see that they placed a cooler, tree, and fire pit in the foreground and a chair and camping stove in the mid-ground. This positioning makes the shot look real because it adds depth that you wouldn't get if it was just the actor and the LED wall.

Experienced virtual production filmmakers will give this rule of thumb: the more layers of fore- and mid-ground you add in front of your virtual background, the more your audience's eyes will be tricked by it.

2. Strategically Use Bokeh & Shallow Depth Of Field

Another reason to add physical set dressing is due to an occasional technology constraint: The processing power required to run big scenes on LED walls can sometimes be so high that your environments may not look photorealistic up close.

When this happens, the camera operator can get around this is by adjusting aperture, or otherwise creating bokeh or a shallow depth of field.

This blurs out the background and mid-ground to a varying degree. It works best when you have actors in the foreground and props or set decoration in the mid-ground. Here's an example of a close up shot where you'd never guess the background was virtual:

3. Use Camera Movement To "Sell" The Illusion

Camera movement is fundamental to film production — whether it's a dolly shot, panning, handheld, or Steadicam. In virtual production, you can add a tracker to the camera and achieve parallax in moving shots, which helps make the shot look and feel natural.

In general, when the camera is moving, the audience's eyes are more easily tricked by the virtual art in the background—if things are moving with the right mathematics (which camera tracking accomplishes automatically).

In the picture below, from the set of the TV show Resignation, you can see one type of camera tracking system.

In the still shot here, you can tell that the background wall is fake. But when you see the shot in-camera with the camera tracking active, it looks like this:

Movement, especially handheld shots, can mask aspects of an environment that doesn't look as real as you want.

4. Use Art To Literally Cover The Seam Between The Floor And Wall

There is usually a literal seam in virtual production between the floor and the LED wall. If you look at it directly, it will look fake. But you can design your scene around anything that naturally would be a seam if this were a real environment.

For example, SMASH Virtual recently showed us a demo of a temple complex where they carted in a bunch of sand and rocks from the abandoned lot across the street from their studio. The image below shows a little wall they built that covers the break between the floor and the wall. Everything below the wall is dirt and debris, whereas everything above is the LED wall.

5. The Big One: Match Your Physical and Virtual Lighting

The #1 thing that can make virtual production look fake is when the lighting in the physical world does not sync with the lighting in the virtual world.

Conversely, the #1 thing you can do to blend your physical and virtual into one world is to get the lighting to match.

The position of the virtual lights should correspond with the placement of the real-world lights. If this does not occur, it could appear strange to viewers, as the illumination would be coming from various directions ,and the shadows would not be positioned correctly. Even if you don't consciously think about this, your brain will register it as phony.

In a traditional green screen shoot, the visual effects department must match the lights in post-production. But with virtual production, you do the matching while you're shooting. The ability to tweak the physical and virtual lights simultaneously benefits the production process. It saves time (and therefore money) because you can see the final outcome in-camera. And it eliminates the headaches caused by guessing what the final VFX will look like based on previsualization.

On the other hand, with a green screen, you'd have to light the physical scene first as best you can and then, maybe months later, do the final VFX lighting. In that scenario, if you really screwed up on the green screen lighting, you'd have to do reshoots.

Below is an image, again from the set of Resignation. This scene has large windows behind the actor at the desk. The windows have sunlight shining through them which are virtual lights. You can see the physical lights above and behind the actor, illuminating him and matching the virtual and physical worlds.

And the result is what we call movie magic:

You can ask Alexa to play your favorite Luis Fonsi song, or turn on the lights in your house. Your doorbell can have a camera that beams video to your smartphone, so you can see who's knocking.

As we've watched the rise of IoT, we've asked ourselves again and again: how can this technology help us do our job as filmmakers?

In filmmaking, time is money. So if a film studio could use IoT to run like a smart home, it could save a lot of time and headaches.

We've learned from personal experience that asking Alexa to pop on some Beyonce to get actors in the mood isn't... a timesaver. But by applying IoT to filmmaking in other ways, we've learned a lot about how to make production go more smoothly. Especially when it comes to the folks managing lighting on set:

Enter SHOWRUNNER:

SHOWRUNNER is a toolkit we built to help you run a film set like a smart home. In particular, SHOWRUNNER helps camera and lighting crews manage all types of devices, like physical and virtual lights, from any smartphone or tablet.

On the face of it, this might not sound groundbreaking. But gaffers will recognize the power of having "virtual lights" and no cables as part of this deal. Until now, the only way to change lighting in virtual production has been to have a coder alter settings in a physics engine console. (And if your eyes just glazed over reading that, you now know why the ability to operate virtual lights from a smartphone is a big deal!)

How Physical Lights Are Normally Managed on Film Sets

Most production lights on a film set are controlled by hand. Walk over, turn the light up. Walk over, turn the color temperature down. Etc.

If the lights are easily accessible, it isn't a huge inconvenience. But it becomes a pain if the lights are high up on C-stands or hanging from the ceiling. And moving a ladder may seem like a small task if you only have to do it once or twice.

But lighting crews often perform this same manual task hundreds of times a day.

It adds up to precious time and reduced productivity.

Of course, newer lights these days come with apps that let you control them remotely. The problem is, rarely will a shoot involve just one brand of light.

If you have five brands of lights on set, you'll need an app for every brand, each with different user interfaces.

Individuals in charge of controlling the lights would need to learn how to use (and then juggle) all five apps. And chances are good those five apps will vary for each production. Depending on how the responsibilities get divvied up, the people who need to learn all these apps might include the director of photography, gaffer, and multiple electricians.

How Physical Lights Are Managed With SHOWRUNNER

SHOWRUNNER's lighting tool is like a mega-app that lets you control all lighting brands from the same interface.

• Control any CRMX, RDM, or DMX-enabled light from the same app.

• Control non-DMX lights via smart plugs, which functionally turns any dumb light into a smart light.

• Dial in any light setting, including the brightness, intensity, RGBW and color temperature.

Below, you can see three different brands of lights (and a fog machine!) all controlled from one tablet interface:

How Virtual Lights Are Normally Managed On Film Sets

Virtual production, using LED walls as scene backgrounds, is an emerging technology. Filmmakers are still settling on the best way to accomplish things. So even with good preparation ahead of a shoot, filmmakers often want small changes to their virtual backgrounds once they see their set decoration and actors in front of their LED walls.

The most common adjustments: the brightness, intensity, color, and direction of virtual lights that show up in the background.

Currently, the typical workflow for adjusting a virtual light is this: the Director of Photography (DP) will ask the Unreal Engine (UE, the most common virtual production physics rendering engine right now) operator to make an adjustment to the virtual lights. They'll go back and forth until the DP is satisfied.

The UE operator needs a desktop PC with a beefy GPU, so they are likely to be set up in an out-of-the-way spot that's not convenient for communicating with the DP. Here's a snapshot of the UE operator's station on a recent shoot that used SHOWRUNNER:

In the typical virtual production lighting process, the Gaffer, the seasoned lighting expert who is responsible for the rest of the lights the set, sits out while the UE operator does mysterious-looking computer stuff on a computer setup that looks like that crazy picture above this paragraph.

Meanwhile, when the UE operator is adjusting lights over and over, they're not able to work on the technical aspects of their job—the things that they're most seasoned in.

How Virtual Lights Are Managed With SHOWRUNNER

SHOWRUNNER's app plugs into UE and lets you adjust virtual light settings, such as the dimmer and temperature, directly from the same tablet or smartphone that you control physical lights with.

This allows the Gaffer to adjust all the lights on their set, and cuts out back and forth between the DP and UE operator, ultimately saving substantial time and money. It allows the lighting experts and the UE experts to do the jobs they're best at—in parallel.

The GIF below shows a gaffer using the SHOWRUNNER tablet app to adjust a virtual background light—in this case, the sun:

A Monitor Is Always With You As You Make Lighting Adjustments

Normally on a film set, when the gaffer makes an adjustment to a light, they're sending someone over to make a manual adjustment, while someone else looks in a monitor to see if it looks right yet. They'll call back and forth—"turn it to 3200... OK, try 3000?"—because the person adjusting the light can't see what the result looks like in-camera.

To help, SHOWRUNNER pipes the camera feed into the tablet or smartphone you're using to adjust the lights, allowing a gaffer or electrician to check the results in real time as they're adjusting lights—from the same app.

The One Thing We Wish SHOWRUNNER Could Do For Gaffers

Unfortunately, the SHOWRUNNER app can only command lights to change the type of light that comes out of them; if your lights are not on robot arms, somebody is going to have to physically move them around if you want them in a different spot. (This isn't the case with virtual lights, of course!)

So while SHOWRUNNER can change lighting settings, you'll still need a person to physically change a light's angle, set up flags, or adjust barn doors.

The good news, though, is with the live camera monitor on your tablet, anyone who's doing manual lighting moves can see how those physical adjustments are affecting the final image

With SHOWRUNNER, someone can be up on a ladder adjusting barn doors while watching in real-time as their change takes effect.

How Will You Manage Lighting On Your Next Virtual Production Shoot?

Virtual production can help you create more fantastic worlds and push more boundaries on the same budget. You can control more things (like the sun), which allows you to control your schedule more than in an on-location shoot. But the workflow is massively different from a standard green screen shoot because visual effects are being captured by the camera, rather than added in later.

This means that physical lighting has to precisely match the virtual background—or the result will look fake in-camera. And that means lots of minor adjustments to dial in the perfect shot.

For the gaffers who've used SHOWRUNNER on major TV and film sets, being able to make those lighting adjustments from one app—seeing the results in real time—has been game-changing.

Jacob Fenn is embracing a new world of creativity

Fennworld's CORE online filmmaking course is—and will forever be—free. Founder Jacob Fenn wants aspiring photographers and filmmakers to know that, while a certain amount of budget and the ability to invest in equipment is helpful, anyone can start learning the basics of virtual production, or VP, with just an iPhone. SHOWRUNNER had the opportunity to talk with him about how FennWorld came to be, and what it's like to embrace the future of filmmaking.

On the cutting edge of education

Since the age of twelve, Jacob Fenn has loved being on the primary frontier of the film industry. He had the good fortune of befriending a neighbor who ran a production company and would invite him on set. This front-row seat introduced Fenn to an industry that would become his life's passion at a far younger age than most of his peers. Fenn knew early on he wanted to spend his life as either a filmmaker or an animator, but animation software was cost prohibitive at the time and to a teenager. That led Fenn to film school.

Fenn recalled film school as an enjoyable networking experience. "But as far as the practical application, one of my big frustrations was the things that I was learning at film school were things that I'd already learned long, long before, just by being on set," he noted.

Thus the question formed in Fenn's mind: How would one make film school practical? How could an educator prepare students for all that the film industry might throw at them?

Fenn became an adjunct professor at Utah Valley University (UVU) and used the position as an opportunity to try to help students in the ways he'd wished he'd been helped in school. Making film school as practical as possible sparked his interest in virtual production, and his own company's needs pushed him to fully embrace the new technology.

Realizing the need for virtual production

In 2020, Fenn and his team shot a project in West Africa. COVID testing was still very much a concern, causing a lot of stress for the team and the talent on the project. Fenn kept thinking about the emerging virtual production space and how much easier it would be to shoot the West African backgrounds in a studio.

At the time, virtual production was still a novelty, and equipment and technology costs were higher.

But for Fenn and the team, it became clear: the time for virtual production had arrived. So, he dove in, believing it was a risk worth taking.

"Any nascent technology always has a long teething period," Fenn reflected. "Because you're an early adopter, you're tasked with showing everybody else how to do it. But I'm glad that we've jumped in, because we've gotten a usable result quicker than I thought we would."

He then started bringing virtual production techniques to UVU students, helping them get ahead in their careers by setting them up with a knowledge of new technology and production possibilities.

"The time is here to be able to bring this to the masses," Fenn thought.

Embracing new methodology

Like any new and still emerging technology, a filmmaker, director of photography (DP), or creative may hesitate to try an entirely new way of performing their craft. Fenn's advice to curious filmmakers is to lean into virtual production's strengths rather than trying to force traditional methodologies on a new way of working.

“If you have a DP that's used to shooting traditionally, this is not a traditional way of shooting and there's no way to sugarcoat that," he said. “And I think it's better to be honest about that. If you think you're just going to shoot traditionally in this environment, you just can't, right? You're still essentially shooting on a sort of a sound stage type environment, with more limitations on top of it. But lean into the things that you can do."

Working within virtual production frameworks will lead to better results and improve the process all around. Fenn also encourages filmmakers to embrace a hybrid approach to shooting. Rather than taking the myopic view that a film must be shot entirely traditionally or entirely with virtual production, Fenn recommends that filmmakers explore, experiment, and mix elements when different technologies are called for. And certain technologies, in particular, are fascinating to him.

“With the neural radiance fields, we finally are starting to see some new technologies that are getting the closest that we've ever been to light field technology," Fenn elaborated. “You combine that with VP, and you can go out on set and essentially shoot in some aspects similar to how you'd shoot in a LED volume. But it's an on-set production experience where you can capture depth. And you can potentially use that to integrate with some other 3D aspects."

Filmmakers can leverage new technology where it's the strongest while utilizing the other tried and true techniques they're familiar with, opening up worlds of potential for creative exploration. However, there's no reason to try to make VP work if that's not what a project calls for, according to Fenn.

“If there's something that you can shoot traditionally and it works better to shoot traditionally, just do that," Fenn advised.

“Don't force VP just because it's cool, and if it's not working for you, don't take a subpar alternative when shooting practically would've been better." Fenn provides a specific example.

"Sometimes people don't have the tracking working correctly, or it'll be too expensive to do the whole 3D tracking system. So, they'll be on a stage that might be fully equipped with a really decent tracking system, but it's cheaper to just shoot with like a projected image background," he said. "But you could have done that in the fifties."

A depth of possibilities

Another element of virtual production that Fenn is most excited about—and eager to witness in his students—is the new range of depth that filmmakers can create.

“A lot of times people think it's just a wall with somebody in front of it," Fenn said. “But once you start to think like a good photographer, and really think of that Z axis, then there's so much you can do with the foreground. This is true even if you're on a budget and can't create set design elements with which the actors can interact."

The more expansive and open-minded filmmakers are willing to think, the more the world of virtual production will open up to them. And this includes new and exciting career possibilities, especially for filmmakers just starting out.

Emerging career paths in a democratizing industry

There are many routes to success for young filmmakers interested in virtual production. From technical expertise to 3D creation, Fenn encourages a new generation of filmmakers to become comfortable working in an ever-expanding field. For those who are naturally tech-minded, Fenn especially recommends embracing that path to VP.

“Some of the most exciting areas right now are the deeply technical ones. So if you've always wanted to be into filmmaking, but you're an introverted person, you may just want to sit in front of a computer and get into machine learning or radiance imaging," he explained.

AR and VR technology, too, is democratizing the industry. "If you're just a storyteller, and don't want to do all this techy stuff, watch for that opportune window where the technology's available enough that you can just use your webcam and scan yourself," he mused. "Or you can use a 3D object that's available and then just use the object recognition software and the skeletal tracking of the visual-based systems."

And for folks just beginning or worried about budget, there's a lot that folks can do with just an iPhone, without needing an expensive tracking system or LED wall. Fenn encourages filmmakers to play around with some iPhone apps and a green screen or plain white wall if they don't have other resources available. That alone can give someone a great introduction to virtual production, and allow creatives to produce stories they're proud of and excited by without needing to break the bank.

“I see these things students are doing and it really is so impressive compared to what you can do with a small team of people," he said. “[At FennWorld], we want to really lead the way and say, 'Here are ways that anybody has access to this technology to tell amazing stories.'"

And as VP continues to evolve and become accessible for more and more filmmakers, Fenn doesn't see the allure of it going away anytime soon. He's constantly inspired by what his team and his students create with these new parameters.

“Just seeing every little piece actually function together is such a magical thing," he said. “You get so lost in the technicality, and it's so rewarding to see the creative part come back. The return of creativity after much, much effort continues to be the most rewarding part."

Since films and series started using LED walls and video game engines to bring faraway locations into studios, virtual production has flashed a ton of promise. Studios don’t have to ship crews around the world. Reflections look real. Actors don’t have to pretend the green wall behind them is more than a green wall. Post-production departments can get a head start on their work.

But anyone who has actually filmed in virtual production will tell you the same thing Hitchcock, Kubrick, and all the classic "rear projection" artists would tell you:

It’s way too easy to make a virtual background look fake.

From Russia with Love, 1963

Why Do Virtual Backgrounds Look That Way? 

The human brain is good at perceiving inauthenticity. Over millennia, human evolution developed a mechanism called “ensemble lifelikeness perception,” which automatically detects minor distortions in the world. 

In more primitive times, these distortions indicated potential dangers. This is why we can detect faces that don’t quite look human (prehistorically indicating potential illness or injury—and nowadays, deepfakes). It's why we can spot false laughter and crying, bad acting, and other potential indications of, well, traps.

In virtual production, there are several common issues that trip the brain’s inauthenticity alarm:

• When the camera moves and the background doesn’t microscopically shift with the proper parallax that the real world does (our brain detects that the background is flat)

• When the background does shift with the proper parallax, but not in real time (the real world doesn’t have latency)

• When there’s a mismatch of perspective between the background and the physical foreground/actors/props (the real world is generally not askew)

• When there’s a mismatch of color balance between the virtual and the real

• When lighting in the foreground changes but doesn’t affect the background (our brains expect light to behave certain ways)

• When production lights reflect off of the plastic LED wall that is projecting the virtual background

• When the camera is too close to the virtual background, and can actually see the pixels generating the image

• When the camera is misaligned with the refresh rate of the background, and scan lines appear

• And of course, when an actor or object should be casting a shadow, but for some reason is not (see also: Why Does The Mandalorian Never Stand Next To Walls?)

These are only some of the more obvious ways a virtual set looks fake. The problem is, these challenges disappear when filming in a real location.

After all, in the real world, the rules of physics can be taken for granted.

Every new field goes through a hype cycle. And these flaws are what threatens to push professionals curious about virtual production into the Trough of Disillusionment. 

When you’ve spent your career framing gorgeous shots and telling compelling stories, it’s hard to be enthusiastic about the idea of making your shot look “not fake.”

Until this changes, virtual production technology will struggle to take off.

Oblivion made virtual production look real in 2013, through manual effort.

The Good News: Data Can Solve All Of These Problems—Automatically

Though our brains are wired to detect inauthenticity, science also shows that when the world lines up with our expectations, we unconsciously encode our observations as factual—even when we know something isn’t physically possible.

This is just a fancy way of saying, “our brains can be tricked into thinking Superman is actually flying.” This is the basis of every good magician’s show and all good special effects in movies.

Good filmmaking—and virtual production—nails this perceptual trickery. It satisfies our brains, so we simply don’t question whether the guy wearing Pedro Pascal’s helmet is actually in a Tunisian desert.

Right now, filmmakers solve the “fake factor” in virtual production through manual tweaks, guess-and-check, and writing around shots that will likely look fake with virtual backgrounds.

But ultimately, many virtual production challenges are about physics—which means we can use math to solve them. And unlike in math class, we can just let the computers do the work this time.

Virtual production already has a standard solution to the parallax problem: By sharing data between the camera and the background wall, we can automatically move backgrounds based on the position of the camera so the human eye perceives the motion as realistic.

We can apply this same kind of data-sharing automation to many of the other problems I mentioned.

For example:

• Data can be used to automatically color-balance an LED wall to match the IRL foreground elements, so both match in camera.

• Data can be used to auto-detect perspective problems, allowing a filmmaker to snap the LED background against the foreground perspective she wants.

• Data can be used to auto-sync the virtual and physical lights when you make changes or have actors move through a space. (This is something SHOWRUNNER automates, for example.)

• Computer vision data can be used to auto-detect moire and flicker and pixelation—and either warn the filmmaker or auto adjust settings to eliminate the issue.

• Similar computer vision data can be used to automatically cast a real actor's shadow into the virtual background.

I believe that the sharing of data between devices and apps is precisely what’ll bring virtual production into the mainstream. 

And once film directors are free from making things look “not fake,” they can focus on the exciting part: using virtual production to do more than we can in the real world.

"The Mandalorian" validated virtual production tech. But these six films and TV series also deserve their share of the limelight.

Any new type of creativity needs validation. For virtual production, that validation came from The Mandalorian. In 2019, viewers tuned in to see otherworldly action, and those other worlds looked great. Think of the intertwining hues of deep oranges, reds and yellows that make up the mesmerizing Tatooine desert planet, or the excitement of flying weaponry exploding near bounty hunter Din Djarin’s Razor Crest M-111 assault ship. 

Today, if you research virtual production technology, you’ll inevitably see someone mention the show. On YouTube, behind-the-scenes featurettes on the technology have racked up millions of hits. Virtual production is the future of filmmaking, and it’s become synonymous with The Mandalorian. However, The Mandalorian is not the only popular work to use virtual production tech. Far from it. 

As virtual production adoption continues to pick up, here are six notable films and TV series that deserve their share of the limelight. 

1. Gravity (2013)

One of the most impressive VFX films ever released, Gravity proved to be lightyears ahead of its time when it broke the record as the highest-earning October and fall openings, grossing $55.8 million in 2013. 

In the movie, Dr. Ryan Stone (Sandra Bullock) is a medical engineer on her first shuttle mission. Her commander is veteran astronaut Matt Kowalsky (George Clooney). During a routine space walk, disaster strikes and the shuttle is destroyed, leaving Bullock and Clooney stranded deep in space with no hope of rescue.

Director Alfonso Cuaron used amazing “Light Box'' technology to bring his vision to life. Rather than figuring out how to move Sandra Bullock through space, the creators realized they could move space around her. To make that technique work, they built the Light Box out of 1.8 million individually controllable LED bulbs to essentially make jumbotron screens that can move 360 degrees. Then they piloted the camera through it using a robot. Ingenious. 

The previs to the films success was primarily done by Framestore, in partnership with The Third Floor. The Gravity team tested a number of systems, with the final solution to have a small number of sets and then the Light Box for zero-gravity work. In other words, it was a minimalist set used to maximum effect. 

2. Snowfall (2017-)

FX’s crime-ridden drama, set against the infancy of the crack cocaine epidemic in 1980s Los Angeles, has garnered national acclaim and awards during its first five seasons. The show uses stylized visuals that have earned Outstanding Achievement in Cinematography honors from the American Society of Cinematographers.

Virtual production is a huge part of that success. Co-producer John Labrucherie brought in Orbital Studios to build out a lavish penthouse, and they didn’t skimp on making this look as extravagant, elegant and exorbitant as possible for the main character. 

After Orbital Studios built out the central set pieces, they utilized a 65-foot LED wall to create the view, which is just as important as everything else staged inside the penthouse. In some on-screen interviews, the crew revealed they’ve been able to reduce their time out on location by at least a third. For nighttime scenes, they shot with 4 Sony Venices and created plates at 12k resolution that dubbed incredibly realistic city night-life and skyline views. After hearing feedback from cast and crew, let’s just say they don’t plan on going back anytime soon. 

3. Solo: A Star Wars Story (2018)

The Star Wars universe seems to have a thing for LED volumes. Aside from The Mandalorian, the 2018 theatrical release Solo: A Star Wars Story also made use of emerging virtual production technology. 

In this prequel, Han Solo meets his future copilot Chewbacca and encounters the notorious gambler Lando Calrssian. It’s a journey that will set the course of one of the Star Wars saga’s most unlikely heroes. 

Directed by Ron Howard, Solo consisted of 1,800 visual effects shot from every variety. From integrating CG elements with liveaction plates to total CG environments and real-time VFX implemented on set, the use of LED screens plays a crucial role in the making of this story. 

When trying to film the iconic Millennium Falcon, production constructed a highly detailed set of the ship’s cockpit. In order to create convincing shots from within the cockpit, the crew put together a quite practical set for filming that could be elevated and surrounded by a massive 160-degree wraparound screen. The volume had the capability to play back effects and content in real time, simply by tapping the interface.  

This is what we’re talking about when we say virtual production is the way of the future. The visual effects work on Solo: A Star Wars Story was recognized by an Academy nomination and three VFX Society Award nominations. 

4. Lion King (2019)

In 1994, Lion King was the highest grossing film of the year. It’s since topped $1 billion in total box office, and basically pioneered the way for nearly every animated feature film that came after it.

Twenty-five years later, Disney rebooted Pride Rock as a live-action movie to pioneer another new type of filmmaking. The 2019 version of The Lion King aimed to solve the challenge of how to direct a story when the director can’t see all of what they are directing. 

In this case, the director is Jon Favreau, who also happens to be the creator, writer, and director of The Mandalorian. Favreau and The Lion King’s creative team took advantage of the revolution in consumer-grade virtual reality technology. Paired with Unit Engine, they were able to produce visual effects and animation that complemented the traditional physical production approach.

MPC was the visual effects and animation company that provided the stunning visuals. Then the Magnopus team, headed by Ben Grossman, started by sketching out ideas around throwing out all the old visual effects based software and switching completely over to game engines. 

They ended up creating a system that allowed Favreau to direct a movie with high-quality, real-time, interactive components while still making a 100% computer-generated film.

And for those keeping score at home, even though The Mandalorian has gotten a lot of virtual production credit over the years, The Lion King actually debuted first by a few months.

5. The Irishman (2019)

The Irishman is another feature that dipped its hands into virtual production. With a run-time of over three hours, Martin Scorcese and co. definitely had the space to incorporate some dynamic environments. 

The film follows Frank Sheeran (De Niro), a truck driver who becomes a hitman involved with mobster Russel Bufalio (Joe Pesci) and his crime family, including his time working for the powerful teamster Jimmy Hoffa (Al Pacino) 

The car interiors were all shot in the studio. To make it look like actors were outside, the crew had to create lighting by using huge LED panels, similar to jumbotrons. They fed onto those screens with images created by the second unit where they had nine cameras filming around one car. With this approach, they were able to capture every possible angle. 

The crew chose one screen as a virtual background (instead of a green screen) and used the others to light the actors’ faces. Remarkably, the actors were literally being lit and shaded by the landscape and buildings passing by. Not bad for a couple of wise guys. 

6. Red Notice (2021)

The most-watched film all time on Netflix is not Bird Box, The Kissing Booth, or The Irishman; it’s 2021’s action comedy Red Notice, starring Dwayne Johnson, Ryan Reynolds, and Gal Gadot. 

The film follows an FBI agent (Johnson) who teams up with the world’s second-best thief (Reynolds) to capture the world’s best thief (Gadot). In true blockbuster form, Red Notice goes on a globe-trotting adventure through Italy, Bali, Spain, Argentina, and Egypt—with a layover in a Russian prison.

Much of Red Notice’s world was built or supplemented with CGI. The film originally was set to have 400-500 VFX shots; when the pandemic hit, that amount nearly tripled. But during production, Netflix didn’t have to fly cast and crew all around the world. In fact, virtual production technology made it easy to shoot everything in Atlanta.

Based on behind-the-scenes photos, it appears the movie relied on virtual production techniques similar to the ones created for The Mandalorian. Judging by the 364 million hours of viewing time the movie generated in the first 28 days, it all paid off.