This article was written with the contribution of Mitchell Block, an Academy Award-winning executive producer and Jon Anderson Endowed Chair in Journalism and Communication at the University of Oregon.
Academy Award-winning executive producer Mitchell Block says he can recall when he fully realized artificial intelligence (AI) would change filmmaking forever. It was shortly after viewing Morgan Neville’s Roadrunner, a documentary on the late Anthony Bourdain.
The film featured AI-generated dialogue of the deceased celebrity. Brock was none the wiser until Neville brought up in an interview that AI had produced that dialogue.
“And that’s really the genius of AI — the idea that you can create something that doesn’t exist without necessarily having an army of artists and effects people to do all that hard work,” he says.
Table of Contents
Transfer Files With No Limitations
Big files? No worries. MASV has no limits on file packages and handles files up to 15TB.
Examples of AI in Filmmaking
The Fourth Industrial Revolution driven by big data and AI has inserted itself into virtually every industry in the world, including filmmaking. AI tools can do more than perform synthetic resurrections of deceased actors. They also help humans scale by performing activities faster and more efficiently.
But questions persist around the ethics, uses, and implications of AI and its offshoots, machine learning (ML) and deep learning (DP). While some experts say it will ultimately be a net job creator, others aren’t so sure — especially when it comes to creative industries once thought safe from AI-related disruption.
Particularly troublesome in the filmmaking and video production space is the nagging (and growing) use of generative AI to create deepfake images and videos.
Still from the music video for ‘The Heart Part 5’ by Kendrick Lamar which uses deepfake technology by Deep Voodoo.
That said, while public awareness of AI in filmmaking has recently ticked upward, the technology has already become well-established in the movie business. AI tools for filmmaking-related tasks such as audio mixing, masking, coloring, language dubs, and video editing are now relatively common.
Indeed, the number of companies offering AI products and services to the video production industry is flourishing.
- Flawless and its TrueSync product for AI dubbing launched in stealth in 2018. Papercup, which performs a similar function, has been around since 2017.
- Digital Domain, Hour One, and Tally Labs create virtual characters and even fake humans.
- The Aigency provide talent management “for robotic personalities” — and by that, they mean virtual humans, not monotonous real people.
And the creators of “South Park” recently scored $20M in venture capital to fund their Deep Voodoo project, which aims to create “leading deepfake technology,” “cost-effective visual effects services,” and “synthetic media projects.”
It all adds up to a frenzy of activity around AI-enabled media, also known as synthetic media. So what’s all the fuss about, exactly?
Get The Best Large File Transfer
Sign up today and get 20GB free to use with MASV’s fast, reliable large file transfer.
How AI Helps Filmmakers and Video Producers
The ways AI helps video production workflows are almost too plentiful to cover fully, so we’ll focus on some of the main areas, with the overarching theme being that AI tools help save time and money in each of them.
1. Pre-Production Planning
AI has already become an everyday part of the pre-production process. It helps plan filming and other schedules, find suitable shooting locations by analyzing real-world data against script requirements, and even organize footage to help editors find specific scenes or camera angles more quickly.
“If you think about simplifying pre-production In terms of logistics and design, of organizing a movie — from casting, to budgeting, to location breakdown, props and wardrobe — all of those things can easily be done and budgeted accurately using AI from the screenplay,” Block explains.
2. Video Editing
AI tools are already well-established in the video editing world. While full-on AI video generators exist, most AI tools for video editing help scale human video editors.
- Marquee tools such as Premiere Pro, for example, offer AI tools to automatically perform color matching, aspect ratio changes, and utilize motion tracking.
- Other tools, such as Runway, allow editors to rotoscope and mask footage in minutes using ML, a major change from traditional methods that typically take several hours.
- Flawless, the company we mentioned above, learns the phonemes and visemes (how they sound and how they look when making sounds) of actors. This allows video editors to “seamlessly alter their facial expressions and mouth movements to match newly recorded dialogue,” according to the Los Angeles Times.
- AI also helps restore old film by automatically removing scratches, dirt, warp, and flicker from prints.
- And AI-enabled video compression can theoretically improve the video quality of lower-resolution productions.
3. Visual Effects (VFX)
We mentioned how AI has helped reanimate deceased actors and how machine learning can dramatically boost the speed of manual work, such as rotoscoping and masking. But it has also helped streamline the VFX pipeline by automating parts of 3D modeling, image processing, color grading, match moving, and other elements.
AI is also regularly used to de-age living actors, build digital backgrounds, and augment lighting, while its ability to scan entire scenes and classify objects can further speed up the VFX process.
4. Casting and Scriptwriting
AI can help make casting decisions by mining rich veins of performance and audience reaction data to determine the best actor for a given role. Casting directors upload screenplays directly into AI tools such as Largo.ai, analyzing which actors could monetize the role most effectively or who may come with higher risk.
But as with any good AI tool, the human makes the final decision. In this case, the casting director. It’s just a more informed decision that requires less time and energy.
The technology is also used to develop scripts. Google’s DeepMind created a tool called Dramatron that leverages large language models to generate rough scripts for movies, TV shows, or plays. The AI-generated scripts come complete with a title, character descriptions, dialogue, and scene descriptions.
5. Marketing and Promotion
Studios like Warner Bros. use AI-enabled software such as Cinelytic or ScriptBook to predict which screenplays have the best chance of commercial success and determine which films get made and which don’t.
“The system can calculate in seconds what used to take days to assess by a human when it comes to general film package evaluation or a star’s worth,” says Cinelytic founder Tobias Queisser in The Hollywood Reporter.
And the Merlin neural network, developed by 20th Century Fox, along with similar tools, help studios perform a range of film management and promotional activities, including:
- Optimizing movie trailers to get the best reaction.
- Optimizing one-sheets for film promotion.
- Planning marketing campaigns by geolocation and audience demographics or interest through micro segmentation.
“If you can identify five million potential viewers to get 500,000 people into the theatre for a fraction of the cost, that’s really advantageous for the movie business,” Block says. “AI can help filmmakers as well as studios, executives, and financiers figure out how to create their film, but at the same time also how to finance and market the film.”
6. Music Scoring
Crews can use AI-enabled software such as Dynascore to compose music for their video production. Instead of stock music and its inherent limitations, AI composition software generates musical scores tailored to synchronize with a screenplay’s cuts and other actions.
And Sony Computer Science Laboratories (CSL) developed Flow Machines, an AI music composer meant to assist human composers come up with various chords and melodies to fit their desired style.
7. Producing
Films or videos entirely produced by AI without much human intervention are still rare, although one computer artist recently won the Cannes Short Film Festival Jury Award for his text-to-video AI-generated film, The Crow.
Another non-filmmaker, Adrian Seltzer, recently won acclaim for Salt, a series of short films released on Twitter and generated through a combination of AI tools such as Stable Diffusion, Midjourney, DALL-E 2, Synthesia, Murf, and GPT-3.
Easily Transfer Virtual Production Video Files
Secure, reliable, and fast transfer of massive 8K or 12K video files with MASV.
Potential Problems With AI and Filmmaking
“We can no longer fully trust what we see,” says Concordia University Wisconsin professor Robert Wahl in the LA Times piece quoted earlier.
And although movie-making has always tried to convince audiences to suspend their disbelief, the deepfake era takes things to a whole new level.
Even Neville’s Roadrunner film garnered its fair share of criticism for not immediately alerting audiences and critics to its AI-generated content — including from the late celebrity’s ex-wife.
Some observers say video- and dialogue-altering technology such as Flawless will likely pave the way for updating or completely changing films several years after the fact, which critics say risks compromising the consent of the original actors.
Indeed, what’s the point of human involvement if AI can write scripts, plan shooting locations, cast actors, and automatically create mind-blowing digital effects — even produce entire films and videos?
Quality is one big reason, according to Block.
“I think it’s like anything else — just because a typewriter replaced the quill, it didn’t mean writing got better,” he says, adding that it’s ultimately people who provide the human creativity and personal touch behind a film, written work, or even a designer good.
“We’re moving to a world where everything can be created or recreated digitally,” he explains. “Including ourselves, as clones. We’ll soon be able to 3D print designer purses or expensive clothing. And if you can create it at home on your 3D printer, then what’s the value of a brand?”
He says it’ll be up to critics, audiences, and even technology platforms to hold filmmakers and studios to account when it comes to the overuse or unethical use of AI in video production.
Will AI Replace Traditional Filmmaking?
The Raindance Film Festival says people can keep themselves relevant in an increasingly automated industry through:
- Focusing on the unique perspective and emotional depth of human storytelling.
- Focusing on tasks that AI currently struggles to achieve, such as those requiring empathy, social skills, and physical dexterity.
- Adapting to and harnessing the power of new technologies to improve their own work.
But Block says he’s not worried about humans being replaced by AI, at least for now.
“AI can’t, by itself, create a brilliant and marketable idea,” he says. “Even in a world where the AI can input the idea into a box and out pops a movie, you still need a human being to make it and look at it and say, ‘this sucks’ or ‘this is terrific.’
“And it’s that process of bringing in the human that makes the work valuable.”
MASV Large File Transfer
Get 20 GB to use with the fastest, secure file transfer service available today, MASV.