What Is AI Video Compression?

by | January 5, 2023

We’ve talked before about several artificial intelligence (AI) tools already making inroads in filmmaking. Another such AI revolution bursting with potential — and backing from big tech — is AI video compression.

Indeed, contemporary video codecs such as H.265 and ProRes are now being challenged by new types of intelligent video compression powered by AI.

These AI video codecs don’t necessarily compress a video file, but rather, reconstruct it based on context and reference images to preserve quality despite a drastic reduction in file size.

Keep reading to learn more about AI video compression below.

Table of Contents

Send Large Video Files With Confidence

Sign up for MASV and start sending and receiving large files quickly and reliably.

Which Video Codecs are Used Today?

Fundamental to any discussion about codecs is the word’s actual meaning: Codec is actually a portmanteau of “compression” and “decompression”. Because that’s what codecs do: Compress and decompress.

Codecs for video and audio offer either “lossy” or “lossless” compression. Lossy compression sacrifices some of the file’s data to make the file more efficient, while lossless compression contains all the original data. Most video codecs are of the lossy variety.

There’s also a difference between the types of codecs available: Delivery, intermediate, and acquisition.

Delivery codecs — used for broadcasting and playback

  • H.264 (AVC)
  • H.265 (HEVC)
  • AV1
  • VP9
  • WMV

Intermediate codecs — used for video editing

  • ProRes 422 and 4444
  • DNxHR
  • DNxHD
  • CineForm

Acquisition codecs — for capturing video:

  • ArriRaw
  • Blackmagic Raw
  • Redcode Raw

Many of us don’t realize just how many codecs are all around us, from phone calls to even the broadcast content we consume. Indeed, watching television is essentially witnessing the compression-decompression process in real time.

There are a lot of codecs out there. Here’s a list of pretty much all of them.

💡 Want a quick way to tell how much compression is in a file? Lay two versions of the same file on your timeline in Premiere Pro or Resolve, then turn one to Difference blend mode. You’ll quickly see where the compression hits are and the differences between the two files.

Transfer Files With No Limitations

Big files? No worries. MASV has no limits on file packages and handles files up to 15 TB.

Why Do We Need Better Video Codecs?

We need better video codecs for a few reasons, but mostly because high-resolution video files have become absurdly large – and are getting bigger all the time, thanks in part to content in 6K, 8K, and even 12K resolutions.

Video Codecs

Streaming 4K video from Netflix, Amazon Prime, HBO Max, or Disney+ also takes up a lot of bandwidth, as does video consumption driven by social media platforms.

Indeed, video is everywhere in the online world:

  • The average person worldwide watches around 84 minutes of video daily.
  • Back in 2018, Cisco predicted that video would make up 82 percent of all network traffic in the world (compared to 75 percent at the time).
  • Video streaming alone accounted for more than 70 percent of all mobile data consumption in 2022. Ericsson says that number will increase to 80 percent by 2027.

The above is great news for video creators and consumers, but the problem is that video takes up a ton of space. Consumers have also become accustomed to having high-quality video available on demand.

Above all, traditional codecs are dumb in that they have no awareness of what they’re doing or what images they’re compressing — they simply compress and decompress pixels, and nothing more.

What is AI Video Compression?

Traditional video compression removes bits of visual information to reduce the overall file size of a video. AI video compression examines the contents of a video to recreate it on a smaller scale. In other words, it creates a hyper-realistic duplicate of the original video.

This is because “AI video compression” is machine learning (ML)-based video compression (sometimes known as neural compression). ML algorithms or models learn to perform tasks and make evaluations based on relevant training data fed by a human — or in this case, a video feed.

These models have learned conceptual patterns of what a human face looks like and how it moves, along with even broader patterns about the environment, such as “boats are usually found on water”.

In fact, several AI video codecs already exist today, including:

  • NVIDIA’s Maxine
  • Google’s VP9
  • AIVC
  • AccMPEG

Let’s examine how exactly AI video compression works.

Send Large Files Without Compression

MASV can handle any file size, so you can spend time creating – not compressing.

How AI Compression Works

One of the easiest ways to illustrate how AI video compression works is to think of it as a piece of vector art.

Vector art is:

  • Created from a reference image.
  • Uses mathematical elements to recreate image assets.
  • Is infinitely scalable.

AI compressors such as NVIDIA’s Maxine video codec operate similarly. They use a reference image (e.g. one frame for every minute of footage) to determine the color balance and isolate other control points such as the background, foreground, the subject’s face, and movement.

The codec maps this data as its compressed, then reconstructs it during decompression by filling in the details through a process called resynthesis.

So why do it this way?

Video context

As mentioned, traditional dumb codecs don’t contextualize the media being compressed. They look at data on a frame-by-frame level to see what can be removed without affecting the overall quality too much. They have no idea if the video is of a talking head, a car driving on a highway, a bear hunting for food, etc. This can result in harsh compression of otherwise important pieces of data.

Bandwidth-friendly

By only sending the base information of each frame — essentially by sending mathematical elements (which are small) and not pixels (which are large) — and rebuilding frames on the fly, AI compression is bandwidth-friendly and can scale infinitely. In this way, AI codecs aren’t all that different from the technology behind thispersondoesnotexist.com, which creates photorealistic images of people who don’t exist.

Upscaling

But these codecs don’t just compress video; they can also improve video quality through their infinite scalability. Video content encoded by an AI codec can be rebuilt from scratch to fit any resolution and frame rate. In this regard AI video compression is a lot like video upscaling, which uses a similar process to improve the quality of older video and film content.

Get The Best Large File Transfer

Sign up today and get 20GB free to use with MASV’s fast, reliably large file transfer.

What’s the Future of Video Compression?

If you’re wondering whether current AI compression tools will push most other codecs into irrelevancy, the answer is: Probably not.

For one, if AI compression recreates an image, it begs the question, is the new output as detailed as the original? It’s a common objection — but solutions are in play. You can learn more about AI compression’s detail preservation here.

But mainly, device manufacturers will always make codecs to suit their use cases. Most cameras have customized acquisition codecs embedded on ASIC- or FGPA-style chips within the device itself. That won’t change anytime soon, so we’ll likely always have Apple codecs, Sony codecs, ARRI codecs, Red codecs, and codecs from other device manufacturers.

The manufacturers will certainly continue to do that because they compress their camera data in a way specific to the camera’s file format.

But as for intermediate and delivery codecs, AI video compression is a sea change that will provide an astronomical performance advantage.

AI Video Compression in a Nutshell

Codecs have been around for a long time, but contemporary encoders such as H.265 have recently come under intense pressure thanks to the ever-growing bandwidth and memory demands of high resolution video.

That’s why a new generation of AI-based codecs that compress video more efficiently, improve video quality through infinite scalability, and — unlike traditional codecs — are aware of the content they’re encoding have now stepped into the fray.

Based on machine learning and deep learning techniques, it’s likely only a matter of time before these codecs become indispensable tools in the video production and post-production workflow.

No matter what kind of compression you use, however, it’s vital for production and post-production teams to share files quickly and reliably — and that’s where MASV comes in.

MASV can send and receive files of virtually any size — up to 15 TB per file — anywhere in the world and is used by teams at Disney+, BBC, PBS, and other major media studios.

Sign-up today and get 20 GB to transfer for free.

MASV Large File Transfer

Get 20 GB to use with the fastest, secure file transfer service available today, MASV.