TLDR:

• AI Research of the Week: Google’s Lumiere Model Looks Amazing

• Leaders: Will The Rabbit Eat The Apple?

🤩 AI Research of the week 🤩

Scott Galloway, the famous business influencer, bet that 2024 would be Google’s AI year.

And it’s starting to look like it.

Now, they have launched Lumiere, a huge breakthrough in text-to-video.

Their approach to AI video synthesis is not only revolutionary, but it also showcases incredible video quality and a wide range of amazing downstream skills like video in/outpainting, image animation, and video styling, making it the new reference in the field.

But how does it work? Let’s get into the weeds.

The Everlasting Problem

Out of all data modalities, video is, without a doubt, the hardest.

Considering videos are simply a concatenation of images, called frames, displayed at a certain frame rate per second (the higher the fps the smoother the video) the logical path to building text-to-video (T2V) is to depart from a text-to-image model (T2I), like DALL-e or Stable Diffusion.

However, T2V adds an extra layer of complexity: Time.

In other words, it’s not enough to generate multiple frames (you can generate as many as you need with the T2I model) but they must be consistent over time.

And this has proven to be a huge problem, as this forces AI videos to be very short, and they nevertheless tend to showcase artifacts, visual imperfections like the orange blob that suddenly appears in this AI-generated video below.

And the reasons for these inconsistencies reside in how these models are built, and that Lumiere revolutionizes.

Originally, the video synthesis process involves three steps:

1. A T2I prior generates a set of key frames set across the complete duration of the video.

2. Next, several TSR (Temporal Super Resolution) models “fill in” the gap between the keyframes with a set of new frames.

3. Then, a set of SSRs (Spatial Super Resolution) will take the low-resolution frames and upscale them to enhance the quality of the video, as most T2V models work in low-resolution pixel space for a more efficient process.

4. Finally, the SSR outputs are “stitched” together and that gives you the video.

Bottom line, AI videos are simply taking an image generator and training it to generate consistent images through time in batches and piece them together.

And these works… kind of.

Just like if you are filming an actor and midway through the sketch he breaks character and you try to finish the rest of the sketch by forcing him into that specific position to continue and avoid losing the first part, chances are that no matter how well you edit it, the cut is going to be visible.

Considering this limitation, video generation didn’t seem quite ready yet. With Google Lumiere however, we might be seeing the start of something big.

Space, Time, and MultiDiffusion

As their image counterparts, T2V are mostly diffusion models.

A diffusion model is a type of AI system that learns to map a noisy distribution of data into the target distribution through a denoising process.

In layman’s terms, they take a noisy image and a text condition (what you want the end result to be) and they gradually take out the noise from the image until you get the result.

However, instead of following the standard procedure we described earlier, Google has found an alternative using an STUnet.

But what is that?

A UNet is an architecture that downsamples, processes, and upsamples images.

It’s standard in image and video generation because it allows you to do the hard processing work in a compressed version of the images and thus save money.

With every step, the input noisy images are downsampled (made smaller) using convolutions that identify spatial patterns in the image.

This way, not only we are making the images more tractable, but we are also capturing the complex structures in them.

Once we have compressed the images and understood their meaning, they are upscaled into pixel space once again, recovering the original size but with the new images.

However, unlike other cases, Lumiere’s process is time-aware, meaning that we not only compress the images… we also compress time.

This allows Lumiere to create all the frames in the video at once (instead of the usual key frame + cascading frame filling we discussed earlier), and thus the STUnet simply needs to focus on capturing the semantics of the frames and upscaling them into the actual video.

However, you still need many SSR models to upscale the image due to memory constraints, meaning that there’s still some ‘stitching’ going on at the end.

Thus, to avoid inconsistencies across the upscaled outputs of each SSR, they apply MultiDiffusion (Bar-Tal et al, 2023).

What this does is ensure consistency across the different generated frame batches by using a MultiDiffuser.

In very simple terms, the MultiDiffuser allows several image generation processes to take place at once over a frame.

For instance, you can create a “blurred image” and at the same time apply new parallel generations to patches of the image, like drawing “a mouse” in a specific part of the image, and “a pile of books”.

The key intuition is that the MultiDiffuser ensures that, no matter what you generate in those fragments of the image through separate diffusion processes, they are consistent with the overall piece.

Consequently, this component ensures that for several frame batches of the video that need to be pieced together, you can recreate the boundaries between the output of the different SSR models so that they are consistent, ensuring a smooth transition between segments.

And the results are, put mildly, very impressive (video and paper).

A New Age for Video

With Lumiere, the world gets a clear view of what the future holds for video generation, editing, and animation, among others.

Soon, anyone will have the power to create impressive videos from scratch in no time and generate a new world of possibilities.

And despite the impressive results, it feels like we are just seeing the tip of the iceberg.

2024 is going to be wild.

🫡 Key contributions 🫡

• Google Lumiere is a new type of text-to-video model that achieves state-of-the-art quality and longer videos by generating the entire video at once and upscaling its quality.

• It also applies MultiDiffusion to ensure that the different segments of the upscaled video transition well.

👾 Best news of the week 👾

🥰 Bard gets image generation

🎥 Google shows off MobileDiffusion

🧐 Stanford lecture on Retrieval-augmented models

🤔 What is Meta’s AI strategy?

🥇 Leaders 🥇

The Fight to Be the New Dominant Computing Platform

Become a paying subscriber of Full Premium package to get access to this post and other subscriber-only content.

Already a paying subscriber? Sign In.

A subscription gets you:

• • NO ADS
• • An additional insights email on Tuesdays
• • Gain access to TheWhiteBox's knowledge base to access four times more content than the free version on markets, cutting-edge research, company deep dives, AI engineering tips, & more