TLDR:

• AI Research of the Week: Sora, OpenAI’s Stunning New Video Model

• Leaders: V-JEPA and World Models

🤩 AI Research of the week 🤩

The feeling that things have changed. That’s what almost everyone who has seen Sora felt.

Last week, we talked about Google’s Lumiere model and how it was better than anything we had seen. Well, a week later, OpenAI’s Sora blows Lumiere completely out of the water.

No words can describe it, so here’s an example:

Thus, today, we are unpacking all that is known about this life-changing model, and what it means for the future of many industries.

Best of both worlds

Sora is OpenAI’s first text-to-video, image-to-video, video-to-video model, a world simulator that generates the most impressive videos we have ever seen, literally miles ahead of anything else.

Similarly to ChatGPT, it has seen Internet-scale data, but of videos and their descriptions, to learn to create up to 1-minute-long high-resolution videos.

However, the similarities with ChatGPT don’t end there.

Just like LLMs treat text as tokens (groups of 3/4 letters on average), Sora treats frame patches as tokens too, calling them “visual spacetime patches”.

But what does that even mean?

In a nutshell, Sora treats videos by decomposing them into two elements:

• Frames. Just like any other video model, the easiest way to build videos is to consider them as concatenations of frames (static images), as Google’s Lumiere does.

• Visual patches. Each frame is then decomposed into groups of pixels. However, these patches are 3D, as they capture more than one frame so that we can account for time too.

Then, depending on the size ratio you want the output to be, those patches are assembled into that shape to generate the video.

But how does it actually generate videos?

As we have described many times, diffusion is the standard approach to synthesizing images and videos.

During training, noise is progressively added into an image, and then the model has to predict how much noise was added, take it out, and reconstruct the original image based on the given text condition that describes the original image, like “a portrait of a relaxed cat”.

That way, during inference, the model is given a noise canvas and it’s capable of creating new images based on the given text description.

So, what’s the end-to-end process then?

Sora’s generation process

During training, the model is given two things, a video, and a text description of that video.

Then, the process follows four steps: compression, patching, diffusion, and decoding. Let’s unpack each one.

First, the video is turned into a compressed, latent representation, using a visual encoder.

However, this transformation has other implications too:

• Efficiency. Sora works videos in compressed form, making the process much cheaper

• Effectiveness. Working in representation space helps the models pay attention to what matters, instead of scrutinizing every pixel of the image.

However, it must be noted that this is a spacetime compression.

But what does that mean? 

Well, for starters, as Sora is a transformer, we know that it processes videos using the attention mechanism.

In very simple terms, attention is the process of making tokens talk and updating the meaning of each token based on its explicit meaning and its surroundings.

Sounds too abstract, but it’s quite intuitive if we think about how ChatGPT works.

If we think about the sentences “The river bank” and “I’m going to the bank to take out some money” the word “bank” has different meanings depending on its surrounding context.

So what ChatGPT does to figure out what bank means in each case is performing the attention mechanism so that each word pays attention to the rest.

That way, it figures out the meaning of “bank” not only by its definition, but also in the context of its surroundings.

In Sora’s case, the attention mechanism is performed at the frame level, making visual patches talk with each other, and also at the sequence level, making frames ‘talk’ with other frames.

The former will indicate what is going on in that particular frame, and the latter will give Sora intuition of what is going on across multiple frames.

Put simply, one tells you there’s a dog in one image, and the other tells you what the dog is doing in the video.

Next, this latent representation is decomposed into the visual spacetime patches we talked earlier about, creating the cube-like shapes we see in the image above, to allow Sora to generate videos in any size, duration, and resolution required.

It is important to note that this token-based mechanism is a key difference between Sora and Lumiere.

Although Lumiere is a diffusion transformer too, the model generated each image in the video all at once. Here, Sora builds every image by concatenating patches.

Furthermore, with the patches placed at the desired arrangement, diffusion is applied to the noisy patches to create the representations of the new video.

Finally, a decoder takes these ‘clean’ new patches and decodes them back into pixel space, giving you the videos.

But that’s not all.

Sora can also be conditioned on images and video, meaning that you can use it to animate images, perform video editing, interpolate videos, and many more.

But how does Sora allow for multiple conditioning?

They don’t elaborate on this, but this isn’t something new. A popular method is classifier-free diffusion guidance. Here, the model is trained to generate outputs both conditionally and unconditionally.

In other words, they sometimes tell it what to generate using a text description or an image, others they simply make it generate “something” with no condition. This way, the model learns to flexibly adapt to every situation, generalizing into several conditional generation processes.

What’s more, this gives you control by tuning a parameter, allowing you to control how much the output must pay attention to one condition or the other.

For instance, in the image below of Michelangelo’s David, depending on the value of each condition (the original image, or the text description “Turn it into a cyborg”, the more weight the text description is given, the less it looks like the original image and more like a cyborg, and vice versa.

The implications? Oh my.

Ten months ago, the state of the art of AI video generation looked like this. Now, we have AI-generated videos that require extensive investigation to uncover whether they are real or not.

Cinema, designers, artists, production companies, no one is safe now. Yet, there’s still much room for improvement.

Seeing some of the generated videos, you quickly realize that while Sora has learned a very good representation of the physics of the world, it still doesn’t understand it, generating really weird, implausible situations at times.

For that, today’s Leaders segment below tackles what Meta feels is the key to actually turning Sora and other models into world models, aka the key to AGI.

But more on that below…

🫡 This Newsletter is Brought to You by AE Studio 🫡

Seeing the speed at which AI develops, dealing with it is becoming unmanageable. And don’t get me started with actually building AI products.

In that case, just let experts do it for you starting with an 80% discount by AE Studio, a company trusted by the likes of Salesforce, Berkshire Hathaway, or Walmart, among others.

Hire a world class AI team for 80% less

Trusted by leading startups and Fortune 500 companies.

Building an AI product is hard. Engineers who understand AI are expensive and hard to find. And there's no way of telling who's legit and who's not.

That's why companies around the world trust AE Studio. We help you craft and implement the optimal AI solution for your business with our team of world class AI experts from Harvard, Princeton, and Stanford.

Our development, design, and data science teams work closely with founders and executives to create custom software and AI solutions that get the job done for a fraction of the cost.

P.S. Wondering how OpenAI DevDay impacts your business? Let’s talk!

👾 Best news of the week 👾

🥰 Apple’s new image-editing model

🧐 Google’s amazing Gemini 1.5, best LLM ever?

🤩 Meta automates unit testing with new model

🎥 Generative AI in a nutshell, by Henrik Kniberg

🥇 Leaders 🥇

V-JEPA, our first true world model?

Become a paying subscriber of Leaders to get access to this post and other subscriber-only content.

Already a paying subscriber? Sign In