TLDR:

• AI Research of the Week: LEGO, the Future of TikTok & Your Kids

• Leaders: The Inevitable Death of Apps

🤩 AI Research of the week 🤩

ByteDance has announced a new model named LEGO that not only has impressive capabilities, but also gives us a clear idea of the future of its products, including the app your kids are glued to right now.

The world-famous TikTok. And you might not like what this future looks like.

At all.

Pay close attention, AI

Even though Multimodal Large Language Models like ChatGPT are all the rage right now, they are surprisingly limited.

Therefore, since the beginning of 2024, researchers have put a lot of focus on making these models better.

The promises of success are so many that some of the largest corporations in the world, like Apple with its recently covered by this newsletter model Ferret, are working on it.

But why are MLLMs so limited?

The attention problem

Most MLLMs today, except some notable exceptions like Fuyu, are based on the combination of two AI components:

• A Large Language Model (LLM), aka the brain of the model. It processes all inputs and generates the response the MLLM will give back to you.

• One or more modality encoders. An encoder is an AI model that takes an input, like text, audio, images, or video and outputs a representation of that input.

Next, researchers, to save money, do what we call ‘grafting’.

They take a pre-trained encoder, like OpenAI’s CLIP model to process images, and an LLM, and connect them using an ‘adapter’.

This adapter is required because, as the encoder and LLM have been trained separately, their respective representations of the same thing, like ‘table’, may be different.

As the generative part of the MLLM always comes through the LLM, the usual practice is to use the adapter to transform the representation of the encoder into the ‘embedding space’—the vector space of representations—of the LLM.

Sounds complicated, but the intuition is simple.

Imagine a Japanese woman and a blind American man facing the sunset.

Even though the Japanese woman can explain what she’s seeing perfectly in Japanese, the blind American man can’t understand her.

Thus, she uses Google to translate its Japanese description into English.

In other words, the encoder and the LLM basically ‘speak different languages’, and the adapter transforms the image representations from the encoder into a language the LLM ‘understands’.

Hence, using a multimodal dataset, we train this adapter to perform this transformation effectively and, this way, LLMs gain the power of sight.

But here’s the thing.

The encoders, be that an image or audio encoder, have not been trained to pay attention to fine-grained details, but to get a global flavor of the input.

And that leads to grounding problems.

Missing key context

MLLM grounding refers to the capability of an MLLM to base the input text on the specific key context provided by other inputs.

If the image portrays a husky and the question is “What is this animal good at?” the model should not only respond based on the fact that it’s a dog, but the fact that it’s a husky indicates the answer should be “sled-pulling” which is certainly not the answer for chihuahuas.

That is grounding.

But for more complex tasks like “What is the animal on the top-left of the image sitting right next to a tree doing with its paws” most AI models, even frontier models like ChatGPT, suffer greatly.

And here’s what LEGO comes to the stage.

One-Stop Shop

LEGO is the first MLLM that achieves multi-modal and fine-grained perception across modalities.

And it’s capable of doing amazing stuff like image grounding:

or previously unseen capabilities like fine-grained video grounding with spatial and temporal awareness (it can pinpoint the exact moment an element in the video does something):

In other words, it’s capable of paying close attention to detail in multiple modalities, not just images, like Apple’s Ferret.

Binding data all in one

Put simply, LEGO simply applies the adapter principle to images, video, and audio… all in one.

To achieve this, instead of fancy localized attention modules like Ferret, it achieved these capabilities by doing it the ‘hard’ way, building a dataset of data exclusively tailored for grounding purposes.

Then, they followed a three-step training process:

1. Multi-modal Pretraining (grafting): Aligns each pre-trained multi-modal encoder with the LLM embedding space using multiple adapters. This stage focuses on enabling the model to comprehend multi-modal inputs.

2. Fine-grained Alignment Tuning: Aims to enhance the model's understanding of spatial coordinates and timestamps. Here, the model learns to draw accurate bounding boxes over objects and signal exact timestamps on a video where the identified event occurs. It involves training the LLM and adapters, while the encoders for each modality remain frozen.

3. Cross-modal Instruction Tuning: Further refines the model using generated data to improve alignment with human preferences and enhance multi-modal interactions. This stage also freezes the modality encoders while training the LLM and adapters.

In summary, in a very cost-effective way, LEGO achieves highly competitive results, despite not being completely revolutionary.

But if the model isn’t a game-changer technologically, why are we talking about it?

Easy, because it means a lot to how Tiktok is going to evolve under the hood soon.

A detective machine

TikTok become the app it is today due to its super-smart AI-based algorithm that glues you to the screen of your smartphone for hours on end.

The algorithm studies your viewing patterns and decides what content is going to make you want to doomscroll until you pass out, as the more content you view, the higher its ad revenues.

Considering one of every four Internet users uses ByteDance’s video-sharing service, it’s fair to say it’s the most successful AI algorithm ever, even surpassing ChatGPT.

Now take these capabilities and add LEGO on top. With LEGO, the algorithm can:

• Clearly understand the content of the video

• Identify what points of the video resonated with you (you paused, you clicked like, etc.)

Et voilà, the most successful and addicting AI algorithm in history now can interpret your viewing patterns during the actual video to become even more successful and addicting.

You see, AI has a lot of good to create in our world, but some of the most addicting products are soon going to be flooded with AI features to better predict our buying patterns, or our content interests, making our already-hinged kids the perfect target for ads and a never-ending dopamine hit as the algorithms get to know them better than we do.

Sounds great, huh?

🫡 Key contributions 🫡

• LEGO is the first MLLM that successfully grounds multiple modalities into one while paying attention to key details.

• It has amazing localized temporal and spatial awareness, being on par with many recent optimized MLLMs and reaching new highs in tasks like video grounding.

Artificial Intelligence online short course from MIT

Study artificial intelligence and gain the knowledge to support its integration into your organization. If you're looking to gain a competitive edge in today's business world, then this artificial intelligence online course may be the perfect option for you.

On completion of the MIT Artificial Intelligence: Implications for Business Strategy online short course, you’ll gain:

• Key AI management and leadership insights to support informed, strategic decision making.

• A practical grounding in AI and its business applications, helping you to transform your organization into a future-forward business.

• A road map for the strategic implementation of AI technologies in a business context.

Get More Information

👾 Best news of the week 👾

🥰 Let David Attenborough narrate your life

🧐 Mark Zuckerberg declares his goal of creating AGI

🫱🏼🫲🏾 First-ever OpenAI deal with a University

🤨 BMW is getting a fresh set of new robots that make coffee (and other stuff)

🥇 Leaders 🥇

The Inevitable Death of the App

Become a paying subscriber of Full Premium package to get access to this post and other subscriber-only content.

Already a paying subscriber? Sign In.

A subscription gets you:

• • NO ADS
• • An additional insights email on Tuesdays
• • Gain access to TheWhiteBox's knowledge base to access four times more content than the free version on markets, cutting-edge research, company deep dives, AI engineering tips, & more