TLDR:

• AI Research of the Week: CogVLM, a Revolutionary Multimodal Solution To the MLLM Problem

• Leaders: OpenAI Shows Us the Future of Software Applications… and Much More

🤯 AI Research of the week 🤯

A group of researchers has presented a new model that revolutionizes the current multimodal AI design standards while blowing out of the water almost all competition.

The introduces an innovative concept, Deep Fusion, a new design that mitigates the biggest problem faced by Multimodal Large Language Models (MLLMs) today, the “shallow alignment problem”.

Were to deliver on its potential, the CogVLM model could become a seminal research paper that will draw the attention of researchers around the world to create a new family of MLLMs, Deep Fusion models.

The actual results? Impressive capabilities like coding math problems from images, and many more we’ll see shortly.

But first and foremost, what is the shallow alignment problem?

It couldn’t be that easy

Building an LLM is a cumbersome task.

You need a huge dataset of text documents, a team of world-class researchers, and a powerful GPU cluster. In other words, you need a ‘whole lotta monei’ and talent.

And if you want to make your model commercially available, you need to make it aware of what to say or not say.

For that alone, you will also need an army of human annotators to perform Reinforcement Learning from Human Feedback (RLHF), which in turn needs even more capital.

The result?

But building an MLLM… that is a totally different beast.

Not only do you have to do the above process, but you also have to train an image encoder, the piece in the architecture that processes images, and make them work together.

Hence, unless you are part of the magnificent seven, or one of such companies is fiercely funding your efforts, you simply don’t have the means to train these models.

Consequently, open-source researchers, driving their efforts mainly through University funding, have to be smart about their designs.

Grafting, an elegant answer

To avoid having to build the complete model by scratch, the most common method of creating MLLMs is grafting, the process of connecting an already-trained image encoder and an LLM through an adapter, usually a Q-former (like the one we saw last week in SALMONN), or an MLP layer.

The rationale is clear. If you want to use components that are already trained, you need to align them, i.e. make them talk the same language.

To do so, once the image encoder processes the image, the resulting representation is projected into the embedding space of the LLM.

But there’s a problem.

The inescapable truth is that the weights of the LLM are not meant to process images. They are not meant to capture the salient features of images like color, the absolute and relative position of objects in the space, and so on.

To comprehend this, let’s say we want the model to tell us “What object is placed on the right-hand side of the image?”.

Firstly, the model will process the image through the encoder and the text through the embedding lookup matrix (LLMs don’t have text encoders, they simply learn the embeddings previously for each word in their vocabulary).

The image encoder will identify all the objects pretty well; oranges, frying pans, ceiling lights, and so on, and these objects and their spatial relationships will be infused into the image embedding.

Next, this embedding is projected into the LLM’s ‘world’ with the adapter and processed by the LLM’s weights.

But here’s the thing, the LLM weights have not been trained to process features inherent to images, but only text-related features, meaning that the image-specific features are ‘lost in translation’, leading to hallucinations.

You can try to fine-tune the LLM with the image data, but that leads to catastrophic forgetting for NLP tasks as suggested by PaLM-E (Driess et al, 2023).

But with Deep Fusion, that is not a problem anymore.

Same cost, much smarter model

Put simply, CogVLM solves the “shallow alignment problem” by adding a visual expert module.

To the left-hand side, the process is the same as until now, an MLP Adapter transforms the outputs of the image encoder into the same dimension the word embedding has.

But here is where things get interesting.

If we look at the right-hand side, we see that the researchers have essentially duplicated the attention weights of the LLM.

The key intuition here is that, while we want to avoid updating the weights of the LLM, we duplicate these weights and train the new ones with the image features.

Therefore, these new weights will focus on the unique features of images, while the original weights that were trained on millions of text passages continue focusing purely on text.

And, more importantly, the results of both attentions are summed up, creating a new, combined representation defined as Deep Fusion by the researchers.

This new representation is like taking the image and the text and merging them into 'one thing', one unique multimodal representation that encapsulates the information from both modalities into a unique set of features.

Using the previous example, the LLM inside CogVLM not only can process the text request, but it now has specialized weights that have captured image representations otherwise lost like color, space, and absolute and relative position of objects.

This all makes questions like “What is the object on the right-hand side of the image?” a very simple task.

In a way, Deep Fusion is analogous to how the human brain integrates visual and textual information. Just as we naturally combine what we see with relevant textual or spoken context, this model fuses visual and textual data to create a more comprehensive understanding of the entire input sequence.

Amazing results

Unsurprisingly, CogVLM faires well when compared to other similar-sized models, beating all previous state-of-the-art models but PaLI-X-55b, a model three times larger.

Naturally, the model is capable of performing highly impressive tasks like complex counting:

Or advanced image understanding:

Innovation, innovation, innovation

In short, CogVLM is not simply another state-of-the-art solution that marginally improves previous models.

It represents a new way of creating multimodality, one that not only is computationally efficient, it also allows a fairly small model to excel at complex cross-modality tasks where traditional MLLMs struggle greatly.

Seeing CogVLM capabilities, I’m sure the entire world will be paying attention to this paper and soon the deep fusion architecture could become ubiquitously related to ‘multimodality’.

In the meantime, check their research paper for more info: Link to paper

🫡 Key contributions 🫡

• CogVLM introduces Deep Fusion, a new architecture design that allows AI models to build hybrid representations that incorporate features from multiple modalities

• It not only excels at common MLLM tasks, it greatly surpasses most MLLMs in cross-modality tasks

🔮 Practical implications 🔮

• MLLMs have eyes and mouth, and will soon have ears (not to be confused with ChatGPT, which isn’t audio multimodal)

• Truly multimodal MLLMs are a step forward toward embodied intelligence, where robots will have similar sensorial capacities to humans

👾 Best news of the week 👾

👀 Neuralink’s new chip wants to cure blindness

🧐 The Financial Times discusses if GenAI will transform business

🤨 Bill Gates’ view on the future of computing with LLM agents

😟 Elon Musk’s Grok model isn’t shy about responding to extreme questions

🦾 Controlling robots with thoughts, a reality today

🥇 Leaders 🥇

OpenAI Just Showed Us the Future of Software Applications… and Much More

Become a paying subscriber of Full Premium package to get access to this post and other subscriber-only content.

Already a paying subscriber? Sign In.

A subscription gets you:

• • NO ADS
• • An additional insights email on Tuesdays
• • Gain access to TheWhiteBox's knowledge base to access four times more content than the free version on markets, cutting-edge research, company deep dives, AI engineering tips, & more