TLDR:

• AI Research of the Week: Microsoft presents LLaVa 1.5, a new open-source paradigm

• Leaders: Everyone will Manage Teams in the future, only most of the team members won’t be human; The MultiAgent Paradigm is now a reality

🤯 AI Research of the week 🤯

I’ve talked about the AI war between open-source and private models many times, and always the outcome seems to be the same.

Open-source seems great and full of promises, but is often simply wishful thinking and impracticality.

But now things might have changed.

Microsoft, alongside the Universities of Wisconsin-Madison and Columbia, has presented the newest version of the LLaVa model, LLaVa 1.5.

LLaVa, one of the first truly performant Large Multimodal Models (LMMs), has been upgraded and the results are highly impressive, considering it is orders of magnitude, and I mean hundreds of times, smaller than models like GPT-4 Vision, OpenAI’s newest release that is taking the world by storm.

The recently released paper not only gives us huge insight on how state-of-the-art multimodal models are built, but also manages to prove the entire industry wrong.

Yes, everybody was wrong about open-source, including myself.

Oh my Sweet, Sweet Grafting

First and foremost, we must clarify what multimodality is, as this word gets thrown around aimlessly.

What really is multimodality

In a strict and concise form, multimodality represents the capacity of a model to process at least two different modalities, a modality being a type of input data (words, images, sounds, et al).

However, this definition can easily be fooled. If I get a speech-to-text model and an LLM, I can easily create a solution that processes both speech and text.

But does that make my solution multimodal?

Yes, but the models used certainly aren’t multimodal.

In its purest form, true multimodality occurs when a model’s different modality components share a common embedding space.

In other words, taking a visual and text model like LLaVa, it happens when the visual and text components encode the input into the same high-dimensional vectorial space.

Therefore, if an image of the Taj Mahal and its Wikipedia text description share very similar vectors in this embedding space, we have achieved multimodality, as the model understands the salient objects (the Taj Mahal in this case) of both modalities as the same ‘thing’, just like a human would.

But how did the researchers achieve this with LLaVa 1.5?

Grafting, a cost-effective and elegant method

There are three ways to achieve multimodality:

• Tool/model-based methods: By combining different models or tools you can allow your solution to handle multiple inputs. While the solution is multimodal, the underlying models aren’t.

• Grafting: Implies using pre-trained image encoders and LLMs and projecting the encoder’s vector embedding into the LLM’s latent space using a projecting matrix or an MLP layer.

• Generalist systems: Most probably how GPT-4V was trained, training an image encoder and an LLM from scratch into the same embedding space. Here, all weights are trained from scratch.

LLaVa was trained using grafting, which is awesome for one key reason:

The image encoder and the LLM’s weights remain frozen, and we simply train the projecting matrix to learn to transform the encoder’s vectors, or ‘grafting’ them, into the LLM’s high-dimensional space.

As you can see above, when an image is sent to the image encoder (a CLIP encoder in LLaVa’s case), it processes it, and then it goes through an ‘adapter’ which in reality is simply a matrix that transforms the output vector of the image encoder into an acceptable vector that the LLM (Vicuna in LLaVa’s case) understands.

After, the model gets fine-tuned using a visual instruction dataset to create a chatbot that will dutifully answer questions regarding images and text.

Grafting not only works great, as LLaVa will prove in a second, but also it makes the whole process very cost-effective, as you only have to learn the weights of the projection matrix, the rest of the components remaining frozen in their pre-trained form.

And what are LLaVa’s results?

Another day, another SOTA

The results obtained from LLaVA 1.5, especially the 13B model, are impressive.

Out of the 12 most popular multimodal benchmarks, LLaVa 1.5 is the new state-of-the-art (SOTA) for 11 out of the 12, beating other popular open-source models by a landslide.

But seeing a model beat others in AI has become business as usual these days.

However, with LLaVa 1.5 things have taken a turn for the unexpected.

LLaVa-1.5 manages to beat all other models, models that use the same base LLM, by using orders of magnitude less data and computational effort, in some cases requiring x2,500 times less data for pre-training ('PT’ column), and 75 times less (‘IT’ column) in the fine-tuning stage.

Consequently, LLaVa-1.5 was trained, literally, in a fricking day with an 8-A100 NVIDIA GPU node for a total of 26 hours.

Ladies and gents, open-source has just become affordable, finally.

Therefore, training high-performant models has finally become attainable for enterprises, that could leverage their data to train open-source models that make them stand above the rest.

In a world driven by money, not everything is about having the best model, sometimes you just need to make it viable.

But, incredibly, LLaVa-1.5 just happens to be both in the world of open-source LMMs.

Try the LLaVa 1.5 13B model here!

🫡 Key contributions 🫡

• LLaVa-1.5 models become the first open-source LMMs that are high-performant while extremely efficient to train

• The paper gives us great insights into how these models are trained, especially regarding the super popular Grafting method

🔮 Practical implications 🔮

• Tech-savvy enterprises can use LLaVa and similar models to fine-tune them with their data and create top models for their niche

• For the first time, private models don’t seem to be the ‘only way’ for companies anymore

👾 Best news of the week 👾

😍 The State of AI Report for 2023

💸 Cleanlab raises $30 million to build trustworthy AI systems

🎆 Meta will now allow you to chat with your favorite celebrity

🎙 ElevenLabs introduces ‘AI Dubbing’ to translate your voice to 20 languages

🤩 Here’s the complete Adobe AI presentation with mind-blowing tools

🥇 Leaders 🥇

The MultiAgent Paradigm: Taking AI to a New Dimension

Become a paying subscriber of Full Premium package to get access to this post and other subscriber-only content.

Already a paying subscriber? Sign In.

A subscription gets you:

• • NO ADS
• • An additional insights email on Tuesdays
• • Gain access to TheWhiteBox's knowledge base to access four times more content than the free version on markets, cutting-edge research, company deep dives, AI engineering tips, & more