🤯 AI Research of the week 🤯

“If you can’t beat it, join it.”

That’s how the saying goes, and that’s precisely what our dear friend Google is doing regarding AI.

In order to remain king of search, Google is purposely contributing to the demise of its ad-base revenue model, one of the most successful businesses in the history of capitalism.

But you shouldn’t fear for Google’s integrity, as we’ll see today… this is the plan all along.

Now, they have released their newest research on ‘AI-enhanced search’ with the release of AVIS, a powerful new framework for efficient question-answering, Image-based search.

The wake-up call

Anyone who follows the AI industry will probably agree on the fact that we’re approaching the death of Internet search as we know it.

And Google knows it too.

Using ChatGPT or Claude is much more quick and convenient than doing link-based searches.

Thus, Google had two options: Milk the dying cash cow while arguing that AI-search is stupid, or acknowledge the obvious and ride the AI wave so that they stay on top once it becomes a thing.

As Google’s execs aren’t dumb, they’ve gone for the second option.

In fact, there’s no company in the world right now more heavily focused on disrupting search with AI than Google.

For instance, a few weeks ago Google Deepmind released WebAgent, an AI autonomous agent capable of searching the web.

Now, as you’ll see, AVIS is another nail in the coffin for standard Internet search.

Of course, this is horrifying for Google’s ad-based model, which relies heavily on the ads displayed to users while they browse the web.

Although this breakdown is from 2022, it’s safe to say that the picture hasn’t changed at all.

Google’s ‘Search advertising’ represents 60% of the revenue! And that figure grows to a staggering 71% if you consider Google AdMob.

But with solutions like AVIS, the ad-model is suddenly completely broken.

The AVIS framework

Autonomous Visual Information Seeking, or AVIS, is a framework developed by Google to leverage Large Language Models (LLMs) to, given an image and a complicated question, have the capacity to dynamically plan, execute tools that provide the required information, and measure its own outputs autonomously until it finds the right answer.

Thus, the AVIS framework is comprised of three distinct elements:

1. LLM planner: An LLM that, based on the current state and observations, plans what tool and query it needs to get the required information from.

2. Working Memory: A log that retains the information from past executions.

3. LLM reasoner: An LLM that, based on the output from the tools, evaluates the obtained information and decides if it needs to progress further or if it’s ready to answer the question.

To better understand, let’s see an example:

Given the image at the top left and the question “When was the drum first used for this event?”, the LLM faces a very complicated task.

For starters, it has no additional information, meaning it’s missing a lot of context not present in the actual image in order to provide an answer.

For instance:

• What type of event is the image actually depicting?

• What’s the date of the event?

• What type of drum is that?

• And, finally, when was that drum first used in the event?

In other words, to answer that question, the LLM needs to search for that additional context.

The first challenge involves drafting the first plan. The LLM planner will define what’s the first tool it needs to start the process. In this case, that process is as follows:

• It starts by leveraging an object detection tool to scrutinize the complete image and crop it into distinct objects.

• Then, it decides to run an image captioning model such as PALI, to obtain a description of each cropped image.

• In parallel, the model chooses the most relevant cropped image based on the question, choosing the image of the drum.

• Next, it runs Google Lens Image Search API to retrieve the most similar images from a dataset of images with their descriptive texts. This way, it now knows that the drum is a ‘Taiko’ usually used in the ‘Aoi Festival’

• Now, the LLM reasoner evaluates the information and decides if it’s enough to answer the question.

• As it still requires the date, it performs an additional search using Google’s Search API, this time including much more relevant info such as the name of the drum and the festival

• With the retrieved information, it’s now finally able to answer, “7th century”.

And the impressive thing is that humans didn’t intervene one time in this whole process.

Additionally, to help the models learn, they performed a user study where they identified the most common actions humans take in such situations, and defined a transition graph for the model to use.

This way, the LLM planner had to decide the next action, it decided among the actions that users usually take in that situation.

But the question that comes to mind is pretty obvious.

Why is Google so adamant in pursuing the disruption of Internet search?

From an ad-based to API-based

In my humble opinion, Google’s plan is pretty clear.

If AI is clearly going to destroy our business model, let’s at least make sure it is us who do it.

Considering the priceless search data that Google holds, it’s almost a guarantee to me that Google will eventually deploy priced search APIs for AI agents to consume.

That way, instead of killing their business model, they are simply changing the way they charge for their data.

From Ads to APIs. Pretty genius if you ask me.

🫡 Key contributions 🫡

• AVIS achieves new state-of-the-art AI solutions in complex visual question-answering benchmarks

• It’s the first autonomous dynamic planner, allowing it to pivot if necessary

• It seamlessly integrates with multiple tools such as image search systems, text search, object detection, image captioning, and with other LLMs, all in one.

🔮 Practical applications 🔮

• Allows for what I believe to be Google’s future, a new range of search products providing data to AI agents through priced APIs

• It allows performing automatic autonomous web searches, liberating the users from having to perform this search manually

👾 Best news of the week 👾

🧐 Study proves that LLaMa 2 is as factually accurate as GPT-4

🥳 We’re now really close to having the first functional watermarking AI detector

😍 Amazing explanation of the opportunities in AI from Andrew Ng

🪐 AI drone beats professional racers

🥸 Google’s AI agent Duet will cost as much as Microsoft’s Copilot, $30

🥇 Leaders 🥇

Become a paying subscriber of Full Premium package to get access to this post and other subscriber-only content.

Already a paying subscriber? Sign In.

A subscription gets you:

• • NO ADS
• • An additional insights email on Tuesdays
• • Gain access to TheWhiteBox's knowledge base to access four times more content than the free version on markets, cutting-edge research, company deep dives, AI engineering tips, & more