TLDR:

• AI Research of the Week: NOIR, Moving Robots with Brain Signals

• Leaders: Envisioning a Future of AI Empowerment and Economic Opportunity

🤯 AI Research of the week 🤯

At this point, my mom thinks some AI applications have become nothing far from magic.

And today we are going to talk about one of those cases, NOIR.

Developed by Stanford University, this is a model that allows humans to move robots with their minds, something not short of extraordinary.

NOIR is a general-purpose model with the capacity to decode imagined motions through brain signals into interpretable robotic motions that execute the action imagined by the person.

Thanks to advancements like NOIR, one can envision a world where physically-impaired humans see their quality of life massively improved.

But first, how does it work?

A Tale of Two Components

To decode the brain signals, the researchers decided to use electroencephalography (EEG).

But EEG is hard to decode, especially considering complex behaviors like:

1. Choosing an object visually

2. Deciding how you’re going to pick it up

3. Deciding what to do with it

4. Perform the action

To fix this, NOIR is comprised of two components:

• A modular goal decoder

• A robot to perform actions

The modular decoder will break down the human goal into three parts: What object, How to interact with it, and Where to interact.

Focusing on the what, NOIR takes in SSVEP signals, a type of brain signal that matches the flickering frequency of certain stimuli.

In layman’s terms, if you watch an object flickering at a certain frequency, your brain will emit SSVEP signals at that exact frequency.

Consequently, if you have objects emitting light at different frequencies, the one with the highest correlation to the frequency of the SSVEP signal will be the object being observed by the human.

Traditionally, researchers incorporated LED lights into objects to make them flicker, but here the Stanford researchers leverage Meta’s DINO computer vision foundation model to simulate this.

When observing a certain scene, DINO segments all objects appearing in the image.

Then, the researchers simply add a virtual flickering mask (every mask at a slightly different flicker frequency to be able to discriminate) to every segmented object.

This way, we know what object the humans are looking at.

Next, to understand what the human wants to do with that object, the how and the where… things get a little bit trickier.

From Imagination to Motion

When we humans think about performing a movement, this thought generates brain signals too. This is what we describe as Motion Imagery (MI).

The challenge here is that we want to be able to see a certain signal and decode —interpret—what precise motion those signals are describing.

However, this is far from easy.

If you process the MI signals when a human thinks about “moving the left hand” or the right hand, being able to discern that from MI signals is not trivial.

In statistical terms, to make the signal interpretable, we want to maximize the variance of a certain motion regarding a certain signal while minimizing the variance of the other possible motions.

In layman’s terms, for NOIR to work properly, we need to be able to discriminate between signals so that each signal is assigned to its correct motion.

For this, researchers applied Common Spatial Pattern (CSP), a technique that identifies the components of a signal that express the most variance toward a certain movement.

I know that made no sense, so let’s see an example: Let’s say we analyze the brain signals of a person thinking about two movements, moving the left hand (red), and then moving the right one (blue).

You are registering these signals along two brain channels at distinct frequencies, each one represented by an axis of the graph.

As you can infer from the above image, the channel depicted in the y-axis shows great variance regarding left-hand movement (red points), with values ranging from 10 to -7 or so.

However, it shows less variance regarding blue points (imagined right-hand movement) with almost all values ranging between -2 and +2.

Long story short, we can infer some information from this graph, but would you be comfortable enough to say that the channel in the y-axis would always refer to left-hand imagined movements (red points)?

Probably not. Thus, you need a clear way to discriminate.

Now, watch the image below with CSP filtering applied:

Now it is a hell lot easier, as left-hand movement (red) is clearly influenced by the signals from the x-axis, and the opposite happens with the right-hand motion.

Next, using other statistical techniques I won’t get into, researchers extract the most important features from each signal that explain every movement.

If you care to know more about CSP, check this.

So, then, what’s the end-to-end process?

1. The user views a certain environment and decides what he wants to do, like “pick a bowl and pour the water into the pan”

2. While the human focuses on the bowl, NOIR uses DINO to segment the video frame and match the SSVEP signal to the frequency of the bowl’s mask

3. Next, the user imagines the movement: move the right arm and use the hand to pick a bowl

4. NOIR processes the MI signals generated from the user imagining that movement and determines what movement the user is thinking about.

5. Then, it uses the required primitive, such as Pick(x,y) and Move(x,y) functions that signal the robot what movement it needs to perform.

Following this pipeline, NOIR is capable of performing various tasks, up to 20, becoming the first Brain-Robot Interface that is general-purpose and, who knows, the future of robotics.

But NOIR had one extra trick up its sleeve: adaptive behavior.

Leading by example

Even though NOIR’s capabilities are amazing, it takes time to make decisions, especially on the MI part.

Here, the researchers propose a method that allows the model to take previous experiences and actions to better interpret the current one.

Using image encoders, the model processes the scene into an image embedding, a vector representation of that image, inserting it into a ‘feature space’ where similar images have similar vectors representing other past experiences.

These past experiences also include the label of the object and movement previously executed.

For instance, if the image the model sees is a table with salt and a steak, the model processes the image, retrieves similar ones, and suggests that, based on previous similar situations, the outcome was picking up the salt and pouring it on the steak.

The results? Over 60% decrease in decoding time and a more adaptive behavior towards the user.

Undeniably, NOIR pulls us much closer to improving the lives of people, like quadriplegics, who desperately need to gain back control over their lives.

Read the full paper here, and video examples here

🫡 Key contributions 🫡

• NOIR is the first general-purpose brain-robot interface that takes in brain signals and decodes them into robot movement

• It also includes an adaptive behavior module that allows NOIR to learn from its user

🔮 Practical implications 🔮

• 50 out of 100k humans are completely paralyzed. Solutions like NOIR could make their lives much better by allowing them to move objects around… or even their own body

• Improved BRI decoders could help humans perform dangerous tasks without actually putting their bodies on the line

👾 Best news of the week 👾

🤨 As Sam Altman was fired, a coup by OpenAI could have him back

😍 Leverage GPT-4V via the “Make it Real” app

🫣 A game that will help autonomous cars understand pedestrians better

🫥 This is how far we are from AGI, according to Google

🥇 Leaders 🥇

Envisioning a Future of AI Empowerment and Economic Opportunity

Become a paying subscriber of Full Premium package to get access to this post and other subscriber-only content.

Already a paying subscriber? Sign In.

A subscription gets you:

• • NO ADS
• • An additional insights email on Tuesdays
• • Gain access to TheWhiteBox's knowledge base to access four times more content than the free version on markets, cutting-edge research, company deep dives, AI engineering tips, & more