AI Gets a Sense of Space: New Research Explores How Machines “Think” in 3D

Artificial intelligence is learning to navigate the world around us, not just through screens, but in the same way we do – by understanding space. A new study titled “Thinking in Space” delves into how multimodal LLMs, the powerful AI models behind the latest advancements in language and image processing, are developing spatial reasoning abilities. This research, a collaboration between NYU, Yale, and Stanford, goes beyond analyzing videos and movies, focusing instead on everyday environments where future AI assistants might operate.

Why is spatial reasoning so crucial for AI? Imagine an AI assistant that can truly understand your home, retrieving objects (“Where are my keys?”), navigating complex layouts (“Can you bring me the book on the shelf behind the armchair?”), or even offering assistance in unfamiliar surroundings (“Guide me to the nearest exit”). This requires more than just recognizing objects; it demands an understanding of spatial relationships, distances, and perspectives.

The study reveals a fascinating gap between how current AI models process visual and spatial information. While some models excel at handling spatial data, like recognizing objects and their positions, multimodal LLMs struggle to integrate this data with logical reasoning. This disconnect highlights the complexity of human spatial-visual thinking, which seamlessly combines perception, memory, and logic.

Researchers tested leading LLMs, including Google’s Gemini Pro, on a variety of spatial intelligence tasks. While these models showed competitive performance, they still lag behind human capabilities, particularly in tasks requiring long-term spatial memory. For example, remembering the location of objects across a sequence of actions or navigating a complex environment over time remains a challenge.

Interestingly, the study found that linguistic prompting, a technique highly effective in general video analysis, actually hinders performance in tasks requiring visual-spatial intelligence. This suggests that spatial reasoning operates on a different cognitive level than language processing, requiring specialized mechanisms.

Another key finding is that current LLMs tend to build “localized” world models, focusing on immediate surroundings rather than forming a comprehensive understanding of the entire space. This limitation impacts their ability to reason about distant objects or navigate complex environments.

This research has significant implications for the future of AI. By improving spatial reasoning capabilities, we can develop AI assistants that truly understand and interact with the physical world. Imagine AI-powered glasses that provide real-time guidance, helping users navigate unfamiliar places, find lost objects, or even assist those with visual impairments.

To encourage further exploration in this critical area, the researchers have made the study’s paper, dataset, and code publicly available. This open approach fosters community involvement and accelerates the development of AI that can “think” in space, bridging the gap between the digital and physical worlds.

https://vision-x-nyu.github.io/thinking-in-space.github.io