Project Astra
NotebookLM shared with system prompt and other contexts
Click Crash Courses for grounding sources in NotebookLM
Project Astra is Google DeepMind’s research prototype for a universal AI assistant designed to see, hear, remember, and interact with the world in real time. Powered by the multimodal capabilities of the Gemini model intelligence, it acts as a context-aware, proactive companion rather than a traditional, reactive voice assistant. [1, 2, 3]
Core Capabilities
- Multimodal Perception: Processes continuous streams of live video and audio input simultaneously.
- Real-Time Latency: Delivers rapid, natural conversations across languages without noticeable lag or interruptions.
- Contextual Memory: Remembers spatial layouts and past events, such as recalling exactly where you left your keys or glasses.
- Action Intelligence: Anticipates user intent and executes multi-step tasks across integrated applications like Search, Gmail, and Google Maps.
- Agent Highlighting: Dynamically identifies real-world objects using live on-screen highlights to guide the user’s attention. [1, 2, 3, 4, 5]
Real-World Applications
- Everyday Assistance: Features have begun rolling out to Gemini Advanced subscribers through Gemini Live, enabling users to share their live camera stream or device screen directly with the AI. For instance, users can point their camera at complex appliances or structural issues to get step-by-step guidance. [4, 6, 7]
- Accessibility: Partnering with organizations like Aira, Project Astra powers specialized visual interpreters within the Aira Explorer App. This enables blind and low-vision users to naturally navigate environments, read physical mail, and describe objects on the go. [8, 9, 10]
- Interactive Learning: Acts as a real-time tutor that can analyze mathematical equations or chemical compounds drawn on a physical board and interactively walk students through calculations. [4, 11]
Watch how Project Astra processes real-time environmental context and interacts across different form factors:
Are you looking to test Project Astra’s capabilities on your device, or are you interested in its underlying technical architecture?
[3] https://www.courseadvisor.asia
[8] https://aira.io
[9] https://aira.io