The Magic Pointer on Googlebook was built with Google DeepMind. The research team behind this underlying capability shared more about the premise of AI-enabled pointers.

DeepMind wants to use AI “to help the pointer not only understand what it’s pointing at, but also why it matters to the user.”

Our goal is to address a common frustration: because a typical AI tool lives in its own window, users need to drag their world into it. We want the opposite: intuitive AI that meets users across all the tools they use, without interrupting their flow. For example, imagine pointing to an image of a building, and requesting “Show me directions”. Nothing more is needed when the AI system already understands the context.

The idea is to replace “text-heavy prompts with simpler, more intuitive interactions.”

An AI-enabled pointer would streamline this process by smoothly capturing the visual and semantic context around the pointer, letting the computer “see” and understand what’s important to the user. 

Similarly, an “AI system that understands this combination of context, pointing and speech would allow users to make complex requests in natural shorthand.”

Example use cases include:

• could point at a PDF and request a bullet-point summary to paste directly into an email
• hover over a table of statistics and request a pie chart version
• highlight a recipe and ask for all the ingredients doubled

In the example below, a “paused frame in a travel video becomes a booking link for that cool-looking restaurant.”

Google has two AI-enabled pointer demos in AI Studio:

• Edit an image
• Find places on the map

Additionally, you will soon have the ability to “use your pointer to ask Gemini in Chrome about the part of the webpage you care about.” This is in the process of rolling out.  

For example, you can select a few products on a page and ask to compare, or point to where you want to visualize a new couch in your living room.