Microsoft releases preview version of 'Copilot Vision' that uses AI to analyze text and images on web pages and provide answers

On December 5, 2024, Microsoft announced that it has released a preview version of Copilot Vision, an AI feature that can analyze web pages being viewed in the Edge web browser and handle tasks such as answering user questions, summarizing and translating text, and highlighting discounted products, to some users.

Copilot Vision now in preview – a new way to browse | Microsoft Copilot Blog
https://www.microsoft.com/en-us/microsoft-copilot/blog/2024/12/05/copilot-vision-now-in-preview-a-new-way-to-browse/?_thumbnail_id=3834

Maybe you shouldn't play favorites with features, but Copilot Vision is an update I've been excited about since day one. It's the first AI experience of its kind—now you can show, not tell, AI what you want help with in real time as you scroll, shop, guess locations...???? pic.twitter.com/DJNmgMuw7D

— Mustafa Suleyman (@mustafasuleyman) December 5, 2024

Copilot Vision, Microsoft's AI tool that can read your screen, launches in preview | TechCrunch
https://techcrunch.com/2024/12/05/copilot-vision-microsofts-ai-tool-that-can-read-your-screen-launches-in-preview/

With Copilot Vision turned on, it can automatically analyze the web page being viewed in Edge and provide appropriate support to the user. In the video below, a woman plans to visit a museum and asks Copilot Vision to introduce her to the museum.

Co-planning with Copilot Vision - YouTube


Copilot Vision can also interact with users to assist them in making purchases.

Co-shopping with Copilot Vision - YouTube


You can also help play the game ' Geoguessr ,' in which you identify a location from a Google Street View image. In the video below, the app provides audio advice to the player, such as 'There is Chinese written in large letters.'

Co-gaming with Copilot Vision - YouTube


According to Microsoft, the most important thing in building Copilot Vision was the security and privacy of users. In fact, Copilot Vision can be turned on or off at the discretion of the user, and when the session with Copilot Vision ends, all data related to what was said and the context is deleted. In addition, the data of users who use Copilot Vision is handled in accordance with the Microsoft Privacy Statement , and only the responses of Copilot Vision are recorded for the purpose of improving the system.

'To provide Copilot Vision, we've partnered with some third-party publishers to help us analyze how users use Copilot Vision to make decisions on their websites,' Microsoft said.

At the time of writing, Copilot Vision is only available to some Copilot Pro users in the United States, and the websites where Copilot Vision is available are limited to a certain few sites. Microsoft said, 'We will carefully expand the websites where Copilot Vision is available to ensure that Copilot Vision does not acquire, store, or use copyrighted content. We will collect feedback over time, iterate and adjust the technology, and gradually expand access to more Copilot Pro users and more websites.'