Apr 17, 2025 13:56:00

Microsoft adds 'computer use' to AI 'Copilot Studio' that can automatically operate PCs, allowing any application running on Windows to be automatically operated

Microsoft has announced a feature called ' computer use ' that allows AI agents to click buttons, select menus, and type text into input fields on websites and apps.

Announcing new computer use in Microsoft Copilot Studio for UI automation | Microsoft Copilot Blog

https://www.microsoft.com/en-us/microsoft-copilot/blog/copilot-studio/announcing-computer-use-microsoft-copilot-studio-ui-automation/

Computer use, which has been added as an Access Research Preview version to

Copilot Studio , a tool for creating custom AI agents, gives AI agents the power to operate GUIs, allowing them to access various buttons and fields on desktop applications.

Copilot Studio's computer use allows you to build AI agents that can recognize and interact with GUIs in both desktop and browser apps. According to Microsoft, 'computer-use allows AI agents to interact with any system that has a GUI,' which allows AI agents to handle tasks even when there is no API to directly connect to the system, allowing AI agents to click buttons, select menus, and enter input into on-screen fields.

In addition, it uses 'inference' to automatically adapt to changes in apps and websites, allowing for flexible responses that simply automated processes could not provide, without interrupting work.

Because computer-use runs on Microsoft-hosted infrastructure, companies don't need to manage their own servers, and while corporate data is stored on Microsoft servers, it is not used to train future AI models. Microsoft touts the potential for faster deployment, less maintenance, and lower infrastructure costs.

It can be used in situations where AI agents can automate processing, such as inputting large amounts of data into a centralized system, automatically collecting and analyzing market data from multiple sources, or extracting data from invoices and inputting it into an accounting system.

No coding is required to set up computer use, you can provide instructions in natural language, test and refine prompts with video, and freely view the activity history, including screenshots and reasoning steps captured by the AI.

Computer use will also be explained in detail at the developer event 'Microsoft Build' to be held in May 2025.