XenonStack

Agentic AI Systems

XenonStack Blog GUI Agents: Computer Use for Enterprise Automation

and | Published | Updated

GUI Agents: Computer Use for Enterprise Automation
12:06

Key takeaway

  • GUI agents operate software by screen, mouse and keyboard. See how they work, where they fit, the risks and controls, and the 3 to 5 year outlook.

GUI agents are AI agents that operate software the way a person does: they read the screen, move a pointer, click, type and scroll. That makes them useful where an application has no usable API, such as legacy ERP screens, vendor portals and desktop tools. They sit one layer below agentic workflows: the workflow decides what should happen, and a GUI agent is one of the tools that carries out a step.

The 2026 picture is more cautious than the 2024 hype. Benchmarks show agents are much better than they were, yet the vendors that sell them still advise isolated environments and human confirmation for consequential actions. This guide covers how GUI agents work, where they fit, what the sourced evidence says, how to control them, and what to expect over the next 3 to 5 years.

What is a GUI agent, and how is it different from an API integration or an RPA bot?

A GUI agent perceives a graphical interface from screenshots, decides the next action with a multimodal model, and acts through a virtual mouse and keyboard. The survey GUI Agents with Foundation Models: A Comprehensive Survey (Wang et al., arXiv, submitted 7 November 2024, revised 13 February 2025) describes this as agents built on large language models and multimodal large language models that carry out tasks by simulating actions such as clicking and typing.

ApproachHow it worksWhen it fitsCurrent status
API or MCP integrationCalls a documented interface directlyThe system exposes an API; the task is high volume and stablePreferred where available, as it is faster and easier to audit
Scripted RPA botFollows recorded steps and element selectorsThe screens rarely change and the rules are fixedMature, but brittle when layouts change; see agentic process automation
GUI agentReads the screen and chooses actions with a multimodal modelNo API exists, screens vary, or the task needs judgementImproving quickly; vendors still advise isolation and human confirmation
Agentic workflow with GUI agent as a toolA workflow plans the task and calls API, MCP or GUI tools as neededMixed estates of modern and legacy systemsThe pattern we recommend; see agentic workflows

In our own solution work, GUI agents have been demonstrated for legacy integration with systems that have no API, while API and MCP integrations are used wherever they exist. That ordering is the practical rule: use the API when there is one, and use the screen when there is not.

Video: how agentic process automation moves beyond scripted bots, from the XenonStack channel.

How do GUI agents work?

Most current GUI agents follow the same loop, and the vendor documents describe it consistently.

Perceive, plan, act, check

The agent takes a screenshot, a multimodal model interprets what is on screen, the model proposes an action such as a click or keystroke, and the result is checked on the next screenshot. According to its documentation, the Anthropic computer use documentation describes a tool that provides screenshot, mouse and keyboard control, where the application runs each requested action in an environment it controls and returns the result. The OpenAI Computer-Using Agent post (23 January 2025) describes its Computer-Using Agent as combining GPT-4o vision with reinforcement-learning-based reasoning, working from screenshots, a virtual mouse and a keyboard without specialised APIs.

Grounding is the hard part

The OSWorld paper (Xie et al., arXiv, April 2024) names GUI grounding and operational knowledge as the main weaknesses of the agents it tested. Grounding means turning an intention such as "open the invoice tab" into the right place on the screen. The Anthropic documentation adds that the model might make mistakes or hallucinate when outputting specific coordinates, so clicks can miss their target.

Where the older building blocks sit

Earlier GUI agent designs were described as a mix of object detection for interface elements, language models for command interpretation and reinforcement learning for task optimisation. Those pieces still exist, but the current vendor tools put a single multimodal model at the centre of the loop. If you are comparing vendors, ask where perception, planning and action are handled, and which parts you can inspect.

What has changed in GUI agents since 2024?

DateWhat happenedSource
April 2024OSWorld introduced 369 real computer tasks; humans completed over 72.36%, the best model 12.24%OSWorld paper (arXiv)
22 October 2024Anthropic released computer use in public beta; Claude 3.5 Sonnet scored 14.9% screenshot-only on OSWorld, 22.0% with more steps allowed; described as at times cumbersome and error-proneAnthropic announcement
23 January 2025OpenAI reported its Computer-Using Agent at 38.1% on OSWorld (human 72.4%), 58.1% on WebArena (human 78.2%) and 87% on WebVoyagerOpenAI post
28 July 2025OSWorld was upgraded to OSWorld-VerifiedOSWorld site
26 June 2026The OSWorld site states that OSWorld 2.0 is available as the latest versionOSWorld site

Read the table as direction, not as a league table. The benchmarks, task sets and scoring rules differ between rows, and the OSWorld site that we read did not display current leaderboard scores, so we do not quote any. For current numbers, check the live OSWorld site before you rely on a figure.

Where do GUI agents fit in the enterprise?

The strongest case is work that people already do by hand across screens that have no API. The table groups the use cases from our earlier guide by what the agent does and how much care each needs. The "fit" column is our assessment, not a measured result.

Use caseWhat the agent doesFitControl to apply
Software testingImitates user steps through an application and checks resultsGood: failures are cheap and visibleRun in a test environment only
Workflow automation in ERP and BI toolsMoves data between applications, enters records, compiles reportsGood where no API existsLog every action; sample outputs
Customer support toolingWorks inside help-desk software to look up and update ticketsMediumHuman approval for customer-facing changes
Healthcare administrationUpdates records, schedules appointments, cross-checks claim dataMedium: sensitive dataLeast privilege, no access to clinical systems without review
Banking and financial servicesHandles routine queries and application data entryCautiousApproval before any payment or account change
HR onboardingExtracts details from forms and enters them in the HR systemGood for low-risk data entryCheck personal data handling
Data migrationMoves records through old interfaces when no export existsMediumReconcile counts and samples after each run
E-commerce and retail operationsUpdates order and stock screensMediumCap actions per run; watch for layout changes
Education administrationHandles timetable and records screensMediumHuman review of grading changes

When a GUI agent hits a case it cannot settle, such as a policy breach or a conflicting rule, route it to a person. The Exception Handling Agent on ElixirData describes this on its site as spotting workflow cases that need human judgement, gathering the background, finding the right approver and logging the resolution in a decision trace.

Key Statistics on GUI Agents

socks

 Market Potential

The global RPA (Robotic Process Automation) market, closely tied to GUI agent growth, was valued at $1.89 billion in 2021 and is expected to reach $13.74 billion by 2028

testing

Software Testing Impact

Automated GUI agents reduce testing time by up to 70% and cut software development costs by 30-40%, making them valuable in agile environments

workplace

Workplace Automation

By 2025, it’s estimated that 60% of large organizations will deploy GUI agents to automate workflows across departments like HR, customer support, finance (Gartner)

productivity

Productivity Boost

Studies show that GUI agents improve task completion speed by up to 50% for repetitive data entry and processing tasks, saving significant labour hours across industries 

The cards above are indicative figures carried over from our earlier edition. The dated, source-linked benchmark figures are in the table in the section on what has changed since 2024.

What are the security and governance risks of GUI agents?

An agent that can click anywhere a user can click can also do harm anywhere a user can. The OWASP prompt injection guidance defines prompt injection as inputs that alter a model's behaviour in unintended ways, notes that injected content can be hidden from humans, and says there may be no foolproof prevention, so the aim is to limit impact. A web page or document on screen is untrusted input to the agent.

Two-column diagram matching on-screen instructions, sensitive data access, malicious sites and irreversible actions to an isolated environment, least privilege, a domain allowlist and human confirmation.
RiskControlSource
Agent follows instructions found in on-screen contentIsolate the environment; classifiers help but do not replace precautionsAnthropic documentation says Claude may follow commands found in content even when they conflict with the operator instructions
Agent reaches sensitive data or systemsDedicated virtual machine or container with minimal privileges; no login credentials in reachAnthropic documentation
Agent reaches malicious sitesLimit internet access to an allowlist of domainsAnthropic documentation
Irreversible action taken in errorHuman confirmation for consequential actions, such as financial transactions or agreeing to termsAnthropic documentation; OpenAI describes confirmation before actions with external effects, and a watch mode on sensitive sites
Excess privilegeLeast privilege and human approval for high-risk actionsOWASP guidance
No record of what the agent didDecision traces and audit logs for each actionVendor site claims, see below

For the policy layer, AgentGRC on ElixirData says on its site that it maps company policies and approval rules onto agent actions so they are checked before an agent reaches a system of record, and that organisations typically start in a shadow mode where agents run alongside people. ElixirClaw says on its site that it provides policies, approval limits and audit trails for agent actions, with four autonomy levels (L1 to L4) that can be set per agent, domain or action. Both are vendor statements; validate them in a pilot. For the privacy side, see data privacy with agentic AI.

What limits GUI agents today?

LimitWhat the sources sayWhat to do about it
SpeedThe Anthropic documentation says computer use might be too slow compared with human-directed actions and suggests tasks where speed is not criticalSchedule batch work; avoid latency-sensitive steps
Accuracy of clicks and editsAnthropic notes possible coordinate errors; OpenAI says text editing is imprecise and error-proneVerify each result on screen; keep steps short
Unfamiliar or varied interfacesOpenAI says the agent struggles with unfamiliar interfaces; Anthropic says reliability might be lower in niche applications or across several applications at onceStart with one stable application; retest after each interface release
Precision and sensitive tasksAnthropic advises against tasks needing perfect precision or sensitive information without human oversight; OpenAI says its agent declines tasks such as banking transactionsKeep a person in the loop for these
Gap to humansOpenAI reports its agent below human performance on OSWorld and WebArenaTreat success rates as something to measure on your own tasks

How should an enterprise adopt GUI agents?

  1. List the systems that have no API. Use API or MCP integration wherever it exists, and keep the GUI agent for the rest.

  2. Pick one low-risk, high-repetition task. Software testing and read-only data lookups are safer first steps than payments.

  3. Run in an isolated environment. Use a virtual machine or container, an allowlist and no stored credentials.

  4. Start in shadow mode. Let the agent record intended actions next to a person, then compare.

  5. Add approvals and logs. Require confirmation for consequential steps and keep a trace of every action.

  6. Measure on your own tasks. Track completion rate, rework and time per task against the manual baseline before extending autonomy.

When not to use one: where a stable API exists, where a mistake cannot be reversed and no human checks it, or where the task needs perfect precision. If the process itself is the problem, fix the process first.

What will change in GUI agents over the next 3 to 5 years?

SignalWhat the source saysSource and date read
Benchmarks keep movingOSWorld went from a best model of 12.24% (2024 paper) to a 2.0 release announced on its site (26 June 2026), so scoring is being made harder as agents improveOSWorld paper and site, read 12 November 2026
Project failure riskGartner forecasts that over 40% of agentic AI projects will be cancelled by the end of 2027 because of rising costs and unclear business valueReuters report as published by The Star (25 June 2025), read 12 November 2026; Gartner page could not be opened
Controls stay in placeVendor documents currently recommend isolation, allowlists and human confirmation for consequential actionsAnthropic documentation and OpenAI post, read 12 November 2026
Open research problemsThe 2025 survey sets out key challenges and future research directions for GUI agents built on foundation modelsSurvey, arXiv v2, 13 February 2025
Physical and visual interfacesAgents that read screens share techniques with vision systems; see computer vision at the edge for the edge side of that shiftXenonStack guide

Our reading, and not a forecast from any of these sources: expect GUI agents to be absorbed into broader agent platforms as one tool among several, with approvals, traces and autonomy limits set by policy. Expect cost and value to decide which programmes survive, as the Gartner forecast suggests, so measure early. See also the future of agentic AI.

To map one GUI-driven task your agents should perform, and the context and policy it depends on, book a Context Workshop. See what happens in the workshop.

Frequently asked questions

What is a GUI agent?

A GUI agent is an AI agent that completes tasks by operating a graphical interface, reading the screen and acting through a virtual mouse and keyboard, in the way a person would. The 2025 survey describes agents that use large language models and multimodal models to simulate actions such as clicking and typing.

When should we use a GUI agent instead of an API?

Use an API or MCP integration whenever one exists, because it is faster and easier to audit. Use a GUI agent for systems that have no usable API, such as legacy screens, where the alternative is manual work.

How accurate are GUI agents?

They have improved but still trail people. OSWorld reported humans above 72% against a best model of 12.24% in 2024, and OpenAI reported 38.1% for its agent in January 2025 against 72.4% for humans. Measure success on your own tasks.

What are the main security risks of GUI agents?

The main risks are prompt injection from on-screen content, access to sensitive data and irreversible actions. The vendor and OWASP guidance recommends isolated environments, least privilege, domain allowlists and human confirmation for consequential actions.

Are GUI agents the same as RPA?

No. A scripted RPA bot follows recorded steps and selectors, while a GUI agent chooses actions from what it sees. In practice the two can be combined inside an agentic workflow, where the workflow decides which tool to call.

What will change in GUI agents over the next 3 to 5 years?

Benchmarks are being replaced by harder versions, as the OSWorld 2.0 release shows, and Gartner expects over 40% of agentic AI projects to be cancelled by the end of 2027 on cost and value grounds. Expect GUI agents to become one governed tool inside larger agent platforms.

Related reading: data privacy with agentic AI, agentic process automation.

Table of Contents

navdeep-singh-gill

Navdeep Singh Gill

Global CEO and Founder of XenonStack

LinkedIn profile

Navdeep Singh Gill is serving as Chief Executive Officer and Product Architect at XenonStack. He holds expertise in building SaaS Platform for Decentralised Big Data management and Governance, AI Marketplace for Operationalising and Scaling. His incredible experience in AI Technologies and Big Data Engineering thrills him to write about different use cases and its approach to solutions.

Get the latest articles in your inbox

Subscribe Now

Interested in Solving your Challenges with XenonStack Team
Get Started

Get Started with your requirements and primary focus, that will help us to make your solution

Please Select your Industry
Banking
Fintech
Payment Providers
Wealth Management
Discrete Manufacturing
Semiconductor
Machinery Manufacturing / Automation
Appliances / Electrical / Electronics
Elevator Manufacturing
Defense & Space Manufacturing
Computers & Electronics / Industrial Machinery
Motor Vehicle Manufacturing
Food and Beverages
Distillery & Wines
Beverages
Shipping
Logistics
Mobility (EV / Public Transport)
Energy & Utilities
Hospitality
Digital Gaming Platforms
SportsTech with AI
Public Safety - Explosives
Public Safety - Firefighting
Public Safety - Surveillance
Public Safety - Others
Media Platforms
City Operations
Airlines & Aviation
Defense Warfare & Drones
Robotics Engineering
Drones Manufacturing
AI Labs for Colleges
AI MSP / Quantum / AGI Institutes
Retail Apparel and Fashion

Proceed Next