In the demo above, Claude installs Python and Git on a fresh Windows machine. For Python, it opens the Windows Store, searches, picks Python 3.13 over 3.12, clicks Get, and waits for installation to complete. For Git, it navigates to the official website in Edge, downloads the installer, asks for a UAC confirmation, runs the installer silently, then launches Git Bash and verifies the installation works, falling back to vision to read the terminal output where UIA has no coverage.

It showcases how an AI agent can move across desktop apps, browsers, and terminals to handle complex tasks.

Why Windows Desktop Automation Is Hard​

The Windows desktop is genuinely difficult to automate programmatically. Unlike the web (where the DOM is structured and designed to be read by code), the Windows desktop is a patchwork of UI frameworks built over decades: Win32, MFC, WPF, UWP, WinUI 3, embedded web views, custom renderers. Each one exposes its internals differently. Dialogs pop up unexpectedly. Apps behave differently across OS versions, display scaling, and language packs. There is no single standard to rely on.

Vision-based computer use is a compelling answer to this complexity. Rather than trying to understand every framework, you just look at the screen. For tasks that require genuine visual reasoning it remains the right tool.

But for automation at any scale, it carries real costs. Each step is a round-trip to an inference API. Pixel coordinates shift when the window moves or the display resolution changes. And a sequence of screenshots is not an auditable record: when something goes wrong, there is no structured trace to diagnose.

UI Automata takes a complementary approach: use the semantic layer that is already there.

Automata Workflow: Shell Scripting for Windows GUI​

Think of a workflow YAML file as the shell script for the Windows desktop. Where a shell script says "run this command, check the exit code, pipe the output to the next step", a workflow YAML says "click this button, wait for this dialog, extract this value."

- intent: click the Save menu item
  action:
    type: Click
    scope: notepad
    selector: ">> [role=menu item][name=Save]"
  expect:
    type: DialogPresent
    scope: notepad
  timeout: 10s

Every step has three parts: an action (what to do), an expect (what state the UI should be in afterwards), and optionally a recovery (what to do if something unexpected happens). The engine executes the action, then watches the UI until the expected state appears. If it does not appear in time, it checks recovery handlers before failing cleanly.

No sleeps. No guessing. No silent failures.

Selectors: CSS for Windows UI​

If the workflow YAML is the shell script, selectors are the file paths. They are how you point to the exact UI element you want to act on.

The Windows UI element tree is messy: a typical app window has hundreds of nested elements, many with no name or identical-looking labels. A single property is rarely enough to pin down the right one. We built a CSS-like selector language that lets you combine every available signal into a precise address:

>> [role=button][name=Save]                    # role AND name
>> [id=TabListView] > [role=tab item]:first    # first tab in a specific list
>> [role=button][name=Settings]:parent         # the container holding this button
>> [role=dialog][name='Confirm Save As'] >> [role=button][name=Yes]  # button inside a specific dialog

Crucially, selectors target semantic properties (what an element is) not pixel coordinates (where it happens to be drawn). They survive window resizes, display scaling changes, theme changes, and most app updates. A selector that worked last week works today.

The Shadow DOM: React in Reverse​

React builds a virtual DOM to avoid expensive real DOM queries. UI Automata does the same for Windows UIA, but in reverse: instead of pushing changes down, it caches what it reads.

Querying Windows UI Automation naively is slow: every element lookup is a cross-process round-trip, like making a network request just to read a variable. A 20-step workflow that re-queries from scratch on every step spends most of its time waiting, not working.

UI Automata solves this with what we call the shadow DOM: a cached map of the live UI. The engine resolves each element once and holds onto it. Subsequent steps that reference the same element are nearly instant.

The shadow DOM also solves identity. On first access, the engine locks the OS-level window handle (HWND). Subsequent resolutions bypass the selector entirely and go directly to that handle, so title changes, focus shifts, and other windows opening mid-flow cannot cause drift. Staleness is detected on every element access via a liveness check, not on a timer. When a cached element goes stale, the engine tries to re-resolve from its cached parent before falling back to a full tree traversal, so a single button refresh is a narrow subtree scan, not a full window walk.

Cross-Framework Coverage​

Windows applications are not a single thing. A modern machine might run a Win32 app from 1998 alongside a WinUI 3 app from last year, a legacy MFC tool, and a web browser, all at once.

UI Automata covers all of them: Win32, MFC, WPF, UWP, WinUI 3, and web browsers via the Chrome DevTools Protocol, giving full access to page structure alongside native UI.

For applications with little UIA support, the vision MCP tool provides OCR and layout recognition; vision and structured automation complement each other in the same agent loop.

Built for AI Agents​

UI Automata is not a scripting framework repurposed for agents. It is designed from the ground up as the interface between an agent and the Windows desktop.

Every step carries an intent field: a plain-English description of what it is trying to do. The engine logs every step with its outcome, giving the agent a full structured trace to read and reason about. When something goes wrong, the trace tells the agent exactly what state the UI was in at the moment of failure.

The MCP interface gives an agent everything it needs:

• desktop: inspect live element trees and test selectors against any window
• app: list installed applications and launch them
• browser: CDP page inspection and navigation
• workflow: run a workflow file, receive structured output
• vision: OCR and layout for windows that lack UIA support
• Schema and linter: validate workflow files before execution, so agents catch errors without running

An agent can explore an unfamiliar UI, author steps interactively, run them immediately, and promote working steps into a reusable workflow file. The entire loop (explore, script, run, verify) happens in a conversation. No human needed to laboriously demonstrate every step.

Industrial-Grade Applications​

Professional desktop software (CAD tools, ERP systems, simulation suites) is where most automation approaches hit a wall. These apps have deeply nested UI structures, lists where off-screen items are invisible to standard queries, and toolbars where dozens of buttons look identical to anything but their internal ID.

UI Automata handles these because it was designed with them in mind. The Invoke action activates elements directly through the accessibility interface, bypassing the need for a visible bounding box entirely. The :parent and :ancestor navigators let you locate a container by identifying a landmark element inside it: the pattern you need when the row you want to click has no unique label of its own.

What We Are Releasing Today​

We are releasing the initial build of UI Automata today, including:

• The workflow engine and YAML format
• The MCP server (automata-agent) for Claude Code and Claude Desktop
• ui-inspector and other CLI tools for interactive UI exploration
• The workflow library: reusable workflows for common Windows applications
• Comprehensive documentation at automata.visioncortex.org
• Source code at visioncortex/ui-automata

Try it, break it, and tell us what you run into. We are building the workflow library based on what people actually need. Open an issue or start a discussion on GitHub.

Prerequisites​

• Windows 10 or Windows 11 (64-bit)
• Claude Code or Claude Desktop (or any MCP-capable client)
• PowerShell

Step 1 — Install​

One PowerShell command installs everything: the workflow engine, the MCP server, and the workflow library.

PowerShell -ExecutionPolicy Bypass -Command "iwr https://raw.githubusercontent.com/visioncortex/ui-automata/refs/heads/main/install/install-windows.ps1 | iex"

The installer places binaries in C:\Users\<you>\.ui-automata\ and adds that directory to your PATH. Open a new PowerShell window and run the self-test to confirm everything is in place:

automata-agent --self-test

Then run the bundled Notepad demo to confirm the workflow engine works end-to-end:

Windows 11​

ui-workflow $env:USERPROFILE\.ui-automata\workflows\win11\notepad\notepad_demo.yml

Windows 10​

ui-workflow $env:USERPROFILE\.ui-automata\workflows\win10\notepad\notepad_demo.yml

You should see Notepad open, some text typed, and the workflow complete successfully.

Step 2 — Connect MCP​

Claude Code​

Launch the MCP server from PowerShell (keep this window open while you work):

automata-agent

It scans ports 3001–4000 for a free one and prints a ready-to-paste config block:

automata-agent started
host : 127.0.0.1
port : 3001

MCP config (paste into .mcp.json, keep this window open):
{
  "mcpServers": {
    "ui-automata": {
      "type": "http",
      "url": "http://127.0.0.1:3001/mcp"
    }
  }
}

Copy the config block from your terminal (the port may differ) and paste it into your .mcp.json. Then run /mcp in Claude Code and click Reconnect next to the ui-automata server. If the tools still don't appear, run Developer: Reload Window from the VS Code command palette (Ctrl+Shift+P) and reconnect again.

Claude Desktop​

Claude Desktop uses stdio instead of HTTP. Add the following to your claude_desktop_config.json, replacing <you> with your Windows username:

{
  "mcpServers": {
    "ui-automata": {
      "command": "C:\\Users\\<you>\\.ui-automata\\automata-agent.exe",
      "args": ["--stdio"]
    }
  }
}

Claude Desktop launches automata-agent automatically. After editing the config, fully quit Claude (right-click the tray icon and choose Quit — closing the window is not enough) and relaunch it. You should see ui-automata listed under Settings → Connectors.

Verify the connection​

Ask Claude: "list the ui-automata tools available to you". It should respond with the full tool list.

Step 3 — Explore the UI​

Before writing a workflow, you need to know the element tree of the application you want to automate. UI Automata ships ui-inspector for this.

Run it from PowerShell:

ui-inspector

Move the mouse over any element in any window. The inspector highlights the element under the cursor and prints its full ancestor chain to the terminal:

[desktop] "Desktop 1" class=#32768 id=
   └─ [window] "Untitled - Notepad" class=Notepad id=
      └─ [pane] "" class=NotepadTextBox id=
         └─ [document] "Text editor" class=RichEditD2DPT id= value= enabled=true rect=(0,0,400,200)

Every line shows the element's role, name, class, AutomationId, value, and bounding rect — exactly what you need to write a selector. Press Ctrl-C to exit.

To find window handles and process names, use list-windows:

list-windows

To test a selector before committing it to a workflow, ask Claude to run desktop find_elements against the window. It will return every match with its role, name, bounds, and full ancestor chain.

You can also dump the full UIA element tree for a window by its HWND.

element-tree 0x1234ab

Step 4 — Your First Workflow​

With the element tree in hand, writing the workflow is straightforward. Here is a minimal example that opens Notepad, types a line, and saves the file:

Windows 11​

# yaml-language-server: $schema=https://raw.githubusercontent.com/visioncortex/ui-automata/main/workflow-schema.json
name: notepad_hello
description: Open Notepad, type a message, and save the file.

defaults:
  timeout: 5s

launch:
  exe: notepad.exe
  wait: new_window

anchors:
  notepad:
    type: Root
    process: Notepad
    selector: "[name~=Notepad]"

  editor:
    type: Stable
    parent: notepad
    selector: ">> [role=document][name='Text editor']"

  saveas_dialog:
    type: Ephemeral
    parent: notepad
    selector: "> [role=dialog][name^=Save]"

phases:

  - name: type_text
    mount: [notepad, editor]  # mounted before steps run
    steps:
      - intent: type text into editor
        action:
          type: TypeText
          scope: editor
          selector: "*"
          text: "Hello Automata"
        expect:
          type: ElementHasText
          scope: editor
          selector: "*"
          pattern:
            contains: "Hello Automata"

  - name: save_file
    mount: [saveas_dialog]
    unmount: [saveas_dialog]
    steps:
      - intent: activate keyboard shortcut for Save As
        action:
          type: PressKey
          scope: notepad
          selector: "*"
          key: "ctrl+shift+s"
        expect:
          type: DialogPresent
          scope: notepad

      - intent: type filename in Save As dialog
        action:
          type: SetValue
          scope: saveas_dialog
          selector: ">> [role=edit][name='File name:']"
          value: "hello-world"
        expect:
          type: ElementHasText
          scope: saveas_dialog
          selector: ">> [role=edit][name='File name:']"
          pattern:
            contains: "hello-world"

      - intent: click Save button
        action:
          type: Invoke
          scope: saveas_dialog
          selector: ">> [role=button][name=Save]"
        expect:
          type: DialogAbsent
          scope: notepad

Windows 10​

# yaml-language-server: $schema=https://raw.githubusercontent.com/visioncortex/ui-automata/main/workflow-schema.json
name: notepad_hello
description: Open Notepad, type a message, and save the file.

defaults:
  timeout: 5s

launch:
  exe: notepad.exe
  wait: new_pid

anchors:
  notepad:
    type: Root
    selector: "[name~=Notepad]"

  editor:
    type: Stable
    parent: notepad
    selector: ">> [role=edit][name='Text Editor']"

  saveas_dialog:
    type: Ephemeral
    parent: notepad
    selector: ">> [role=dialog][name='Save As']"

phases:

  - name: type_text
    mount: [notepad, editor]
    steps:
      - intent: type text into editor
        action:
          type: TypeText
          scope: editor
          selector: "*"
          text: "Hello Automata"
        expect:
          type: ElementHasText
          scope: editor
          selector: "*"
          pattern:
            contains: "Hello Automata"

  - name: save_file
    mount: [saveas_dialog]
    unmount: [saveas_dialog]
    steps:
      - intent: activate keyboard shortcut for Save As
        action:
          type: PressKey
          scope: notepad
          selector: "*"
          key: "ctrl+shift+s"
        expect:
          type: DialogPresent
          scope: notepad

      - intent: type filename in Save As dialog
        action:
          type: SetValue
          scope: saveas_dialog
          selector: ">> [role=combo box][name='File name:'] > [role=edit]"
          value: "hello-world"
        expect:
          type: ElementHasText
          scope: saveas_dialog
          selector: ">> [role=combo box][name='File name:'] > [role=edit]"
          pattern:
            contains: "hello-world"

      - intent: click Save button
        action:
          type: Invoke
          scope: saveas_dialog
          selector: ">> [role=button][name=Save]"
        expect:
          type: DialogAbsent
          scope: notepad

Save this as notepad_hello.yml and run it:

ui-workflow notepad_hello.yml

Reading the logs​

After each run, a detailed structured log is saved to:

$env:USERPROFILE\.ui-automata\logs\notepad_hello\20260403T152954.log

[2026-04-03T14:29:54.074Z] [INFO] log → C:\Users\chris\.ui-automata\logs\notepad_hello\20260403T152954.log
[2026-04-03T14:29:54.568Z] [INFO] launched 'notepad.exe' pid=20828
[2026-04-03T14:29:54.568Z] [INFO] launch: waiting for notepad.exe window (strategy=NewWindow, timeout=5s)
[2026-04-03T14:29:55.450Z] [INFO] phase: type_text
[2026-04-03T14:29:55.904Z] [DEBUG] resolved anchor 'notepad':

[2026-04-03T14:29:59.756Z] [DEBUG] action → Ok
[2026-04-03T14:29:59.797Z] [DEBUG] poll: ElementHasText(saveas_dialog:[role=edit][name=File name:]) → true
[2026-04-03T14:29:59.797Z] [INFO] step 2/3: ok
[2026-04-03T14:29:59.797Z] [INFO] step 3/3 'click Save button'
[2026-04-03T14:29:59.797Z] [DEBUG] action: Click(saveas_dialog:[role=button][name=Save])
[2026-04-03T14:30:00.917Z] [DEBUG] action → Ok
[2026-04-03T14:30:01.459Z] [DEBUG] poll: DialogAbsent(notepad) → true
[2026-04-03T14:30:01.459Z] [INFO] step 3/3: ok
[2026-04-03T14:30:01.460Z] [DEBUG] unmounted node 3 for anchor 'saveas_dialog'
[2026-04-03T14:30:01.460Z] [DEBUG] cleanup depth 0: removed node 1 for key 'notepad'
[2026-04-03T14:30:01.460Z] [DEBUG] cleanup depth 0: removed node 2 for key 'editor'
[2026-04-03T14:30:01.460Z] [INFO] outputs: {}
[2026-04-03T14:30:01.460Z] [INFO] completed successfully

Open it to inspect exactly what happened on every step — actions taken, conditions evaluated, recovery handlers triggered, and the final UI state at the point of failure.

Before Starting a Task with Claude​

Ask Claude to read the library context before starting any automation task:

Read the CLAUDE.md and AGENT.md from the ui-automata resource list before we start.

Where to Go Next​

• Core Concepts — how the engine works, anchors, selectors, conditions, and recovery
• Writing Workflows — hands-on guides for common authoring tasks
• Workflow Library — ready-to-use workflows for Notepad, Explorer, Windows Settings, Word, and more