VNC pipeline
The previous sandbox shipped a full noVNC plus Fluxbox desktop the user could watch over. The Ultron shell does not. This page describes what replaced VNC: a screenshot binary plus Playwright-driven live previews plus an HTTP-streamed deploy preview for built artefacts.
Overview
- Old surface
- Xvfb + Fluxbox + noVNC. One persistent desktop per user.
- New surface
- Headless. Visual outputs via screenshot binary or deploy preview.
- Visual debug
ultron-screenshotbinary in /tools-bin- Browser automation
- Playwright + Chromium installed in the image
- Live preview
- HTTP-served from the container's working directory; surfaced via the dashboard preview pane
Keeping noVNC up reliably across container restarts was the heaviest single source of operational pain in the previous sandbox. Sessions migrated to a different node mid-render, noVNC sockets dropped, the desktop ended up in a half-rendered state. The shell trades that surface for a cleaner set of single-purpose tools that cover each VNC use case explicitly.
What changed from the old sandbox
| Use case | Old (VNC) | New (shell) |
|---|---|---|
| Watch the agent click a button | Live noVNC iframe | ultron-screenshot at key moments, displayed inline in chat |
| Drive a real browser | Manual click in the noVNC session | Playwright script written by the agent + ultron-screenshot for verification |
| See a built UI | Open a localhost URL inside the desktop | ultron-deploy to a hosted preview URL; the dashboard preview pane points at it |
| Debug a Python script | Run + observe via the desktop | Run in container, capture stdout/stderr in the chat |
| Fill out a form | Manual click in the desktop | Playwright + ultron-screenshot to verify |
Screenshot path
The most-used visual primitive in the new world.
The binary uses the same Playwright Chromium that the rest of the shell uses. Output PNGs are written to /work; the chat agent can attach them to its reply, and they survive session snapshots like any other file in /work.
Playwright sessions
For agent flows that need an interactive browser session (form submission, OAuth handshake, drag-and-drop testing), the agent CLI spawns a Playwright session directly. The session is headless by default; the agent uses ultron-screenshot at each meaningful step to produce a frame the operator can review.
Live preview surfaces
When the agent builds a web artefact (a landing page, a small app, a static export of a deck), the right pattern is to call ultron-deploy and surface a hosted preview URL. The dashboard's preview pane points at that URL. The artefact is reachable from the operator's own browser, not from inside a desktop session, which avoids every flavour of VNC failure mode.