# World of Bits

> Open-domain platform for web-based reinforcement learning agents

- Type: Paper
- By: Shi et al.
- Year: 2017
- Topics: Reinforcement learning, Web agents, Benchmark, Multimodal
- Page: https://aipedia.org/world-of-bits

## Resources

- [World of Bits: An Open-Domain Platform for Web-Based Agents](http://proceedings.mlr.press/v70/shi17a.html): Paper, Shi et al., ICML 2017
- [Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration](https://arxiv.org/abs/1802.08802): Paper, Liu et al., 2018
- [WebGPT: Browser-assisted question-answering with human feedback](https://arxiv.org/abs/2112.09332): Paper, Nakano et al., 2021

**World of Bits (WoB)** is a platform introduced by Shi et al. (2017) where AI agents learn to complete tasks on the web using the same interface humans use: looking at rendered pages and performing low-level keyboard and mouse actions.

> This page assumes basic [Reinforcement Learning](https://aipedia.org/reinforcement-learning) vocabulary (agent, environment, reward). If those terms are new, start there.

## Why Students Should Care

- WoB is the direct ancestor of today's computer-use and browser agents — the idea of "an AI that clicks around the web" starts here.
- It shows how to turn a messy real-world domain (the web) into a well-defined RL environment.
- The benchmark gap it exposed (agents far below human performance on multi-step web tasks) drove years of follow-up work and is still not fully closed.

## Core Idea

WoB treats the web as an open-domain reinforcement learning environment. At each timestep, an agent receives:

- **Pixels** $\mathcal{I} \in \mathbb{R}^{W \times H \times 3}$ — rendered webpage screenshot
- **DOM** $\mathcal{D}$ — Document Object Model (the page's element tree) with element coordinates
- **Reward** $r$ — task completion signal

The agent outputs mouse coordinates $(x, y)$ and keyboard actions to interact with the webpage. Nothing about the interface is simplified for the agent — it sees what you see and acts how you act.

The deeper point: by using the web as an environment, WoB bridges the gap between simulated benchmarks and real-world tasks. Agents learn from the same rich, semantic content that humans create and interact with daily.

## Why the Web?

Three key benefits make the web an ideal learning platform:

| Benefit | Description |
|---------|-------------|
| **Open-domain** | Unlimited websites provide diverse tasks and real-world semantics |
| **Open-source** | HTML/CSS/JS is inspectable and modifiable |
| **Data collection** | Human demonstrations can be crowdsourced easily |

Unlike robotics, web environments are fully digital — enabling fast iteration and massive scaling.

## Interactive Demo

Explore how a WoB agent perceives the DOM and takes actions to complete web tasks:

*Interactive demo: https://aipedia.org/world-of-bits*

## Benchmark Tasks

WoB introduces three task categories of increasing complexity.

### MiniWoB

100 hand-crafted web tasks with small synthetic pages:

- **click-button** — click a specific button
- **enter-text** — type text into an input field
- **use-slider** — adjust a slider to a target value
- **book-flight** — complete a multi-step booking form

These tasks feature clean reward functions and controlled complexity — the "gym exercises" of web interaction.

### FormWoB

Real flight booking websites (United, Alaska, etc.) packaged as reproducible environments:

- Live HTTP traffic is cached via a man-in-the-middle proxy
- Enables offline training while approximating real web dynamics
- Tests generalization to production websites

### QAWoB

Crowdsourced question-answering tasks on live websites. Queries are built from templates with fillable slots:

$$
\text{Query} = \text{Template}(\text{slot}_1, \text{slot}_2, \ldots)
$$

Examples:

- *"What is the population of **Paris**?"* (Wikipedia)
- *"Find flights from **NYC** to **LA** on **Dec 25**"* (Flight sites)

Workers provide demonstrations of how to answer queries using keyboard and mouse.

## Agent Architecture

The baseline WoB agent uses a CNN-LSTM architecture:

```
Observation → CNN(pixels) + MLP(DOM text) → LSTM → Policy(actions)
```

Key design choices:

- **Multimodal input**: combines visual features with semantic DOM information
- **Recurrent memory**: an LSTM tracks state across multiple interaction steps (see [Understanding LSTMs](https://aipedia.org/understanding-lstms))
- **Policy output**: coordinates $(x, y)$ for mouse, one-hot for keyboard

## Training Approaches

| Method | Description |
|--------|-------------|
| **Behavioral Cloning** | Supervised learning on human demonstrations |
| **REINFORCE** | Policy gradient with sparse task rewards |
| **Guided RL** | Warm-start with BC, then fine-tune with RL |

Behavioral cloning alone achieves reasonable performance but struggles to recover from errors it never saw in demonstrations. RL enables adaptation but requires reward shaping for complex tasks. (For the RL side, see [Policy Gradient](https://aipedia.org/policy-gradient).)

## Performance Gap

Even with demonstrations and RL, significant gaps remain:

| Task Type | Agent Success | Human Success |
|-----------|---------------|---------------|
| Simple clicks | ~80% | 100% |
| Multi-step forms | ~40% | 100% |
| Open-domain QA | ~20% | 100% |

This gap motivated years of continued research on web agents.

## Impact & Legacy

WoB pioneered the study of web-based agents and inspired subsequent benchmarks:

- **MiniWoB++** — extended tasks with improved reward signals
- **WebShop** — e-commerce navigation benchmark
- **WebArena** — realistic web task environment
- **Mind2Web** — large-scale web agent dataset

Modern LLM-based agents (GPT-4V, Gemini) are now evaluated on these benchmarks.

## Common Confusion

- **WoB vs. MiniWoB**: World of Bits is the whole platform; MiniWoB is just its easiest task suite (and the part that survived longest, as MiniWoB++).
- **Pixels-and-DOM vs. text-only agents**: WoB agents act through raw screen coordinates and keystrokes. Later LLM web agents often act on the DOM or text directly — an easier but less general interface.
- **This is 2017-era RL, not LLMs**: the baseline agent is a CNN-LSTM trained with behavioral cloning and REINFORCE. Modern browser agents solve these tasks very differently, but on benchmarks descended from this one.

## Where To Go Next

- Read [Reinforcement Learning](https://aipedia.org/reinforcement-learning) for the framework WoB is built on.
- Read [Policy Gradient](https://aipedia.org/policy-gradient) for the REINFORCE algorithm used to train the baseline agent.
- Read [Understanding LSTMs](https://aipedia.org/understanding-lstms) for the recurrent memory in the agent architecture.
- Read [DQN](https://aipedia.org/dqn) for the other landmark "RL from pixels" line of work, in Atari games.