What the agent can do¶
What the agent can do¶
goto, click, type, select (native dropdowns), press, scroll, back, wait, read (text on screen), find (jump to a word on a long page), switch_tab, close_tab, remember, ask_human and done.
- Dropdowns: a native
<select>is listed with its current value and options, andselectpicks one by its text or value. - Frames and web components: controls inside iframes (same-origin and cross-origin) and open shadow roots appear in the same numbered list and are clicked the same way.
- Tabs: when a link or popup opens a new tab the agent follows it, and the page list starts with
[tabs: 1 "Shop" ยท 2 "Details" (current)]so the model knows. Not supported yet: closed shadow roots, drag and drop, file uploads, custom (non-<select>) dropdowns beyond clicking them. See ROADMAP.md.
Each tool schema is sent on every model call, so tools are kept few and short (about 790 tokens for all of them on a typical step).
How learning works¶
Knowledge is stored per site in ~/.selfbrowse/sites/<domain>.json.
- Memo: short facts the model chose to save with the
remembertool. - Flows: click sequences that worked, keyed by page type (for example
/*/dp/*). A flow is replayed without a model call only when it is trusted: - generic: it succeeded for at least two different tasks (a cookie banner, a sign-in step), so it is replayed for every task;
- repeat: it succeeded at least twice for the exact same task, so it is replayed for that task only.
- Only successful runs are learned from. Steps after typing, or after a click that changes the page in place (such as "Add to cart"), are never recorded for that page; only clicks that open another page or dismiss a banner or dialog are replayed. Risky clicks (buy, pay, delete, send, sign out and similar) are never recorded. Failed actions and detours (clicks followed by going back) are not learned. A flow that stops working is dropped, and flows unused for 90 days expire.
When learning is used, the command line says so on that step: (learned: replayed from saved flow, no model call) for a replayed click, (learned: used saved notes about this site) when the model chose the step with saved notes in front of it. In Python, check step.from_cache and step.used_notes.
Each page type keeps up to eight distinct click sequences (variants), and trust is worked out for the start of a sequence. If the cookie-banner click came first in two different tasks, it is replayed for any task even though what followed differed. A different path on a later run is kept as another variant instead of erasing a trusted flow. Files written by earlier versions still load.
Replaying goes by page type and element label: the agent finds the stored control again by role and text on a fresh page list (an exact text match, or a single partial match). If it is missing or ambiguous the flow records a miss and the model takes over, and every replayed click still passes the guardrails.
Measured (local model qwen3.8, a local demo shop, one run per cell, success judged by the shop's own state; benchmarks/compare/learning_eval.py):
| Scenario | Learning off | Learning on |
|---|---|---|
| The same multi-page lookup, 4 runs in a row | 2/4 correct, 6.8 calls, 10,942 tokens on average | 4/4 correct, 5.0 calls, 7,645 tokens on average |
| Once trusted (runs 3 and 4) | 7 to 9 calls, 55 to 79 s | 3 calls, about 4,000 tokens, 10 to 12 s |
| Different tasks on one site (add-to-cart, different SKUs) | no change | no change: no shared start was found |
Learning helps routine jobs you repeat, from the third run, and did not help across different tasks in these tests. Details and the earlier results that led to the in-place-click rule are in benchmarks/compare/results/learning-qwen3.8*.
Known limit: research-style tasks learn little, because most of their clicks are specific to the question. The biggest wins are generic steps and routine jobs you repeat.