https://github.com/nicobailon/pi-annotate
Visual feedback from browser to AI. Click elements, add comments, fix code.
https://github.com/nicobailon/pi-annotate
ai-tools annotation chrome-extension pi
Last synced: about 2 months ago
JSON representation
Visual feedback from browser to AI. Click elements, add comments, fix code.
- Host: GitHub
- URL: https://github.com/nicobailon/pi-annotate
- Owner: nicobailon
- License: mit
- Created: 2026-01-23T07:58:58.000Z (5 months ago)
- Default Branch: main
- Last Pushed: 2026-04-04T02:13:29.000Z (2 months ago)
- Last Synced: 2026-04-12T09:06:30.569Z (about 2 months ago)
- Topics: ai-tools, annotation, chrome-extension, pi
- Language: JavaScript
- Size: 5.37 MB
- Stars: 114
- Watchers: 1
- Forks: 10
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# Pi Annotate
**Visual annotation for AI. Click elements, capture screenshots, fix code.**
[](LICENSE)
[]()
```
/annotate
```
Figma-like annotation experience with floating inline note cards. DevTools-like element picker in vanilla JS.
Click elements, add comments, submit. The agent gets selectors, box model, accessibility, screenshots — everything it needs to fix your UI.
https://github.com/user-attachments/assets/115b10ca-86e8-4b1c-b8a4-492c68759c58
## Quick Start
### 1. Install Pi Extension
```bash
pi install npm:pi-annotate
```
Restart pi to load the extension.
### 2. Load Supported Browser Extension
1. Open the extensions page in Google Chrome, Google Chrome for Testing, or Chromium, and enable **Developer mode**
2. Click **Load unpacked** → select the `chrome-extension/` folder inside the installed package
3. Click the **Pi Annotate icon** in the toolbar
### 3. Install Native Host
The popup shows your extension ID. Click **Copy** next to the install command, then run it from `chrome-extension/native/` in the installed package:
```bash
./install.sh
```
This installs the native messaging manifest for Google Chrome, Google Chrome for Testing, and Chromium on macOS, plus the default/current config-home locations for those browsers on Linux. Fully quit and reopen that browser. The popup will show **Connected** when ready.
## Usage
```bash
/annotate # Current browser tab
/annotate https://x.com # Opens URL first
```
| Action | How |
|--------|-----|
| Select element | Click on page |
| Cycle ancestors | Alt/⌥+scroll while hovering |
| Multi-select | Toggle "Multi" or Shift+click |
| Add comment | Type in note card textarea |
| Toggle screenshot | 📷 button in note card header |
| Reposition note | Drag by header |
| Scroll to element | Click selector in note card |
| Toggle note | Click numbered badge |
| Expand/collapse all | ▼/▲ buttons in toolbar |
| Toggle edit capture | "Etch" toggle in toolbar |
| Toggle annotation UI | `⌘/Ctrl+Shift+P` |
| Close | `ESC` |
## Features
**Context Capture** — Each element automatically gets box model breakdown (padding, border, margin), accessibility info (role, name, focusable, ARIA states), all HTML attributes, and key CSS styles (display, position, overflow, colors, typography). Enable **Debug mode** for computed styles (40+ properties), parent context, and CSS variables.
**Inline Note Cards** — Draggable floating cards with per-element comments, SVG connectors linking notes to elements, click-to-scroll, and per-element screenshot toggles.
**Screenshots** — Individual crops per element (20px padding) or full-page mode with numbered badges drawn on the screenshot to identify elements. Toggle per element with the 📷 button.
**Edit Capture** — Toggle "Etch" in the toolbar to record DevTools edits. Change inline styles, modify CSS rules, add/remove classes, edit text — everything is tracked via MutationObserver. A pulsing red dot and badge counter show recording status. At submit, the extension takes before/after screenshots by briefly undoing visual changes, and produces structured property-level diffs the agent can map to source code. Works alongside element selection or standalone.
**Restricted Tabs** — If the current tab is `chrome://` or other restricted URLs, providing a URL opens a new tab automatically. Popup button and keyboard shortcut auto-inject the content script on fresh tabs.
## Output
```markdown
## Page Annotation: https://example.com
**Viewport:** 1440×900
**Context:** Fix the styling issues
### Selected Elements (2)
1. **button**
- Selector: `#submit-btn`
- ID: `submit-btn`
- Classes: `btn, btn-primary`
- Text: "Submit"
- **Box Model:** 120×40 (content: 96×24, padding: 8 16, border: 1, margin: 0 8)
- **Attributes:** type="submit", data-testid="submit"
- **Styles:** display: flex, backgroundColor: rgb(59, 130, 246)
- **Accessibility:** role=button, name="Submit", focusable=true, disabled=false
- **Comment:** Make this blue with rounded corners
2. **div**
- Selector: `.error-message`
- Classes: `error-message, hidden`
- Text: "Please fill required fields"
- **Box Model:** 300×20 (content: 300×20, padding: 0, border: 0, margin: 0 0 8)
- **Accessibility:** focusable=false, disabled=false
- **Comment:** This should appear in red, not hidden
### Screenshots
- Element 1: /var/folders/.../pi-annotate-...-el1.png
- Element 2: /var/folders/.../pi-annotate-...-el2.png
## Edit Capture (2 changes, 35s)
### Inline Style Changes
**`#submit-btn`**
- `background-color`: `rgb(59, 130, 246)` → `rgb(37, 99, 235)`
- `border-radius`: added `8px`
### CSS Rule Changes
**`.btn-primary:hover`** (styles.css)
- `background-color`: `rgb(37, 99, 235)` → `rgb(29, 78, 216)`
### Before/After Screenshots
- Before: /var/folders/.../pi-annotate-...-before.png
- After: /var/folders/.../pi-annotate-...-after.png
```
Debug mode adds computed styles, parent context, and CSS variables per element. Edit capture appears when the Etch toggle is enabled and changes are detected.
## Architecture
```
Pi Extension (index.ts)
↕ Unix Socket (/tmp/pi-annotate.sock)
Native Host (host.cjs)
↕ Browser Native Messaging
Browser Extension (background.js → content.js)
```
| File | Purpose |
|------|---------|
| `index.ts` | Pi extension — `/annotate` command + tool |
| `types.ts` | TypeScript interfaces |
| `chrome-extension/content.js` | Element picker UI (vanilla JS) |
| `chrome-extension/background.js` | Native messaging, screenshots, tab routing |
| `chrome-extension/native/host.cjs` | Socket ↔ native messaging bridge |
| `chrome-extension/popup.html` | Connection status + setup |
Auth token generated per-run at `/tmp/pi-annotate.token`. Socket and token files use 0600 permissions.
## Development
No build step. Edit `content.js` or `background.js` directly, reload at `chrome://extensions`. Pi extension (TypeScript) loads via jiti — restart pi after changes.
```bash
tail -f /tmp/pi-annotate-host.log # Native host logs
# chrome://extensions → Pi Annotate → service worker # Background logs
# DevTools on target page # Content script logs
```
## Troubleshooting
| Issue | Fix |
|-------|-----|
| UI doesn't appear | Refresh page, check service worker console |
| "restricted URL" error | Provide a URL: `/annotate https://example.com` |
| Native host not connecting | Click extension icon → check status, re-run install, fully restart the supported browser |
| "Extension ID mismatch" | Copy install command from popup, re-run |
| Socket errors | `ls -la /tmp/pi-annotate.sock` |
**Verify native host:**
- macOS Google Chrome: `cat ~/Library/Application\ Support/Google/Chrome/NativeMessagingHosts/com.pi.annotate.json`
- macOS Google Chrome for Testing: `cat ~/Library/Application\ Support/Google/ChromeForTesting/NativeMessagingHosts/com.pi.annotate.json`
- macOS Chromium: `cat ~/Library/Application\ Support/Chromium/NativeMessagingHosts/com.pi.annotate.json`
- Linux Google Chrome (default path): `cat ~/.config/google-chrome/NativeMessagingHosts/com.pi.annotate.json`
- Linux Google Chrome for Testing (default path): `cat ~/.config/google-chrome-for-testing/NativeMessagingHosts/com.pi.annotate.json`
- Linux Chromium (default path): `cat ~/.config/chromium/NativeMessagingHosts/com.pi.annotate.json`
- Linux with custom config home: `echo "${CHROME_CONFIG_HOME:-${XDG_CONFIG_HOME:-$HOME/.config}}"`
If your Linux browser uses a different XDG config root, export `CHROME_CONFIG_HOME` or `XDG_CONFIG_HOME` before running `./install.sh `. Custom `--user-data-dir` layouts are not handled by this installer.
## License
MIT