Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/reidbarber/webmarker
Mark web pages for use with vision-language models
https://github.com/reidbarber/webmarker
claude gemini gpt4o gpt4v llms playwright prompt prompt-engineering qwen-vl set-of-mark som vision-language-model
Last synced: 3 months ago
JSON representation
Mark web pages for use with vision-language models
- Host: GitHub
- URL: https://github.com/reidbarber/webmarker
- Owner: reidbarber
- License: mit
- Created: 2024-04-29T04:41:27.000Z (9 months ago)
- Default Branch: main
- Last Pushed: 2024-09-22T02:48:13.000Z (5 months ago)
- Last Synced: 2024-10-13T18:50:06.966Z (4 months ago)
- Topics: claude, gemini, gpt4o, gpt4v, llms, playwright, prompt, prompt-engineering, qwen-vl, set-of-mark, som, vision-language-model
- Language: TypeScript
- Homepage: https://webmarkerjs.com/
- Size: 662 KB
- Stars: 15
- Watchers: 1
- Forks: 2
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Mark web pages for use with vision-language models.
## Overview
**WebMarker** adds visual markings with labels to elements on a web page. This can be used for [Set-of-Mark](https://github.com/microsoft/SoM) prompting, which improves visual grounding abilities of vision-language models such as GPT-4o, Claude 3.5, and Google Gemini 1.5.
## How it works
**1. Call the `mark()` function**
This marks the interactive elements on the page, and returns an object containing the marked elements, where each key is a mark label string, and each value is an object with the following properties:
- `element`: The interactive element that was marked.
- `markElement`: The label element that was added to the page.
- `boundingBoxElement`: The bounding box element that was added to the page.
You can use this information to build your prompt for the vision-language model.
**2. Send a screenshot of the marked page to a vision-language model, along with your prompt**
Example prompt:
```javascript
let markedElements = mark();
let prompt = `The following is a screenshot of a web page.
Interactive elements have been marked with red bounding boxes and labels.
When referring to elements, use the labels to identify them.
Return an action and element to perform the action on.
Available actions: click, hover
Available elements:
${Object.keys(markedElements)
.map((label) => `- ${label}`)
.join("\n")}
Example response: click 0
`;
```
**3. Programmatically interact with the marked elements.**
In a web browser (i.e. via Playwright), interact with elements as needed.
For prompting or agent ideas, see the [WebVoyager](https://github.com/MinorJerry/WebVoyager) paper.
## Playwright example
```javascript
// Inject the WebMarker library into the page
await page.addScriptTag({
url: "https://cdn.jsdelivr.net/npm/webmarker-js/dist/main.js",
});
// Mark the page and get the marked elements
let markedElements = await page.evaluate(async () => await WebMarker.mark());
// Click a marked element
await page.locator('[data-mark-label="0"]').click();
// (Optional) Check if page is marked
let isMarked = await page.evaluate(async () => await WebMarker.isMarked());
// (Optional) Unmark the page
await page.evaluate(async () => await WebMarker.unmark());
```
## Options
### selector
A custom CSS selector to specify which elements to mark.
- Type: `string`
- Default: `"button, input, a, select, textarea"`
### getLabel
Provide a function for generating labels. By default, labels are generated as integers starting from 0.
- Type: `(element: Element, index: number) => string`
- Default: `(_, index) => index.toString()`
### markAttribute
A custom attribute to add to the marked elements. This attribute contains the label of the mark.
- Type: `string`
- Default: `"data-mark-label"`
### markPlacement
The placement of the mark relative to the element.
- Type: `'top' | 'top-start' | 'top-end' | 'right' | 'right-start' | 'right-end' | 'bottom' | 'bottom-start' | 'bottom-end' | 'left' | 'left-start' | 'left-end'`
- Default: `'top-start'`
### markStyle
A CSS style to apply to the label element. You can also specify a function that returns a CSS style object.
- Type: `Partial | (element: Element) => Partial`
- Default: `{backgroundColor: "red", color: "white", padding: "2px 4px", fontSize: "12px", fontWeight: "bold"}`
### boundingBoxStyle
A CSS style to apply to the bounding box element. You can also specify a function that returns a CSS style object. Bounding boxes are only shown if showBoundingBoxes is true.
- Type: `Partial | (element: Element) => Partial`
- Default: `{outline: "2px dashed red", backgroundColor: "transparent"}`
### showBoundingBoxes
Whether or not to show bounding boxes around the elements.
- Type: `boolean`
- Default: `true`
### containerElement
Provide a container element to query the elements to be marked. By default, the container element is document.body.
- Type: `Element`
- Default: `document.body`
### viewPortOnly
Only mark elements that are visible in the current viewport.
- Type: `boolean`
- Default: `false`
### Advanced example
```typescript
const markedElements = mark({
// Only mark buttons and inputs
selector: "button, input",
// Use test id attribute for marker labels
markAttribute: "data-test-id",
// Use a blue mark with white text
markStyle: { color: "white", backgroundColor: "blue", padding: 5 },
// Use a blue dashed outline with a transparent and slighly blue background
boundingBoxStyle: { outline: "2px dashed blue", backgroundColor: "rgba(0, 0, 255, 0.1)"},
// Place the mark at the top right corner of the element
markPlacement: "top-end";
// Show bounding boxes over elements (defaults to true)
showBoundingBoxes: true,
// Generate labels as 'Element 0', 'Element 1', 'Element 2'...
// Defaults to '0', '1', '2'... if not provided.
getLabel: (element, index) => `Element ${index}`,
// A custom container element to query the elements to be marked.
// Defaults to the document.body.
containerElement: document.body.querySelector("main"),
// Only mark elements that are visible in the current viewport
viewPortOnly: true,
});
```