{"id":25193847,"url":"https://github.com/donguillotine/langraph_browser_agent","last_synced_at":"2025-06-28T11:03:03.277Z","repository":{"id":276484988,"uuid":"929424579","full_name":"DonGuillotine/langraph_browser_agent","owner":"DonGuillotine","description":"A web automation agent driven by annotated browser screenshots and language model predictions","archived":false,"fork":false,"pushed_at":"2025-02-08T14:29:23.000Z","size":11,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-07T23:43:54.252Z","etag":null,"topics":["agents","langchain","langraph","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DonGuillotine.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-08T14:21:11.000Z","updated_at":"2025-02-11T03:00:00.000Z","dependencies_parsed_at":"2025-02-08T15:38:27.746Z","dependency_job_id":null,"html_url":"https://github.com/DonGuillotine/langraph_browser_agent","commit_stats":null,"previous_names":["donguillotine/langraph_browser_agent"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/DonGuillotine/langraph_browser_agent","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DonGuillotine%2Flangraph_browser_agent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DonGuillotine%2Flangraph_browser_agent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DonGuillotine%2Flangraph_browser_agent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DonGuillotine%2Flangraph_browser_agent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DonGuillotine","download_url":"https://codeload.github.com/DonGuillotine/langraph_browser_agent/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DonGuillotine%2Flangraph_browser_agent/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262419701,"owners_count":23308095,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agents","langchain","langraph","python"],"created_at":"2025-02-09T23:22:54.476Z","updated_at":"2025-06-28T11:03:03.233Z","avatar_url":"https://github.com/DonGuillotine.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Vision-Enabled Web Browsing Agent\n\n**Vision-Enabled Web Browsing Agent** is a Python project that implements a web automation agent driven by annotated browser screenshots and language model predictions. The agent leverages state-of-the-art browser automation and large language models (LLMs) to simulate human-like browsing behavior by controlling the mouse and keyboard.\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Features](#features)\n- [Requirements](#requirements)\n- [Installation](#installation)\n- [Usage](#usage)\n- [Project Structure](#project-structure)\n- [Customization](#customization)\n- [Contributing](#contributing)\n- [License](#license)\n- [Acknowledgments](#acknowledgments)\n- [Contact](#contact)\n\n## Overview\n\nThis project integrates several powerful libraries and tools:\n- **Playwright**: For browser automation and interaction.\n- **LangChain and LangGraph**: To integrate LLMs (via OpenAI Chat models) and manage the conversation flow using a state graph.\n- **PIL (Pillow)**: For processing and displaying screenshots.\n- **Custom JavaScript (`mark_page.js`)**: To annotate pages with bounding boxes that highlight actionable UI elements.\n\nThe agent works by taking a screenshot of the browser, identifying interactive elements, and then deciding which action to take next (e.g., clicking, typing, scrolling) based on LLM predictions. This simulation of human behavior is designed to help avoid detection (e.g., CAPTCHAs) during web automation.\n\n## Features\n\n- **Vision-Enabled Navigation:**  \n  The agent captures screenshots of the browser, overlays bounding boxes on UI elements, and uses these annotations to inform action decisions.\n\n- **LLM-Powered Decision Making:**  \n  Integrates ChatGPT (via OpenAI’s API) to analyze the current page state and decide on the next best action.\n\n- **Tool Execution:**  \n  Supports a variety of actions:\n  - **Click:** Simulate mouse clicks on identified UI elements.\n  - **Type Text:** Enter text with realistic, human-like typing delays.\n  - **Scroll:** Scroll the window or specific elements.\n  - **Wait:** Pause execution to mimic human reading time.\n  - **Go Back:** Navigate back in browser history.\n  - **Navigate to Google:** Quickly jump to `google.com`.\n\n- **State Graph Management:**  \n  Uses LangGraph to manage and transition between different agent states and actions.\n\n- **Human-Like Typing Simulation:**  \n  The `type_text` function simulates realistic typing by introducing delays between keystrokes, reducing the risk of automated detection.\n\n## Requirements\n\n- Python 3.8 or higher\n- [Playwright](https://playwright.dev/python/)\n- [LangChain](https://github.com/hwchase17/langchain) and [LangGraph](https://github.com/langgraph/langgraph)\n- [Pillow](https://python-pillow.org/)\n- Other dependencies as listed in `requirements.txt`\n\nYou will also need an OpenAI API key, which should be available as the environment variable `OPENAI_API_KEY`.\n\n## Installation\n\n1. **Clone the repository:**\n\n   ```bash\n   git clone https://github.com/DonGuillotine/langraph_browser_agent.git\n   cd langraph_browser_agent\n   ```\n\n2. **Create and activate a virtual environment:**\n\n   ```bash\n   python -m venv venv\n   source venv/bin/activate  # On Windows: venv\\Scripts\\activate\n   ```\n\n3. **Install the Python dependencies:**\n\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n4. **Install the Playwright browsers:**\n\n   ```bash\n   playwright install\n   ```\n\n5. **Set the OpenAI API Key:**\n\n   You can either set the environment variable manually:\n   \n   ```bash\n   export OPENAI_API_KEY=\"your-api-key\"  # On Windows use set instead of export\n   ```\n\n   Or simply run the script; it will prompt you for the API key if it isn’t set.\n\n## Usage\n\nThe main entry point is the `main()` function in your project. When executed, the agent will launch a Chromium browser window, navigate to a starting page (e.g., `google.com`), and execute actions based on the LLM’s predictions.\n\nTo run the agent:\n\n```bash\npython main.py\n```\n\nWhile running, the agent will display:\n- Annotated browser screenshots (using PIL)\n- A console log of the actions taken\n\nExample prompt for the agent might be:  \n`\"Find me the latest news on CNN\"`\n\n## Project Structure\n\n- **`main.py`**  \n  The main script that sets up the agent, defines the state graph, and orchestrates the LLM-driven actions.\n\n- **`mark_page.js`**  \n  A JavaScript file used by Playwright to annotate the current page with bounding boxes for interactive elements.\n\n- **`requirements.txt`**  \n  Lists all the required Python packages for the project.\n\n- **`README.md`**  \n  This file.\n\n## Customization\n\n- **Typing Simulation:**  \n  The `type_text` function simulates human-like typing. You can adjust the delay parameters (or use randomized delays) to further mimic natural typing speeds.\n\n- **LLM Prompt and Output Parsing:**  \n  The agent uses a prompt (pulled from `wfh/web-voyager`) and a custom parser to interpret LLM responses. Feel free to modify these to better suit your needs.\n\n- **Tool Configuration:**  \n  You can add, remove, or modify tools in the `tools` dictionary in `main.py` to change the agent’s behavior.\n\n## Contributing\n\nContributions are welcome! If you have suggestions for improvements, bug fixes, or new features, please open an issue or submit a pull request.\n\nWhen contributing, please ensure:\n- Your code follows the existing style.\n- Any new features are accompanied by relevant tests or documentation updates.\n\n## License\n\nThis project is licensed under the [MIT License](LICENSE).\n\n## Acknowledgments\n\n- [Playwright](https://playwright.dev/python/)\n- [OpenAI](https://openai.com/)\n- [LangChain](https://github.com/hwchase17/langchain)\n- [LangGraph](https://github.com/langgraph/langgraph)\n- [Pillow](https://python-pillow.org/)\n\n## Contact\n\nFor questions, suggestions, or contributions, please open an issue in this repository or contact [infect3dlab@gmail.com](mailto:infect3dlab@gmail.com).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdonguillotine%2Flangraph_browser_agent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdonguillotine%2Flangraph_browser_agent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdonguillotine%2Flangraph_browser_agent/lists"}