{"id":17967151,"url":"https://github.com/lx-0/computer-use-nodejs-demo","last_synced_at":"2025-08-16T15:31:47.063Z","repository":{"id":259862339,"uuid":"879641732","full_name":"lx-0/computer-use-nodejs-demo","owner":"lx-0","description":"🤖 LLM-powered computer control through local and Docker environments. Features VNC integration, automated interactions, and a chat interface for natural language system control.","archived":false,"fork":false,"pushed_at":"2024-12-01T15:12:02.000Z","size":1221,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-12-01T16:23:13.225Z","etag":null,"topics":["ai","computer-use","docker","function-calling","llm"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lx-0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-28T09:39:50.000Z","updated_at":"2024-12-01T15:12:06.000Z","dependencies_parsed_at":"2024-10-28T13:52:04.484Z","dependency_job_id":"d00ac16a-3f65-4a0d-8b5d-0313471bdd2c","html_url":"https://github.com/lx-0/computer-use-nodejs-demo","commit_stats":{"total_commits":33,"total_committers":2,"mean_commits":16.5,"dds":0.06060606060606055,"last_synced_commit":"12da10bed088ad98373ab537a39cd02b9b163fd4"},"previous_names":["lx-0/computer-use-nodejs-demo"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lx-0%2Fcomputer-use-nodejs-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lx-0%2Fcomputer-use-nodejs-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lx-0%2Fcomputer-use-nodejs-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lx-0%2Fcomputer-use-nodejs-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lx-0","download_url":"https://codeload.github.com/lx-0/computer-use-nodejs-demo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230043146,"owners_count":18163966,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","computer-use","docker","function-calling","llm"],"created_at":"2024-10-29T14:04:19.251Z","updated_at":"2024-12-17T00:13:40.510Z","avatar_url":"https://github.com/lx-0.png","language":"TypeScript","readme":"# LLM-Controlled Computer\n\nA Next.js application that uses a large language model to control a computer through both local system control and virtual machine (Docker) environments.\n\n[![Next.js](https://img.shields.io/badge/Next.js-black?style=for-the-badge\u0026logo=next.js\u0026logoColor=white)](https://nextjs.org)\n[![TypeScript](https://img.shields.io/badge/TypeScript-007ACC?style=for-the-badge\u0026logo=typescript\u0026logoColor=white)](https://www.typescriptlang.org)\n[![Langchain](https://img.shields.io/badge/Langchain-000000?style=for-the-badge\u0026logo=langchain\u0026logoColor=white)](https://langchain.com)\n[![Docker](https://img.shields.io/badge/Docker-2496ED?style=for-the-badge\u0026logo=docker\u0026logoColor=white)](https://www.docker.com)\n[![shadcn/ui](https://img.shields.io/badge/shadcn--ui-000000?style=for-the-badge\u0026logo=shadcnui\u0026logoColor=white)](https://ui.shadcn.com)\n[![Tailwind_CSS](https://img.shields.io/badge/Tailwind_CSS-38B2AC?style=for-the-badge\u0026logo=tailwind-css\u0026logoColor=white)](https://tailwindcss.com)\n[![Electron](https://img.shields.io/badge/Electron-47848F?style=for-the-badge\u0026logo=electron\u0026logoColor=white)](https://www.electronjs.org)\n[![Built with Cursor](https://img.shields.io/badge/Built_With-Cursor-5C4EE5?style=for-the-badge)](https://cursor.com)\n\n![Screenshot](./public/images/screenshot.png)\n\n\u003e **🚧 Work in Progress**\n\u003e\n\u003e This project is under active development. Some features may be incomplete or subject to change.\n\u003e\n\u003e The overall goal is to create a tool that allows a user to control their computer with any large language model in nodejs. Anthropics [Computer Use Demo](https://github.com/anthropics/anthropic-quickstarts) is the main inspirational source for this project.\n\u003e\n\u003e Roadmap:\n\u003e\n\u003e - ✅ Docker container management\n\u003e - ✅ VNC integration\n\u003e - ✅ Chat interface\n\u003e - 🔳 (Generic) LLM integration\n\u003e   - ✅ Base architecture\n\u003e   - ✅ Model selection\n\u003e   - ✅ Model tracking\n\u003e   - ✅ Message history\n\u003e   - ✅ Local model support\n\u003e   - ✅ Model download tracking\n\u003e   - 🔳 Context management\n\u003e   - 🔳 Function calling\n\u003e   - ⬜ Streaming support\n\u003e - ⬜ Computer use tooling\n\u003e   - ⬜ File management\n\u003e   - ⬜ Screenshot analysis\n\u003e   - ⬜ Mouse and keyboard control\n\u003e   - ⬜ Bash command execution\n\u003e - 🔳 Launch options\n\u003e   - ⬜ CLI\n\u003e   - ✅ Web server\n\u003e   - ⬜ Electron app\n\u003e - 🔳 Computer Use modes\n\u003e   - ✅ Virtual (Docker)\n\u003e   - ⬜ Local (direct control)\n\u003e - ⬜ Conversation history\n\u003e - ⬜ Multi Agent support\n\u003e - ⬜ Memory management\n\u003e\n\u003e Please check back later for updates or **feel free to contribute!**\n\n## Features\n\n### Core Capabilities\n\n- Screenshot analysis\n- Mouse and keyboard control\n- Bash command execution\n- File management\n- Chat interface for LLM interaction\n- VNC-based graphical interactions\n\n### Operation Modes\n\n- **Local Mode**: Direct system control\n- **Docker Mode**: Virtual machine control via Docker containers\n- **Multiple Launch Options**:\n  - Web browser (Next.js server)\n  - Desktop application (Electron)\n  - CLI for specific LLM tasks\n\n### Docker Integration\n\n- Real-time container management\n- Build progress streaming\n- Container lifecycle control (start, stop, delete)\n- Status monitoring and detailed logging\n- NoVNC integration for web-based access\n- Automated environment setup\n\n### User Interface\n\n- Responsive split-view layout\n- Settings sidebar\n- Real-time Docker status indicators\n- Expandable log entries\n- Copy-to-clipboard functionality\n- Auto-scrolling chat interface\n\n## Tech Stack\n\n- **Frontend**: Next.js with TypeScript\n- **UI Components**: Radix UI, Tailwind CSS\n- **Container Management**: Dockerode\n- **Remote Access**: VNC, SSH2\n- **LLM Integration**: Langchain.js\n- **Desktop Packaging**: Electron\n- **Terminal**: node-pty, xterm.js\n\n## Prerequisites\n\n- Node.js (LTS version)\n- Docker\n- Python 3.11.6 (for certain features)\n- Ollama (for local models) - See [Ollama Setup](#ollama-setup) section\n\n## Installation\n\n### 1. Clone the repository\n\n```bash\ngit clone [repository-url]\ncd llm-controlled-computer\n```\n\n### 2. Install dependencies\n\n```bash\nnpm install\n```\n\n### 3. Set up environment variables\n\n```bash\ncp .env.example .env\n```\n\nEdit `.env` with your configuration.\n\n## Development\n\nStart the development server:\n\n```bash\nnpm run dev\n```\n\n### Building\n\nFor production build:\n\n```bash\nnpm run build\n```\n\nFor Electron desktop app:\n\n```bash\nnpm run build:electron\n```\n\n## Docker Usage\n\nThe application includes a custom Docker environment with:\n\n- Ubuntu 22.04 base\n- Python environment with pyenv\n- Desktop environment with VNC access\n- Firefox ESR with pre-configured extensions\n- Various utility applications\n\n## Ollama Setup\n\n### Installation\n\n#### macOS\n\n```bash\n# Using Homebrew\nbrew install ollama\n\n# Start Ollama service\nollama serve\n```\n\n#### Linux\n\n```bash\n# Install Ollama\ncurl -fsSL https://ollama.com/install.sh | sh\n\n# Start Ollama service\nsystemctl start ollama\n```\n\n#### Windows\n\n1. Install WSL2 if not already installed:\n\n```bash\nwsl --install\n```\n\n2. Install Ollama in WSL2:\n\n```bash\ncurl -fsSL https://ollama.com/install.sh | sh\n```\n\n3. Start Ollama service in WSL2:\n\n```bash\nollama serve\n```\n\n### Configuration\n\nAdd the following to your `.env` file:\n\n```env\n# Ollama Configuration\nNEXT_PUBLIC_OLLAMA_URL=http://localhost:11434\n```\n\n### Troubleshooting\n\n1. Check if Ollama is running:\n\n```bash\ncurl http://localhost:11434/api/health\n```\n\n2. If not running, start the service:\n\n```bash\n# macOS/Linux\nollama serve\n\n# Windows (in WSL2)\nwsl -d Ubuntu -u root ollama serve\n```\n\n3. Common issues:\n   - Port 11434 is already in use\n   - Insufficient disk space\n   - GPU drivers not properly installed (for GPU acceleration)\n\n## Contributing\n\n1. Ensure you follow the project's coding standards:\n   - Use TypeScript with strict typing\n   - Follow clean code principles\n   - Write comprehensive tests\n   - Add proper documentation\n\n2. Submit pull requests with:\n   - Clear description of changes\n   - Test coverage\n   - Documentation updates\n\n## License\n\nISC\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flx-0%2Fcomputer-use-nodejs-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flx-0%2Fcomputer-use-nodejs-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flx-0%2Fcomputer-use-nodejs-demo/lists"}