{"id":28423746,"url":"https://github.com/moses000/aidesktoppilot","last_synced_at":"2026-04-27T18:35:05.895Z","repository":{"id":295969475,"uuid":"991863367","full_name":"moses000/AIDesktopPilot","owner":"moses000","description":"AI-driven desktop automation","archived":false,"fork":false,"pushed_at":"2025-05-28T22:26:28.000Z","size":40254,"stargazers_count":0,"open_issues_count":0,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-25T18:45:50.379Z","etag":null,"topics":["ai","automation","computer-vision","desktop-automation","opencv","pytesseract","python","rpa","screenshots","tkinter"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/moses000.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-28T09:03:27.000Z","updated_at":"2025-05-28T22:26:33.000Z","dependencies_parsed_at":"2025-05-28T10:25:03.025Z","dependency_job_id":"e96ad5f4-4468-4868-b3b8-66dd3b4765c9","html_url":"https://github.com/moses000/AIDesktopPilot","commit_stats":null,"previous_names":["moses000/aidesktoppilot"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/moses000/AIDesktopPilot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moses000%2FAIDesktopPilot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moses000%2FAIDesktopPilot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moses000%2FAIDesktopPilot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moses000%2FAIDesktopPilot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/moses000","download_url":"https://codeload.github.com/moses000/AIDesktopPilot/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moses000%2FAIDesktopPilot/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32349898,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-27T17:12:42.749Z","status":"ssl_error","status_checked_at":"2026-04-27T17:12:41.658Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","automation","computer-vision","desktop-automation","opencv","pytesseract","python","rpa","screenshots","tkinter"],"created_at":"2025-06-05T09:09:23.857Z","updated_at":"2026-04-27T18:35:05.882Z","avatar_url":"https://github.com/moses000.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AI Desktop Mentor\n\nAI Desktop Mentor is an advanced Python-based desktop automation tool designed to emulate human-like interactions with a computer system. It leverages cutting-edge AI technologies, including **YOLOv8** for UI element detection, **Vosk** for offline speech recognition, and **DistilBERT** for natural language processing (NLP), to perform tasks such as opening applications, navigating websites, logging in, and processing screenshots. With a **Tkinter GUI**, it supports voice commands, task scripting, and automated workflows—ideal for business automation and personal productivity.\n\n---\n\n## ✨ Features\n\n- **Automation**: Open apps (e.g., Chrome, Notepad), type text, navigate URLs, and log in to websites.\n- **Screenshots**: Capture manually (`Ctrl+Shift+S`) or auto (every 15 minutes).\n- **AI Navigation**: YOLOv8 detects UI elements (e.g., login fields); OCR reads screen text.\n- **Task Scripting**: Execute sequences defined in `tasks.json`.\n- **Voice Control**: Use offline voice commands via Vosk.\n- **NLP Understanding**: Parse natural language with DistilBERT.\n- **Context Awareness**: Detect CAPTCHAs/pop-ups with OCR.\n- **Cross-Platform**: Works on Windows, macOS, and Linux.\n\n---\n\n## 📁 Directory Structure    \n\n```\n\nAIDesktopMentor/\n├── automation/\n│   └── automation\\_tool.py\n├── config/\n│   └── tasks.json\n├── docs/\n│   ├── README.md\n│   └── requirements.txt\n├── models/\n│   ├── yolo\\_ui\\_model.pth\n│   └── vosk-model-small-en-us/\n├── outputs/\n│   └── screenshots/\n├── dataset/\n│   ├── images/\n│   │   ├── train/\n│   │   └── val/\n│   ├── labels/\n│   │   ├── train/\n│   │   └── val/\n│   └── data.yaml\n\n````\n\n\n## 🔁 Technical Workflow\n\n```mermaid\ngraph TD\n    A[User Input] --\u003e B[GUI Tkinter]\n    A --\u003e C[Voice Listener Vosk]\n    C --\u003e D[NLP Parser DistilBERT]\n    D --\u003e E[Command Processor]\n    B --\u003e E\n    E --\u003e F[Automation Engine PyAutoGUI]\n    E --\u003e G[UI Detection YOLOv8]\n    E --\u003e H[Screenshot Module]\n    E --\u003e I[OCR Pytesseract]\n    E --\u003e J[Check Popups]\n    F --\u003e K[OS Interaction]\n    H --\u003e L[Save to screenshots/]\n    I --\u003e M[Context Feedback]\n    K --\u003e N[Screen Output]\n    M --\u003e N\n```\n\n## 📊 Business Workflow\n\n```mermaid\ngraph TD\n    A[Business User] --\u003e B[Define Task]\n    B --\u003e|Manual| C[GUI Interaction]\n    B --\u003e|Automated| D[Configure tasks.json]\n    B --\u003e|Voice| E[Voice Command]\n    C --\u003e F[Execute Task]\n    D --\u003e F\n    E --\u003e F\n    F --\u003e|Open App| G[Access System]\n    F --\u003e|Login| H[Authenticate]\n    F --\u003e|Navigate| I[Access Resource]\n    F --\u003e|Screenshot| J[Generate Report]\n    H --\u003e|YOLO Detection| I\n    I --\u003e K[Perform Business Function]\n    J --\u003e L[Save Output]\n    K --\u003e M[Business Outcome]\n    L --\u003e M\n```\n\n## ✅ Prerequisites\n\n* Python **3.8+**\n* Tesseract OCR\n\n  * Windows: [Install](https://github.com/tesseract-ocr/tesseract/wiki)\n  * macOS: `brew install tesseract`\n  * Linux: `sudo apt-get install tesseract-ocr`\n* Vosk Model\n\n  * [Download](https://alphacephei.com/vosk/models) `vosk-model-small-en-us`\n  * Extract into `models/vosk-model-small-en-us/`\n* YOLO Model\n\n  * Use `yolov8n.pt` or custom-trained model saved as `yolo_ui_model.pt`\n* Python dependencies\n\n  ```bash\n  pip install -r requirements.txt\n  ```\n* Microphone access + Permissions (macOS/Linux screen recording/input).\n\n---\n\n## ⚙️ Installation\n\n```bash\n# Clone the repo\ngit clone https://github.com/moses000/AIDesktopMentor.git\ncd AIDesktopMentor\n\n# Set up folder structure\nmkdir -p outputs/screenshots dataset/images/train dataset/images/val dataset/labels/train dataset/labels/val\n\n# Install dependencies\npip install -r requirements.txt\n```\n\n\u003e **Don't forget to install Tesseract OCR, Vosk model, and YOLO model.**\n\n---\n\n## 🧠 YOLO Setup\n\n### Option 1: Pre-trained\n\n```python\nfrom ultralytics import YOLO\nmodel = YOLO(\"yolov8n.pt\")\n```\n\n```bash\nmv yolov8n.pt models/yolo_ui_model.pt\n```\n\n\u003e Accuracy for UI tasks may be limited.\n\n---\n\n### Option 2: Train Your Own\n\n1. **Capture screenshots**\n\n```python\nimport pyautogui, time\nfor i in range(100):\n    pyautogui.screenshot(f\"dataset/images/train/login_{i}.png\")\n    time.sleep(2)\n```\n\n2. **Label with LabelImg**\n\n```bash\npip install labelImg\nlabelImg dataset/images/train dataset/labels/train\n```\n\n3. **Create `data.yaml`**\n\n```yaml\ntrain: dataset/images/train/\nval: dataset/images/val/\nnc: 3\nnames: ['username_field', 'password_field', 'login_button']\n```\n\n4. **Train**\n\n```python\nfrom ultralytics import YOLO\nmodel = YOLO(\"yolov8n.pt\")\nmodel.train(data=\"dataset/data.yaml\", epochs=50, imgsz=640, batch=16)\n```\n\n5. **Save model**\n\n```bash\ncp runs/train/exp/weights/best.pt models/yolo_ui_model.pt\n```\n\n---\n\n## 🚀 Usage\n\n```bash\npython automation/automation_tool.py\n```\n\n### GUI Tasks:\n\n* Open Notepad \u0026 type\n* Execute `tasks.json`\n* Take screenshot\n* OCR read screen\n* Login via GUI\n* Enable voice commands\n\n### Voice Commands:\n\n* \"open Chrome\"\n* \"go to example.com\"\n* \"log in to example.com\"\n* \"type hello world\"\n* \"take screenshot\"\n* \"read text\"\n* \"execute tasks\"\n* \"stop listening\"\n\n---\n\n## 🧾 Sample `tasks.json`\n\n```json\n[\n    {\n        \"action\": \"open\",\n        \"app\": \"chrome\"\n    },\n    {\n        \"action\": \"navigate\",\n        \"url\": \"https://example.com\"\n    },\n    {\n        \"action\": \"login\",\n        \"url\": \"https://example.com\"\n    },\n    {\n        \"action\": \"screenshot\",\n        \"prefix\": \"login_task\"\n    }\n]\n```\n\n---\n\n## 🛠 Notes\n\n* **Permissions**: macOS/Linux may need screen/microphone/input access.\n* **YOLO**: Required for login automation.\n* **Vosk**: Ensure correct folder structure in `models/`.\n* **Performance Tip**: Keep automation interval ≥ 5s to avoid resource strain.\n\n---\n\n## 📦 Deployment\n\n```bash\npip install pyinstaller\npyinstaller --onefile automation/automation_tool.py\n```\n\n---\n\n## 🧯 Troubleshooting\n\n* **YOLO Errors**: Check `yolo_ui_model.pt` \u0026 class IDs\n* **Vosk Errors**: Confirm model directory/mic permissions\n* **GUI Not Working**: Verify Python/Tkinter setup\n\n---\n\n## 🧠 Future Improvements\n\n* Expand YOLO UI detection classes\n* Add reinforcement learning for adaptive workflows\n* CAPTCHA solvers\n* Larger NLP models (e.g., BERT)\n* GUI task builder for `tasks.json`\n\n---\n\n## 📜 License\n\nMIT License\n\n---\n\n## 🤝 Contributing\n\nOpen issues or submit pull requests on GitHub.\n\n---\n\n## 📬 Contact\n\nFor support, [create an issue](https://github.com/moses000/AIDesktopMentor/issues) or email [im.imoleayomoses@gmail.com](mailto:im.imoleayomoses@gmail.com)\n\n---\n\n## 🙏 Acknowledgements\n\n* [Ultralytics](https://github.com/ultralytics/ultralytics) for YOLOv8\n* [Vosk](https://alphacephei.com/vosk/) for speech recognition\n* [Hugging Face](https://huggingface.co/) for transformers\n\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoses000%2Faidesktoppilot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmoses000%2Faidesktoppilot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoses000%2Faidesktoppilot/lists"}