{"id":31678904,"url":"https://github.com/notedit/fastapi-mp-agent","last_synced_at":"2026-08-05T06:31:10.598Z","repository":{"id":283603792,"uuid":"951960522","full_name":"notedit/fastapi-mp-agent","owner":"notedit","description":"fastapi multiprocess agent","archived":false,"fork":false,"pushed_at":"2025-03-21T04:40:41.000Z","size":6,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-21T05:27:00.921Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/notedit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-20T14:10:21.000Z","updated_at":"2025-03-21T04:40:44.000Z","dependencies_parsed_at":"2025-03-21T05:37:02.133Z","dependency_job_id":null,"html_url":"https://github.com/notedit/fastapi-mp-agent","commit_stats":null,"previous_names":["notedit/fastapi-mp-agent"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/notedit/fastapi-mp-agent","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/notedit%2Ffastapi-mp-agent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/notedit%2Ffastapi-mp-agent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/notedit%2Ffastapi-mp-agent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/notedit%2Ffastapi-mp-agent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/notedit","download_url":"https://codeload.github.com/notedit/fastapi-mp-agent/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/notedit%2Ffastapi-mp-agent/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278899231,"owners_count":26065152,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-08T06:19:55.024Z","updated_at":"2025-10-08T06:19:58.970Z","avatar_url":"https://github.com/notedit.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# aiohttp异步多进程任务处理服务\n\n一个使用aiohttp作为HTTP服务器、asyncio处理异步任务监控、multiprocessing.Process处理CPU密集型工作的多进程服务。每个工作进程启动自己的HTTP服务器，用于接收主进程的控制请求。\n\n## 功能\n\n- 接收HTTP请求并启动长时间运行的任务\n- 每个任务在单独的进程中运行\n- 每个工作进程启动自己的HTTP服务器，用于接收控制命令\n- 使用asyncio处理主服务器的异步逻辑\n- 使用Python multiprocessing进行进程间通信\n- 支持查询任务状态、列出所有任务和终止任务\n- 支持暂停和恢复任务执行\n- 定期（每10秒）在日志中打印所有任务及其状态\n- 支持多种任务类型，包括CPU密集型任务\n\n## 技术特点\n\n- 使用aiohttp作为异步Web服务器\n- 每个工作进程有自己的aiohttp服务器实例，在随机可用端口上运行\n- 通过HTTP接口实现进程间通信，提供更丰富的交互功能\n- 使用asyncio处理主服务器的事件循环和异步任务\n- 使用multiprocessing.Process创建独立工作进程执行CPU密集型任务\n- 通过multiprocessing.Queue进行基本进程间通信\n- 异步任务状态监控实时更新任务状态\n- 支持通过HTTP接口优雅地终止任务\n- 优雅的应用关闭，确保所有资源被正确释放\n\n## 设计思路\n\n该应用结合了asyncio和multiprocessing的优势：\n\n1. **主服务进程（aiohttp）**：使用asyncio处理所有HTTP请求和内部事件循环\n2. **异步监控**：使用asyncio协程监控所有工作进程的状态\n3. **工作进程（multiprocessing）**：使用multiprocessing.Process在独立进程中执行CPU密集型任务\n4. **工作进程HTTP服务器**：每个工作进程启动自己的aiohttp服务器，用于接收控制命令\n5. **双重进程间通信**：\n   - 通过multiprocessing.Queue进行快速状态更新\n   - 通过HTTP接口进行丰富的交互控制\n\n## 设置环境\n\n```bash\n# 创建虚拟环境\npython3 -m venv venv\n\n# 激活虚拟环境\nsource venv/bin/activate\n\n# 安装依赖\npip install -r requirements.txt\n```\n\n## 运行应用\n\n```bash\n# 从虚拟环境中运行\npython run.py\n```\n\n应用将在 http://localhost:8000 上启动。\n\n## API端点\n\n- `POST /tasks` - 创建新任务\n- `GET /tasks` - 列出所有任务\n- `GET /tasks/{task_id}` - 获取特定任务的状态\n- `GET /tasks/{task_id}?detail=true` - 获取任务的详细状态（从工作进程HTTP服务器获取）\n- `POST /tasks/{task_id}/control` - 控制任务（暂停/恢复等）\n- `DELETE /tasks/{task_id}` - 立即终止任务（强制）\n- `DELETE /tasks/{task_id}?use_http=true` - 优雅地终止任务（通过HTTP请求）\n\n## 支持的任务类型\n\n1. `process_data` - 数据处理任务（IO密集型）\n2. `generate_report` - 报告生成任务（IO密集型）\n3. `cpu_intensive` - CPU密集型计算任务\n\n## 工作进程HTTP端点\n\n每个工作进程启动的HTTP服务器提供以下端点：\n\n- `GET /status` - 获取工作进程的详细状态\n- `POST /control` - 接收控制命令（如暂停/恢复）\n- `POST /terminate` - 接收终止请求\n\n## 示例请求\n\n创建数据处理任务:\n\n```bash\ncurl -X POST http://localhost:8000/tasks \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"task_type\": \"process_data\", \"task_data\": {\"items\": [1, 2, 3, 4, 5]}}'\n```\n\n创建报告生成任务:\n\n```bash\ncurl -X POST http://localhost:8000/tasks \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"task_type\": \"generate_report\", \"task_data\": {\"report_type\": \"detailed\"}}'\n```\n\n创建CPU密集型任务:\n\n```bash\ncurl -X POST http://localhost:8000/tasks \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"task_type\": \"cpu_intensive\", \"task_data\": {\"complexity\": 2000000, \"batches\": 5}}'\n```\n\n暂停一个正在运行的任务:\n\n```bash\ncurl -X POST http://localhost:8000/tasks/{task_id}/control \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"action\": \"pause\"}'\n```\n\n恢复一个已暂停的任务:\n\n```bash\ncurl -X POST http://localhost:8000/tasks/{task_id}/control \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"action\": \"resume\"}'\n```\n\n优雅地终止任务:\n\n```bash\ncurl -X DELETE \"http://localhost:8000/tasks/{task_id}?use_http=true\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnotedit%2Ffastapi-mp-agent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnotedit%2Ffastapi-mp-agent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnotedit%2Ffastapi-mp-agent/lists"}