{"id":15352010,"url":"https://github.com/joddiy/multiprocess","last_synced_at":"2025-03-27T14:46:12.891Z","repository":{"id":119316725,"uuid":"114609755","full_name":"joddiy/multiprocess","owner":"joddiy","description":"common multiprocess model for python","archived":false,"fork":false,"pushed_at":"2018-06-01T17:48:50.000Z","size":498,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-01T18:45:05.905Z","etag":null,"topics":["muti-process","python","python-multiprocessing","python3"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joddiy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-12-18T07:33:13.000Z","updated_at":"2018-06-01T17:48:51.000Z","dependencies_parsed_at":null,"dependency_job_id":"a25927a6-5bea-4037-85c3-1d30f108f93a","html_url":"https://github.com/joddiy/multiprocess","commit_stats":{"total_commits":10,"total_committers":1,"mean_commits":10.0,"dds":0.0,"last_synced_commit":"51ad54a79399d39d5645a623c35ff3be7710e382"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joddiy%2Fmultiprocess","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joddiy%2Fmultiprocess/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joddiy%2Fmultiprocess/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joddiy%2Fmultiprocess/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joddiy","download_url":"https://codeload.github.com/joddiy/multiprocess/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245864804,"owners_count":20685108,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["muti-process","python","python-multiprocessing","python3"],"created_at":"2024-10-01T12:08:29.010Z","updated_at":"2025-03-27T14:46:12.866Z","avatar_url":"https://github.com/joddiy.png","language":"Python","readme":"# Common Multiprocess Module\n\n## Usage\n\n### Example\n\n```python\nimport os\nimport random\nimport time\n\nfrom multiprocess import MultiProcess, InfoTask, Model\n\n\nclass Demo(Model):\n    \"\"\"\n    Task Model\n    \"\"\"\n    \n    def runnable(self, *args):\n        \"\"\"\n        main function will be executed by Worker\n        :param args:\n        :return:\n        \"\"\"\n        print(\"run pid\", os.getpid(), end=\", \")\n        print(\"get first args\", args[0], end=\", \")\n        print(\"get second args\", args[1])\n        time.sleep(random.uniform(0, 2))\n        return args[0], args[1]\n\n    def callback(self, result):\n        \"\"\"\n        callback function when success\n        PLEASE NOTE: don't raise any Exception here, otherwise the process cannot exit normally\n        :param result:\n        :return:\n        \"\"\"\n        print(\"pid\", os.getpid(), \"result\", result[0], result[1])\n\n    def error_callback(self, error):\n        \"\"\"\n        callback function when fail\n        PLEASE NOTE: don't raise any Exception here, otherwise the process cannot exit normally\n        :param error:\n        :return:\n        \"\"\"\n        print(\"error_callback \", os.getpid(), error)\n\n\nif __name__ == '__main__':\n    # get a instance of CM module (specify the process-pool size)\n    mp = MultiProcess(5)\n    # start the CM module\n    mp.start()\n    for i in range(10):\n        # create a Task instance (specify Task module, timeout, args)\n        task = InfoTask(\"example.Demo\", 1, {\"test\": i}, {\"error\": i})\n        # push Task to process-pool\n        mp.push(task)\n    # no more new Task, wait exit\n    mp.stop()\n\n```\n\n\n## 架构和设计\n\n### 架构图\n\n![架构图](asset/structure.png)\n\n### 组成部分\n\n#### 任务 *Task*\n\n任务由父进程 *1* 通过管道推送给子进程 *2*，其中指定了任务模块所在命名空间 *module*（其中需要包含 *runnable，callback 和 error_callback*）三个方法，该任务的超时时间 \u0026*Timeout*，和需要传递给 runnable 方法的参数 *Args*。\n\n\n#### 父进程 *1*\n\n即入口进程，与调用入口在统一进程，负责开启模块，在开启的同时，指定允许的进程池容量，父进程通过管道 *Pipe* 向子进程 *2* 推送任务消息。\n\n#### 子进程 *2*\n\n即调度进程，该进程维护了一个进程池 *Process Pool*，并通过管道 *Pipe* 和父进程 *1* 通讯，如果接收到了父进程 *1* 推送的任务，就分配给进程池 *Process Pool* 去执行，如果接收到退出消息，就等待进程池 *Process Pool* 中所有的进程退出后退出。\n\n#### 进程池 *Process Pool*\n\n其容量在父进程 *1* 开启模块时指定，采用带 callback 和 error_callback 的异步执行模式。\n\n#### 任务进程 *3-7*\n\n真正执行任务的 Worker 进程，采用 **双线程模式**，一个线程 *Thread1* 执行任务，另一个线程 *Thread2* 负责监控超时。如果任务成功，则返回结果并通知**子进程2**调用 callback 方法，如果超时或者出错，通知**子进程2**调用 error_callback 方法\n\n\n### 设计思想\n\n#### 线程 OR 进程\n\nPython 的线程由于 Global Interpreter Lock（GIL，全局解释锁）的存在，使其实际上只能串行工作。另外，线程由于共享内存空间，会存在并行不安全性的问题。相反，进程是真正意义上的并发执行，且进程与进程之间是隔离的，不共享内存空间，因此一个进程崩溃并不会影响其他进程。\n\n#### 由子进程维护进程池\n\n为了实现解耦，不是由父进程直接维护进程池，而是新建一个子进程来维护，父进程只需要将消息通过管道推送给子进程即可。\n\n\n#### 分配任务\n\n子进程分配任务给进程池中的进程时，采用的是非阻塞的方式，即消息一旦推送给子进程，子进程就会立刻将任务分配给进程池（如果进程池有空闲的话）。在这种模式下，接收一个任务就会立即执行一个任务，能够提高效率。\n\n#### 双线程监督超时\n\n为了避免进程池中的进程出现假死而一直无法退出的情况，对于所有任务，当分配给它一个进程的时候，都会采用双线程的方式执行，一个线程用于执行任务，另一个线程用于监督是否超时，如果超时就会杀死任务并回调。\n\n#### 带有正常和异常的回调\n\n任务支持指定正常和异常回调函数，当函数正常退出时，会由子进程调用正常回调函数，当超时或者异常退出时，会由子进程调用异常回调函数。因此可以对任务的执行进行监控和后续操作。\n\n### 注意\n\n**一定要 catch 掉【正常回调】和【异常回调】抛出的异常**，否则父进程会无法退出。\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoddiy%2Fmultiprocess","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoddiy%2Fmultiprocess","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoddiy%2Fmultiprocess/lists"}