{"id":19220487,"url":"https://github.com/timgoll/multiprocessing","last_synced_at":"2026-05-16T06:33:33.126Z","repository":{"id":123885886,"uuid":"154477528","full_name":"TimGoll/multiprocessing","owner":"TimGoll","description":"code example code of a multiprocessing project with queue handling","archived":false,"fork":false,"pushed_at":"2019-10-11T16:17:49.000Z","size":13,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-23T09:23:35.761Z","etag":null,"topics":["library","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TimGoll.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-24T09:51:05.000Z","updated_at":"2022-02-16T11:09:59.000Z","dependencies_parsed_at":null,"dependency_job_id":"8f86aa6a-0f52-4519-9e06-665dfdc3e3e6","html_url":"https://github.com/TimGoll/multiprocessing","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TimGoll/multiprocessing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimGoll%2Fmultiprocessing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimGoll%2Fmultiprocessing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimGoll%2Fmultiprocessing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimGoll%2Fmultiprocessing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TimGoll","download_url":"https://codeload.github.com/TimGoll/multiprocessing/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimGoll%2Fmultiprocessing/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33092688,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-16T04:41:52.686Z","status":"ssl_error","status_checked_at":"2026-05-16T04:41:52.009Z","response_time":115,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["library","python"],"created_at":"2024-11-09T14:35:19.265Z","updated_at":"2026-05-16T06:33:33.110Z","avatar_url":"https://github.com/TimGoll.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Python Multiprocessing\nMultiprocessing is vastly superior to threading because it creates real parallel running threads. Therefore you have to pass the data through queues. These two given classes will handle everything for you.\n\nTested with `Python 2.7` and `Python 3.7`.\n\n## Multiprocessing Basics\nPython multiprocessing works as follows: As soon as you start the process, all variables of the previously created class get copied into the new process. Therefore you can define as many vairables as you like before you start the process, but you can't change the values afterwards. This is the reason why queues are needed. They are threadsafe and allow communication between different processes.\n\n## Example\nIn the following text we're discussing the topic based on a simple read/write program, where one thread sets the data and another thread reads it.\n\nYou start with inheriting the baseclass and setting up the initial data, the init function is defined as `__init__(self, obj_ptr_queue_handler, str_queue_name_1, str_queue_name_2, ... , user_args)`.\n\n```python\nimport queue_handler, multiprocessing_base\n\nclass ClassReader(multiprocessing_base.MultiprocessingBase):\n    def __init__(self, queue_handler, main_queue_name):\n        super(ClassReader, self).__init__(queue_handler)\n        self.main_queue_name = main_queue_name\n        self.flag = 'REDR'\n        \nclass ClassWriter(multiprocessing_base.MultiprocessingBase):\n    def __init__(self, queue_handler, main_queue_name):\n        super(ClassInserter, self).__init__(queue_handler)\n        self.main_queue_name = main_queue_name\n        self.flag = 'WRTR'\n        \n        self.counter = 0\n```\n\nIn both classes the parents class `__init__` mathod has to get called first via `super`. `queue_handler` is a necessary argument, because it is the object pointer to the queue handler which is needed, even if you don't plan to use queues, because the process uses an internal queue to communicate, especially to stop the process when you call the stop function. `main_queue_name` is a string and not necessary. You can pass as many queue_names and user specific arguments as you'd like to.\n\nIn the next step you have to overwrite `init_process`, `run_process` and `deinit_process`. None of them is mendatory, but at least `run_process` should be used and is already a loop.\n\nTherefore the finished classes look like this:\n```python\nclass ClassReader(multiprocessing_base.MultiprocessingBase):\n    def __init__(self, queue_handler, main_queue_name):\n        super(ClassReader, self).__init__(queue_handler)\n        self.main_queue_name = main_queue_name\n        self.flag = 'REDR'\n\n    def run_process(self):\n        data = self.queue_handler.get(self.main_queue_name)\n        \n        if data == None:\n            return\n\n        print('read data: ' + str(data))\n        sleep(0.1)\n\nclass ClassInserter(multiprocessing_base.MultiprocessingBase):\n    def __init__(self, queue_handler, main_queue_name):\n        super(ClassInserter, self).__init__(queue_handler)\n        self.main_queue_name = main_queue_name\n        self.flag = 'INST'\n\n        self.counter = 0\n\n    def run_process(self):\n        self.queue_handler.put(self.main_queue_name, self.counter)\n        self.counter += 1\n        sleep(0.25)\n```\n\nA small note on `super`. `__init__` needs the super notation because it extends the code of the base class. The other three methods don't have anything to be overwritten and therefore need no `super`.\n\nThis code runs perfectly fine, but there is an easy way to make the program more performant, while also making sure that the process will never be too slow to handle all the data received over queues: the `wait()` function. While the multiprocessing library alredy already comes with a wait implementation, the multiprocessing base comes with its own wrapper to make sure the internal queues aren't blocked as well.\n\n```python\nclass ClassReader(multiprocessing_base.MultiprocessingBase):\n    def __init__(self, queue_handler, main_queue_name):\n        super(ClassReader, self).__init__(queue_handler)\n        self.main_queue_name = main_queue_name\n        self.flag = 'REDR'\n\n    def run_process(self):\n        # wait until new data is available without using a sleep()\n        self.wait([\n            self.queue_handler.get_reader(self.main_queue_name)\n        ])\n\n        data = self.queue_handler.get(self.main_queue_name)\n\n        # make sure data is valid\n        if data == None:\n            return\n\n        print('read data: ' + str(data))\n```\n\nThis wait function blocks the code until new data is available in the queue. If multiple queues are used, multiple queues can be added to this array. You have to make sure to use the reader of the queue, not the queue itself! A secondary optional parameter of this wait function is a timeout variable. Setting this timeout variable will stop the wait function after a certain amount of time.\n\nTo run your code, you have to create instances of your classes and then start the processes. **Important note:** Always stop processes after you've started them or you get ghost processes running forever in the background.\n```python\nqueueHandler = queue_handler.QueueHandler()\nqueueHandler.initQueue('main_queue')\n\nclassReader   = ClassReader(queue_handler = queueHandler, main_queue_name = 'main_queue')\n#create another instance to demonstrate the uniqueness of the internal queue name (address)\nclassReader2  = ClassReader(queue_handler = queueHandler, main_queue_name = 'main_queue')\nclassInserter = ClassInserter(queue_handler = queueHandler, main_queue_name = 'main_queue')\n\nclassReader.start_process()\nclassReader2.start_process()\nclassInserter.start_process()\n\n#...\n\nclassReader.stop_process()\nclassReader2.stop_process()\nclassInserter.stop_process()\n```\n\nThe console output of this code looks like this:\n```\n[QHDL] queue with the name main_queue initialised!\n[QHDL] queue with the name ClassReader_0x30174d0__internal_queue initialised!\n[QHDL] queue with the name ClassReader_0x3017510__internal_queue initialised!\n[QHDL] queue with the name ClassInserter_0x3017530__internal_queue initialised!\n[REDR] process initialized!\n[REDR] process initialized!\n[INST] process initialized!\nread data: 0\nread data: 1\nread data: 2\nread data: 3\nread data: 4\nread data: 5\nread data: 6\nread data: 7\nread data: 8\n[REDR] stop flag received\n[REDR] stop flag received\n[REDR] process successfully stopped!\n[REDR] process successfully stopped!\n[INST] stop flag received\n[INST] process successfully stopped!\n```\n\nYou can find this example as an executable python script in this repository.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimgoll%2Fmultiprocessing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftimgoll%2Fmultiprocessing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimgoll%2Fmultiprocessing/lists"}