{"id":13725147,"url":"https://github.com/jruere/multiprocessing-logging","last_synced_at":"2025-10-21T20:46:18.624Z","repository":{"id":26236567,"uuid":"29683398","full_name":"jruere/multiprocessing-logging","owner":"jruere","description":"Handler for logging from multiple processes","archived":false,"fork":false,"pushed_at":"2023-10-25T19:50:43.000Z","size":77,"stargazers_count":390,"open_issues_count":8,"forks_count":58,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-04-21T00:03:25.430Z","etag":null,"topics":["linux","logging","multiprocessing","python"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jruere.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2015-01-22T14:49:01.000Z","updated_at":"2025-03-27T19:09:29.000Z","dependencies_parsed_at":"2023-11-12T12:37:16.577Z","dependency_job_id":"2b6bcacd-04c3-4c1d-92bb-38637177e947","html_url":"https://github.com/jruere/multiprocessing-logging","commit_stats":{"total_commits":95,"total_committers":11,"mean_commits":8.636363636363637,"dds":0.1473684210526316,"last_synced_commit":"c75d088dcbdd3ec52d164e94707d6d534a449e52"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jruere%2Fmultiprocessing-logging","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jruere%2Fmultiprocessing-logging/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jruere%2Fmultiprocessing-logging/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jruere%2Fmultiprocessing-logging/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jruere","download_url":"https://codeload.github.com/jruere/multiprocessing-logging/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252943906,"owners_count":21829336,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["linux","logging","multiprocessing","python"],"created_at":"2024-08-03T01:02:14.204Z","updated_at":"2025-10-21T20:46:18.520Z","avatar_url":"https://github.com/jruere.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# multiprocessing-logging\n\n[![Supported Python versions](https://img.shields.io/pypi/pyversions/multiprocessing-logging.svg)](https://pypi.python.org/pypi/multiprocessing-logging/)\n[![License](https://img.shields.io/pypi/l/multiprocessing-logging.svg)](https://pypi.python.org/pypi/multiprocessing-logging/)\n\n\nWhen using the `multiprocessing` module, logging becomes less useful since\nsub-processes should log to individual files/streams or there's the risk of\nrecords becoming garbled.\n\nThis simple module implements a `Handler` that when set on the root\n`Logger` will handle tunneling the records to the main process so that\nthey are handled correctly.\n\nIt's currently tested in Linux and Python 2.7 \u0026 3.6+.\n\nPypy3 hangs on the tests so I don't recommend using it.\n\nPypy appears to be working, recently.\n\nOnly works on POSIX systems and only Linux is supported. It does not work on Windows.\n\n# Origin\n\nThis library was taken verbatim from a [StackOverflow post](http://stackoverflow.com/questions/641420/how-should-i-log-while-using-multiprocessing-in-python)\nand extracted into a module so that I wouldn't have to copy the code in every\nproject.\n\nLater, several improvements have been contributed.\n\n# Usage\n\nBefore you start logging but after you configure the logging framework (maybe with `logging.basicConfig(...)`), do the following:\n\n```py\nimport multiprocessing_logging\n\nmultiprocessing_logging.install_mp_handler()\n```\n\nand that's it.\n\n## With multiprocessing.Pool\n\nWhen using a Pool, make sure `install_mp_handler` is called before the Pool is instantiated, for example:\n\n```py\nimport logging\nfrom multiprocessing import Pool\nfrom multiprocessing_logging import install_mp_handler\n\nlogging.basicConfig(...)\ninstall_mp_handler()\npool = Pool(...)\n```\n\n# Problems\nThe approach of this module relies on\n[fork](https://docs.python.org/3.9/library/multiprocessing.html#multiprocessing.set_start_method)\nbeing used to create new processes. This start method\n[is basically unsafe when also using threads](https://bugs.python.org/issue37429),\nas this module does.\n\nThe consequence is that there's a low probability of the application hanging\nwhen creating new processes.\n\nAs a palliative, don't continuously create new processes. Instead, create a\nPool once and reuse it.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjruere%2Fmultiprocessing-logging","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjruere%2Fmultiprocessing-logging","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjruere%2Fmultiprocessing-logging/lists"}