{"id":26183814,"url":"https://github.com/mikata-project/parallel_write","last_synced_at":"2026-04-24T10:32:11.857Z","repository":{"id":57450729,"uuid":"436679207","full_name":"Mikata-Project/parallel_write","owner":"Mikata-Project","description":"Writes to many open file objects in parallel","archived":false,"fork":false,"pushed_at":"2022-01-14T14:59:43.000Z","size":53,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-06T11:19:20.565Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Mikata-Project.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-12-09T16:07:58.000Z","updated_at":"2022-01-03T13:49:52.000Z","dependencies_parsed_at":"2022-09-26T17:31:34.521Z","dependency_job_id":null,"html_url":"https://github.com/Mikata-Project/parallel_write","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":"joaomcteixeira/python-project-skeleton","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mikata-Project%2Fparallel_write","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mikata-Project%2Fparallel_write/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mikata-Project%2Fparallel_write/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mikata-Project%2Fparallel_write/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Mikata-Project","download_url":"https://codeload.github.com/Mikata-Project/parallel_write/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243125593,"owners_count":20240276,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-11T22:49:20.567Z","updated_at":"2025-12-24T10:16:27.533Z","avatar_url":"https://github.com/Mikata-Project.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"================================\nParallel write\n================================\n\n.. image:: https://github.com/Mikata-Project/parallel_write/workflows/Tests/badge.svg?branch=master\n    :target: https://github.com/Mikata-Project/parallel_write/actions?workflow=Tests\n    :alt: Test Status\n\n.. image:: https://github.com/Mikata-Project/parallel_write/workflows/Package%20Build/badge.svg?branch=master\n    :target: https://github.com/Mikata-Project/parallel_write/actions?workflow=Package%20Build\n    :alt: Package Build\n\n.. image:: https://codecov.io/gh/Mikata-Project/parallel_write/branch/master/graph/badge.svg\n    :target: https://codecov.io/gh/Mikata-Project/parallel_write\n    :alt: Codecov\n\n.. image:: https://img.shields.io/readthedocs/parallel-write/latest?label=Read%20the%20Docs\n    :target: https://parallel-write.readthedocs.io/en/latest/index.html\n    :alt: Read the Docs\n\nSummary\n=======\n\nParallel write is a Python module for distributing writes between an arbitrary number of open\nfile(like) objects.\n\nFeatures:\n\n* Distributes each calls to the proxy object to each passed file objects, so all of them\n  should be in the same state\n* Writes are done in a configurable length thread pool, so you can have slower underlying\n  objects, their slowness won't add up\n* Compares results from the methods, so despite its name, you can actually read from many objects\n  at once and fail if any of them return different data\n\nMotivation\n==========\n\nWe often write the same data to local disk (for later caching) and remote (S3 for persistence).\nThe files must be the same, but the tool we're using may produce binary-different outputs for\ntwo subsequent writes (either because `PYTHONHASH` shuffles things or it includes time-stamps\ninto the compressed output's metadata, doesn't matter).\n\nWe could write the file locally first, then copy it to S3, but that would take more time and\ncomplexity in code. It's easier to write them at the same time.\n\nHow to use this module\n======================\n\nSee the `documentation`_.\n\n\nIssues and Discussions\n======================\n\nAs usual in any GitHub based project, raise an `issue`_ if you find any bug or room for\nimprovements.\n\nVersion\n=======\n\nv0.0.10\n\n.. _documentation: https://parallel-write.readthedocs.io/en/latest/\n.. _issue: https://github.com/Mikata-Project/parallel_write/issues\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmikata-project%2Fparallel_write","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmikata-project%2Fparallel_write","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmikata-project%2Fparallel_write/lists"}