{"id":30344947,"url":"https://github.com/dankinder/qpipe","last_synced_at":"2025-08-18T13:25:58.606Z","repository":{"id":57459034,"uuid":"56812493","full_name":"dankinder/qpipe","owner":"dankinder","description":"A python library for simple flow-based multiprocessing","archived":false,"fork":false,"pushed_at":"2016-04-26T15:27:14.000Z","size":28,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-25T12:53:06.413Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dankinder.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-04-21T23:50:12.000Z","updated_at":"2022-08-11T17:36:05.000Z","dependencies_parsed_at":"2022-08-27T21:00:17.782Z","dependency_job_id":null,"html_url":"https://github.com/dankinder/qpipe","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/dankinder/qpipe","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dankinder%2Fqpipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dankinder%2Fqpipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dankinder%2Fqpipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dankinder%2Fqpipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dankinder","download_url":"https://codeload.github.com/dankinder/qpipe/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dankinder%2Fqpipe/sbom","scorecard":{"id":321054,"data":{"date":"2025-08-11","repo":{"name":"github.com/dankinder/qpipe","commit":"b115f6302074d78f300de19a9bfc81d5a085220c"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":0,"reason":"Found 0/9 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-18T01:22:28.070Z","repository_id":57459034,"created_at":"2025-08-18T01:22:28.070Z","updated_at":"2025-08-18T01:22:28.070Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270999783,"owners_count":24682520,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-18T02:00:08.743Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-18T13:25:54.694Z","updated_at":"2025-08-18T13:25:58.589Z","avatar_url":"https://github.com/dankinder.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# QPipe\n\nQPipe (\"Quick-Pipe\", or \"Queue-Pipe\") is a syntatically clean and generic way\nof writing concurrent programs using a flow model (like a graph of pipes).\nThere are currently many ways of writing [Flow Based\nPrograms](https://wiki.python.org/moin/FlowBasedProgramming) in python, but\nnone seemed to have the simplicity, elegance, and generlizeability that QPipe\ndoes (the closest probably being\n[pipedream](https://github.com/tgecho/pipedream/) and maybe\n[pypes](https://bitbucket.org/diji/pypes/wiki/Home)).\n\n```bash\nsudo pip install qpipe\n```\n\n## Overview\n\nOne of the simplest ways to parallelize in python is the Pool class:\n\n```python\nfrom multiprocessing import Pool\n\ndef f(x):\n    return x*x\n\nif __name__ == '__main__':\n    p = Pool(5)\n    print(p.map(f, [1, 2, 3]))\n```\n\nThis is simple but inflexible. It becomes orders of magnitude more complicated\nto stream data while processing or inter-communicate with other distinct\nprocesses that may be performing related tasks.\n\nImplemented instead with the qpipe library:\n\n```python\nfrom qpipe import Pipe, Fn\n\ndef f(x):\n    return x*x\n\nif __name__ == '__main__':\n    print(Iter([10, 20, 30]).into(Fn(f, processes=4)).results())\n```\n\nHere is the implementation of a more complicated pipeline using qpipe, the\npython equivalent of `tail -f myapp.log | grep error`:\n\n```python\nimport re\nfrom qpipe import Pipe, Print\n\nclass Tail(Pipe):\n    def setup(self, filename):\n        for line in sh.tail(\"-f\", filename, _iter=True):\n            self.emit(line)\n\nclass Grep(Pipe):\n    def setup(self, grepstring):\n        self.regex = re.compile(grepstring)\n    def do(self, text):\n        if(self.regex.search(text)):\n            self.emit(text)\n\nif __name__ == '__main__':\n    Tail(\"myapp.log\").into(Grep(\"error\")).into(Print()).execute()\n```\n\n## API\n\n### Using Pipes\n\nConstructing any qpipe builds a pipeline. Use `into` to attach multiple\ncomponents together.\n\n```python\nOpen(\"myfile.txt\").into(Print())\n```\n\nNothing happens until the pipeline is executed.\n\n```python\n# Starts printing lines from myfile.txt\nOpen(\"myfile.txt\").into(Print()).execute()\n```\n\nPipes like Open can start generating output using just constructor\narguments, but others (like Print) won't do anything unless values are piped\ninto them. Passing a list (or any iterable) to the provided Iter qpipe works\nfor this.\n\n```python\n# Prints the list contents on separate lines\nIter([\"hello world\", 4, []]).into(Print()).execute()\n```\n\nPipes can easily be parallelized by specifying a `processes` keyword\nargument.\n\n```python\n# Runs the \"complex math\" in 4 processes\nIter(range(10000)).into(SomeComplexMath(processes=4)).into(Print()).execute()\n```\n\nPipelines can be started with the following calls:\n\n- `execute()`: start processing, block until completion, return nothing.\n- `start()`: start processing, do not block, return nothing.\n- `results()`: start processing (if not started already), block until completion, return results as a list.\n\nSee the [examples](examples) for a few more qpipe program examples.\n\n### Developing Custom Pipes\n\nTo create your own qpipe, subclass Pipe and override any of these methods:\n- `setup`: called once per process at start. Constructor arguments passed to\nyour qpipe that aren't used by the Pipe superclass will be passed along\nto this method.\n- `do`: called once for each input value.\n- `teardown`: called once per process at the end.\n\nNote that all of these are called on the processing thread. If you run a\nPipe with multiple processes then each one will call setup/teardown.\nInstance variables are not shared.\n\nWithin your code, to send values to the next pipe (or results), call\n`self.emit(value)`. `setup`, `do`, and `teardown` can all call `emit` 0 or\nmore times.\n\n### Other relevant API calls\n\nThe backend qpipe uses determines how tasks are parallelized. The current choices are:\n- `Backend.MULTIPROCESSING`: the default backend. The pipes use\n  [multiprocessing.Process](https://docs.python.org/2/library/multiprocessing.html#multiprocessing.Process)\n  to execute tasks. If the `processes=X` keyword argument is used this causes\n  that pipe to have X processes executing tasks at once.\n- `Backend.THREADING`: similar to MULTIPROCESSING, but instead uses\n  [threading.Thread](https://docs.python.org/2/library/threading.html#threading.Thread)\n  objects. The amount of parallelism with this backend will be limited by\n  python's Global Interpreter Lock (only a problem for CPU-intensive tasks),\n  but does not have to serialize(pickle) data like the MULTIPROCESSING backend does.\n- `Backend.DUMMY`: a simple backend for testing pipelines, not concurrent. It\n  simply executes each stage of the pipeline and completes it before moving on\n  to the next one.\n\nThese can be checked and set using the following functions. This setting is\ncurrently global and should not be changed while pipeline is executing.\n- `set_backend(Backend.THREADING)`\n- `get_backend()`\n- `is_backend(Backend.MULTIPROCESSING)`\n\n## A word of caution\n\nI have created this library because I have found myself wanting to write\nconcurrent python many times but *not* wanting to deal with the complexity.\n\nThis implementation does work, but isn't very robust. It doesn't have any fancy\nfeatures like multiple inputs and outputs (multiplexing), pipelines cannot be\nmodified during execution, it hasn't been carefully tested for long-running\ncases like listening for messages on a queue or connections on a port, and\nexceptions can easily cause the pipeline to never complete. I would love to see\nothers be inspired by the API and contribute or implement it more robustly than\nI have. Though bear in mind: the lack of fancy features is also deliberate.\nQPipe is intended to be a library optimized for human understanding. This\nmeans keeping complexity down.\n\nAll this to say: this software should be considered in alpha and the API could\neasily change, especially of the tools in [tools.py](qpipe/tools.py).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdankinder%2Fqpipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdankinder%2Fqpipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdankinder%2Fqpipe/lists"}