{"id":18691052,"url":"https://github.com/ffissore/pipes","last_synced_at":"2026-03-27T04:51:57.023Z","repository":{"id":145833793,"uuid":"1140786","full_name":"ffissore/pipes","owner":"ffissore","description":"Pipes feed your threads","archived":false,"fork":false,"pushed_at":"2020-10-13T09:53:27.000Z","size":9,"stargazers_count":5,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-12T06:17:58.388Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ffissore.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2010-12-05T17:52:54.000Z","updated_at":"2020-10-13T09:53:22.000Z","dependencies_parsed_at":null,"dependency_job_id":"ed27729e-dcf4-42a6-970f-04b61eee39ea","html_url":"https://github.com/ffissore/pipes","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ffissore%2Fpipes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ffissore%2Fpipes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ffissore%2Fpipes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ffissore%2Fpipes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ffissore","download_url":"https://codeload.github.com/ffissore/pipes/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248525141,"owners_count":21118620,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T10:53:04.226Z","updated_at":"2026-03-27T04:51:56.990Z","avatar_url":"https://github.com/ffissore.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pipes\nPipes is an easy way to parallelize long running pieces of code that produce objects that other long running pieces of code will use as input.\n\nFor example, think about a web crawler, a parsing logic and a service layer.\nWhat you do at the beginning is a repetition of these three steps: crawl, parse, save, for each URL you have to crawl. You soon understand you are wasting time: you are using just one core of your quad-core server. These steps can run independently: the crawler will crawl and feed the parsing logic ASAP; the parsing logic, as new web pages can be parsed, will produce objects for the service layer.\n\n## What's in the name\n\n                         in-pipe          out-pipe   in-pipe          out-pipe\n    \u003e\u003e INCOMING DATA \u003e\u003e  ======= [ PUMP ] ======== - ======= | PUMP | ======== \u003e\u003e OUTGOING DATA \u003e\u003e\n                                               pipes link\n\nThe building block of Pipes is the **pipe**: each pipe has a **pump** in the middle. The pump draws data from the incoming part of the pipe, elaborating it, to the outgoing part.\nEach middle pipe has its outgoing part connected with the incoming part of the next pipe.\n\nYou can assemble the pipes on your own, or ask a **plumber** to do that for you: you need to give the plumber the beginning and ending parts of your pipes, and the collection of pumps that will elaborate/transform the data. The plumber will assemble the pipes and the pumps.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fffissore%2Fpipes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fffissore%2Fpipes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fffissore%2Fpipes/lists"}