{"id":18620551,"url":"https://github.com/simonsobs/bbpipe","last_synced_at":"2025-07-01T16:07:14.690Z","repository":{"id":38950414,"uuid":"156411648","full_name":"simonsobs/BBPipe","owner":"simonsobs","description":"B-modes pipeline constructor","archived":false,"fork":false,"pushed_at":"2023-05-10T10:30:48.000Z","size":15187,"stargazers_count":5,"open_issues_count":7,"forks_count":6,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-11T12:18:43.471Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simonsobs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-11-06T16:15:55.000Z","updated_at":"2023-12-25T06:49:25.000Z","dependencies_parsed_at":"2025-04-11T02:41:44.725Z","dependency_job_id":"13fdb9d3-a0f1-41bf-8d4f-dab49f8a1a51","html_url":"https://github.com/simonsobs/BBPipe","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/simonsobs/BBPipe","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2FBBPipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2FBBPipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2FBBPipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2FBBPipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simonsobs","download_url":"https://codeload.github.com/simonsobs/BBPipe/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2FBBPipe/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262996402,"owners_count":23396903,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T04:06:54.107Z","updated_at":"2025-07-01T16:07:14.646Z","avatar_url":"https://github.com/simonsobs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Building Beautiful pipelines - A pipeline constructor\n----------------------------\nA framework for creating pipelines for the Simons Observatory. Documentation is still under development, but details about how to use this package to create new pipelines can be found below and in [CONTRIBUTING.md](CONTRIBUTING.md). \n\n## Dependencies\n- python\u003c=3.9.x\n- numpy\n- healpy\n- ipython_genutils\n- pyyaml\n- parsl\u003c0.6.0\n\n## Installation\nTo install `BBPipe`, just clone this repository and run\n```bash\npython setup.py install\n```\n(add `--user` if you don't have permissions, which is probably the case at e.g. NERSC).\n\nOnce installed, you can test the installation by running it on the current [test power-spectrum pipeline](bbpower_test) (mostly made out of placeholders). To do so, type:\n```bash\nbbpipe test/test_ini.yml\n```\n\n## Creating a pipeline\nTo create a new pipeline, you must create its corresponding pipeline stages, and link them together through a [yaml](http://yaml.org/) file.\n\nCreating a new pipeline stage involves creating a python module. Note that this module doesn't have to live in this repo, it just has to be accessible by `bbpipe` when you run it. The new repo must:\n\n- Have an `__init__.py` file that imports from `.` all the stages used by your pipeline.\n- Have a `__main__.py` file with the same contents as those from the example `bbpower_test` [directory](bbpower_test).\n- Each stage is defined by a class which must inherit from `bbpipe.PipelineStage`. Each class must have its own `name`, `inputs` and `outputs` attributes (essentially the names of the expected input and output data), and a `run` method that executes the stage.\n- The `run` method should use the parent methods from `PipelineStage` to get its inputs and outputs etc.\n\nHave a look at the extended comments in [bbpower_test/mask_preproc.py](bbpower_test/mask_preproc.py) for more details on the structure of any pipeline stage.\n\nTo create the yaml file that puts your pipeline together, have a look at the [test file](test/test.yml). This file should contain:\n- A list of modules where the different pipeline stages are to be found.\n- The launcher type (to be used by PARSL to launch each stage). Currently the only defined launcher type is the `local` one (i.e. launch jobs serially in your machine), but more will be defined. They will be located in [`bbpipe/sites`](bbpipe/sites).\n- The list of stages that define your pipeline. Note that this list is not related to the order in which the different stages will be executed. This order is automatically determined from the inputs and outputs of each pipeline stage.\n- The overall inputs of the pipeline (accessible to all pipeline stages).\n- A path to another yaml file (`config`) containing configuration options for each individual pipeline stage as well as global options. Have a look at [`test/config.yml`](test/config.yml) to see an example for our test power spectrum pipeline.\n- A value for the `resume` parameter, which determines whether a given stage is run if its outputs already exist.\n- An output directory where the pipeline outputs will be stored.\n\n\n## Credit\n`BBPipe` is heavily inspired by `ceci`, a pipeline constructor designed within the LSST DESC by Joe Zuntz, Francois Lanusse and others.\n`BBPipe` uses [PARSL](http://parsl-project.org/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonsobs%2Fbbpipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimonsobs%2Fbbpipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonsobs%2Fbbpipe/lists"}