{"id":37080989,"url":"https://github.com/clowdr/clowdr","last_synced_at":"2026-01-14T09:50:07.634Z","repository":{"id":57417330,"uuid":"121551982","full_name":"clowdr/clowdr","owner":"clowdr","description":"Command-line utility for iteratively developing pipelines, deploying them at scale, and sharing data and derivatives","archived":false,"fork":false,"pushed_at":"2020-06-15T13:43:50.000Z","size":838,"stargazers_count":10,"open_issues_count":10,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2026-01-05T15:54:05.102Z","etag":null,"topics":["data-science","docker","hpc-applications","pipelines","python","singularity"],"latest_commit_sha":null,"homepage":"https://clowdr.github.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/clowdr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-02-14T19:27:35.000Z","updated_at":"2022-09-25T22:58:07.000Z","dependencies_parsed_at":"2022-08-30T04:11:01.724Z","dependency_job_id":null,"html_url":"https://github.com/clowdr/clowdr","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/clowdr/clowdr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clowdr%2Fclowdr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clowdr%2Fclowdr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clowdr%2Fclowdr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clowdr%2Fclowdr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/clowdr","download_url":"https://codeload.github.com/clowdr/clowdr/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clowdr%2Fclowdr/sbom","scorecard":{"id":293494,"data":{"date":"2025-08-11","repo":{"name":"github.com/clowdr/clowdr","commit":"346263ee806ae7c992a37dfc9bd9b87db3fa48c3"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":1.8,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Code-Review","score":0,"reason":"Found 0/5 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: Dockerfile:1: pin your Docker image by updating alpine:edge to alpine:edge@sha256:115729ec5cb049ba6359c3ab005ac742012d92bbaa5b8bc1a878f1e8f62c0cb8","Warn: pipCommand not pinned by hash: Dockerfile:7","Warn: pipCommand not pinned by hash: Dockerfile:7","Warn: pipCommand not pinned by hash: Dockerfile:10","Warn: pipCommand not pinned by hash: Dockerfile:11","Info:   0 out of   1 containerImage dependencies pinned","Info:   0 out of   4 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":2,"reason":"8 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2024-35 / GHSA-547x-748v-vp6p","Warn: Project is vulnerable to: PYSEC-2021-856 / GHSA-5545-2q6w-2gh6","Warn: Project is vulnerable to: GHSA-6p56-wp2h-9hxr","Warn: Project is vulnerable to: PYSEC-2019-108 / GHSA-9fq2-x9r6-wfmf","Warn: Project is vulnerable to: PYSEC-2021-857 / GHSA-f7c7-j99h-c22f","Warn: Project is vulnerable to: GHSA-fpfv-jqm9-f5jm","Warn: Project is vulnerable to: PYSEC-2020-73","Warn: Project is vulnerable to: PYSEC-2019-41 / GHSA-qfc5-mcwq-26q8"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 30 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-17T18:54:06.224Z","repository_id":57417330,"created_at":"2025-08-17T18:54:06.224Z","updated_at":"2025-08-17T18:54:06.224Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28416120,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T08:38:59.149Z","status":"ssl_error","status_checked_at":"2026-01-14T08:38:43.588Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","docker","hpc-applications","pipelines","python","singularity"],"created_at":"2026-01-14T09:50:06.846Z","updated_at":"2026-01-14T09:50:07.629Z","avatar_url":"https://github.com/clowdr.png","language":"Python","readme":"# Clowdr\n\nClowd is a command-line utility for iteratively developing pipelines, deploying them at scale, and sharing data and derivatives.\n\n[![](https://img.shields.io/pypi/v/clowdr.svg)](https://pypi.python.org/pypi/clowdr)\n[![Build Status](https://travis-ci.org/clowdr/clowdr.svg?branch=master)](https://travis-ci.org/clowdr/clowdr)\n[![Coverage Status](https://coveralls.io/repos/github/clowdr/clowdr/badge.svg?branch=master)](https://coveralls.io/github/clowdr/clowdr?branch=master)\n[![Documentation Status](https://readthedocs.org/projects/clowdr/badge/?version=latest)](https://clowdr.readthedocs.io/en/latest/?badge=latest)\n[![DOI](https://zenodo.org/badge/121551982.svg)](https://zenodo.org/badge/latestdoi/121551982)\n\n## Contents\n\n- [Overview](#overview)\n- [System Requirements](#system-requirements)\n  - [Installation Instructions](#installation-instructions)\n  - [Docker](#docker)\n  - [Singularity](#singularity)\n- [Usage](#usage)\n  - [Local](#local)\n  - [Cluster](#cluster)\n  - [Cloud](#cloud)\n  - [Share](#share)\n- [Documentation](#documentation)\n- [License](#license)\n- [Issues](#issues)\n\n## Overview\n*Clowdr* can be thought of as a cloud execution utility for [Boutiques](http://boutiques.github.io), the JSON-based\ndescriptive command-line framework. As Boutiques and the [Boutiques tools](https://github.com/boutiques/boutiques) allow\nthe encapsulation, validation, evaluation, and deployment of command-line routines, *Clowdr* inherits and extends this \nfunctionality to remote datasets and computational resources.\n\n*Clowdr* exposees several levels of evaluation: `local`, `cluster`, `cloud`, and `share`. The `local` runs tasks using the\nsystem scheduler, and paired with the `-dev` flag can enable the rapid prototyping of tools, descriptors, and invocations.\nThe `cluster` mode generates the exact same executions as in `local` but submits them through a cluster's scheduler for parallel\nexecution. Similarly, `cloud` runs the tasks on a remote cloud such as Amazon. Finally, the `share` mode launches a light-weight\nwebserver, ultimately generating a static HTML page which can be stored and redistributed that documents provenance and run\ninformation for the launched tasks.\n\n## System Requirements\nClowdr requires Python3 and either Docker or Singularity. It has only been tested on Mac OSX and Linux, though no requirements\nare specific to these operating systems and suggest that it may also function properly on Windows.\n\n### Installation Instructions\nInstallation is quite simple - just run:\n\n```\npip install clowdr\n```\n\n### Docker\nClowdr is available on Docker Hub, and can be downloaded with:\n\n```\ndocker pull clowdr/clowdr\n```\n\n### Singularity\nClowdr is also available on Singularity Hub, and can be downloaded with:\n\n```\nsingularity pull clowdr/clowdr\n```\n\n## Usage\n(*For up to date command-lines please check out our [documentation](https://clowdr.rtfd.io)*)\n\nBelow we'll explore each of the main three modes of operation for Clowdr. If in doubt, always feel free to turn back to the help-text:\n\n```\nclowdr -h\n```\n\n### Local\nFrom this directory, assuming the [BIDS dataset `ds114`](https://github.com/INCF/BIDS-examples) is installed at `/data/ds114`, and\nyour system has Docker installed, run:\n\n```\nclowdr local examples/descriptor_d.json examples/invocation.json examples/task/ /data/ds114/ -v /data/ds114/:/data/ds114 -b\n```\n\nWhat you just did was launched `clowdr` in `local` mode, with the tool `examples/descriptor_d.json`, invocation at `examples/invocation.json`,\nwhere outputs will be stored in the Clowdr directory, `examples/task`, from data stored at `/data/ds114`, and being mounted to that same container\nlocation, `-v /data/ds114:/data/ds114`, that happens to be organized according to the BIDS specification, `-b`. If you also wanted verbose output,\n`-V`, or to develop, `-d`, as well as some other options, there are flags that can be discovered with the help flag, `-h`.\n\nIf the data wasn't organized in BIDS format, we could provide a directory of invocations in place of `examples/invocation.json`, or of course a\nsingle invocation and omit the `-b` flag in both cases to run either a group of tasks or single task, respectively.\n\nYou can now look in the Clowdr directory to see the outputs of this pipeline.\n\n### Cluster\nIF you want to scale up your analysis, you can then turn to the cluster mode. The arguments supplied are exactly the same, with some minor\nadditions such as adding the hostname to your data location, specifying your cluster type, here `slurm`, and your account identifier, job\nnames, etc.\n\n```\nclowdr cluster ./examples/descriptor_s.json ./examples/invocation.json ./examples/task/ server.hostname.ca:/path/to/data/ slurm -v /path/to/data/:/data/ --account my-account-id --jobname clowdr-taskname -b\n```\n\nThe execution takes place here exactly like in the local mode, where here we specified a singularity version of the descriptor. Flags such as\n`-d` for development/single-execution mode also are consistent in this mode and helpful for prototyping analyses prior to large executions.\n\n\n### Cloud\nPresuming you ran locally and were happy with the results, but have larger collections of data you'd like to process, and don't have access to a\ncluster, you can turn to the cloud. If you've uploaded the same dataset to Amazon Web Services S3 at `s3://mybucket/ds114/`, and have your\ncredentials stored in this directory at `credentials.csv`, run:\n\n```\nclowdr cloud examples/descriptor_d.json examples/invocation.json s3://mybucket/clowdr/ s3://mybucket/ds114/ aws credentials.csv -bv -r us-east-1\n```\n\nHere, you also did the same as above, except in `cloud` mode, with remote data on S3, specifying the Amazon endpoint, `aws`, and setting your Amazon\nregion to `us-east-1`.\n\n### Share\nOnce Clowdr tasks are launched, they will return a directory which will be home to the output task information - either on Amazon S3 or local, depending\non the parameters provided. The share mode allows you to quickly inspect and explore the launched tasks, give updates on their status, and ultimately\nprovides a static HTML page which can be downloaded and shared with the processed derivatives as provenance information about the execution. You can\npoint the share service at either your Clowdr output directory, or in the case of an example packaged with the repository, the line below:\n\n```\nclowdr share ./examples/task/bids-example/clowdr/ -d\n```\n\n## Documentation\nFor detailed and up-to-date documentation, check out our read-the-docs page, at [clowdr.rtfd.io](http://clowdr.rtfd.io).\n\n## License\nThis project is covered under the [MIT License](https://github.com/clowdr/clowdr/blob/master/LICENSE).\n\n## Issues\nIf you're having trouble, notice a bug, or want to contribute (such as a fix to the bug you may have just found) feel free to open a\n[git issue](https://github.com/clowdr/clowdr/issues/new) or pull request. Enjoy!\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclowdr%2Fclowdr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclowdr%2Fclowdr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclowdr%2Fclowdr/lists"}