{"id":23442327,"url":"https://github.com/repronim/reproman","last_synced_at":"2025-08-28T16:51:03.446Z","repository":{"id":10800262,"uuid":"57322104","full_name":"ReproNim/reproman","owner":"ReproNim","description":"ReproMan (AKA NICEMAN, AKA ReproNim TRD3)","archived":false,"fork":false,"pushed_at":"2023-06-16T15:45:37.000Z","size":3395,"stargazers_count":24,"open_issues_count":156,"forks_count":14,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-03-27T02:21:25.311Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://reproman.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ReproNim.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2016-04-28T17:41:42.000Z","updated_at":"2023-10-25T15:46:02.000Z","dependencies_parsed_at":"2023-02-11T01:45:33.417Z","dependency_job_id":"5c3b242b-9718-41c4-9ec0-6961308524e4","html_url":"https://github.com/ReproNim/reproman","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReproNim%2Freproman","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReproNim%2Freproman/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReproNim%2Freproman/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReproNim%2Freproman/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ReproNim","download_url":"https://codeload.github.com/ReproNim/reproman/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248703198,"owners_count":21148118,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-23T17:29:24.923Z","updated_at":"2025-08-28T16:51:03.435Z","avatar_url":"https://github.com/ReproNim.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ReproMan\n\n[![Supports python version](https://img.shields.io/pypi/pyversions/datalad)](https://pypi.org/project/datalad/)\n[![GitHub release](https://img.shields.io/github/release/ReproNim/reproman.svg)](https://GitHub.com/ReproNim/reproman/releases/)\n[![PyPI version fury.io](https://badge.fury.io/py/reproman.svg)](https://pypi.python.org/pypi/reproman/)\n[![Tests](https://github.com/ReproNim/reproman/workflows/Tests/badge.svg)](https://github.com/ReproNim/reproman/actions?query=workflow%3ATests)\n[![codecov.io](https://codecov.io/github/ReproNim/reproman/coverage.svg?branch=master)](https://codecov.io/github/ReproNim/reproman?branch=master)\n[![Documentation](https://readthedocs.org/projects/reproman/badge/?version=latest)](https://reproman.readthedocs.io/en/latest/?badge=latest)\n\n\nReproMan aims to simplify creation and management of computing environments\nin Neuroimaging.  While concentrating on Neuroimaging use-cases, it is\nby no means is limited to this field of science and tools will find\nutility in other fields as well.\n\n# Status\n\nReproMan is under rapid development. While\nthe code base is still growing the focus is increasingly shifting towards\nrobust and safe operation with a sensible API. There has been no major public\nrelease yet, as organization and configuration are still subject of\nconsiderable reorganization and standardization. \n\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) if you are interested in\ninternals and/or contributing to the project.\n\n# Installation\n\nReproMan requires Python 3 (\u003e= 3.9).\n\n## Linux'es and OSX (Windows yet TODO) - via pip\n\nBy default, installation via pip (`pip install reproman`) installs core functionality of reproman\nallowing for managing datasets etc.  Additional installation schemes\nare available, so you could provide enhanced installation via\n`pip install 'reproman[SCHEME]'` where `SCHEME` could be\n\n- tests\n     to also install dependencies used by unit-tests battery of the reproman\n- full\n     to install all of possible dependencies, e.g. [DataLad](http://datalad.org)\n\nFor installation through `pip` you would need some external dependencies\nnot shipped from it (e.g. `docker`, `singularity`, etc.) for which please refer to\nthe next section.  \n\n## Debian-based systems\n\nOn Debian-based systems we recommend to enable [NeuroDebian](http://neuro.debian.net)\nfrom which we will soon provide recent releases of ReproMan.  We will also provide backports of\nall necessary packages from that repository.\n\n\n## Dependencies\n\nPython 3.9+ **with header files** possibly needed to build some extensions without wheels. They are provided by `python3-dev` on\ndebian-based systems or `python-devel` on Red Hat systems.\n\nOur `setup.py` and corresponding packaging describes all necessary python dependencies.\nOn Debian-based systems we recommend to enable [NeuroDebian](http://neuro.debian.net)\nsince we use it to provide backports of recent fixed external modules we\ndepend upon.  Additionally, if you would\nlike to develop and run our tests battery see [CONTRIBUTING.md](CONTRIBUTING.md)\nregarding additional dependencies.\n\n# A typical workflow for `reproman run`\n\nThis example is heavily based on the [\"Typical workflow\"](https://github.com/ReproNim/containers/#a-typical-workflow)\nexample created for [///repronim/containers](https://github.com/ReproNim/containers/)\nwhich we refer you to discover more about YODA principles etc.  In this reproman example we will\nfollow exactly the same goal -- running MRIQC on a sample dataset -- but this time utilizing\nReproMan's ability to run computation remotely. DataLad and `///repronim/containers` will\nstill be used for data and containers logistics, while reproman will establish a little [HTCondor](https://research.cs.wisc.edu/htcondor/)\ncluster in the AWS cloud, run the analysis, and fetch the results.\n\n## Step 1: Create the HTCondor AWS EC2 cluster\n\nIf it is the first time you are using ReproMan to interact with AWS cloud services, you should first provide\nReproMan with secret credentials to interact with AWS. For that edit its configuration file\n(`~/.config/reproman/reproman.cfg` on Linux, `~/Library/Application Support/reproman/reproman.cfg` on OSX)\n\n    [aws]\n    access_key_id = ...\n    secret_access_key = ...\n\n**Disclaimer/Warning: Never share or post those secrets publicly.**\n\nfilling out the `...`s.  If `reproman` fails to find this information, error message `Unable to locate credentials` will appear.\n\nRun (need to be done once, makes resource available for `reproman login` or `reproman run`):\n\n```shell\nreproman create aws-hpc2 -t aws-condor -b size=2 -b instance_type=t2.medium\n```\nto create a new ReproMan resource: 2 AWS EC2 instances, with HTCondor installed (we use [NITRC-CE](https://www.nitrc.org/projects/nitrc_es/) instances).\n\n**Disclaimer/Warning: It is important to monitor your cloud resources in the cloud provider dashboard(s)\nto ensure absent run away instances etc. to help avoid incuring heavy cost for used cloud services.**\n\n## Step 2: Create analysis DataLad dataset and run computation on aws-hpc2\n\nFollowing script is an exact replica from [///repronim/containers](https://github.com/ReproNim/containers/#a-typical-workflow)\nwhere only the `datalad containers-run` command, which fetches data locally and runs computation locally and serially, is replaced with\n`reproman run` which publishes dataset (without data) to the remote resource, fetches the data, runs computation\nvia HTCondor in parallel across 2 nodes, and then fetches results back:\n\n```shell\n#!/bin/sh\n(  # so it could be just copy pasted or used as a script\nPS4='\u003e '; set -xeu  # to see what we are doing and exit upon error\n# Work in some temporary directory\ncd $(mktemp -d ${TMPDIR:-/tmp}/repro-XXXXXXX)\n# Create a dataset to contain mriqc output\ndatalad create -d ds000003-qc -c text2git\ncd ds000003-qc\n# Install our containers collection:\ndatalad install -d . ///repronim/containers\n# (optionally) Freeze container of interest to the specific version desired\n# to facilitate reproducibility of some older results\ndatalad run -m \"Downgrade/Freeze mriqc container version\" \\\n    containers/scripts/freeze_versions bids-mriqc=0.16.0\n# Install input data:\ndatalad install -d . -s https://github.com/ReproNim/ds000003-demo sourcedata\n# Setup git to ignore workdir to be used by pipelines\necho \"workdir/\" \u003e .gitignore \u0026\u0026 datalad save -m \"Ignore workdir\" .gitignore\n# Execute desired preprocessing in parallel across two subjects\n# on remote AWS EC2 cluster, creating a provenance record\n# in git history containing all condor submission scripts and logs, and\n# fetching them locally\nreproman run -r aws-hpc2 \\\n   --sub condor --orc datalad-pair \\\n   --jp \"container=containers/bids-mriqc\" --bp subj=02,13 --follow \\\n   --input 'sourcedata/sub-{p[subj]}' \\\n   --output . \\\n   '{inputs}' . participant group -w workdir --participant_label '{p[subj]}'\n)\n```\n[ReproMan: Execute](https://reproman.readthedocs.io/en/latest/execute.html) documentation section\nprovides more information on the underlying principles behind [`reproman run`](https://reproman.readthedocs.io/en/latest/generated/man/reproman-run.html)\ncommand.\n\n## Step 3: Remove resource\n\nWhenever everything is computed and fetched, and you are satisfied with the results, use `reproman delete aws-hpc2` to terminate\nremote cluster in AWS, to not cause unnecessary charges.\n\n# License\n\nMIT/Expat\n\n\n# Disclaimer\n\nIt is in a beta stage -- majority of the functionality is usable but\nDocumentation and API enhancements is WiP to make it better.  Please do not be\nshy of filing an issue or a pull request. See [CONTRIBUTING.md](CONTRIBUTING.md)\nfor the guidance.\n\n[Git]: https://git-scm.com\n[Git-annex]: http://git-annex.branchable.com\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frepronim%2Freproman","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frepronim%2Freproman","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frepronim%2Freproman/lists"}