{"id":26111244,"url":"https://github.com/scisweeper/scisweeper","last_synced_at":"2026-04-29T00:04:43.058Z","repository":{"id":57464493,"uuid":"193897457","full_name":"scisweeper/scisweeper","owner":"scisweeper","description":"scisweeper - scientific parameter sweeper, with easy debugging capabilities and an interface for most scientific queuing systems.","archived":false,"fork":false,"pushed_at":"2023-05-18T15:19:08.000Z","size":91,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-08-09T06:36:52.845Z","etag":null,"topics":["lfs","moab","pandas","parameter","python","science","sge","slurm","sweeper","torque"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scisweeper.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-06-26T12:11:50.000Z","updated_at":"2020-01-24T07:58:27.000Z","dependencies_parsed_at":"2022-09-19T05:20:56.410Z","dependency_job_id":null,"html_url":"https://github.com/scisweeper/scisweeper","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scisweeper%2Fscisweeper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scisweeper%2Fscisweeper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scisweeper%2Fscisweeper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scisweeper%2Fscisweeper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scisweeper","download_url":"https://codeload.github.com/scisweeper/scisweeper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242774985,"owners_count":20183189,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lfs","moab","pandas","parameter","python","science","sge","slurm","sweeper","torque"],"created_at":"2025-03-10T01:49:34.259Z","updated_at":"2026-04-29T00:04:43.022Z","avatar_url":"https://github.com/scisweeper.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# scisweeper\nscisweeper - scientific parameter sweeper\n\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/caa562772b6944a2bcc3bd4d33ec62b0)](https://www.codacy.com/app/jan-janssen/scisweeper?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=scisweeper/scisweeper\u0026amp;utm_campaign=Badge_Grade)\n[![Build Status](https://travis-ci.org/scisweeper/scisweeper.svg?branch=master)](https://travis-ci.org/scisweeper/scisweeper)\n[![Coverage Status](https://coveralls.io/repos/github/scisweeper/scisweeper/badge.svg?branch=master)](https://coveralls.io/github/scisweeper/scisweeper?branch=master)\n[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/scisweeper/scisweeper/master?filepath=notebooks%2Fdemo.ipynb)\n\nscisweeper is a utility for parameter sweeps in a scientific environment. By defining the `write_input` and \n`collect_output` function, as well as a link to the `executable` the users can link scisweeper to their own code. \nAfterwards scisweeper can execute the parameter sweep either locally using a given number of parallel threads or submit \nthe individual calculations to a queuing system like `LFS`, `MOAB`, `SGE`, `SLURM`, or `TORQUE` using the built-in \ninterface to `pysqa`. After the calculations are finished the results are summarized in a `pandas.DataFrame`, which makes \nthem easily accessible for machine learning tools like `tensorflow` or `pytorch`. Coming from a scientific environment \nscisweeper was develop with a focus on debugging. Broken jobs can be identified, executed again manually or deleted and \nin case you need to update the `collect_output` function there is no need to execute the calculations again, the output \ncan be parsed separatly to update the results in the `pandas.DataFrame`. \n\n# Installation\nThe scisweeper can either be installed via pip using:\n\n    pip install scisweeper\n\nOr via anaconda from the conda-forge channel\n\n    conda install -c conda-forge scisweeper\n\n\n# Usage \nFor a simple executable like:\n\n    #!/bin/bash\n    awk '{ print $1+$2*$3 }1{print 1}' input_file \u003e output.log\n\nWhich is located in your home directory as `~/test.sh` a parameter sweep can be conducted using the following steps. \nStart with importing `scisweeper` : \n\n    from scisweeper import SciSweeper\n    ssw = SciSweeper()\n    \nDefine your `write_input` and `collect_output` function in a derived class. The important part about these `write_input` and `collect_output` function is that all the necessary import statements have to be nested within the function. This might be counterintuitive in the beginning, but is essential as it allows scisweeper to transfer the function to the compute node where the code is going to be executed: \n\n    class BashSciSweeper(ssw.job):\n        @property\n        def executable(self): \n            return 'bash ~/test.sh'\n        \n        @staticmethod\n        def write_input(input_dict, working_directory='.'):\n            import os \n            from jinja2 import Template\n            template = Template(\"{{value_1}} {{value_2}} {{value_3}}\")\n            template_str = template.render(value_1=input_dict['value_1'],\n                                           value_2=input_dict['value_2'],\n                                           value_3=input_dict['value_3'])\n            with open(os.path.join(working_directory, 'input_file'), 'w') as f:\n                f.writelines(template_str)\n    \n        @staticmethod\n        def collect_output(working_directory='.'):\n            import os \n            with open(os.path.join(working_directory, 'output.log'), 'r') as f:\n                output = f.readlines()\n            return {'result': int(output[1])}\n\nAnd apply this function to our `scisweeper` instance: \n\n    ssw.job_class = BashSciSweeper \n\nAfterwards we build a list of dictionaries with the input values we want to iterate over:\n\n    input_lst = []\n    for i in range(10):\n        for j in range(10):\n            for k in range(10):\n                input_lst.append({'value_1': i, 'value_2': j, 'value_3': k}) \n\nOptionally we can define a function to generate custom job names for the individual calculations. This can help \nidentifying broken calculation: \n\n    def job_name(input_dict, counter):\n        return 'job_' + str(counter) + '_' + str(input_dict['value_1'])\n    \n    ssw.job_name_function = job_name\n    \nThen we execute the caluclation in parallel using twenty cores:\n    \n    ssw.run_jobs_in_parallel(input_dict_lst=input_lst, cores=20)\n\nAnd collect the results: \n\n    ssw.collect()\n    ssw.results\n    \nIf we want to update our output parser, because we want to parse more quantities, we can do so by replacing the \ninterface:\n\n    class BashSciSweeper2(ssw.job):\n        @staticmethod\n        def collect_output(working_directory='.'):\n            with open(os.path.join(working_directory, 'output.log'), 'r') as f:\n                output = f.readlines()\n            return {'result': int(output[0])}\n            \nAnd repeat the steps from above:\n\n    ssw.job_class = BashSciSweeper2\n    ssw.run_collect_output()\n    ssw.collect()\n    ssw.results\n    \nOr we can identify broken calculations using: \n\n    ssw.broken_jobs\n    \nFor more information feel free to look at the unit tests and the example notebooks.\nhttps://github.com/scisweeper/scisweeper/blob/master/notebooks/demo.ipynb \n\n# Queuing system\nTo interface with the queuing system we use pysqa - https://github.com/pyiron/pysqa - which is constructed around the idea that even though modern queuing systems allow for an wide range of different configuration, most users submit the majority of their jobs with very similar parameters. Sample configurations for the specific queuing systems are availabe in the pysqa tests:\n\n* lsf - https://github.com/pyiron/pysqa/tree/master/tests/config/lsf\n* moab - https://github.com/pyiron/pysqa/tree/master/tests/config/moab\n* SGE - https://github.com/pyiron/pysqa/tree/master/tests/config/sge\n* slurm - https://github.com/pyiron/pysqa/tree/master/tests/config/slurm\n* torque - https://github.com/pyiron/pysqa/tree/master/tests/config/torque\n\n# License\nThe scisweeper is released under the BSD license https://github.com/scisweeper/scisweeper/blob/master/LICENSE . \nIt is a spin-off of the pyiron project https://github.com/pyiron/pyiron therefore if you use the scisweeper for your \npublication, please cite: \n\n    @article{pyiron-paper,\n      title = {pyiron: An integrated development environment for computational materials science},\n      journal = {Computational Materials Science},\n      volume = {163},\n      pages = {24 - 36},\n      year = {2019},\n      issn = {0927-0256},\n      doi = {https://doi.org/10.1016/j.commatsci.2018.07.043},\n      url = {http://www.sciencedirect.com/science/article/pii/S0927025618304786},\n      author = {Jan Janssen and Sudarsan Surendralal and Yury Lysogorskiy and Mira Todorova and Tilmann Hickel and Ralf Drautz and Jörg Neugebauer},\n      keywords = {Modelling workflow, Integrated development environment, Complex simulation protocols},\n    }\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscisweeper%2Fscisweeper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscisweeper%2Fscisweeper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscisweeper%2Fscisweeper/lists"}