{"id":30416875,"url":"https://github.com/kirilenkobm/py_nf","last_synced_at":"2025-08-22T05:10:37.440Z","repository":{"id":49388530,"uuid":"295126008","full_name":"kirilenkobm/py_nf","owner":"kirilenkobm","description":"Python library to execute batches of jobs using nextflow.","archived":false,"fork":false,"pushed_at":"2022-09-06T18:34:44.000Z","size":51,"stargazers_count":5,"open_issues_count":2,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-08-06T17:19:27.027Z","etag":null,"topics":["cluster","controls-nextflow","nextflow","nextflow-executor","nextflow-pipeline","nextflow-script","nextflow-wrapper","parallel-computing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kirilenkobm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-09-13T10:17:47.000Z","updated_at":"2023-08-09T09:04:44.000Z","dependencies_parsed_at":"2022-09-04T16:10:57.549Z","dependency_job_id":null,"html_url":"https://github.com/kirilenkobm/py_nf","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/kirilenkobm/py_nf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kirilenkobm%2Fpy_nf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kirilenkobm%2Fpy_nf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kirilenkobm%2Fpy_nf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kirilenkobm%2Fpy_nf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kirilenkobm","download_url":"https://codeload.github.com/kirilenkobm/py_nf/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kirilenkobm%2Fpy_nf/sbom","scorecard":{"id":561230,"data":{"date":"2025-08-11","repo":{"name":"github.com/kirilenkobm/py_nf","commit":"0cf9cb92c846a25f2fa6d74f85b4e54b0a83b79f"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.6,"checks":[{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Pinned-Dependencies","score":10,"reason":"all dependencies are pinned","details":["Info:   1 out of   1 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-20T13:45:35.210Z","repository_id":49388530,"created_at":"2025-08-20T13:45:35.210Z","updated_at":"2025-08-20T13:45:35.210Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271588743,"owners_count":24785751,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-22T02:00:08.480Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cluster","controls-nextflow","nextflow","nextflow-executor","nextflow-pipeline","nextflow-script","nextflow-wrapper","parallel-computing"],"created_at":"2025-08-22T05:10:31.764Z","updated_at":"2025-08-22T05:10:37.430Z","avatar_url":"https://github.com/kirilenkobm.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# py_nf library\n\nPython library to run batches of jobs using Nextflow.\n\nIf you write computational workflows in Python and need Nextflow only\nas an abstraction layer to execute batches of jobs, most likely py_nf is\nwhat you are looking for.\n\nPy_nf is a wrapper around nextflow: you provide a job list and batch parameters,\nthen py_nf creates and executes a temporary Nextflow pipeline.\n\n## Installation\n\n### Nextflow\nFirst of all install Nextflow and make sure you can call it.\nPlease see [nextflow.io](https://nextflow.io) for details.\n\nMost likely one of these commands will help:\n\n```shell script\ncurl -fsSL https://get.nextflow.io | bash\n# OR\nconda install -c bioconda nextflow\n```\n\nAlternatively, you can use utils.install_nf_if_not_installed function:\n\n```python\nfrom py_nf.utils import install_nf_if_not_installed\n# nf_executable -\u003e absolute path to Nextflow executable\nnf_executable = install_nf_if_not_installed()\n```\n\n#### Nextflow install troubleshooting\n\nIf you work on MacOS and nextflow, being installed with conda, crashes with the following message:\n\n```\nThe operation couldn’t be completed. Unable to locate a Java Runtime.\nPlease visit http://www.java.com for information on installing Java.\n\n/Users/XXX/opt/anaconda3/bin/nextflow: line 284: /bin/java: No such file or directory\nNOTE: Nextflow is trying to use the Java VM defined by the following environment variables:\n JAVA_CMD: /bin/java\n NXF_OPTS: \n```\n\nPlease follow the recipe [here](https://github.com/nextflow-io/nextflow/issues/1523). Nextflow has a pretty sophisticated procedure to identify Java runtime. Commenting a couple of lines in this procedure may help. \n\n### Py_nf library\n\nTo install py_nf just do:\n\n```shell script\npip3 install py_nf\n```\n\nIf it doesn't work please add --user flag.\n\n## Usage\n\nBasic usage scenario looks like this:\n\n```python\n# import py_nf library:\nfrom py_nf.py_nf import Nextflow\n\n# initiate nextflow handler\n# here you can define parameters such as executor, amount of memory per job\n# and so on, see Nextflow class parameters section for details\nnf = Nextflow(executor=\"local\", project_name=\"project\", cpus=4)\n\n# then generate a joblist\n# this must be a list (or other iterable such as tuple or even a generator) of strings\n# where each string is a command\njoblist = [\"script.py in/1.txt out/1.txt\",\n           \"script.py in/2.txt out/2.txt\"\n           \"script.py in/3.txt out/3.txt\"\n           \"script.py in/4.txt out/4.txt\"\n           \"script.py in/5.txt out/5.txt\"]\n\n# then execute this joblist using Nextflow\nstatus = nf.execute(joblist)\n\n# look at return status, there are 2 cases: either 0 or something else\nif status == 0:\n    # 0 means that Nextflow pipeline was executed without errors\n    # enjoy your results\n    pass\nelse:\n    # sadly, the pipeline failed\n    # py_nf doesn't terminate the program in this case to let user\n    # to do some cleanup, for example\n    # please read nextflow output messages and logs to figure out\n    # what exactly happened\n    # do_some_cleanup()\n    exit(1)\n```\n\nImportant:\n\nplease use absolute pathways in your commands!\n\nYou can use utils.paths_to_abspaths_in_joblist function:\n\n```python\nfrom py_nf.utils import paths_to_abspaths_in_joblist\n\njoblist = [\"script.py in/1.txt out/1.txt -p\",\n           \"script.py in/2.txt out/2.txt -p\"\n           \"script.py in/3.txt out/3.txt -p\"\n           \"script.py in/4.txt out/4.txt -p\"\n           \"script.py in/5.txt out/5.txt -p\"]\n\nabs_path_joblist = paths_to_abspaths_in_joblist(joblist)\n```\n\nThis function will search for file or directory paths in your batch and\nthen replace them with absolute paths.\n\nFor example, this command:\n\n```text\nscript.py in/1.txt out/1.txt -p\n```\n\nWill be replaced with something like this:\n\n```text\n/home/user/proj/script.py /home/user/proj/in/1.txt /home/user/proj/out/1.txt -p\n```\n\n### Read more about nextflow executors\n\nNextflow supports a wide range of cluster schedulers, please read about them and\nacceptable parameters [here](https://www.nextflow.io/docs/latest/executor.html).\n\n### Nextflow class parameters\n\nYou can initiate Nextflow() class with the parameters listed here.\nMost of these options reproduce Nextflow process parameters, you can read\nabout them in the [documentation](https://www.nextflow.io/docs/latest/process.html).\n\n1) *nextflow_executable*, \"nextflow\" is default.\nDefine the path to nextflow executable you like to use:\nnf = Nextflow(nextflow_executable=\"/home/user/nf_v20/nextflow\").\nMake sure you have the nextflow executable in a directory accesable \n2) *executor*, \"local\" is default.\nPlease read more about Nextflow executor in the corresponding section.\nTo use \"slurm\" executor do the following:\nnf = Nextflow(executor=\"slurm\")\n3) *error_strategy*, \"retry\" is default.\nYou can define errorStrategy nextflow parameter.\nPlease look for available errorStrategy options\n[here](https://www.nextflow.io/docs/latest/process.html#errorstrategy).\nUsage example:\nnf = Nextflow(error_strategy=\"ignore\")\n4) *max_retries*, default 3.\nControls Nextflow [maxRetries parameter](https://www.nextflow.io/docs/latest/process.html#maxretries).\nnf = Nextflow(max_retries=5).\n5) *queue*, default \"batch\".\nControls Nextflow process queue parameter.\nTo set \"long\" queue do:\nnf = Nextflow(queue=\"long\").\nA list of available queues depends on your scheduler.\n6) *memory*, default \"10\".\nAmount of memory each process is allowed to use.\nPlease find [here](https://www.nextflow.io/docs/latest/process.html#memory)\nhow to format the memory amount.\nTo set \"memory\" parameter to 100 do the following:\nnf = Nextflow(memory=\"100\", memory_units=\"GB\").\n7) *memory_units*, default \"GB\"\nThe parameter controls memory parameter units.\nFor example, with --memory 3500 and --time_units MB the process will be limited to 3500 megabytes.\nThe full list of available memory units is available at [here](https://www.nextflow.io/docs/latest/process.html#memory)\n8) *time*, default \"1\".\nThis parameter controls how long a process is allowed to run.\nPlease read formatting rules [here](https://www.nextflow.io/docs/latest/process.html#time).\nTo set the time limit to 1 day:\nnf = Nextflow(time=\"1d\").\n9) *time_units*, default \"h\"\nThis parametes allows to specify the time value units.\nFor example, with --time 11 and --time_units m the process will be limited to 11 minutes.\nThe full list of available time units is available at [here](https://www.nextflow.io/docs/latest/process.html#time)\n10) *cpus*, default 1.\nControls the number of CPUs required by each job.\nA usage example:\nnf = Nextflow(cpus=8).\n11) *queue_size*, default 100.\nControls nextflow process \"queue_size\" parameter.\nThe number of tasks the executor will handle in a parallel manner.\n12) *remove_logs*, default False.\nIf set to True and Nextflow executes jobs successfully, all intermediate and log files\nwill be removed.\nThis might be important because Nextflow produces a whole bunch of files which might\nbe not welcome at some file systems.\nTo set this parameter do the following:\nnf = Nextflow(remove_logs=True).\n13) *force_remove_logs*, default False.\nThe only difference with \"remove_logs\" parameter is that Nextflow logs and intermediate \nfiles will be removed in any case.\nnf = Nextflow(force_remove_logs=True).\n14) *wd*, cwd is default (directory you call the script from).\nThis is the directory where the library creates the project directory.\nThen the library saves nextflow script, configuration and all intermediate files to\nthe created project directory.\nUsage example:\nnf = Nextflow(wd=\"/tmp/project/\").\nMight be useful if the filesystem where you run your pipeline doesn't support file\nlocks and you have to run the nextflow pipeline outside the filesystem.\nPlease see Troubleshooting section case 1 for details.\n15) *project_name*, default: \"nextflow_project_at_{timestamp}\".\nBasically a project directory name where the library keeps all nextflow-related data and files.\nnf = Nextflow(project_name=\"test_project\").\nIf not set, the library will automatically generate the project name using timestamp.\n16) *no_nf_check*, default False.\nNormally the library checks whether there is an accessible nextflow executable and\nraises an error if that's not the case.\nIf you set it to True:\nnf = Nextflow(no_nf_check=True)\nthen library will not check for that on the stage of Nextflow class initiation.\n17) *switch_to_local*, default False.\nNormally the library checks, whether the executor which the user set is accessible.\nFor instance, if user set the \"executor\" parameter to \"slurm\", but there is not \"sbatch\"\nexecutable accessible, then program raises an error.\nHowever, if you set switch_to_local parameter:\nnf = Nextflow(switch_to_local=True),\nthe library will just replace \"slurm\" executor to \"local\".\n18) *retry_increase_mem*, default False\nIn case of job failure, retry with x2 memory.\n19) *retry_increase_time*, default False\nIn case of job failure, retry with x2 time.\n20) *executor_queuesize*, default 100\nControls executor.queueSize parameter\n21) *cluster_options*, default None\nControls nextflow clusterOptions parameter. A string to extend scheduler call. Please see:\nhttps://www.nextflow.io/docs/latest/process.html#process-clusteroptions\n[clusterOptions parameter](https://www.nextflow.io/docs/latest/process.html#process-clusteroptions)\nOption example: \"-S /bin/bash -l mem_free=20G -q all.q@compute-0-${key}\"\n\n\n### execute function parameters\n\nInput: list/tuple or other iterable of strings.\nEach string is a separate shell script command, such as:\n\n```shell script\npython3 script.py in_dir/file_1.txt out_dir/file_1.txt --some_option 1 --other_option 2\n```\n\nOutput: 0 or 1\n- If 0: nextflow pipeline executed successfully.\n- Otherwise, if 1: nextflow pipeline crashed.\nPlease have a look at logs.\n\nPlease note that if pipeline crashed, py_nf does not raise an error!\nUser should decide what to do in this case, for instance do some cleanup before or so.\n\nThe proper usage would be:\n\n```python\nstatus = nf.execute(job_list)\nif status == 1:\n    # pipeline failed, need to do some cleanup\n    do_some_cleanup()\n    sys.exit(1)\n```\n\n#### config_file option\n\nIf you already have a robust configuration file then you can use it with py_nf library.\nTo do so:\n\n```python\nstatus = nf.execute(job_list, config_file=\"/path/to/your/config/file\")\nif status == 1:\n    # pipeline failed, need to do some cleanup\n    do_some_cleanup()\n    sys.exit(1)\n```\n\n\n## Troubleshooting\n\nCase 1, you see an error message like this:\n\n```txt\nCan't open cache DB: /lustre/projects/project-xxx/.nextflow/cache/a80d212d-5a68-42b0-a8a5-d92665bdc492/db\n\nNextflow needs to be executed in a shared file system that supports file locks.\nAlternatively you can run it in a local directory and specify the shared work\ndirectory by using by `-w` command line option.\n```\n\nThat means your filesystem doesn't file locks: maybe it's lustre and your system\nadministrator disabled the locks.\nThe simplest way to override this is to pick some directory outside lustre filesystem and\ncall nextflow from there.\nYou can use \"wd\" parameter to do so:\n\n```python\nfrom py_nf.py_nf import Nextflow\n\nsome_dir = \"/home/user/nextflow_stuff\"\nnf = Nextflow(wd=some_dir)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkirilenkobm%2Fpy_nf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkirilenkobm%2Fpy_nf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkirilenkobm%2Fpy_nf/lists"}