{"id":13689648,"url":"https://github.com/khanlab/neuroglia-helpers","last_synced_at":"2026-02-05T07:32:59.869Z","repository":{"id":26642567,"uuid":"109436815","full_name":"khanlab/neuroglia-helpers","owner":"khanlab","description":"Helper and wrapper scripts for using Singularity locally and on compute canada","archived":false,"fork":false,"pushed_at":"2023-09-28T13:49:36.000Z","size":3782,"stargazers_count":5,"open_issues_count":1,"forks_count":7,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-11-12T13:39:30.869Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/khanlab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-11-03T19:49:13.000Z","updated_at":"2023-11-06T04:04:00.000Z","dependencies_parsed_at":"2024-11-12T13:38:59.272Z","dependency_job_id":"a32e79d1-8bd3-455f-83fb-680ad033ff99","html_url":"https://github.com/khanlab/neuroglia-helpers","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khanlab%2Fneuroglia-helpers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khanlab%2Fneuroglia-helpers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khanlab%2Fneuroglia-helpers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khanlab%2Fneuroglia-helpers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/khanlab","download_url":"https://codeload.github.com/khanlab/neuroglia-helpers/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251966431,"owners_count":21672666,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T15:01:59.118Z","updated_at":"2026-02-05T07:32:59.855Z","avatar_url":"https://github.com/khanlab.png","language":"Shell","funding_links":[],"categories":["Shell"],"sub_categories":[],"readme":"# neuroglia-helpers\n\nHelper \u0026 wrapper scripts for using Singularity images locally and remotely on graham (compute canada) - or any other SLURM system\n\nFeatures:\n* wrapper scripts for submitting single and batch jobs using SLURM arrays\n* variants of scripts that use (neuroglia*) and do not use (regular*) singularity\n* scripts for interactive job submission \n* BIDS-App support with parallelization over subjects with SLURM arrays\n* Deployment via Singularity (pulling containers from shub://, docker://, or file://)\n\n## Install:\n\nTo set-up neuroglia-helpers on graham, run the following:\n```\ngit clone http://github.com/khanlab/neuroglia-helpers ~/neuroglia-helpers\n~/neuroglia-helpers/setup.sh\n```\n\nIf you want to use a different configuration profile, e.g. `khanlab`, add it as an argument to `setup.sh`:\n```\n~/neuroglia-helpers/setup.sh khanlab\n```\n\n\nNOTE: You should remove any lines that source `00_init.sh` or set `SINGULARITY_*` variables in your `.bashrc` file. If the above lines are in your .bashrc file, then you will have problems using sftp or scp clients. If you need to use your .bashrc file, use the following:\n```\necho \"source ~/neuroglia-helpers/00_initrc.sh\" \u003e\u003e ~/.bashrc\n```\n\n## Configuration\n\nThe `cfg/graham_default.cfg` file contains environment variables defining Compute Canada resource allocation accounts for compute jobs, and for defining where your Singularity container folder is placed.  \n\n## Job templates\n\nThe wrappers make use of the job templates to define resources for submitted jobs, e.g. the number of CPUs, size of memory, and length of time. The predefined job templates are designed based on the size of Compute Canada (graham cluster) nodes and queues to make optimal use of the resources. You should use the template that best fits your job type. The `Regular` job type is default in most cases (8core, 32gb, 24hrs), but the full list of job templates can be found by using the `-J` option with any wrapper.\n\n## BIDS Apps:\n\nThe bidsBatch wrapper uses SLURM to parallelize over *subjects*, and by default will run a job for every subject in your participants.tsv file. \nbidsBatch uses SLURM Arrays, so it groups all the different jobs under a single job ID, with each array job indicated as `\u003cjobid\u003e_\u003cindex\u003e`.\n\n\nEach deployed BIDS app has a name (unique identifier), a URI (location of the container), default options, and default job template. These are all stored in the `bids-apps.tsv` file. Each line is a different deployment, which may use the same container (URI) as another deployement, but e.g. with different default options and/or job templates.\n\nThe list of BIDS app names is given when you run `bidsBatch`.\n\nTo get a list of names for all BIDS apps deployed, along with URI and default job template, use `bidsBatch -B`.\n\nTo get usage information for a particular BIDS app, use `bidsBatch \u003capp_name\u003e`. This will also display the default options for that deployment.\n\nNote: when you run bidsBatch to process a dataset located at `BIDS_DIR`, a `BIDS_DIR/code/jobs` folder is created containing the SLURM log files.\n\n\n### Example: Running fmriprep on your bids dataset\n```\nbidsBatch fmriprep_1.0.4 ~/my-bids-dataset ~/my-bids-dataset/derivatives/fmriprep-v1.0.4 participant \n```\nTo get usage/options for a particular bids app, leave out the bids-app required arguments, e.g.:\n\n```\nbidsBatch fmriprep_1.0.4 \n```\n\n\n\n\n\n## Wrappers:\n\nWrappers for singularity and job submission that make use of the neuroglia configuration.\n\n* neuroglia\n\nSimple wrapper for singularity exec (does not submit a job)\n\n* neurogliaShell\n\nSimple wrapper for singularity shell (does not submit a job)\n\n* neurogliaSubmit\n\nCreates and submits a singularity exec job, e.g.: `neurogliaSubmit bet t1.nii` will submit a job to perform bet skull stripping.\nEnter `neurogliaSubmit` for more details.\n\n* neurogliaBatch\n\nWrapper for submitting batch cluster jobs, using a singularity container, looping through a subject list. User provides the command to run, and the arguments that come before and after the subjid.  Enter `neurogliaBatch` for more details.\n\n* regularSubmit, regularBatch\n\nThese are the analogous to neurogliaSubmit and neurogliaBatch, but run the commands directly and do not use `singularity exec`.\n\n* regularInteractive\n\nWrapper for submitting an interactive cluster job, this gives you command-line access to a compute node terminal for a short period of time (up to 3 hours), to run jobs directly. Remember that you are not supposed to run any jobs directly on the *login* nodes. \n\nNote: interactive job default is 3hours, 8cores, 32GB memory\n\nWith `regularInteractive` you can also run GUI applications from Compute Canada, if you use `ssh -Y` when connecting to graham.\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkhanlab%2Fneuroglia-helpers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkhanlab%2Fneuroglia-helpers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkhanlab%2Fneuroglia-helpers/lists"}