{"id":19541883,"url":"https://github.com/imcf/snijder","last_synced_at":"2025-07-08T20:38:04.962Z","repository":{"id":150382561,"uuid":"89288826","full_name":"imcf/snijder","owner":"imcf","description":":scissors: Single Node Inhomogeneous Job Dispatcher, Executor \u0026 Reporter","archived":false,"fork":false,"pushed_at":"2020-10-15T09:05:19.000Z","size":987,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-01-08T18:48:31.582Z","etag":null,"topics":["executor","python","queue-manager","scheduler"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/imcf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-24T21:28:41.000Z","updated_at":"2024-06-21T09:41:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"ce432a96-1a64-4ebf-a32c-d3f130c43140","html_url":"https://github.com/imcf/snijder","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imcf%2Fsnijder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imcf%2Fsnijder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imcf%2Fsnijder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imcf%2Fsnijder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/imcf","download_url":"https://codeload.github.com/imcf/snijder/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240795453,"owners_count":19858821,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["executor","python","queue-manager","scheduler"],"created_at":"2024-11-11T03:12:15.768Z","updated_at":"2025-02-26T05:19:21.179Z","avatar_url":"https://github.com/imcf.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ![Snijder logo][img_snijder_logo] Snijder\n\nThe _Snijder_ project provides a Python package managing one or more job-queues\nfor multiple users in a simple, accessible way. Flexibility and maintainability\nhave priority over performance, as the targeted jobs will run in the range of\nminutes to hours, so being able to process thousands of queue-requests per\nsecond is just not important here.\n\n_Snijder_ is an acronym for\n\n* **`S`** ingle\n* **`N`** ode\n* **`I`** nhomogeneous\n* **`J`** ob\n* **`D`** ispatcher,\n* **`E`** xecutor and\n* **`R`** eporter\n\nInitially it was created to replace the queue manager of the [Huygens Remote\nManager (HRM)](http://huygens-rm.org/) but from a technical perspective\n_Snijder_ is completely independent of that project and is actually used for\nother tasks as well.\n\n## Under The Hood\n\n[![Code style: black][img_codestyle_black]](https://github.com/psf/black)\n\n_Snijder_ makes use of the [GC3Pie](https://github.com/imcf/gc3pie) Python\npackage dealing with job dispatching, monitoring, data transfer and cluster\nqueueing systems.\n\n## Installation\n\n### GC3Pie Setup\n\nTo install GC3Pie a few additional packages are required. On Debian / Ubuntu\nsystems, simply run this command to prepare the installation:\n\n```bash\nsudo apt install \\\n    gcc \\\n    make \\\n    git \\\n    time \\\n    python-virtualenv \\\n    python-dev \\\n    libffi-dev \\\n    libssl-dev\n```\n\nThen follow the instructions below to set up GC3Pie in a directory structure\nunderneath `/opt/snijder`.\n\nFirst, make sure the base directory is there and writable to the installatoin\nuser. Run the following commands as `root` or use `sudo`, depending on your\npreferences:\n\n```bash\nBASE_DIR=\"/opt/snijder\"\nSNIJDER_USER=\"snijder\"\nSNIJDER_GROUP=\"snijder\"\n\nmkdir -pv $BASE_DIR\nchown $SNIJDER_USER:$SNIJDER_GROUP $BASE_DIR\n```\n\nThen, as the above configured `$SNIJDER_USER` run:\n\n```bash\nGC3VER=\"2.5.0\"\nGC3HOME=\"$BASE_DIR/venvs/gc3pie_$GC3VER\"\n\nvirtualenv --system-site-packages $GC3HOME\nsource $GC3HOME/bin/activate\n\npip install --upgrade pip\npip install --upgrade pycli prettytable docutils\n\nCURDIR=$(pwd)\n\ncd $BASE_DIR\ngit clone https://github.com/uzh/gc3pie.git gc3pie.git\n\ncd gc3pie.git\ngit checkout -b tag-$GC3VER tags/v$GC3VER\nenv CC=gcc ./setup.py install\n\ncd $CURDIR\n```\n\n### Snijder Setup\n\nTo set up _Snijder_ you need to create a base directory where all the spooling /\nqueueing will take place, then you're good to clone the repository:\n\n```bash\nsudo apt install python-pyinotify\n\nSPOOL_BASE=\"/opt/spool\"  # adapt as you like, e.g. \"/scratch/spool\" or similar\n\nmkdir -pv \"$SPOOL_BASE/snijder\"\nmkdir -pv \"$SPOOL_BASE/gc3/resourcedir\"\n\ncd $BASE_DIR\ngit clone https://github.com/imcf/snijder.git\n\n\nif [ -z \"$VIRTUAL_ENV\" ] ; then\n    source $BASE_DIR/venvs/gc3pie_2.5.0/bin/activate\nfi\n\npip install --upgrade psutil\n\n# TEMPORARY SETUP using a symlink:\ncd $VIRTUAL_ENV/lib/python2.7\nln -s $BASE_DIR/snijder/src/snijder\n```\n\nFor testing purposes, there is a symlink provided to an example configuration in the\n`config/` directory. If you need to use a different configuration, simply remove the\nlink and/or replace the file.\n\n## Example\n\nThe testing scripts mentioned below also serve as a very nice example to see\n_Snijder_ in action. To run the spooler / queue manager manually, use the\nfollowing command (from within the Python virtualenv created above):\n\n```bash\nif [ -z \"$VIRTUAL_ENV\" ] ; then\n    source $BASE_DIR/venvs/gc3pie_2.5.0/bin/activate\nfi\ncd $BASE_DIR/snijder\n\nbin/snijder-queue --spooldir $SPOOL_BASE/snijder --config config/gc3pie/localhost.conf -v\n```\n\nFrom there on you're ready to submit jobs through the configured spooling\ndirectories, e.g. like so:\n\n```bash\ncp -v tests/snijder-queue/jobfiles/decon_it-3_user01.cfg $SPOOL_BASE/snijder/spool/new/\n```\n\n## Testing\n\nTo run the tests provided in `tests/snijder-queue` you need some sample input\nfiles which are not part of this repository, as they are large binary files. See\nthe next section on how to get them.\n\n### HuCore Test Images\n\nThe test images for deconvolution are a set of images which can be downloaded\nfrom the [SVI website](https://svi.nl/DemoImages) (requires registration).\nSimply place them in `resources/sample_data/hucore/` to run the tests.\n\n### Running the tests\n\nOnce the sample images are there, you can just launch the test runner. Make sure\nto have the Python virtualenv activated that was created above, then:\n\n```bash\ncd $BASE_DIR/snijder\ntests/snijder-queue/run_tests.sh\n```\n\n## Contributing\n\nPlease see the details in the [Development And Contribution\nGuide](CONTRIBUTING.md).\n\n[img_snijder_logo]: https://raw.githubusercontent.com/imcf/snijder/master/resources/artwork/snijder-logo-blue-64.png\n[img_codestyle_black]: https://img.shields.io/badge/code%20style-black-000000.svg\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimcf%2Fsnijder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fimcf%2Fsnijder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimcf%2Fsnijder/lists"}