{"id":13738725,"url":"https://github.com/Computational-Plant-Science/plantit","last_synced_at":"2025-05-08T17:32:14.100Z","repository":{"id":37550776,"uuid":"137381489","full_name":"Computational-Plant-Science/plantit","owner":"Computational-Plant-Science","description":null,"archived":false,"fork":false,"pushed_at":"2023-04-04T14:51:51.000Z","size":76683,"stargazers_count":11,"open_issues_count":61,"forks_count":5,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-10-31T07:42:51.712Z","etag":null,"topics":["hpc","phenomics","phenotyping","science-gateways","singularity"],"latest_commit_sha":null,"homepage":"https://plantit.cyverse.org","language":"Vue","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Computational-Plant-Science.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null}},"created_at":"2018-06-14T16:13:43.000Z","updated_at":"2024-04-26T23:34:04.000Z","dependencies_parsed_at":"2023-02-15T21:15:36.161Z","dependency_job_id":"ed00577a-22fc-4748-bac9-523b8cd38bd4","html_url":"https://github.com/Computational-Plant-Science/plantit","commit_stats":null,"previous_names":[],"tags_count":35,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Computational-Plant-Science%2Fplantit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Computational-Plant-Science%2Fplantit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Computational-Plant-Science%2Fplantit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Computational-Plant-Science%2Fplantit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Computational-Plant-Science","download_url":"https://codeload.github.com/Computational-Plant-Science/plantit/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224747111,"owners_count":17363192,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hpc","phenomics","phenotyping","science-gateways","singularity"],"created_at":"2024-08-03T03:02:35.051Z","updated_at":"2025-05-08T17:32:14.079Z","avatar_url":"https://github.com/Computational-Plant-Science.png","language":"Vue","funding_links":[],"categories":["Vue"],"sub_categories":[],"readme":"\n\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"https://github.com/Computational-Plant-Science/plantit/blob/master/plantit/front_end/src/assets/logo.png?raw=true\" style=\"position:relative; top: 40px; height: 150px\" /\u003e\n\n![build](https://github.com/Computational-Plant-Science/plantit/workflows/build/badge.svg)\n![release](https://github.com/Computational-Plant-Science/plantit/workflows/release/badge.svg)\n[![Documentation Status](https://readthedocs.org/projects/plantit/badge/?version=latest)](https://plantit.readthedocs.io/en/latest/?badge=latest)\n[![Coverage Status](https://coveralls.io/repos/github/Computational-Plant-Science/plantit/badge.svg?branch=master)](https://coveralls.io/github/Computational-Plant-Science/plantit?branch=master)\n\n\u003c/div\u003e\n\n\u003c!-- START doctoc generated TOC please keep comment here to allow auto update --\u003e\n\u003c!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --\u003e\n**Contents**\n\n- [About](#about)\n  - [Status](#status)\n  - [Motivation](#motivation)\n  - [Overview](#overview)\n- [Development](#development)\n  - [Requirements](#requirements)\n  - [Installation](#installation)\n    - [Setting up a development environment](#setting-up-a-development-environment)\n      - [Running tests](#running-tests)\n      - [Reverse proxying with `ngrok`](#reverse-proxying-with-ngrok)\n    - [Deploying to production](#deploying-to-production)\n      - [SSL Certificates](#ssl-certificates)\n  - [Configuring environment variables](#configuring-environment-variables)\n  - [Configuring deployment targets](#configuring-deployment-targets)\n    - [Agent requirements](#agent-requirements)\n    - [Authenticating with Docker](#authenticating-with-docker)\n  - [Building the documentation](#building-the-documentation)\n  - [Testing DIRT migrations](#testing-dirt-migrations)\n\n\u003c!-- END doctoc generated TOC please keep comment here to allow auto update --\u003e\n\n## About\n\nPlantIT is a framework for deploying research apps to high-performance/-throughput clusters. Specifically, it's a science gateway for image-based plant phenotyping. Future work may generalize the platform for any domain.\n\n### Status\n\nPlantIT debuted (with [pre-release v0.1.0](https://github.com/Computational-Plant-Science/plantit/releases/tag/v0.1.0)) at [NAPPN 2022](https://www.plantphenotyping.org/conference-home) (Feb. 22-25). See [Releases](https://github.com/Computational-Plant-Science/plantit/releases) for changelog and [Roadmap](https://github.com/Computational-Plant-Science/plantit/wiki/Changelog-\u0026-Roadmap) for planned features and fixes. Capabilities will likely continue to evolve for some time, with \"official\" releases following an eventually forthcoming publication.\n\n### Motivation\n\nHigh-throughput phenotyping is resource-intensive and often demands virtual computing resources. This presents deployment challenges related to packaging and portability, and raises barriers to entry. Research software should:\n\n- be highly configurable when necessary\n- automate deployment details where possible\n- let users (and developers) focus on the problem domain\n\n### Overview\n\nPlantIT aims to bridge two user groups: researchers and developers. Of course one may wear both hats. The idea is an open-source conveyor belt for scientific software: make it easier to 1) package and share research applications, and 2) deploy them to clusters.\n\nPlantIT is just glue between version control, data storage, container engine, and cluster scheduler. To publish an app, containerize it (e.g., write a Dockerfile) and add a `plantit.yaml` file to the GitHub repository. Then run it from the browser with a few clicks.\n\n## Development\n\nRead on if you're interested in contributing to `plantit` or hosting your own instance somewhere.\n\n### Requirements\n\nThe following are required to develop or deploy `plantit` in a Unix environment:\n\n- [Docker](https://www.docker.com/)\n- [npm](https://www.npmjs.com/get-npm)\n\n### Installation\n\nFirst, clone the repository:\n\n```bash\ngit clone https://github.com/Computational-Plant-Science/plantit.git\n```\n\n#### Setting up a development environment\n\nTo set up a new (or restore a clean) development environment, run `scripts/bootstrap.sh` from the project root (you may need to use `chmod +x` first). You can use the `-n` option to disable the Docker build cache. This command will:\n\n- Stop and remove project containers and networks\n- If an `.env` file (to configure environment variables) does not exist, generate one with default values\n- Build the Vue front end\n- Build Docker images\n- Run migrations\n\nThen bring everything up with `docker-compose -f docker-compose.dev.yml up` (`-d` for detached mode).\n\nThis will start a number of containers:\n\n- `plantit`: Django web application (`http://localhost:3000`)\n- `postgres`: PostgreSQL database\n- `celery`: Celery prefork worker\n- `celerye`: Celery [eventlet](https://docs.celeryq.dev/en/latest/userguide/concurrency/eventlet.html) worker\n- `flower`: Flower web UI for Celery (`http://localhost:5555`)\n- `redis`: Redis instance (caching, Celery message broker)\n- `sandbox`: Ubuntu test environment\n\nThe Django admin interface is at `http://localhost:3000/admin/`. To use it, you'll need to log into the site at least once (this will create a Django account for you), then shell into the `plantit` container, run `./manage.py shell`, and update your profile with staff/superuser privileges. For instance:\n\n```python\nfrom django.contrib.auth.models import User\nuser = User.objects.get(username=\u003cyour CyVerse username\u003e)\nuser.is_staff = True\nuser.is_superuser = True\nuser.save()\n```\n\nYou can also run `./scripts/configure-superuser.sh -u \u003cyour CyVerse username\u003e` to accomplish the same thing.\n\nNote that the bootstrap script will not clear migrations. To restore to a totally clean database state, you will need to remove all `*.py` files from the `plantit/plantit/migrations` directory (except for `__init__.py`).\n\n##### Running tests\n\nOnce the containers are up, tests can be run with `docker-compose -f docker-compose.dev.yml exec plantit ./manage.py test`.\n\n##### Reverse proxying with `ngrok`\n\nTo test remote job submissions using a local development version of the `plantit` web application, you will need a service of some kind to accept and forward job completion signals to your machine's `localhost`. One convenient tool is [`ngrok`](https://ngrok.com/). After downloading and adding the `ngrok` executable to your path, run `ngrok http 3000` to start a tunnel, then set the `DJANGO_API_URL` variable in `.env` to the URL `ngrok` reports it's listening on.\n\n#### Deploying to production\n\nIn production configuration, NGINX serves static assets and reverse-proxies Django via Gunicorn (both in the same container).\n\nTo configure PlantIT for deployment, first clone the repo, then, from the root directory, run:\n\n```\nchmod +x /scripts/deploy.sh\n./scripts/deploy.sh \u003cconfiguration ('rc' or 'prod')\u003e \u003chost IP or FQDN\u003e \u003cadmin email address\u003e\n```\n\nThis script is idempotent and may safely be triggered to run by e.g., a CI/CD server. This will:\n\n- Bring containers down\n- Fetch the latest version of the project\n- Pull the latest versions of Docker containers\n- Build the Vue front end\n- Collect static files\n- Configure NGINX (replace `localhost` in `config/ngnix/conf.d/local.conf` with the host's IP or FQDN, configured via environment variable)\n- Update environment variables (disable debugging, enable SSL and secure cookies, etc)\n- Bring containers up\n- Run migrations\n\nAt this point the following containers should be running:\n\n- `nginx`: NGINX server (reverse proxy)\n- `plantit`: Django web application behind Gunicorn (`http://localhost:80`)\n- `postgres`: PostgreSQL database\n- `celery`: Celery background worker\n- `redis`: Redis instance\n\n##### SSL Certificates\n\nPlantIT uses [Let's Encrypt](https://letsencrypt.org/) and [Certbot](https://certbot.eff.org/) for SSL certification. The production configuration includes a `certbot` container which can be used to request new or renew existing certificates from Let's Encrypt. Standard certificates last 90 days.\n\nIn production the `certbot` container is configured by default to automatically renew certs when necessary:\n\n```yaml\ncertbot:\n  image: certbot/certbot\n  volumes:\n    - ./config/certbot/conf:/etc/letsencrypt/\n    - ./config/certbot/www:/var/www/certbot\n  entrypoint: \"/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 24h \u0026 wait $${!}; done;'\"\n```\n\nTo manually request a new certificate, run:\n\n```shell\ndocker-compose -f docker-compose.prod.yml run certbot\n```\n\nTo renew an existing certificate, use the `renew` command, then restart all containers:\n\n```shell\ndocker-compose -f docker-compose.prod.yml run certbot renew\ndocker-compose -f docker-compose.prod.yml restart\n```\n\nUse the `--dry-run` flag with any command to test without writing anything to disk.\n\n### Configuring environment variables\n\nDocker will read environment variables in the following format from a file named `.env` in the project root directory (if the file exists):\n\n```\nkey=value\nkey=value\n...\n```\n\n`bootstrap.sh` will generate an `.env` file like the following if one does not exist:\n\n```\nVITE_TITLE=plantit\nMAPBOX_TOKEN=\u003cyour Mapbox token\u003e\nMAPBOX_FEATURE_REFRESH_MINUTES=60\nCYVERSE_REDIRECT_URL=http://localhost:3000/apis/v1/idp/cyverse_handle_temporary_code/\nCYVERSE_CLIENT_ID=\u003cyour cyverse client id\u003e\nCYVERSE_CLIENT_SECRET=\u003cyour cyverse client secret\u003e\nCVVERSE_USERNAME=\u003cyour cyverse username\u003e\nCYVERSE_PASSWORD=\u003cyour cyverse password\u003e\nCYVERSE_TOKEN_REFRESH_MINUTES=60\nNODE_ENV=development\nDJANGO_SETTINGS_MODULE=plantit.settings\nDJANGO_SECRET_KEY=\u003cyour django secret key\u003e\nDJANGO_DEBUG=True\nDJANGO_API_URL=http://plantit:3000/apis/v1/\nDJANGO_SECURE_SSL_REDIRECT=False\nDJANGO_SESSION_COOKIE_SECURE=False\nDJANGO_CSRF_COOKIE_SECURE=False\nDJANGO_ALLOWED_HOSTS=*\nDJANGO_ADMIN_USERNAME=\u003cyour django admin username\u003e\nDJANGO_ADMIN_PASSWORD=\u003cyour django admin password\u003e\nDJANGO_ADMIN_EMAIL=\u003cyour django admin email\u003e\nCELERY_EVENTLET_QUEUE=eventlet\nUSERS_CACHE=/code/users.json\nUSERS_REFRESH_MINUTES=60\nUSERS_STATS_REFRESH_MINUTES=10\nSTATS_WINDOW_WIDTH_DAYS=30\nMORE_USERS=/code/more_users.json\nAGENT_KEYS=/code/agent_keys\nWORKFLOWS_CACHE=/code/workflows.json\nWORKFLOWS_REFRESH_MINUTES=60\nTASKS_LOGS=/code/logs\nTASKS_TIMEOUT_MULTIPLIER=2\nTASKS_STEP_TIME_LIMIT_SECONDS=20\nLAUNCHER_SCRIPT_NAME=launch\nINPUTS_FILE_NAME=inputs.txt\nICOMMANDS_IMAGE=computationalplantscience/icommands\nSQL_ENGINE=django.db.backends.postgresql\nSQL_HOST=postgres\nSQL_PORT=5432\nSQL_NAME=postgres\nSQL_USER=postgres\nSQL_PASSWORD=\u003cyour database password\u003e\nGITHUB_AUTH_URI=https://github.com/login/oauth/authorize\nGITHUB_REDIRECT_URI=http://localhost:3000/apis/v1/users/github_handle_temporary_code/\nGITHUB_SECRET=\u003cyour github secret\u003e\nGITHUB_CLIENT_ID=\u003cyour github client ID\u003e\nDOCKER_USERNAME=\u003cyour docker username\u003e\nDOCKER_PASSWORD=\u003cyour docker password\u003e\nNO_PREVIEW_THUMBNAIL=/code/plantit/front_end/src/assets/no_preview_thumbnail.png\nAWS_ACCESS_KEY=\u003cyour AWS access key\u003e\nAWS_SECRET_KEY=\u003cyour AWS secret key\u003e\nAWS_REGION=\u003cyour AWS region\u003e\nAWS_FEEDBACK_ARN=\u003cyour AWS feedback ARN\u003e\nAGENTS_HEALTHCHECKS_MINUTES=5\nAGENTS_HEALTHCHECKS_SAVED=12\nTUTORIALS_FILE=/code/tutorials.pdf\nFEEDBACK_FILE=/code/feedback.pdf\nCELERY_AUTH=user:password\nHTTP_TIMEOUT=15\nCURL_IMAGE=curlimages/curl\nGH_USERNAME=\u003cyour github username\u003e\nFIND_STRANDED_TASKS=True\n```\n\nNote that the following environment variables must be supplied manually:\n\n- `MAPBOX_TOKEN`\n- `CYVERSE_CLIENT_ID`\n- `CYVERSE_CLIENT_SECRET`\n- `CVVERSE_USERNAME`\n- `CYVERSE_PASSWORD`\n- `GITHUB_CLIENT_ID`\n- `GITHUB_SECRET`\n- `AWS_ACCESS_KEY`\n- `AWS_SECRET_KEY`\n- `AWS_REGION`\n- `AWS_FEEDBACK_ARN`\n  \nSeveral others will be auto-generated by `scripts/bootstrap.sh` in a clean install directory:\n\n- `DJANGO_ADMIN_PASSWORD`\n- `DJANGO_SECRET_KEY`\n- `SQL_PASSWORD`\n\nSome variables must be reconfigured for production environments (`scripts/deploy` will automatically do so):\n\n- `NODE_ENV` should be set to `production`\n- `DJANGO_DEBUG` should be set to `False`\n- `DJANGO_SECURE_SSL_REDIRECT` should be set to `True`\n- `DJANGO_API_URL` should point to the host's IP or FQDN\n\n### Configuring deployment targets\n\nAn agent is an abstraction of a computing resource, such as a cluster or supercomputer. `plantit` interacts with agents via key-authenticated SSH and requires the SLURM scheduler to be installed. (Support for additional schedulers is in development.)\n\nDeployment targets may be configured programmatically or with the Django admin interface. To configure an agent via the Django admin site, make sure you're logged into `plantit`, then navigate to `http://localhost:3000/admin/` (`https://\u003chost\u003e/admin/` in production). Select the `Agents` tab on the left side of the screen, then `Add Agent`.\n\nOn many clusters it is customary to configure dependencies on a per-user basis with a module system, e.g. `module load \u003csome software\u003e`. The `pre_commands` agent property is the place for commands like these: when provided, they will be prepended to all commands `plantit` sends to the cluster for job orchestration.\n\n#### Agent requirements\n\n`plantit` deployment targets must run some Linux distribution with either the `sh` or `bash` shells available. Only 2 dependencies are required:\n\n- SLURM\n- Singularity\n\n`plantit` tasks expect standard SLURM commands (e.g., `sbatch`, `scancel`) to be available. Singularity must also be installed and available on the `$PATH`.\n\n#### Authenticating with Docker\n\nDocker Hub applies rate limits to unauthenticated users. These are easy to meet or exceed, since Singularity queries the Docker API on each `singularity exec docker://\u003csome container\u003e`. It is recommended to use `singularity remote login --username \u003cyour Docker username\u003e docker://docker.io` with a paid Docker account: this will cache your Docker credentials on the deployment target for Singularity to use thereafter.\n\n### Building the documentation\n\nTo build the `sphinx` documentation locally, use:\n\n```shell\ndocker run -v $(pwd):/opt/dev -w /opt/dev computationalplantscience/plantit sphinx-build -b html docs docs_output\n```\n\n### Testing DIRT migrations\n\nThe DIRT migration feature allows users of the original [DIRT](http://dirt.cyverse.org/) web application to migrate their data to `plantit`. To test this feature, you will need to have access to the DIRT server and database. The following environment variables must be set:\n\n- `DIRT_MIGRATION_DATA_DIR`: the directory on the DIRT server where DIRT data is stored\n- `DIRT_MIGRATION_HOST`: the hostname of the DIRT server\n- `DIRT_MIGRATION_PORT`: the SSH port of the DIRT server\n- `DIRT_MIGRATION_USERNAME`: the SSH username for the DIRT server\n- `DIRT_MIGRATION_DB_HOST`: the hostname of the DIRT database server\n- `DIRT_MIGRATION_DB_PORT`: the port of the DIRT database server\n- `DIRT_MIGRATION_DB_USER`: the username of the DIRT database user\n- `DIRT_MIGRATION_DB_DATABASE`: the name of the DIRT database\n- `DIRT_MIGRATION_DB_PASSWORD`: the DIRT database password\n\nAn SSH tunnel must also be opened to the DIRT server, as the database is not open to external connections. For instance, to open a tunnel from port 5678 on the DIRT server to port 3306 on a development machine:\n\n```shell\nssh -L 3306:localhost:3306 -p \u003cDIRT server SSH port\u003e \u003cyour cyverse username\u003e@\u003cDIRT server IP or FQDN\u003e\n```\n\nOn some Linux systems it may be necessary to substitute the loopback IP address `127.0.0.1` for `localhost`\n\nBe sure to set `DIRT_MIGRATION_DB_HOST=host.docker.internal` to point the Docker containers to the host's loopback/localhost address.\n\nSome extra configuration is necessary for Linux systems to allow containers to access services running on the local host. The `docker-compose.dev.yml` configuration file configures the `plantit`, `celery`, and `celerye` containers with the `extra_hosts` option:\n\n```yaml\n  extra_hosts:\n    - \"host.docker.internal:host-gateway\"\n```\n\nThis is only necessary on Linux systems. On Mac and Windows, the `host.docker.internal` hostname is automatically configured. See [this post](https://stackoverflow.com/a/43541732/6514033) for more information.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FComputational-Plant-Science%2Fplantit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FComputational-Plant-Science%2Fplantit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FComputational-Plant-Science%2Fplantit/lists"}