{"id":19749471,"url":"https://github.com/mskcc/beagle","last_synced_at":"2025-04-30T09:31:13.513Z","repository":{"id":37476700,"uuid":"193136920","full_name":"mskcc/beagle","owner":"mskcc","description":"Voyager Backend","archived":false,"fork":false,"pushed_at":"2025-04-28T20:07:33.000Z","size":76607,"stargazers_count":2,"open_issues_count":115,"forks_count":5,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-28T20:43:01.581Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mskcc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-06-21T17:30:34.000Z","updated_at":"2025-04-17T18:13:56.000Z","dependencies_parsed_at":"2023-10-04T22:24:53.378Z","dependency_job_id":"e9f765f7-6b03-4ed8-9b84-9175e1a1b84b","html_url":"https://github.com/mskcc/beagle","commit_stats":null,"previous_names":[],"tags_count":39,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mskcc%2Fbeagle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mskcc%2Fbeagle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mskcc%2Fbeagle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mskcc%2Fbeagle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mskcc","download_url":"https://codeload.github.com/mskcc/beagle/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251676715,"owners_count":21626057,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T02:26:39.170Z","updated_at":"2025-04-30T09:31:12.745Z","avatar_url":"https://github.com/mskcc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Beagle\n\nBeagle is a backend service for managing files, pipelines and runs.\n\n![alt text](docs/pics/voyager.png \"Diagram of Voyager project\")\n\n## Beagle Responsibilities\n\n- Users\n  - Authentication using MSKCC LDAP\n  - Every user will have same permissions\n- Files\n  - List files in Beagle DB\n  - Search files (filename, metadata, file-type, file-group)\n  - Create File in Beagle DB\n- FileMetadata\n  - Metadata is associated with file.\n  - Metadata versioning. Changes are tracked, and can be reverted.\n  - Metadata validation using JsonSchema.\n- Pipelines\n  - Using pipelines hosted on github\n  - Creating RUNs from pipelines\n- Run\n  - Creating run (choosing pipeline, choosing inputs)\n  - Submitting job to rabix executor\n  - Receiving updates about job status from rabix\n  - List outputs generated from run\n- LIMS integration\n  - Periodically fetch new samples from LIMS and create File objects in Beagle DB\n  - Try to pair fails, and create runs\n  - Notify if there are some errors with files or file metadata\n\n## `beagle_cli.py`\n\n- Command line utility which helps handles authentication and accessing beagle endpoints.\n\n## Setup\n\n- Requirements\n  - PostgreSQL==11\n  - RabbitMQ\n  - python 3\n\n- Instructions\n  - virtualenv beagle\n  - pip install -r requirements.txt\n  - setup your environment using the [environment page](docs/ENVIRONMENT_VARIABLES.md)\n\n  - python manage.py migrate\n  - python manage.py runserver\n\n- Async\n  - Celery is used for scheduling tasks related to ETL from LIMS and submission to CWL Executor\n  - celery -A beagle_etl beat -l info -f beat.log (starting the periodic task)\n  - celery -A beagle_etl worker -l info -Q \u003cbeagle_default_queue\u003e -f beagle-worker.log (starting the worker)\n  - celery -A beagle_etl worker --concurrency 1 -l info -Q \u003cbeagle_job_scheduler_queue\u003e -f scheduler-worker.log\n  - celery -A beagle_etl worker -l info -Q \u003cbeagle_runner_queue\u003e -f beagle-runner.log\n\nRead more detailed specification on [wiki page](https://github.com/mskcc/beagle/wiki/Beagle).\n\n# Development Instance\n\nA development instance can be easily set up using `conda` with the following commands:\n\n- Clone this repo:\n\n```\ngit clone https://github.com/mskcc/beagle.git\ncd beagle\n```\n\n- Install dependencies in the current directory with `conda`:\n\n```\nmake install\n```\n\n- If using a m1 mac, install with: \n```\nmake install-m1\n```\n\nand activate the conda environment: \n\n```\nconda activate beagle\n```\n\n- Initialize the PostgreSQL database:\n\n```\nmake db-init\n```\n\n- Initialize the Django database and set an admin ('superuser') account:\n\n```\nmake django-init\n```\n\n- Start Postgres, RabbitMQ, and Celery servers:\n\n```\nmake start-services\n```\n\n- Start the main Django development server:\n\n```\nmake runserver\n```\n\nThe included Makefile will pre-populate most required environment variables needed for Beagle to run, using default settings. These settings can be changed when you invoke `make` on the command line by including them as keyword args, for example:\n\n```\nmake db-init BEAGLE_DB_NAME=db-dev\n```\n\nSome environment variables needed for full functionality are not included; you should save these separately and `source` them before running the Makefile. These variables are:\n\n```\nBEAGLE_LIMS_USERNAME\nBEAGLE_LIMS_PASSWORD\nBEAGLE_LIMS_URL\nBEAGLE_AUTH_LDAP_SERVER_URI\n```\n\nBeagle can run without these, but it will not be able to access IGO LIMS and LDAP server for authentication.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmskcc%2Fbeagle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmskcc%2Fbeagle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmskcc%2Fbeagle/lists"}