{"id":24610964,"url":"https://github.com/scilifelabdatacentre/dds_web","last_synced_at":"2026-03-08T06:33:18.487Z","repository":{"id":37460188,"uuid":"209742486","full_name":"ScilifelabDataCentre/dds_web","owner":"ScilifelabDataCentre","description":"A cloud-based system for the delivery of data from SciLifeLab Facilities to their users (e.g. research group).","archived":false,"fork":false,"pushed_at":"2026-03-06T01:12:41.000Z","size":52873,"stargazers_count":9,"open_issues_count":119,"forks_count":8,"subscribers_count":4,"default_branch":"dev","last_synced_at":"2026-03-06T05:22:19.281Z","etag":null,"topics":["flask","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ScilifelabDataCentre.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.rst","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2019-09-20T08:27:29.000Z","updated_at":"2026-02-27T16:20:24.000Z","dependencies_parsed_at":"2023-12-18T11:30:48.462Z","dependency_job_id":"5bc9903d-277a-4ec9-92d5-455efbe6f019","html_url":"https://github.com/ScilifelabDataCentre/dds_web","commit_stats":null,"previous_names":[],"tags_count":48,"template":false,"template_full_name":null,"purl":"pkg:github/ScilifelabDataCentre/dds_web","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScilifelabDataCentre%2Fdds_web","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScilifelabDataCentre%2Fdds_web/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScilifelabDataCentre%2Fdds_web/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScilifelabDataCentre%2Fdds_web/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ScilifelabDataCentre","download_url":"https://codeload.github.com/ScilifelabDataCentre/dds_web/tar.gz/refs/heads/dev","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScilifelabDataCentre%2Fdds_web/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30247379,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-08T05:41:50.788Z","status":"ssl_error","status_checked_at":"2026-03-08T05:41:39.075Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flask","python"],"created_at":"2025-01-24T19:19:51.906Z","updated_at":"2026-03-08T06:33:18.461Z","avatar_url":"https://github.com/ScilifelabDataCentre.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp\u003e\n   \u003ch1 align=\"center\"\u003eData Delivery System Web / API\u003c/h1\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n   \u003cimg width=\"70%\" src=\"https://github.com/ScilifelabDataCentre/dds_web/blob/master/dds_web/templates/components/hero_image.svg\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n   \u003cimg alt=\"Release\" src=\"https://img.shields.io/github/v/release/SciLifeLabDataCentre/dds_web\"\u003e\n   \u003ca href=\"https://opensource.org/licenses/MIT\"\u003e\n      \u003cimg alt=\"Licence: MIT\" src=\"https://img.shields.io/badge/License-MIT-yellow.svg\"\u003e\n   \u003c/a\u003e\n   \u003ca href=\"[https://opensource.org/licenses/MIT](https://github.com/psf/black)\"\u003e\n      \u003cimg alt=\"Code style: black\" src=\"https://img.shields.io/badge/code%20style-black-000000.svg\"\u003e\n   \u003c/a\u003e\n   \u003ca href=\"https://prettier.io/\"\u003e\n      \u003cimg alt=\"Code style: prettier\" src=\"https://img.shields.io/badge/code_style-prettier-ff69b4.svg\"\u003e\n   \u003c/a\u003e\n   \u003cbr /\u003e\n   \u003cimg alt=\"Linting\" src=\"https://github.com/ScilifelabDataCentre/dds_web/actions/workflows/python-black.yml/badge.svg\"\u003e\n   \u003cimg alt=\"CodeQL\" src=\"https://github.com/ScilifelabDataCentre/dds_web/actions/workflows/codeql-analysis.yml/badge.svg\"\u003e\n   \u003ca href=\"https://codecov.io/github/ScilifelabDataCentre/dds_web\" \u003e \n      \u003cimg src=\"https://codecov.io/github/ScilifelabDataCentre/dds_web/graph/badge.svg?token=r5tM6o08Sd\"/\u003e \n   \u003c/a\u003e\n   \u003cimg alt=\"Tests\" src=\"https://github.com/ScilifelabDataCentre/dds_web/actions/workflows/docker-compose-tests.yml/badge.svg\"\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n   \u003cb\u003eLinks\u003c/b\u003e\n   \u003cbr /\u003e\n   \u003ca href=\"https://scilifelabdatacentre.github.io/dds_cli/\"\u003e\n      \u003cimg alt=\"Documentation\" src=\"https://img.shields.io/badge/-Documentation-222222?logo=github-pages\"\u003e\n   \u003c/a\u003e\n   \u003ca href=\"https://github.com/ScilifelabDataCentre/dds_web/blob/master/doc/technical-overview.md\"\u003e\n      \u003cimg alt=\"Technical Overview\" src=\"https://img.shields.io/badge/-Technical%20Overview-informational?logo=github\"\u003e\n   \u003c/a\u003e\n   \u003ca href=\"https://github.com/ScilifelabDataCentre/dds_web/wiki/Architecture-Decision-Record,-ADR\"\u003e\n      \u003cimg alt=\"Architecture Decision Record\" src=\"https://img.shields.io/badge/-ADR-000000?logo=github\"\u003e\n   \u003c/a\u003e\n   \u003ca href=\"https://github.com/ScilifelabDataCentre/dds_web/blob/master/doc/troubleshooting.md\"\u003e\n      \u003cimg alt=\"Troubleshooting\" src=\"https://img.shields.io/badge/-Troubleshooting%20Guide-red?logo=github\"\u003e\n   \u003c/a\u003e\n   \u003ca href=\"https://github.com/ScilifelabDataCentre/dds_cli\"\u003e\n      \u003cimg alt=\"CLI\" src=\"https://img.shields.io/badge/-CLI-yellow?logo=github\"\u003e\n   \u003c/a\u003e\n\u003c/p\u003e\n\n## About\n\n**The Data Delivery System (DDS) is a cloud-based system for all SciLifeLab platforms where data generated throughout each project can be delivered to the research groups in a fast, secure and simple way. The Web / API is the backend, handling the requests and the logic behind the scenes.**\n\n\u003e _The Data Delivery System is developed and maintained by the SciLifeLab Data Centre. National Genomics Infrastructure (NGI) Stockholm has been a part of the development team during 2021 and early 2022._\n\u003e\n\u003e _This project is supported by EIT Digital, activity number 19390. This deliverable consists of design document and implementation report of application and validation of VEIL.AI technology in SciLifeLab context in Sweden._\n\n---\n\n## Table of Contents\n\n- [Development Setup](#development-setup)\n  - [Profiles](#profiles)\n  - [Debugging inside docker](#python-debugger-inside-docker)\n  - [Config settings](#config-settings)\n  - [Database changes](#database-changes)\n- [Production Instance](#production-instance)\n\n## Development Setup\n\nWhen developing this software, we recommend that you run the web server locally using Docker.\nYou can download Docker here: \u003chttps://docs.docker.com/get-docker/\u003e\n\nThen, fork this repository and clone to your local system.\nIn the root folder of the repo, run the server with one of the following profiles (_plain_, _dev_, _full-dev_, _cli_) depending on your needs.\n\n### Profiles\n\n#### Application \u0026 Database: Plain\n\n```bash\ndocker-compose up\n```\n\nThis command will orchestrate the building and running of two containers: one for the SQL database (`mariadb`) and one for the application.\n\n#### Mailcatcher: `dev`\n\n```bash\ndocker-compose --profile dev up\n```\n\nThis will give you the above two containers, but also `mailcatcher` that will allow you to read\nany sent emails by going to `localhost:1080` in a web browser.\n\n#### Minio S3 Storage \u0026 Limiter: `full-dev`\n\n```bash\ndocker-compose --profile full-dev up\n```\n\nWill also activate minio for s3 storage (clearly not functional with cli) and redis to enable a persistent limiter for the API.\nYou also need to uncomment `RATELIMIT_STORAGE_URI` in `docker-compose.yml` to enable redis.\n\nIf you prefer, you can run the web servers in 'detached' mode with the `-d` flag, which does not block your terminal.\nIf using this method, you can stop the web server with the command `docker-compose down`.\n\n#### CLI development against local environment: `cli`\n\n```bash\ndocker-compose --profile cli up\n```\n\nWill start database, backend, minio, and mailcatcher. Will also start an extra container prepared for working with the CLI.\n\nRequires that dds_cli is checked out in `../dds_cli` (otherwise adapt the volume path in `docker-compose.yml`).\n\n1. Start docker-compose with the `cli` profile\n2. Inject into the `dds_cli` container:\n\n   ```bash\n   docker exec -it dds_cli /bin/sh\n   ```\n\nThen you can freely use the dds cli component against the local development setup in the active CLI.\n\n### Python debugger inside docker\n\nIt's possible to use the interactive debugging tool `pdb` inside Docker with this method:\n\n1. Edit the `docker-compose.yml` and for the `backend` service, add:\n\n   ```yaml\n   tty: true\n   stdin_open: true\n   ```\n\n   just under\n\n   ```yaml\n   ports:\n     - 127.0.0.1:5000:5000\n   ```\n\n2. Put `import pdb; pdb.set_trace()` in the python code where you would like to activate the debugger.\n3. Run with docker-compose as normal.\n4. Find out the id of the container running the `backend`.\n\n   ```bash\n   docker container ls\n   ```\n\n5. Attach to the running backend container:\n\n   ```bash\n   docker container attach \u003ccontainer_id/name\u003e\n   ```\n\n### Config settings\n\nWhen run from the cloned repo, all settings are set to default values.\nThese values are publicly visible on GitHub and **should not be used in production!**\n\n\u003e ❗️\n\u003e **At the time of writing, upload within projects created in the development database will most likely not work.**\n\u003e To use the upload functionality with the `CLI`, first create a project.\n\nThe following test usernames ship in the development setup:\n\n- `superadmin`\n- `unituser_1`\n- `unituser_2`\n- `researchuser_1`\n- `researchuser_2`\n\nAll have the password: `password`\n\n### Database changes\n\nIf you modify the database models (e.g. tables or indexes), you must create a migration for the changes. We use `Alembic` (via `flask-migrate`) which compares our database models with the running database to generate a suggested migration.\n\nFor instructions on how to do this, see [the README in the migrations directory](./migrations/README.md).\n\n## Run tests\n\nTests run on github actions on every pull request and push against master and dev. To run the tests locally, use this command:\n\n```bash\ndocker-compose -f docker-compose.yml -f tests/docker-compose-test.yml up --build --exit-code-from backend\n```\n\nThis will create a test database in the mariadb container called `DeliverySystemTest` which will be populated before a test and emptied after a test has finished.\n\nIt's possible to supply arguments to pytest via the environment variable `$DDS_PYTEST_ARGS`.\nFor example to only run the `test_x` inside the file `tests/test_y.py` you would set this variable as follows: `export DDS_PYTEST_ARGS=tests/test_y.py::test_x`.\n\nTo run interactively, use the following command:\n\n```bash\ndocker-compose -f docker-compose.yml -f tests/docker-compose-test-interactive.yml up --build --exit-code-from backend\n```\n\nThen in a new terminal, shell into the container and run pytest:\n\n```bash\ndocker exec -it dds_backend /bin/sh\n```\n\n```bash\npytest\n```\n\nIf you want to run tests quickly, without rebuilding the database each time, set the `SAVE_DB` environment variable:\n\n```bash\nSAVE_DB=1 pytest\n```\n\nNote that this stops the database from being deleted, so it will speed up the _next_ run.\nEqually, if you want to tear down you need to run pytest _twice_ without it, as it only affects the tear down.\n\n---\n\n## Production Instance\n\nThe production version of the backend image is published at the [GitHub Container Registry (GHCR, ghcr.io/scilifelabdatacentre/dds-backend)](https://github.com/scilifelabdatacentre/dds_web/pkgs/container/dds-backend). It can also be built by running:\n\n```bash\ndocker build --target production -f Dockerfiles/backend.Dockerfile .\n```\n\nUse `docker-compose.yml` as a reference for the required environment.\n\n### Configuration\n\nThe environment variable `DDS_APP_CONFIG` defines the location of the config file, e.g. `/code/dds_web/dds_app.cfg`. The config values are listed in `dds_web/config.py`. Add them to the file in the format:\n\n```python\nMAX_CONTENT_LENGTH = 0x1000000\nMAX_DOWNLOAD_LIMIT = 1000000000\n```\n\n\u003e ❗ It is recommended that you redefine all values in `config.py` in your config file to avoid using default values by mistake.\n\n### Initialise the database\n\nBefore you can use the system, you must run `flask db upgrade` to initialise the database schema and prepare for future database migrations. You can also add a superuser by running `flask init-db production`. In order to customize the user, make sure to set the `SUPERADMIN*` config options.\n\n### Upgrades\n\nWhenever you upgrade to a newer version, start by running `flask db upgrade` to make sure that the database schema is up-to-date.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscilifelabdatacentre%2Fdds_web","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscilifelabdatacentre%2Fdds_web","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscilifelabdatacentre%2Fdds_web/lists"}