{"id":20038690,"url":"https://github.com/defra/cdp-portal-backend","last_synced_at":"2026-05-13T06:02:47.953Z","repository":{"id":226237300,"uuid":"681143570","full_name":"DEFRA/cdp-portal-backend","owner":"DEFRA","description":"Git repository for service cdp-portal-backend","archived":false,"fork":false,"pushed_at":"2026-05-05T12:08:47.000Z","size":1656,"stargazers_count":0,"open_issues_count":1,"forks_count":2,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-05-05T13:08:09.473Z","etag":null,"topics":["backend","cdp","dotnet","service"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DEFRA.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-08-21T11:12:06.000Z","updated_at":"2026-05-05T12:06:37.000Z","dependencies_parsed_at":"2024-05-01T17:43:16.931Z","dependency_job_id":"8619d818-edf2-4f42-9d8b-6e013bbe125c","html_url":"https://github.com/DEFRA/cdp-portal-backend","commit_stats":null,"previous_names":["defra/cdp-portal-backend"],"tags_count":453,"template":false,"template_full_name":null,"purl":"pkg:github/DEFRA/cdp-portal-backend","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Fcdp-portal-backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Fcdp-portal-backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Fcdp-portal-backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Fcdp-portal-backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DEFRA","download_url":"https://codeload.github.com/DEFRA/cdp-portal-backend/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Fcdp-portal-backend/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32970190,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-13T05:31:44.146Z","status":"ssl_error","status_checked_at":"2026-05-13T05:31:23.775Z","response_time":115,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backend","cdp","dotnet","service"],"created_at":"2024-11-13T10:31:56.852Z","updated_at":"2026-05-13T06:02:47.947Z","avatar_url":"https://github.com/DEFRA.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CDP Portal Backend\n\nA service for tracking deployments (what's deployed and where) across environments and for discovering, persisting and\nproviding access to all deployable artifacts on the platform (docker and ECR images).\n\n[![Publish](https://github.com/DEFRA/cdp-portal-backend/actions/workflows/publish.yml/badge.svg)](https://github.com/DEFRA/cdp-portal-backend/actions/workflows/publish.yml)\n[![Journey Tests](https://github.com/DEFRA/cdp-portal-backend/actions/workflows/journey-tests.yml/badge.svg)](https://github.com/DEFRA/cdp-portal-backend/actions/workflows/journey-tests.yml)\n\n- [How does it work?](#how-does-it-work)\n- [APIs](#apis)\n    - [Find Services](#find-services)\n    - [Get Artifacts](#get-artifacts)\n    - [Getting content of a file](#getting-content-of-a-file)\n    - [Getting lists of deployable artifacts](#getting-lists-of-deployable-artifacts)\n    - [Admin backend](#admin-backend)\n    - [Getting generic lists of deployments](#getting-generic-lists-of-deployments)\n    - [Getting lists of deployments for each environment](#getting-lists-of-deployments-for-each-environment)\n- [Local Development](#local-development)\n    - [Install MongoDB](#install-mongodb)\n        - [Start MongoDB](#start-mongodb)\n            - [Run it natively](#run-it-natively)\n            - [Running it in docker](#running-it-in-docker)\n            - [Running it with Homebrew on Mac](#running-it-with-homebrew-on-mac)\n        - [Inspect MongoDB](#inspect-mongodb)\n    - [Run CDP Portal Backend application](#run-cdp-portal-backend-application)\n    - [Setup a local docker registry](#setup-a-local-docker-registry)\n    - [Install and run LocalStack AWS](#install-and-run-localstack-aws)\n    - [Create local SQS queues](#create-local-sqs-queues)\n    - [Simulating ECR SQS messages locally](#simulating-ecr-sqs-messages-locally)\n    - [Generate fake deployments](#generate-fake-deployments)\n\n## How does it work\n\nTL;DR the CDP Portal Backend listens to SQS events for ECR image uploads and ECS deployments, populates a mongodb\ndatabase with that information,\nand provides a RESTful API to access that information for the frontend.\n\n**Deployments - the \"what is where\"**\n\n`cdp-portal-backend` listens for AWS ECS deployment events on an SQS queue.\n\nWhen an event comes in it will do the following:\n\n- looks up the aws account id against a list of known environment names\n- extracts the container image name and version from the event\n- discards any containers that dont already exist\n- writes the event to the deployments collection\n\n**Artifacts and images - what we can deploy on the portal**\n\n`cdp-portal-backend` listens for SQS events from the AWS ECR repository. It keeps track of what versions/tags are \navailable and to deploy. If a tag is deleted from ECR it is also deleted from cdp-portal-backend.\n\n## APIs\n\nFor the most up-to-date APIs and schema, run `cdp-portal-backend` locally, as instructed below, and navigate to\n[the Swagger page at http://localhost:5094/swagger](http://localhost:5094/swagger).\n\n### Find Entities\n\nWhen an image is scanned its 'service name' is extracted from a docker label (`defra.cdp.service.name` see\nArtifactScanner.cs).\n\n`GET /entities`\n\n```json\n[\n    {\n        \"name\": \"cdp-portal-backend\",\n        \"type\": \"Microservice\",\n        \"subType\": \"Backend\",\n        \"primaryLanguage\": \"JavaScript\",\n        \"created\": \"2016-12-05T11:21:25Z\",\n        \"teams\": [\n            {\n                \"teamId\": \"aabe63e7-87ef-4beb-a596-c810631fc474\",\n                \"name\": \"Platform\"\n            }\n        ],\n        \"status\": \"Success\",\n        \"tags\": [\"live\", \"beta\"]\n    },\n    {\n        \"name\": \"cdp-portal-frontend\",\n        \"type\": \"Microservice\",\n        \"subType\": \"Frontend\",\n        \"primaryLanguage\": \"JavaScript\",\n        \"created\": \"2016-12-05T11:21:25Z\",\n        \"teams\": [\n            {\n                \"teamId\": \"aabe63e7-87ef-4beb-a596-c810631fc474\",\n                \"name\": \"Platform\"\n            }\n        ],\n        \"status\": \"Success\",\n        \"tags\": [\"live\"]\n    }\n]\n\n```\n\n`GET /entities/cdp-portal-backend`\n\n```json\n{\n    \"name\": \"cdp-portal-frontend\",\n    \"type\": \"Microservice\",\n    \"subType\": \"Frontend\",\n    \"primaryLanguage\": \"JavaScript\",\n    \"created\": \"2016-12-05T11:21:25Z\",\n    \"teams\": [\n        {\n            \"teamId\": \"aabe63e7-87ef-4beb-a596-c810631fc474\",\n            \"name\": \"Platform\"\n        }\n    ],\n    \"status\": \"Success\",\n    \"tags\": [\"live\"]\n}\n```\n\n### Get Artifacts\n\n`GET /artifacts`\n\n`GET /artifacts/foo/1.2.3`\n\n```json\n\n{\n    \"created\": \"2023-05-16T11:03:47.732Z\",\n    \"repo\": \"cdp-portal-backend\",\n    \"tag\": \"v0.21.0\",\n    \"sha256\": \"sha256:157e63cedba182003e9831047e3f611c516e8d67a4425dfcb15d7d7295c17872\",\n    \"githubUrl\": \"https://github.com/DEFRA/cdp-portal-backend\",\n    \"serviceName\": \"cdp-portal-backend\",\n    \"scannerVersion\": 1,\n    \"files\": [\n        {\n            \"fileName\": \"Defra.Cdp.Deployments.deps.json\",\n            \"path\": \"app/Defra.Cdp.Deployments.deps.json\",\n            \"layerSha256\": \"sha256:115ae824b21b9f671e304112f034096061d8d56b655509561207084fd1f3ccde\"\n        }\n    ]\n}\n```\n\n### Getting content of a file\n\n`GET /files/sha256:115ae824b21b9f671e304112f034096061d8d56b655509561207084fd1f3ccde?path=app%2FDefra.Cdp.Deployments.deps.json`\n(you'll likely need to url encode the path parameter)\n\n```json\n{\n    \"name\": \"cdp-node-frontend-exemplar\",\n    \"version\": \"0.1.0\",\n    \"lockfileVersion\": 3,\n    \"requires\": true,\n    \"packages\": {\n        ...\n```\n\nTakes the layerSha256 value from the artifact data and the fileName as a query param.\nReturns a 404 if the filename is invalid.\nIf you can retrieve the artifact, you have all the data required to access the file (digest and path).\n\n### Getting lists of deployable artifacts\n\n`GET /deployables`\n\n`GET /deployables/image-name`\n\nBoth provide a list of docker image names and tags for that image.\n\n### Admin backend\n\n`POST /admin/scan?repo=foo\u0026tag=1.2.3`\nTriggers a scan of the given image/tag. This will grab the Docker image from ECR, scan the manifest and updates the\n`artifacts` collection.\n\n`POST /admin/backfill`\nRescan everything in the docker registry. This will take a long time!\nCurrently it will *not* drop the existing records, that should be done manually.\n\n### Getting generic lists of deployments\n\n`GET /deployments` returns the most recent deployment events, 1 per task. For services with more than one instance\nyou'll get 1 event per instance rather than the whole deployment\n\n`GET /deployments/:deploymentId` returns details about a specific deployment\n\n### Getting lists of deployments for each environment\n\n`GET /running-services` return the most recent deployment for each service in each environment\n\n`GET /running-services/:service` return the most recent deployment for a given service in each environment\n\n### Get list of all GitHub repositories\n\n`GET /repositories` returns all repositories for the organisation\n\n`GET /repositories?team=exampleTeam` returns repositories owned by this team\n\n`GET /templates` returns all repositories that are templates in the Portal\n\n`GET /templates?team=exampleTeam` returns templates for a specific team\n\n## Local Development\n\n### Set up pre-commit script:\n```bash\ngit config core.hooksPath .githooks\n```\n\n### Install MongoDB\n\n- Install [MongoDB](https://www.mongodb.com/docs/manual/tutorial/#installation) on your local machine\n    - Consider docker if running on Linux or WSL2 for Windows 10/11\n    - Consider Homebrew for MacOS as Mongodb in docker for Apple Silicon can be temperamental\n\n#### Start MongoDB\n\n##### Run it natively\n\n```bash\nsudo mongod --dbpath ~/mongodb-cdp\n```\n\n##### Running it in docker\n\n\u003e Note: using the ubuntu image to avoid licensing issues with the ubi8 (redhat linux) image\n\n```bash\ndocker run --name mongodb-cdp -d -p 27017:27017 mongodb/mongodb-community-server:6.0.7-ubuntu2204\n```\n\n##### Running it with Homebrew on Mac\n\n\u003e Note: if installing mongodb community edition\n\n```bash\nbrew services start mongodb-community@6.0\n```\n\n#### Inspect MongoDB\n\nTo inspect the Database and Collections locally:\n\n```bash\nmongosh\n```\n\n### Run CDP Portal Backend application\n\n- Set optional `ASPNETCORE_ENVIRONMENT` environment variable:\n\n```bash\nexport ASPNETCORE_ENVIRONMENT=Development \nexport GITHUB__TOKEN=\u003ctoken_from_secret_manager\u003e\n```\n\n- To run the `cdp-portal-backend` application:\n\n```bash\ndotnet run --project Defra.Cdp.Backend.Api --no-launch-profile\n```\n\nIf you use Jetbrains Rider, make sure you create a configuration for `.NET Project` and not\n`.NET Launch Setting Profile` so you can add your own environment variable there.\nIt should look something like this:\n\n![rider-settings](images/rider-settings.png)\n![rider-env](images/rider-env.png)\n### Setup a local docker registry\n\nRunning a local docker registry enables the `cdp-portal-backend` to obtain the manifests from local registry images.\n\n```bash\ndocker run -d -p 5000:5000 --restart=always --name registry registry:2\n```\n\n\u003e Note: If you need to update the external port also change it in the `Docker.RegistryUrl` property within\n`appsettings.Development.json`\n\nYou will then need to pull an image from ECR. This assumes you have AWS CLI setup correctly.\n\n```bash\naws ecr get-login-password --region eu-west-2 --profile management | docker login --username AWS --password-stdin \u003caccount_number\u003e.dkr.ecr.eu-west-2.amazonaws.com\n\ndocker pull  \u003caccount_number\u003e.dkr.ecr.eu-west-2.amazonaws.com/cdp-portal-frontend:\u003cversion\u003e\n```\n\nThen tag the pulled image\n\n```bash\ndocker tag \u003caccount_number\u003e.dkr.ecr.eu-west-2.amazonaws.com/cdp-portal-frontend:\u003cversion\u003e localhost:5000/cdp-portal-frontend:\u003cversion\u003e\n```\n\nThe push the tagged image to your local docker registry\n\n```bash\ndocker push localhost:5000/cdp-portal-frontend:\u003cversion\u003e \n```\n\nThis will make sure that the `cdp-portal-backend` can obtain the manifest for the versions in our local MongoDB.\n\n### Install and run LocalStack AWS\n\n- Install [LocalStack AWS CLI](https://github.com/localstack/awscli-local)\n- Run AWS LocalStack Docker container:\n\n```bash\ndocker run --pull=always -d -p 4566:4566 -p 4510-4559:4510-4559 localstack/localstack:latest\n```\n\n### Create local SQS queues\n\n- Create `ecs-deployments` local SQS queue:\n\n```bash\nawslocal sqs create-queue --queue-name ecs-deployments\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdefra%2Fcdp-portal-backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdefra%2Fcdp-portal-backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdefra%2Fcdp-portal-backend/lists"}