{"id":24417231,"url":"https://github.com/defra/cdp-uploader","last_synced_at":"2026-04-22T12:32:57.963Z","repository":{"id":227677433,"uuid":"772125845","full_name":"DEFRA/cdp-uploader","owner":"DEFRA","description":"Git repository for service cdp-uploader","archived":false,"fork":false,"pushed_at":"2024-04-12T18:40:16.000Z","size":1204,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-14T06:45:25.285Z","etag":null,"topics":["cdp","frontend","node","service"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DEFRA.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-03-14T15:28:20.000Z","updated_at":"2024-04-17T05:25:27.825Z","dependencies_parsed_at":"2024-04-17T05:24:56.890Z","dependency_job_id":null,"html_url":"https://github.com/DEFRA/cdp-uploader","commit_stats":null,"previous_names":["defra/cdp-uploader"],"tags_count":40,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Fcdp-uploader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Fcdp-uploader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Fcdp-uploader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Fcdp-uploader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DEFRA","download_url":"https://codeload.github.com/DEFRA/cdp-uploader/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243402285,"owners_count":20285192,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdp","frontend","node","service"],"created_at":"2025-01-20T08:15:56.102Z","updated_at":"2025-12-27T16:03:25.223Z","avatar_url":"https://github.com/DEFRA.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cdp-uploader\n\nCore delivery platform Node.js Frontend Template.\n\n- [Requirements](#requirements)\n  - [Node.js](#nodejs)\n- [API](#api)\n  - [POST /initiate](#post-initiate)\n    - [HTTP Headers:](#http-headers)\n    - [Body parameters:](#body-parameters)\n    - [Example response](#example-response)\n  - [POST /upload-and-scan/{uploadId}](#post-upload-and-scanuploadid)\n    - [Path Parameters](#path-parameters)\n    - [Response](#response)\n  - [GET /status/{uploadId}](#get-statusuploadid)\n    - [Path Parameters](#path-parameters-1)\n    - [Query Parameters](#query-parameters)\n    - [Response Payload with Debug](#response-payload-with-debug)\n    - [Response Payload without Debug](#response-payload-without-debug)\n    - [File field in form](#file-field-in-form)\n    - [Error Handling](#error-handling)\n  - [Callback](#callback)\n    - [Intended use](#intended-use)\n- [S3](#s3)\n  - [Object metadata on S3 Objects](#object-metadata-on-s3-objects)\n- [Configuration](#configuration)\n  - [App config](#app-config)\n  - [Secrets](#secrets)\n  - [Docker compose](#docker-compose)\n  - [Local dev env var](#local-dev-env-var)\n  - [Configuration names](#configuration-names)\n- [Local development](#local-development)\n  - [Developing services that use the CDP-Uploader](#developing-services-that-use-the-cdp-uploader)\n    - [Docker Compose](#docker-compose-1)\n    - [Relative vs Absolute URLs](#relative-vs-absolute-urls)\n    - [Test Harness (mock scanning)](#test-harness-mock-scanning)\n    - [EICAR files and local development](#eicar-files-and-local-development)\n  - [Setup for developing CDP-Uploader](#setup-for-developing-cdp-uploader)\n  - [Development](#development)\n    - [Updating dependencies](#updating-dependencies)\n    - [Tests](#tests)\n      - [Unit tests](#unit-tests)\n      - [End-to-end tests](#end-to-end-tests)\n      - [Smoke tests](#smoke-tests)\n  - [AWS CLI](#aws-cli)\n  - [AWS Local](#aws-local)\n    - [AWS local alias](#aws-local-alias)\n  - [LocalStack](#localstack)\n    - [Docker](#docker)\n  - [Localstack CLI](#localstack-cli)\n    - [Setup local S3 buckets](#setup-local-s3-buckets)\n    - [List local buckets](#list-local-buckets)\n    - [View bucket contents](#view-bucket-contents)\n    - [Empty bucket contents](#empty-bucket-contents)\n    - [Setup local queues](#setup-local-queues)\n    - [Purge local queues](#purge-local-queues)\n    - [Setup local test harness](#setup-local-test-harness)\n  - [Local JSON API](#local-json-api)\n  - [Production](#production)\n  - [Npm scripts](#npm-scripts)\n- [Docker](#docker-1)\n  - [Development image](#development-image)\n  - [Production image](#production-image)\n- [Licence](#licence)\n  - [About the licence](#about-the-licence)\n\n# Requirements\n\n## Node.js\n\nPlease install [Node.js](http://nodejs.org/) `\u003e= v18` and [npm](https://nodejs.org/) `\u003e= v9`. You will find it\neasier to use the Node Version Manager [nvm](https://github.com/creationix/nvm)\n\nTo use the correct version of Node.js for this application, via nvm:\n\n```bash\ncd cdp-uploader\nnvm use\n```\n\n# API\n\n## POST /initiate\n\nExample `/initiate` request:\n\n```json\n{\n  \"redirect\": \"https://myservice.com/nextPage\",\n  \"callback\": \"https://myservice.com/callback\",\n  \"s3Bucket\": \"myservice\",\n  \"s3Path\": \"scanned\",\n  \"metadata\": {\n    \"customerId\": \"1234\",\n    \"accountId\": \"1234\"\n  }\n}\n```\n\nExample `/initiate` download request:\n\n```json\n{\n  \"downloadUrls\": [\"https://myservice.com/file\"],\n  \"callback\": \"https://myservice.com/callback\",\n  \"s3Bucket\": \"myservice\",\n  \"s3Path\": \"scanned\",\n  \"metadata\": {\n    \"customerId\": \"1234\",\n    \"accountId\": \"1234\"\n  }\n}\n```\n\n### HTTP Headers:\n\n| Header name | Description                                       | Required |\n| ----------- | ------------------------------------------------- | -------- |\n| User-Agent  | Identifier of the service that calls cdp-uploader | TBD      |\n\n### Body parameters:\n\n| Parameter name | Description                                                           | Required |\n| -------------- | --------------------------------------------------------------------- |----------|\n| redirect       | URL to redirect to after file has been successfully uploaded. Cannot be used together with downloadUrls.         | no       |\n| downloadUrls | List of URLs pointing to files that should be downloaded and scanned. Cannot be used together with redirect. | no |\n| s3Bucket       | S3 bucket that file will be moved to once the scanning is complete    | yes      |\n| s3Path         | 'Folder' in bucket where scanned files will be placed                 | no       |\n| callback       | Url that will be called once all files in upload have been scanned    | no       |\n| metadata       | Map of additional information related to upload                       | no       |\n| mimeTypes      | List of accepted mimeTypes                                            | no       |\n| maxFileSize    | Maximum size in bytes that a file can be (10MB is 10 \\* 1000 \\* 1000) | no       |\n\n\u003e [!NOTE]\n\u003e We will generate an uploadId for the upload and fileIds for files in the upload request. Files (objects) will be moved\n\u003e to destination bucket under the path uploadId/fileId which will be prefixed with the bucket path if it has been\n\u003e provided.\n\u003e\n\u003e The /initiate endpoint now supports initiating uploads via download URLs.\nYou can either specify a redirect or downloadUrls, but not both.\nWhen downloadUrls are provided, cdp-uploader downloads and scans the files asynchronously on behalf of the caller.\n\n### Example response\n\n```json\n{\n  \"uploadId\": \"b18ceadb-afb1-4955-a70b-256bf94444d5\",\n  \"uploadUrl\": \"/upload-and-scan/b18ceadb-afb1-4955-a70b-256bf94444d5\",\n  \"statusUrl\": \"https://cdp-uploader/status/b18ceadb-afb1-4955-a70b-256bf94444d5\"\n}\n```\n\n| Parameter name | Description                                                                                                   | Required |\n| -------------- |---------------------------------------------------------------------------------------------------------------|----------|\n| uploadId       | Identifier used for the upload                                                                                | yes      |\n| uploadUrl      | Url which must be used for the upload. If the initiate was a download request, this field will not be present | no       |\n| statusUrl      | Endpoint that can be polled for status of upload                                                              | yes      |\n\n## POST /upload-and-scan/{uploadId}\n\n### Path Parameters\n\n| Parameter Name | Description |\n| -------------- | ------------------------------------------------------------------------ |\n| `uploadId` | Unique ID for the upload. The `uploadId` is provided via the `/initiate` call. |\n\n### Description\n\nThe `/upload-and-scan/{uploadId}` endpoint supports two types of POST requests:\n\n1. **Browser uploads (multipart/form-data):** For public-facing forms or user-facing applications where a file is\nsubmitted directly from the browser.\n\n2. **API uploads (binary or JSON body):** For integrations from other government departments or data centres that need\nto programmatically upload files. These can use a direct `POST` request with an appropriate `Content-Type` header\n(e.g. `application/octet-stream`) and an optional `x-filename` header for metadata.\n\nThis flexibility allows simple integration with automated or API-based workflows as well as traditional web forms.\n\n---\n\n### Example: Browser (multipart/form-data)\n\n```html\n\u003cform\n  action=\"/upload-and-scan/b18ceadb-afb1-4955-a70b-256bf94444d5\"\n  method=\"post\"\n  enctype=\"multipart/form-data\"\n\u003e\n  \u003clabel for=\"file\"\u003eFile\u003c/label\u003e\n  \u003cinput id=\"file\" name=\"file\" type=\"file\" /\u003e\n  \u003cbutton\u003eUpload\u003c/button\u003e\n\u003c/form\u003e\n```\n\n---\n\n### Example: API (binary upload)\n\n```bash\ncurl -X POST \"https://example.gov.uk/upload-and-scan/b18ceadb-afb1-4955-a70b-256bf94444d5\"   -H \"Content-Type: application/octet-stream\"   -H \"x-filename: report.pdf\"   --data-binary \"@report.pdf\"\n```\n\n---\n\n### Response\n\nOnce the upload is accepted, the user (or API client) will be redirected to the redirect URL provided in the `/initiate`\nresponse.\nThe CDP Uploader will then asynchronously perform a **virus scan** on the uploaded file.\n\nFor **API integrations**, this redirect can be ignored — frontend clients should prevent automatic redirect following\n(for example, by using fetch with redirect: 'manual' or an equivalent mechanism).\n\nInstead, the integration can rely on one of the following approaches:\n- Callback (optional): If a callback URL is configured, the CDP Uploader will send a callback once the virus scan is complete, including the scan result and file metadata.\n- Polling: If no callback URL is configured, the tenant or API client can poll the status endpoint to check the progress and outcome of the upload.\n\n## GET /status/{uploadId}\n\nThe status API provides information about uploaded files, virus scan status and S3 location.\nThe API is intended to be polled by the frontend services, it is not public and cannot be called direct from the\nbrowser.\n\n### Path Parameters\n\n| Parameter Name | Description                                                              |\n| -------------- | ------------------------------------------------------------------------ |\n| uploadId       | Unique id for that upload. UploadId is provided via the `/initiate` call |\n\n### Query Parameters\n\n| Parameter Name | Description                                                                     |\n| -------------- | ------------------------------------------------------------------------------- |\n| debug          | set to 'true' to debug information. Currently contains initiate request payload |\n\n\u003e [!NOTE]\n\u003e Debug should not be used in production\n\n### Response Payload with Debug\n\n```json\n{\n  \"debug\": {\n    \"request\": {\n      \"redirect\": \"http://localhost:3000/creatures/e90fee47-ead0-45c7-8319-4388cca9ebbc/upload-status-poller?uploadId=ba477d67-0005-4c26-a673-fa5139a2adf5\",\n      \"s3Bucket\": \"cdp-example-bucket\",\n      \"metadata\": {\n        \"example-id\": \"id\"\n      }\n    }\n  },\n  \"uploadStatus\": \"ready\",\n  \"metadata\": {\n    \"example-id\": \"id\"\n  },\n  \"form\": {\n    \"a-form-field\": \"some value\",\n    \"a-file-upload-field\": {\n      \"fileId\": \"9fcaabe5-77ec-44db-8356-3a6e8dc51b13\",\n      \"filename\": \"dragon-b.jpeg\",\n      \"contentType\": \"image/jpeg\",\n      \"fileStatus\": \"complete\",\n      \"contentLength\": 11264,\n      \"checksumSha256\": \"bng5jOVC6TxEgwTUlX4DikFtDEYEc8vQTsOP0ZAv21c=\",\n      \"detectedContentType\": \"image/jpeg\",\n      \"s3Key\": \"3b0b2a02-a669-44ba-9b78-bd5cb8460253/9fcaabe5-77ec-44db-8356-3a6e8dc51b13\",\n      \"s3Bucket\": \"cdp-example-node-frontend\"\n    },\n    \"another-form-field\": \"foobazbar\"\n  },\n  \"numberOfRejectedFiles\": 0\n}\n```\n\n### Response Payload without Debug\n\n```json\n{\n  \"uploadStatus\": \"ready\",\n  \"metadata\": {\n    \"example-id\": \"id\"\n  },\n  \"form\": {\n    \"a-form-field\": \"some value\",\n    \"a-file-upload-field\": {\n      \"fileId\": \"9fcaabe5-77ec-44db-8356-3a6e8dc51b13\",\n      \"filename\": \"dragon-b.jpeg\",\n      \"contentType\": \"image/jpeg\",\n      \"fileStatus\": \"complete\",\n      \"contentLength\": 11264,\n      \"checksumSha256\": \"bng5jOVC6TxEgwTUlX4DikFtDEYEc8vQTsOP0ZAv21c=\",\n      \"detectedContentType\": \"image/jpeg\",\n      \"s3Key\": \"3b0b2a02-a669-44ba-9b78-bd5cb8460253/9fcaabe5-77ec-44db-8356-3a6e8dc51b13\",\n      \"s3Bucket\": \"cdp-example-node-frontend\"\n    },\n    \"another-form-field\": \"foobazbar\"\n  },\n  \"numberOfRejectedFiles\": 0\n}\n```\n\n### Example response with downloadUrls\n\nIf a single download URL was provided:\n```json\n{\n  \"uploadStatus\": \"ready\",\n  \"metadata\": {\n    \"customerId\": \"1234\",\n    \"accountId\": \"1234\"\n  },\n  \"form\": {\n    \"file\": {\n      \"fileId\": \"16463a29-a040-4921-8c98-b1adf3ff09ec\",\n      \"filename\": \"example-document.pdf\",\n      \"contentType\": \"application/pdf\",\n      \"downloadUrl\": \"https://my-bucket.s3.eu-west-2.amazonaws.com/scanned/example-document.pdf\",\n      \"fileStatus\": \"complete\",\n      \"contentLength\": 204800,\n      \"checksumSha256\": \"czZ6B/jFbdYI+uhuGpnQ097MjOwdsW4s0kEYL6vwMMo=\",\n      \"detectedContentType\": \"application/pdf\",\n      \"s3Key\": \"2Fcb87c080-f5d0-49e5-9afe-712bab4f4ab4/16463a29-a040-4921-8c98-b1adf3ff09ec\",\n      \"s3Bucket\": \"my-bucket\"\n    }\n  },\n  \"numberOfRejectedFiles\": 0\n}\n```\n\nIf multiple download URLs were provided:\n\n```json\n{\n  \"uploadStatus\": \"ready\",\n  \"metadata\": {\n    \"customerId\": \"1234\",\n    \"accountId\": \"1234\",\n    \"some-metadata\": \"example\"\n  },\n  \"form\": {\n    \"files\": [\n      {\n        \"fileId\": \"16463a29-a040-4921-8c98-b1adf3ff09ec\",\n        \"filename\": \"document-1.pdf\",\n        \"contentType\": \"application/pdf\",\n        \"downloadUrl\": \"https://my-bucket.s3.eu-west-2.amazonaws.com/scanned/document-1.pdf\",\n        \"fileStatus\": \"complete\",\n        \"contentLength\": 204800,\n        \"checksumSha256\": \"czZ6B/jFbdYI+uhuGpnQ097MjOwdsW4s0kEYL6vwMMo=\",\n        \"detectedContentType\": \"application/pdf\",\n        \"s3Key\": \"2Fcb87c080-f5d0-49e5-9afe-712bab4f4ab4/16463a29-a040-4921-8c98-b1adf3ff09ec\",\n        \"s3Bucket\": \"my-bucket\"\n      },\n      {\n        \"fileId\": \"4549f0dc-48d2-4bf3-955e-ae0fcef0ea41\",\n        \"filename\": \"document-2.pdf\",\n        \"contentType\": \"application/pdf\",\n        \"downloadUrl\": \"https://my-bucket.s3.eu-west-2.amazonaws.com/scanned/document-2.pdf\",\n        \"fileStatus\": \"complete\",\n        \"contentLength\": 104800,\n        \"checksumSha256\": \"bng5jOVC6TxEgwTUlX4DikFtDEYEc8vQTsOP0ZAv21c=\",\n        \"detectedContentType\": \"application/pdf\",\n        \"s3Key\": \"2Fcb87c080-f5d0-49e5-9afe-712bab4f4ab4/4549f0dc-48d2-4bf3-955e-ae0fcef0ea41\",\n        \"s3Bucket\": \"my-bucket\"\n      }\n    ]\n  },\n  \"numberOfRejectedFiles\": 0\n}\n```\n\n| Scenario                                    | Behaviour                                                                              |\n| ------------------------------------------- |----------------------------------------------------------------------------------------|\n| `redirect` provided                         | Uploader waits for browser uploads - scanned asynchronously                            |\n| `downloadUrls` provided                     | Files are downloaded and scanned **automatically and asynchronously** by cdp-uploader. |\n| Both `redirect` and `downloadUrls` provided | Request is rejected with an error message (mutually exclusive).                        |\n| Multiple `downloadUrls` provided            | Response includes an array under `form.files`.                                         |\n| Single `downloadUrl` provided               | Response includes a single `form.file` object.                                         |\n\n\n| Parameter Name        | Description                                                                                                                                                  |\n| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |\n| uploaderStatus        | Have all scans completed, can be `initiated`, `pending` or `ready`                                                                                           |\n| metadata              | Extra data and identified set by the requesting service in the /initialize call. Returned exactly as they were presented                                     |\n| form                  | An object representing each field in the multipart request. Text fields are preserved exactly as they were sent, file fields contain details about the file. |\n| numberOfRejectedFiles | Total number of files that have been rejected by the uploader                                                                                                |\n| debug.request         | When set to true, the initiate request payload received by cdp-uploader                                                                                      |\n\n### File field in form\n\n| Parameter Name      | Description                                                                                                                        |\n| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |\n| fileId              | uuid of the file.                                                                                                                  |\n| filename            | filename of file uploaded, if present                                                                                              |\n| contentType         | The mime type as declared in the multipart upload                                                                                  |\n| fileStatus          | `complete` or `rejected` if the virus scan has completed, `pending` if its still in progress                                       |\n| contentLength       | Size of file in bytes                                                                                                              |\n| checksumSha256      | SHA256 check sum of file recieved by cdp-uploader before uploading to S3 bucket                                                    |\n| detectedContentType | The mime type as detected by the CDP-Uploader                                                                                      |\n| s3Bucket            | S3 bucket where scanned file is moved. Only set if file status is `complete`                                                       |\n| s3Key               | S3 Path where scanned file is moved. Includes path prefix if set. Only set when fileStatus is `complete`                           |\n| hasError            | `true/false` Only set to true if the file has been rejected or could not be delivered. Reason is supplied in `errorMessage` field. |\n| errorMessage        | Reason why file was rejected. Error message is based on GDS design guidelines and can be show directly to the end-user.            |\n\n### Error Handling\n\nA file can be rejected by uploader for a number of reasons. When this happens no file will be delivered to the\ndestination S3 bucket.\nA rejected file has the following data set:\n\n- fileStatus: rejected\n- hasError: true\n- errorMessage: string\n\nThe `errorMessage` field is a test description of why the file was rejected.\n\n| Cause                                                                                       | errorMessage                                            |\n|---------------------------------------------------------------------------------------------|---------------------------------------------------------|\n| Virus detected                                                                              | `The selected file contains a virus`                    |\n| File is empty                                                                               | `The selected file is empty`                            |\n| File size exceeds max size (either set in the /init call or the uploaders max default 100M) | `The selected file must be smaller than $MAXSIZE`       |\n| File doesn't match the mime types set in the init call                                      | `The selected file must be a $MIMETYPES`                |\n| Any server side error in CDP-Uploader                                                       | `The selected file could not be uploaded – try again`   |\n| Failed download                                                                             | `The selected file could not be downloaded`             |\n\n\nThe messages are based on the [GDS File Upload guidelines](https://design-system.service.gov.uk/components/file-upload/)\n\nThe intention of the `errorMessage` field is that the content can be displayed directly to the end user.\n\n## Callback\n\nIf a callback url has been provided in the initiate request, we will POST a callback to your service once scanning is\ncomplete and files have been moved to your services bucket. The payload is exactly the same as the response from\nthe [Status](#get-statusuploadid) endpoint (without debug enabled).\n\n### Intended use\n\nFrontend services will poll the /status endpoint after a user has uploaded a file.\n\nThe payload contains all of the fields submitted in the HTML form with the values preserved with the exception of any\nfields that contained files. These fields are replaced with an object showing if they passed the scan and where in S3\nthey can be accessed.\n\nFrontend services are expected to validate the content of the payload and present any errors back to the user.\nAny files uploaded by a user will never be sent directly to the frontend service.\n\n# S3\n\n## Object metadata on S3 Objects\n\nThe objects placed in your buckets contain the following metadata fields and will be returned by AWS as response headers\n\n| Parameter Name             | Description                                                                                                                                      |\n| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |\n| Content-Type               | AWS system defined metadata of what AWS deem the content type to be.                                                                             |\n| x-amz-meta-contenttype     | What cdp-uploader deemed the content-type to be from inspecting the file.                                                                        |\n| x-amz-meta-encodedfilename | Name of uploaded file when it contained non ascii characters. This is RFC-2047 encoded. Optional.                                                |\n| x-amz-meta-filename        | Name of uploaded file when it does not contain non ascii characters. x-amz-meta-encodedfilename will be returned instead when it does. Optional. |\n| x-amz-meta-fileid          | uuid of the file.                                                                                                                                |\n| x-amz-meta-uploadid        | Unique id for that upload. UploadId is provided via the `/initiate` call.                                                                        |\n\n# Configuration\n\nRuntime environment variables required or optional.\n\nAdd these where relevant to:\n\n## App config\n\nFor configurations that are not sensitive and per staging and prod environments.\nhttps://github.com/DEFRA/cdp-app-config\n\n## Secrets\n\nFor sensitive config for staging and prod environments\nthis can be self-serviced in your portal service page's **secrets** tab.\n\nhttps://portal.cdp-int.defra.cloud/services/YOURSERVICE/secrets\n\n## Docker compose\n\nFor local docker compose setups add these to your setup,\ne.g. modify locally your `compose/aws.env` or similar.\n\n## Local dev env var\n\nSet as environment variables locally.\nE.g. as `export` commands if using **bash**, or `.envrc` if using [**direnv**](https://direnv.net/)\n\n## Configuration names\n\n| Config name            | ENV_VAR                 | Default | Required | Purpose                                                                                               |\n| ---------------------- | ----------------------- | ------- | -------- | ----------------------------------------------------------------------------------------------------- |\n| `bucketsAllowlist`     | CONSUMER_BUCKETS        | []      | [x]      | A comma separated list of buckets uploader can write to. Can not be empty if not in development mode. |\n| `mockVirusScanEnabled` | MOCK_VIRUS_SCAN_ENABLED | false   |          | Boolean. Useful in local development                                                                  |\n\nThere are several other configs, such as AWS details, SQS queue names, polling times etc.\nFor more details and other service configuration look in `src/config/index.js`\n\n# Local development\n\n## Developing services that use the CDP-Uploader\n\nIf your service is going to use the CDP-Uploader to receive files you may want to start by running the uploader locally.\nThe easiest way to do this is using `docker compose`.\n\n### Docker Compose\n\nThe CDP-Uploader project provide as base [compose.yml](compose.yml) file to get you started.\n\nCopy [compose.yml](compose.yml) as well as the [./compose](./compose) folder into your own project and running\n`docker compose pull` and then `docker compose up`.\n\nThis will start:\n\n- redis\n- localstack (a local AWS emulator)\n- cdp-uploader\n\nIt will also inject a [start-up script](./compose/start-localstack.sh) into the localstack container that automatically\ncreates the queues and buckets needed by the uploader as well as a test bucket named 'my-bucket'.\nIf your service requires its own bucket you can add a line to [start-localstack.sh](./compose/start-localstack.sh)\nscript to create one.\n\n```yaml\naws --endpoint-url=http://localhost:4566 s3 mb s3://cdp-uploader-quarantine\naws --endpoint-url=http://localhost:4566 s3 mb s3://my-bucket\n\n  ## Insert your bucket here, e.g.\naws --endpoint-url=http://localhost:4566 s3 mb s3://your-service-bucket\n```\n\n\u003e [!NOTE]\n\u003e The script sets the mock AWS region to be `eu-west-2`. See [aws.env](./compose/aws.env) for the other localstack\n\u003e environment variables.\n\u003e If your service is also talking to S3 then it will need to use the same region and credentials when talking to\n\u003e localstack.\n\u003e The can be done by simply setting environment variables before running your sevices locally:\n\u003e\n\u003e ```bash\n\u003e export AWS_REGION=eu-west-2\n\u003e export AWS_ACCESS_KEY_ID=test\n\u003e export AWS_SECRET_ACCESS_KEY=test\n\u003e ```\n\nIf everything has worked as expected the CDP-Uploader will be available on `localhost` port `7337`.\n\nAny other supporting services can be added to the compose file as required.\n\n### Relative vs Absolute URLs\n\nIn a real environment relative urls work fine since all the services are behind the same host, however locally they're\ngoing to be running on different ports so relative redirect URLs won't work!\n\nWhen run in `development` mode the cdp-uploader will convert relative redirect urls into absolute urls using the '\nreferer' header. This should make uploader behave more or less the same locally as it would in a real environment.\n\n### Test Harness (mock scanning)\n\nWhen running locally the CDP-Uploader will be running with its mock virus scanner enabled.\n\nThis _does not_ actually virus scan files, rather it simulates a response based on filename. If you submit a file with\nthe word `virus` in the name it will be flagged as infected.\n\n### EICAR files and local development\n\nThe test harness does not support [EICAR](https://www.eicar.org/download-anti-malware-testfile/) virus test files yet,\nso if you are submitting one and getting a `CLEAN` response back from the uploader's test harness, this is expected. In\nthe real environments EICAR files will work.\n\n## Setup for developing CDP-Uploader\n\nInstall application dependencies:\n\n```bash\nnpm install\n```\n\n## Development\n\nTo run the application in `development` mode run:\n\n```bash\nnpm run dev\n```\n\n### Updating dependencies\n\nTo update dependencies, globally install https://www.npmjs.com/package/npm-check-updates. Then run the below script,\nrun tests, test the application and commit the altered `package.json` and `package-lock.json` files. For more\noptions around updates check the package docs.\n\n```bash\nncu -i\n```\n\n### Tests\n\n#### Unit tests\n\nTo run unit tests:\n\n```bash\nnpm test\n```\n\n#### End-to-end tests\n\nTo run end-to-end tests, you will need to have a few docker compose services running locally:\n\n```bash\nexport AWS_REGION=eu-west-2\nexport AWS_ACCESS_KEY_ID=test\nexport AWS_SECRET_ACCESS_KEY=test\n\ndocker compose pull\ndocker compose up -d localstack redis\nnpm run test:e2e\n```\n\n#### Smoke tests\n\nTo run smoke tests start up the `cdp-uploader` via docker compose:\n\n```bash\ncd cdp-uploader\ndocker compose pull\ndocker compose up\n```\n\nThen clone https://github.com/DEFRA/cdp-uploader-smoke-tests and run it:\n\n```bash\ncd cdp-uploader-smoke-tests\nENVIRONMENT=local npm run test\n```\n\n## AWS CLI\n\n- Install the AWS CLI https://aws.amazon.com/cli/\n\n## AWS Local\n\n- _awslocal_ is a wrapper around the _AWS CLI_ that talks to your local _localstack_\n- You can install a PIP https://github.com/localstack/awscli-local\n  But note it only works with the older v1 of _AWS CLI_\n\n### AWS local alias\n\n- Alternatively just alias it locally:\n  - E.g in _fish_\n\n```fish\nfunction awslocal\n  aws --profile localstack $argv --endpoint-url http://localhost:4566\nend\n\n```\n\n- And add to your `.aws/credentials`\n\n```\n[localstack]\naws_access_key_id = test\naws_secret_access_key = test\nregion=eu-west-2\n```\n\n## LocalStack\n\nEither run _localstack_ directly via _docker_ and _AWS CLI_, or via the _localstack CLI_.\n\n### Docker\n\n- Run AWS LocalStack Docker container:\n\n```bash\ndocker run --pull=always -d -p 4566:4566 -p 4510-4559:4510-4559 localstack/localstack:latest\n```\n\nNote the exposed endpoint is `http://localhost:4566`\n\n## Localstack CLI\n\n- Install [LocalStack CLI](https://docs.localstack.cloud/getting-started/installation/#localstack-cli)\n- Run the CLI\n\n```\nlocalstack start\n```\n\nNote the exposed endpoint is `https://localhost:4566`\n\n### Setup local S3 buckets\n\nYou need local buckets setup in localstack\n\n```bash\n\naws --endpoint-url=http://localhost:4566 s3 mb s3://my-bucket\naws --endpoint-url=http://localhost:4566 s3 mb s3://cdp-uploader-quarantine\n```\n\nThe `--endpoint-url http://localhost:4566` may not be needed depending on how your `awslocal` is set up.\nAlso note depending on how your _localstack_ is running the endpoint may be `http` or `https`.\n\n### List local buckets\n\n```bash\naws --endpoint-url=http://localhost:4566 s3 list-buckets\n```\n\n### View bucket contents\n\n```bash\naws --endpoint-url=http://localhost:4566 s3 ls s3://cdp-uploader-quarantine\naws --endpoint-url=http://localhost:4566 s3 ls s3://my-bucket\n```\n\nOf view in your browser:\n\n```\nhttp://localhost:4566/cdp-uploader-quarantine/\nhttp://localhost:4566/my-bucket/\n```\n\n### Empty bucket contents\n\n```bash\naws --endpoint-url=http://localhost:4566 s3 rm s3://cdp-uploader-quarantine --recursive\naws --endpoint-url=http://localhost:4566 s3 rm s3://my-bucket --recursive\n```\n\n### Setup local queues\n\n```bash\naws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name cdp-clamav-results\naws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name cdp-uploader-scan-results-callback.fifo --attributes \"{\\\"FifoQueue\\\":\\\"true\\\",\\\"ContentBasedDeduplication\\\": \\\"true\\\"}\"\n```\n\n### Purge local queues\n\n```bash\naws --endpoint-url=http://localhost:4566 sqs purge-queue --region eu-west-2 --queue-url http://localhost:4566/000000000000/cdp-clamav-results\naws --endpoint-url=http://localhost:4566 sqs purge-queue --region eu-west-2 --queue-url http://localhost:4566/000000000000/cdp-uploader-scan-results-callback.fifo\n```\n\n### Setup local test harness\n\nWhen running locally there is a built-in test harness to simulate scan results. This requires an extra setup step\n\n```bash\naws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name mock-clamav\naws --endpoint-url=http://localhost:4566 s3api put-bucket-notification-configuration\\\n    --bucket $BUCKET_NAME\\\n    --notification-configuration '{\n                                      \"QueueConfigurations\": [\n                                         {\n                                           \"QueueArn\": \"arn:aws:sqs:eu-west-2:000000000000:mock-clamav\",\n                                           \"Events\": [\"s3:ObjectCreated:*\"]\n                                         }\n                                       ]\n\t                                }'\n```\n\nWhen running from the IDE the test harness is enabled by default. It can be enabled/disabled via\nthe `MOCK_VIRUS_SCAN_ENABLED` environment variable.\nThe test harness will listen for files being uploaded to the quarantine bucket.\nWhen a file arrives, it checks if the original filename matches the regex set in `MOCK_VIRUS_REGEX` (defaults to\nchecking if the file has 'virus' in the name).\nThere is a short delay (set via `MOCK_VIRUS_RESULT_DELAY`) to simulate scan time before the response is sent.\n\n## Local JSON API\n\nWhilst the APIs are being developed this app uses a local JSON mock API. To start this locally run:\n\n```bash\nnpm run mockApi\n```\n\n## Production\n\nTo mimic the application running in `production` mode locally run:\n\n```bash\nnpm start\n```\n\n## Npm scripts\n\nAll available Npm scripts can be seen in [package.json](./package.json)\nTo view them in your command line run:\n\n```bash\nnpm run\n```\n\n# Docker\n\n## Development image\n\nBuild:\n\n```bash\ndocker build --target development --no-cache --tag cdp-uploader:development .\n```\n\nRun:\n\n```bash\ndocker run -p 3000:3000 cdp-uploader:development\n```\n\n## Production image\n\nBuild:\n\n```bash\ndocker build --no-cache --tag cdp-uploader .\n```\n\nRun:\n\n```bash\ndocker run -p 3000:3000 cdp-uploader\n```\n\n# Licence\n\nTHIS INFORMATION IS LICENSED UNDER THE CONDITIONS OF THE OPEN GOVERNMENT LICENCE found at:\n\n\u003chttp://www.nationalarchives.gov.uk/doc/open-government-licence/version/3\u003e\n\nThe following attribution statement MUST be cited in your products and applications when using this information.\n\n\u003e Contains public sector information licensed under the Open Government license v3\n\n## About the licence\n\nThe Open Government Licence (OGL) was developed by the Controller of Her Majesty's Stationery Office (HMSO) to enable\ninformation providers in the public sector to license the use and re-use of their information under a common open\nlicence.\n\nIt is designed to encourage use and re-use of information freely and flexibly, with only a few conditions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdefra%2Fcdp-uploader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdefra%2Fcdp-uploader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdefra%2Fcdp-uploader/lists"}