{"id":20038656,"url":"https://github.com/defra/forms-submission-api","last_synced_at":"2026-05-13T20:33:53.137Z","repository":{"id":250170151,"uuid":"833684333","full_name":"DEFRA/forms-submission-api","owner":"DEFRA","description":"Git repository for service forms-submission-api","archived":false,"fork":false,"pushed_at":"2025-02-27T18:09:51.000Z","size":1518,"stargazers_count":0,"open_issues_count":11,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-28T01:57:39.895Z","etag":null,"topics":["backend","cdp","node","service"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DEFRA.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-25T14:34:57.000Z","updated_at":"2024-12-31T11:22:09.000Z","dependencies_parsed_at":"2024-11-13T19:23:16.421Z","dependency_job_id":"1354fd1f-7d30-4f21-82f2-978f230adddf","html_url":"https://github.com/DEFRA/forms-submission-api","commit_stats":null,"previous_names":["defra/forms-submission-api"],"tags_count":41,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Fforms-submission-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Fforms-submission-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Fforms-submission-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Fforms-submission-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DEFRA","download_url":"https://codeload.github.com/DEFRA/forms-submission-api/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241469095,"owners_count":19967988,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backend","cdp","node","service"],"created_at":"2024-11-13T10:31:34.201Z","updated_at":"2026-05-13T20:33:53.132Z","avatar_url":"https://github.com/DEFRA.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# forms-submission-api\n\nAPI to track form submissions. Currently tracks file submissions only.\n\nSee [docs/](docs/) for documentation.\n\n- [Requirements](#requirements)\n  - [Node.js](#nodejs)\n- [Local development](#local-development)\n  - [Setup](#setup)\n  - [Development](#development)\n  - [Production](#production)\n  - [Npm scripts](#npm-scripts)\n  - [Database Migrations](#database-migrations)\n- [Integration Tests](#integration-tests)\n  - [Prerequisites](#prerequisites)\n  - [Running Integration Tests](#running-integration-tests)\n    - [Quick Start](#quick-start)\n    - [Manual Step-by-Step Execution](#manual-step-by-step-execution)\n  - [What the Integration Tests Cover](#what-the-integration-tests-cover)\n  - [Test Environment Details](#test-environment-details)\n  - [Test Reports](#test-reports)\n  - [Troubleshooting](#troubleshooting)\n    - [Common Issues:](#common-issues)\n    - [Viewing Logs:](#viewing-logs)\n- [API endpoints](#api-endpoints)\n- [Docker](#docker)\n  - [Development Image](#development-image)\n  - [Production Image](#production-image)\n- [Swagger](#swagger)\n- [Licence](#licence)\n  - [About the licence](#about-the-licence)\n\n## Requirements\n\n### Node.js\n\nPlease install the Node.js version in [.nvmrc](.nvmrc) using [Node Version Manager `nvm`](https://github.com/creationix/nvm) via:\n\n```bash\ncd forms-submission-api\nnvm use\n```\n\n## Local development\n\n### Setup\n\n1. Install Docker\n\n2. Bring up runtime dependencies\n\n```bash\ndocker compose up\n```\n\n3. Create a `.env` file with the following mandatory environment variables populated at root level:\n\n```text\nMONGO_URI=\"\"\nMONGO_DATABASE=\"\"\nOIDC_JWKS_URI=\"\"\nOIDC_VERIFY_AUD=\"\"\nOIDC_VERIFY_ISS=\"\"\nCOGNITO_JWKS_URI=\"\"\nCOGNITO_CLIENT_ID=\"\"\nCOGNITO_VERIFY_ISS=\"\"\nHTTP_PROXY=\nHTTPS_PROXY=\nNO_PROXY=\n```\n\nFor proxy options, see https://www.npmjs.com/package/proxy-from-env which is used by https://github.com/TooTallNate/proxy-agents/tree/main/packages/proxy-agent. It's currently supports Hapi Wreck only, e.g. in the JWKS lookup.\n\n4. **Database setup**: See [Database Migrations](#database-migrations) for information on how database migrations work in this project.\n\n### Development\n\nTo run the application in `development` mode run:\n\n```bash\nnpm run dev\n```\n\n### Production\n\nTo mimic the application running in `production` mode locally run:\n\n```bash\nnpm start\n```\n\n### Npm scripts\n\nAll available Npm scripts can be seen in [package.json](./package.json)\nTo view them in your command line run:\n\n```bash\nnpm run\n```\n\n## Integration Tests\n\nThe integration tests use Docker Compose to create a complete testing environment with MongoDB, LocalStack (for S3 emulation), OIDC mock server, and Newman (Postman CLI) for API testing.\n\n### Prerequisites\n\n- Docker and Docker Compose installed\n- Node.js (version as specified in `.nvmrc`)\n\n### Running Integration Tests\n\n#### Quick Start\n\nTo run the full integration test suite:\n\n```bash\nnpm run test:integration\n```\n\nThis command will:\n\n1. Start all required services (MongoDB, OIDC mock, app)\n2. Wait for services to be ready\n3. Run the Postman collection tests\n4. Clean up all services\n\n#### Manual Step-by-Step Execution\n\nFor debugging or development purposes, you can run the integration tests step by step:\n\n1. **Start infrastructure services:**\n\n   ```bash\n   npm run test:integration:setup\n   ```\n\n2. **Start the application:**\n\n   ```bash\n   npm run test:integration:start\n   ```\n\n3. **Wait for services to be ready:**\n\n   ```bash\n   npm run test:integration:wait\n   ```\n\n4. **Run the tests:**\n\n   ```bash\n   npm run test:integration:run\n   ```\n\n5. **Clean up:**\n   ```bash\n   npm run test:integration:stop\n   ```\n\n### What the Integration Tests Cover\n\nThe integration tests verify all the key functionality of the forms-submission-api:\n\n1. **Health Check** - Verifies the API is running\n2. **File Ingestion** - Tests the CDP callback endpoint (`POST /file`)\n3. **File Status Check** - Tests file existence verification (`GET /file/{fileId}`)\n4. **Presigned Link Generation** - Tests secure file access (`POST /file/link`)\n5. **File Persistence** - Tests moving files from 7-day to 30-day storage (`POST /files/persist`)\n6. **Form Submission** - Tests complete form submission flow (`POST /submit`)\n7. **Error Handling** - Tests validation failures and invalid requests\n8. **Batch Operations** - Tests multiple file handling\n\n### Test Environment Details\n\nThe integration test environment includes:\n\n- **MongoDB**: Replica set configuration for transaction support\n- **OIDC Mock Server**: Authentication provider for secure endpoints\n- **Newman**: Postman CLI for running API test collections\n\n### Test Reports\n\nWhen running on the main branch, HTML reports are generated:\n\n- Location: `./newman-reports/newman-report.html`\n- Includes detailed request/response logs and test results\n- Available as CI artifacts in GitHub Actions\n\n### Troubleshooting\n\n#### Common Issues:\n\n1. **Port conflicts**: The tests use ports 3002 (API), 5556, and 27018. Make sure these are available.\n\n2. **Docker resources**: The integration tests require sufficient Docker resources. Increase Docker memory if needed.\n\n3. **Services not ready**: If tests fail due to services not being ready, increase the wait time in the `test:integration:wait` script.\n\n4. **Clean up**: If services are left running, use:\n   ```bash\n   npm run test:integration:stop\n   ```\n\n#### Viewing Logs:\n\nTo view service logs:\n\n```bash\ndocker compose -f docker-compose.integration-test.yml logs [service_name]\n```\n\nAvailable services: `mongo_test`, `oidc`, `app_test`, `newman`\n\n### Database Migrations\n\nThis project uses [migrate-mongo](https://www.npmjs.com/package/migrate-mongo) to manage database migrations.\n\n#### Production\n\nIn production, migrations run automatically when the Docker container starts via the `scripts/run-migrations-and-start.sh` shell script. This script:\n\n1. Runs all pending migrations (`migrate-mongo up`)\n2. Starts the application server\n3. Logs migration progress to the container output\n\n**No manual intervention is required** - migrations execute automatically on container startup.\n\n#### Local Development\n\nFor local development, you have two options:\n\n##### Option 1: Using Docker (Recommended)\n\nMigrations run automatically when using Docker:\n\n```bash\ndocker compose up --build forms-submission-api\n```\n\nThis mimics the production environment and runs migrations via the same shell script.\n\n##### Option 2: Manual Migration Commands\n\nTo work with migrations manually, you can install migrate-mongo globally:\n\n```bash\nnpm install -g migrate-mongo\n```\n\nAvailable migration commands:\n\n```bash\n# Check migration status\nnpm run migrate:status\n\n# Run all pending migrations\nnpm run migrate:up\n\n# Rollback the last migration\nnpm run migrate:down\n\n# Create a new migration\nnpx migrate-mongo create \u003cmigration-name\u003e -f migrate-mongo-config.js\n```\n\n**Important**: When running migrations manually, ensure your `.env` file contains the correct `MONGO_URI` and `MONGO_DATABASE` values that match your local MongoDB instance.\n\n## API endpoints\n\n| Endpoint               | Description                                                                                     |\n| :--------------------- | :---------------------------------------------------------------------------------------------- |\n| `GET: /health`         | Health                                                                                          |\n| `POST: /file`          | Ingests a file with a 7 day expiry. Called by the CDP uploader as a callback (upon file upload) |\n| `GET: /file/{fileId}`  | Checks that a file has been ingested.                                                           |\n| `POST: /file/link`     | Creates a link to a file which can be accessed by a user. Valid for 60 minutes.                 |\n| `POST: /files/persist` | Extends the expiry to 30 days. Called upon form submission.                                     |\n\n## Docker\n\n### Development image\n\nBuild:\n\n```bash\ndocker build --target development --no-cache --tag forms-submission-api:development .\n```\n\nRun:\n\n```bash\ndocker run -e GITHUB_API_TOKEN -p 3008:3008 forms-submission-api:development\n```\n\n### Production image\n\nBuild:\n\n```bash\ndocker build --no-cache --tag forms-submission-api .\n```\n\nRun:\n\n```bash\ndocker run -e GITHUB_API_TOKEN -p 3002:3002 forms-submission-api\n```\n\n## Swagger\n\nWhen running locally, you can view swagger documentation on http://localhost:3002/documentation this is built automatically from the joi schema and Hapi endpoints using Swagger Hapi.\n\n## Licence\n\nTHIS INFORMATION IS LICENSED UNDER THE CONDITIONS OF THE OPEN GOVERNMENT LICENCE found at:\n\n\u003chttp://www.nationalarchives.gov.uk/doc/open-government-licence/version/3\u003e\n\nThe following attribution statement MUST be cited in your products and applications when using this information.\n\n\u003e Contains public sector information licensed under the Open Government license v3\n\n### About the licence\n\nThe Open Government Licence (OGL) was developed by the Controller of Her Majesty's Stationery Office (HMSO) to enable\ninformation providers in the public sector to license the use and re-use of their information under a common open\nlicence.\n\nIt is designed to encourage use and re-use of information freely and flexibly, with only a few conditions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdefra%2Fforms-submission-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdefra%2Fforms-submission-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdefra%2Fforms-submission-api/lists"}