https://github.com/onsdigital/sds
Supplementary Data Service
https://github.com/onsdigital/sds
Last synced: 2 months ago
JSON representation
Supplementary Data Service
- Host: GitHub
- URL: https://github.com/onsdigital/sds
- Owner: ONSdigital
- License: mit
- Created: 2022-12-21T11:46:09.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2026-04-08T11:17:58.000Z (2 months ago)
- Last Synced: 2026-04-08T11:22:36.675Z (2 months ago)
- Language: Python
- Size: 2.81 MB
- Stars: 0
- Watchers: 1
- Forks: 2
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: CODEOWNERS
Awesome Lists containing this project
README
# Supplementary Data Service (sds)
More information on this service can be found on Confluence:
- https://officefornationalstatistics.atlassian.net/wiki/spaces/SDC/pages/48739999/SDS+Supplementary+Data+Service
## Dockerized
The docker-compose will launch the SDS application, two storage emulators(firebase and bucket). The SDS application will also support hot reloading within the `/app` directory.
```
docker-compose up
```
Once, loaded you can do the following:
- View the API service docs [localhost:3033/docs](http://localhost:3033/docs).
- See files put into cloud storage within `devtools/gcp-storage-emulator/data/default-bucket`.
- Utilize the firestore emulator [localhost:4001/firestore](http://localhost:4001/firestore).
- Simulate the SDX publish process, invoked with a dataset as follows. (deprecated)
```
curl -X POST localhost:3006 \
-H "Content-Type: application/cloudevents+json" \
-d '{ "survey_id": "NRX",
"period_id": "ttt",
"title": "Which side was better?",
"schema_version": "v1.0.0",
"form_types": [
"klk",
"xyz",
"tzr"
],
"data": [
{
"identifier": "43532",
"unit_data":
{
"runame": "Pipes and Maps Ltd",
"local_unit": [
{
"identifier": "2012763A",
"luname": "Maps Factory"
},
{
"identifier": "20127364B",
"luname": "Pipes R Us Subsidiary"
}
]
}
}
]
}'
```
---
## Running locally
To run this service locally, you will need the following:
- Python 3.13
- Docker or credentials for GCloud
It is also strongly recommended you install the Google SDK (`brew install --cask google-cloud-sdk`)
You will need to make a choice to run with either GCP service emulators or the real thing.
Instructions for setting up both are included below.
### Setting up a virtual environment
Check that you have the correct version of Python installed and then run the following commands:
```bash
make setup
```
### Running SDS locally with services running in GCloud
In order to connect to real services in GCloud, you will need a GCP test project or make
use of the sandbox project. Instructions for setting this up are included in the IaC repo.
Once you have set up your project, you will need to add the role "Service Account Token Creator" to your user account for the project.
This is needed to allow you to impersonate the service account that the SDS API service uses to connect to Firestore and Cloud Storage when running integration tests on locally using cloud resources.
To do this:
- Go the IAM page and select Service accounts
- Go to your user account under your email
- Click "Edit principal" and add the role "Service Account Token Creator"
### Running the SDS API service locally with real GCloud services
```bash
gcloud auth application-default login
make start-cloud-dev
```
### Running the SDS with service emulators
The Firestore and Cloud Storage emulators run in Docker. To connect to the Firestore emulator running locally in Docker,
run the following commands:
```bash
docker-compose up --build
```
## Running linting and unit tests
As part of the CI pipeline we ensure the code is linted and tested. To run the linting run:
```bash
make lint
```
To automatically fix any linting issues run:
```bash
make lint-fix
```
To run the unit tests run:
```bash
make unit-tests
```
## Setting up GPG Key
- For signing commits to the git repository, create a new GPG key if you don't have an existing key. Follow the [link](https://docs.github.com/en/authentication/managing-commit-signature-verification/generating-a-new-gpg-key) for creating a new GPG Key
- For the adding the new key to the account, follow the [link](https://docs.github.com/en/authentication/managing-commit-signature-verification/adding-a-gpg-key-to-your-github-account)
- For telling Git about the Signing Key(Only needed once),follow the [link](https://docs.github.com/en/authentication/managing-commit-signature-verification/telling-git-about-your-signing-key)
## OpenAPI Specification
The openapi spec file in gateway/openapi.yaml should not be edited manually as it can be autogenerated using FastAPI utilities. This file should be regenerated every time the app changes. To autogenerate the file:
```bash
make generate-spec
```
## Running the integration tests
The integration tests will work in a number of different ways depending on how you want to test the SDS API service
and SDS cloud function. The following sections describe a number of combinations
### Everything running in the cloud
In this configuration, the integration test uses the SDS API service running in Cloud Run and the new-dataset-function
on Cloud Functions of your test/dev GCP project. Please note that the SDS and cloud function are not the updated version unless
run after creating a PR and gone through the pipeline. These services both talk to Firestore and Cloud Storage running on the same project.
This test configuration is also what is run at the end of the Cloud Build deployment.
```bash
PROJECT_NAME=
gcloud auth login
gcloud config set project $PROJECT_NAME
gcloud auth application-default login
make integration-tests-sandbox
```
### Running integration tests locally
This configuration makes use of the firestore and storage services running in Docker.
```bash
docker-compose up
make integration-tests-local
```
# Contact
- [sds.cir.team@ons.gov.uk](mailto:sds.cir.team@ons.gov.uk)
-