https://github.com/moj-analytical-services/data-engineering-exports
Infrastructure to allow data from the Analytical Platform to be accessed by other services
https://github.com/moj-analytical-services/data-engineering-exports
analytical-platform aws data-engineering pulumi python
Last synced: about 1 year ago
JSON representation
Infrastructure to allow data from the Analytical Platform to be accessed by other services
- Host: GitHub
- URL: https://github.com/moj-analytical-services/data-engineering-exports
- Owner: moj-analytical-services
- License: mit
- Created: 2021-02-24T15:58:03.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2025-03-12T15:51:08.000Z (over 1 year ago)
- Last Synced: 2025-03-12T16:37:58.612Z (over 1 year ago)
- Topics: analytical-platform, aws, data-engineering, pulumi, python
- Language: Python
- Homepage:
- Size: 217 KB
- Stars: 3
- Watchers: 19
- Forks: 2
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
# Data engineering exports
This repository lets you export data from the Analytical Platform to other platforms. For now, this means the HMPPS Performance Hub or the Ministry of Justice Cloud Platform. Message us on the #ask-data-engineering Slack channel if you want to connect to somewhere else.
You must have a data protection impact assessment (DPIA), data sharing agreement or similar documentation confirming you can transfer the data to the destination.
## Ways to export
There are 2 ways to export your data:
- a 'push bucket', which will send any data you put in it to a bucket in another account
- a 'pull bucket', which is on the Analytical Platform but whose contents are visible to a specific external user
Both are fine in principle - your team may prefer one or the other. Contact us on the #ask-data-engineering Slack channel to discuss it.
At the moment pull buckets only work if the other platform is also on Amazon Web Services. It isn't currently an option for Microsoft Azure or Google Cloud Platform. But contact us if you need one of these - we can add it, it just hasn't been needed yet.
## Setting up exports
You can request either bucket type through this repository. The Analytical Platform has advice on [working with GitHub branches](https://user-guidance.services.alpha.mojanalytics.xyz/github.html#working-on-a-branch).
This guide uses an example dataset called <>. Wherever this appears, insert the name of your own dataset. Make sure you remove the <<>> - this is just to show you what to change.
Only use lower case and underscores in your dataset name.
1. Create a new branch in this repository called either `push_dataset/<>` or `pull_dataset/<>`
2. Create a new file in either `pull_datasets` or `push_datasets` called `new_project.yaml`
3. Add your project name, target bucket (the bucket you want the files to go to **without** the `aws:arn:s3:::` prefix), list of Analytical Platform usernames, and link to your DPIA, like this:
``` yaml
name: new_project
bucket: target-bucket-name
users:
- alpha_user_one
- alpha_user_two
paperwork: link to your DPIA
```
4. For a pull dataset, you must also add the Amazon Web Services 'ARNs' of the **roles** (not the buckets) that should have access to the bucket. Talk to your Cloud Platform team to get these - or contact us to discuss it. Your config should end up looking like this:
``` yaml
name: new_project
pull_arns:
- arn:aws:iam::1234567890:role/role-name-1
- arn:aws:iam::1234567890:role/role-name-2
users:
- alpha_user_one
- alpha_user_two
paperwork: link to your DPIA
```
5. Commit the file and push it to GitHub
6. Create a new pull request and request a review from the data engineering team. Once this is approved, you can merge your PR: this doesn't happen automatically, so don't forget.
7. Once your changes are in the `main` branch, request a data engineer to `pulumi up` which deploys your changes to the infrastructure. They will tell you when it's ready. If you have access to the data engineering SSO role, then you can do this yourself [following the instructions](./CONTRIBUTING.md).
8. If you can't see your new role in IAM (in our example it's `export_new_project-move`) then your changes haven't been deployed. You may need to wait 24 hours.
## Exporting from your bucket
How often you can export is limited by AWS Lambda, which has a complex rate limiting and burst quota system. When you upload a file to the export bucket, this triggers a Lambda function to send your file to the recipient and delete it from the export bucket. If you are likely to export more than ~1,000 files per second please contact data engineering.
How large a file you can export is limited by how the files are sent. The maximum file size in S3 is 5,000 GB (5 TB): this is the largest file you can store in the bucket. However, it may not export: the file size limit for a single `PUT` operation is 5 GB. If you need to export files larger than 5 GB please contact data engineering,
If your project causes `500` or `503` status errors see [here](https://repost.aws/knowledge-center/http-5xx-errors-s3): you may be close to the [limits](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html) of 3,500 `COPY` or `PUT` operations per second.
### Exporting data from a push bucket
The users in your dataset file will now have permission to add files to any path beginning with: `s3://mojap-hub-exports/new_project/`.
You **must** include the project name in the path. For example, you could write a file to `s3://mojap-hub-exports/new_project/my_data.csv`, but not to `s3://mojap-hub-exports/my_data.csv`. In the management console, you will not see the folder `s3://mojap-hub-exports/new_project/` until there is at least one file in it.
For how to move files, see the Analytical Platform guide to [writing to an S3 bucket](https://user-guidance.services.alpha.mojanalytics.xyz/data/data-faqs/#how-do-i-read-write-data-from-an-s3-bucket).
The owner of the receiving bucket must add permission to their bucket policy for the exporter to write (`PutObject` in AWS language) to that bucket. Ther service role for the project `new_project` is `arn:aws:iam::593291632749:role/service-role/export_new_project-move`.
After you send files to this location they will be copied to your target bucket, then deleted from `mojap-hub-exports`.
### Exporting data from a pull bucket
You will be given a bucket called `mojap-new-project` - the name of your project, prefixed with `mojap` (this means it's managed by data engineering rather than the Analytical Platform team).
The users in your dataset file will have read and write access to this bucket.
The 'pull ARNs' in your dataset file have read-only access to the bucket.
### Use with Cloud Platform
This tool can be used to allow data from the Analytical Platform buckets to be read by the Cloud Platform. In order to do this, you need to setup a cross IAM role using terraform in the [cloud-platform-environments](https://github.com/ministryofjustice/cloud-platform-environments) repository (an example is [here](https://github.com/ministryofjustice/cloud-platform-environments/blob/main/namespaces/live.cloud-platform.service.justice.gov.uk/ops-pilot-test/resources/cross-iam-role-sa.tf)). The key part is:
```
resource "kubernetes_secret" "irsa" {
metadata {
name = "irsa-output"
namespace = var.namespace
}
data = {
role = module.irsa.aws_iam_role_name
serviceaccount = module.irsa.service_account_name.name
}
```
Once deployed, you can then get the IAM role using the following:
`cloud-platform decode-secret -n -s irsa-name`
Once you have the `data` from the response. You can use this as your ARN in your `.yaml` file (note, the CP Account is `754256621582`)
## Reporting bugs and asking for features
Open an issue in this repository or contact the #ask-data-engineering Slack channel.