{"id":23421308,"url":"https://github.com/nhsdigital/nrlf-lambda-pipeline","last_synced_at":"2025-08-25T11:31:15.882Z","repository":{"id":141917129,"uuid":"542544519","full_name":"NHSDigital/nrlf-lambda-pipeline","owner":"NHSDigital","description":"Robust implementation of step chaining for AWS Lambda executions","archived":false,"fork":false,"pushed_at":"2024-02-29T17:28:56.000Z","size":61,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-04-13T07:14:21.333Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NHSDigital.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-09-28T10:59:30.000Z","updated_at":"2022-09-29T08:47:05.000Z","dependencies_parsed_at":"2024-02-29T18:49:10.871Z","dependency_job_id":null,"html_url":"https://github.com/NHSDigital/nrlf-lambda-pipeline","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NHSDigital%2Fnrlf-lambda-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NHSDigital%2Fnrlf-lambda-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NHSDigital%2Fnrlf-lambda-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NHSDigital%2Fnrlf-lambda-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NHSDigital","download_url":"https://codeload.github.com/NHSDigital/nrlf-lambda-pipeline/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230903875,"owners_count":18297817,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-23T02:14:45.508Z","updated_at":"2024-12-23T02:14:45.990Z","avatar_url":"https://github.com/NHSDigital.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# lambda-pipeline\n\nRobust implementation of step chaining for AWS Lambda executions.\n\n# For users\n\n## Installation\n\nBleeding edge:\n\n```\npip install git+https://github.com/NHSDigital/nrlf-lambda-pipeline.git\n```\n\nor a specific tag:\n\n```\npip install git+https://github.com/NHSDigital/nrlf-lambda-pipeline.git@v0.1.0\n```\n\n## Usage\n\n### 1. Define a list of steps\n\nThe list of steps indicates to `make_pipeline` the order in which to apply sequential steps on to the source event, e.g.\n\n```python\nsteps = [\n    authorise,\n    validate_x_request_url,\n    a_flaky_step,\n    intermediate_step,\n    read_document_from_db,\n]\n```\n\n### 2. Define your pipeline steps as functions with the required signature\n\nAll pipeline steps must be annotated with and adhere to the following signature:\n\n```python\ndef func(data: PipelineData, context: LambdaContext, event: EventModel, dependencies: FrozenDict[str, Any]) -\u003e PipelineData\n```\n\nNoting that:\n\n- `make_pipeline` will explicitly enforce this signature internally.\n- You provide the `EventModel` class. It is recommended to use one of the predefined models from [aws-lambda-powertools](https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/parser/#built-in-models).\n- `PipelineData` is used to pass data between sequential steps\n- `PipelineData` objects are `FrozenDict` objects internally, and are therefore immutable and so you must create a new `PipelineData` in the response of each step,\n- `make_pipeline` will force both `event` and `dependencies` to be immutable, so that they can be shared deterministically between steps (and in the case of `dependencies` between lambda invocations).\n- While `context` is technically mutable within a step, changes to `context` are not persisted between steps.\n\n### 3. Wrap up any external functions to match the function signature\n\nFor example:\n\n```python\ndef _validate_x_request_url(x_request_url: str):\n    \"\"\"Doesn't match the required step signature!\"\"\"\n    if x_request_url == \"something\":\n        raise ValueError(\"Invalid value for 'x_request_url'\")\n\n\ndef validate_x_request_url(\n    data: PipelineData,\n    event: EventModel,\n    context: LambdaContext,\n    dependencies: FrozenDict[str, Any],\n) -\u003e PipelineData:\n\n    \"\"\"An example of standardising an unstandardised third party tool by wrapping\"\"\"\n    try:\n        _validate_x_request_url(x_request_url=event.headers.get(\"x-request-url\"))\n    except ValueError as exc:\n        raise PipelineError(str(exc))\n    return data\n```\n\n### 4. Import your steps from your handler module to build your pipeline\n\n```python\nfrom example.api.handler import EventModel, build_shared_dependencies, steps\nfrom lambda_pipeline.pipeline import make_pipeline\nfrom lambda_pipeline.types import PipelineData, LambdaContext\n\nshared_dependencies = build_shared_dependencies()\n\ndef handler(event: dict, context: LambdaContext = None) -\u003e dict[str, str]:\n    if context is None:\n        context = LambdaContext()\n\n    pipeline = make_pipeline(\n        steps=steps,\n        event=EventModel(**event),\n        context=context,\n        dependencies=shared_dependencies,\n    )\n\n    return pipeline(data=PipelineData()).to_dict()\n```\n\n## Examples from this repo\n\nSet yourself up with (for example with `ipython`):\n\n```python\nfrom example.api.index import handler\nfrom example.api.tests import example_event\n```\n\n### 1. Happy path\n\n```python\nevent = example_event(headers={\"auth_level\": 10, \"x-request-url\": \"example.com\"})\nhandler(event=event)\n\n\u003e\u003e\u003e [... some logging ...]\n{\n    'status_code': '200',\n    'body': '{\"id\": 123, \"content-type\": \"application/json\", \"message\": \"hello, world\"}'\n}\n```\n\n### 2. Authorisation fails\n\n```python\nevent = example_event(headers={\"auth_level\": 1, \"x-request-url\": \"example.com\"})\nhandler(event=event)\n\n\u003e\u003e\u003e [... some logging ...]\n{\n    'status_code': '400',\n    'body': '{\"message\": \"Minimum authorisation not satisfied\"}'\n}\n```\n\n### 3. Simulate a transient error\n\n```python\nimport os\nos.environ[\"FLAKE_OUT\"] = \"True\"\nevent = example_event(headers={\"auth_level\": 10, \"x-request-url\": \"example.com\"})\nhandler(event=event)\n\n\u003e\u003e\u003e [... some logging ...]\n{\n    'status_code': '500',\n    'body': '{\"message\": \"Internal Server Error\"}'\n}\n```\n\n# For Developers\n\n## Setup\n\nInstall dependencies with `poetry`:\n\n```\npoetry config virtualenvs.in-project true\npoetry install\nsource .venv/bin/activate\n```\n\nHook-up pre-commit hooks:\n\n```\npre-commit install\n```\n\n## Tests\n\n### Unit\n\n```\npython -m pytest -m 'not integration'\n```\n\n### Integration\n\nThis will run tests against the lambda(s) in `example` by deploying to localstack. There is an assumed dependency on docker client, which you should\ninstall against the instructions for your operating system. [Docker Desktop](https://www.docker.com/products/docker-desktop/)\nis a good place to start if you don't have opinions on the matter.\n\n```\nlocalstack start -d\n```\n\n```\npython -m pytest -m 'integration'\n```\n\n### Build\n\nCreate a build of this package\n\n```\npoetry build\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnhsdigital%2Fnrlf-lambda-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnhsdigital%2Fnrlf-lambda-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnhsdigital%2Fnrlf-lambda-pipeline/lists"}