{"id":27201097,"url":"https://github.com/gregl83/pac-man","last_synced_at":"2025-04-09T21:55:08.942Z","repository":{"id":57651820,"uuid":"332008718","full_name":"gregl83/pac-man","owner":"gregl83","description":"AWS Lambda URI consumer  ᗧ • • •","archived":false,"fork":false,"pushed_at":"2021-03-03T07:49:36.000Z","size":169,"stargazers_count":7,"open_issues_count":5,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-09T21:55:05.097Z","etag":null,"topics":["api-client","aws","aws-lambda","consumer","rust","secrets","serverless"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gregl83.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-01-22T16:57:28.000Z","updated_at":"2022-07-17T00:12:09.000Z","dependencies_parsed_at":"2022-08-28T02:22:24.649Z","dependency_job_id":null,"html_url":"https://github.com/gregl83/pac-man","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gregl83%2Fpac-man","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gregl83%2Fpac-man/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gregl83%2Fpac-man/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gregl83%2Fpac-man/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gregl83","download_url":"https://codeload.github.com/gregl83/pac-man/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248119394,"owners_count":21050754,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api-client","aws","aws-lambda","consumer","rust","secrets","serverless"],"created_at":"2025-04-09T21:55:08.233Z","updated_at":"2025-04-09T21:55:08.936Z","avatar_url":"https://github.com/gregl83.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://github.com/gregl83/pac-man/workflows/CI/badge.svg?branch=main)](https://github.com/gregl83/pac-man/actions?query=workflow%3ACI+branch%3Amain)\n[![Crates.io](https://img.shields.io/crates/v/pac-man.svg)](https://crates.io/crates/pac-man)\n[![MIT licensed](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/gregl83/pac-man/blob/master/LICENSE)\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"/assets/pac-man.png\" /\u003e\u003c/p\u003e\n\n# pac-man\n\nAWS Lambda URI consumer.\n\nUse this generic Lambda service to consume APIs and persist response data in S3. Once the data is in S3, it can be consumed by any number of AWS cloud services.\n\n- [Architecture](#architecture)\n- [Service Dependencies](#service-dependencies)\n- [Usage](#usage)\n  - [Lambda Event](#lambda-event)\n  - [Modifiers](#modifiers)\n    - [Chunks](#chunks)\n    - [Secrets](#secrets)\n    - [Uuid](#uuid)\n- [Testing](#testing)\n- [Deployment](#deployment)\n- [References](#references)\n\n## Stability\n\nExperimental\n\n## Architecture\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"/assets/pac-man-components.png\" /\u003e\u003c/p\u003e\n\n## Service Dependencies\n\n**Required**\n\n- [AWS Lambda](https://aws.amazon.com/lambda/) as executor of pac-man tasks.\n- [AWS S3](https://aws.amazon.com/s3/) as Destination for data.\n\n**Optional**\n\n- [GitHub Actions CI/CD](https://github.com/features/actions) to deploy pac-man.\n- [AWS Secrets Manager](https://aws.amazon.com/secrets-managser/) for sensitive request or persistence parameters.\n- [AWS EventBridge](https://aws.amazon.com/eventbridge/) to trigger pac-man runs.\n\n## Usage\n\nAWS Lambda deployments can be triggered in a variety of ways. Configuring events to trigger your Lambda functions involves parameters specific to each use-case.\n\nUncomfortable with sensitive values in plaintext?\n\n*Good, you should be!*\n\nAWS Secrets Manager is supported, using a modifier, with the following value expression: `{:secrets:\u003cname\u003e:\u003ckey\u003e}` (see Modifiers).\n\n**Caution**\n\nLambda functions are naturally `ephemeral` or `stateless`. By design, pac-man expects to be executed as a single-use `serverless` function. Some modifiers maintain a cache in the event they are utilized more than once for a given request. Using pac-man in `stateful` applications can have unexpected consequences.\n\n### Lambda Event\n\n**Minima Lambda Event**\n\n```json\n{\n  \"source\": {\n    \"scheme\": \"https\",\n    \"hostname\": \"example.com\"\n  },\n  \"destination\": {\n    \"region\": \"us-east-1\",\n    \"collection\": \"bucket-name\",\n    \"name\": \"key\"\n  }\n}\n```\n\n**Maxima Lambda Event**\n\n```json\n{\n  \"mods\": [\n    {\n      \"name\": \"chunks\",\n      \"start\": 0,\n      \"end\": 100,\n      \"chunk\": {\n        \"length\": 10      \n      },\n      \"bytes\": 100\n    },\n    {\n      \"name\": \"secrets\",\n      \"region\": \"us-east-1\"\n    },\n    {\n      \"name\": \"uuid\"    \n    }\n  ],\n  \"source\": {\n    \"headers\": {\n      \"Accepts\": [\n        \"text/json\"\n      ]\n    },\n    \"scheme\": \"https\",\n    \"username\": \"pseudo\",\n    \"password\": \"{:secrets:pac-man:pw}\",\n    \"hostname\": \"example.com\",\n    \"port\": 8080,\n    \"path\": \"/follow/the\",\n    \"params\": {\n      \"from\": \"{:chunks:chunk:start}\",\n      \"to\": \"{:chunks:chunk:end}\"    \n    },\n    \"fragment\": \"/yellow/brick/road\"\n  },\n  \"destination\": {\n    \"region\": \"us-east-1\",\n    \"collection\": \"bucket-name\",\n    \"name\": \"key-{:uuid}\"\n  }\n}\n```\n\n**Optional Event Fields**\n\n`delta( minima.fields, maxima.fields )`\n\n### Modifiers\n\nModifiers or `mods` implement functionality that modifies placeholders in pac-man event fields.\n\nBy default, `mods` aren't active and must be toggled respectively within the `mods` configuration body of the Lambda Event.\n\n#### Chunks\n\n`{:chunks:chunk:start}`\n`{:chunks:chunk:end}`\n`{:chunks:chunk:index}`\n`{:chunks:chunk:page}`\n```json\n{\n  \"name\": \"chunks\",\n  \"start\": 0,\n  \"end\": 100,\n  \"chunk\": {\n    \"length\": 10      \n  },\n  \"bytes\": 100\n}\n```\n\n`bytes` is a special configuration that denotes the minimum Content-Length header required to continue. It can be used to decide when a last page has been reached.\n\n#### Secrets\n\n`{:secrets:\u003cname\u003e:\u003ckey\u003e}`\n```json\n{\n  \"name\": \"secrets\",\n  \"region\": \"us-east-1\"\n}\n```\n\n#### Uuid\n\n`{:uuid}`\n`{:uuid:\u003ckey\u003e}`\n```json\n{\n  \"name\": \"uuid\"\n}\n```\n\nUsing a key with the `uuid` modifier allows reference to the same `uuid` using a HashMap cache for a given run.\n\n## Testing\n\nLambda functions can be executed with the help of [Docker](https://github.com/awslabs/aws-lambda-rust-runtime#docker).\n\n### Docker Containers\n\n[docker-test.sh](/docker-test.sh) launches a Lambda build using Docker.\n\n#### Environment Variables\n\n- AWS_SECRET_ACCESS_KEY\n- AWS_ACCESS_KEY_ID\n\n#### Build and Run\n\n1. Build Package\n\n```bash\n../pac-man$ docker run --rm \\\n      -v ${PWD}:/code \\\n      -v ${HOME}/.cargo/registry:/root/.cargo/registry \\\n      -v ${HOME}/.cargo/git:/root/.cargo/git \\\n      softprops/lambda-rust\n```\n\n2. Unzip Package\n\n```bash\n../pac-man$ unzip -o \\\n      target/lambda/release/pac-man.zip \\\n      -d /tmp/lambda\n```\n\n3. Run Unzipped Package\n```bash\n../pac-man$ docker run \\\n      -i -e DOCKER_LAMBDA_USE_STDIN=1 \\\n      -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \\\n      -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \\\n      --rm \\\n      -v /tmp/lambda:/var/task \\\n      lambci/lambda:provided\n```\n\n## Deployment\n\n1. Build Lambda Package Using Docker\n```bash\n../pac-man$ docker run --rm -ti -v ${PWD}:/code softprops/lambda-rust\n```\n\n2. Deploy Lambda Using AWS CLI\n```bash\n../pac-man$ aws lambda create-function --function-name pac-man \\\n  --handler doesnt.matter \\\n  --zip-file fileb://target/lambda/release/pac-man.zip \\\n  --runtime provided \\\n  --region \u003clambda-region\u003e \\\n  --role arn:aws:iam::\u003clambda-create-role\u003e \\\n  --timeout \u003clambda-timeout-seconds\u003e \\\n  --environment Variables={RUST_BACKTRACE=1} \\\n  --tracing-config Mode=Active\n```\n\n## References\n\n- [W3 URI Specification](https://www.w3.org/Addressing/URL/uri-spec.html)\n- [AWS Lambda Runtime](https://github.com/awslabs/aws-lambda-rust-runtime)\n- [AWS Glue](https://aws.amazon.com/glue/)\n\n## License\n\n[MIT](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgregl83%2Fpac-man","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgregl83%2Fpac-man","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgregl83%2Fpac-man/lists"}