{"id":22428497,"url":"https://github.com/kwame-mintah/aws-lambda-model-evaluation","last_synced_at":"2026-05-07T18:32:07.341Z","repository":{"id":229716085,"uuid":"777442096","full_name":"kwame-mintah/aws-lambda-model-evaluation","owner":"kwame-mintah","description":"A lambda to invoke inference requests to AWS SageMaker endpoints, using test data previous created. Helping with monitoring model performance.","archived":false,"fork":false,"pushed_at":"2024-08-30T13:17:04.000Z","size":73,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-04T12:43:52.704Z","etag":null,"topics":["aws","aws-lambda","model-evaluation","python","python312"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kwame-mintah.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-25T21:24:23.000Z","updated_at":"2024-08-30T13:17:08.000Z","dependencies_parsed_at":"2025-03-27T06:44:12.655Z","dependency_job_id":"e177b730-4829-4a32-a9ca-8e7874c859fe","html_url":"https://github.com/kwame-mintah/aws-lambda-model-evaluation","commit_stats":null,"previous_names":["kwame-mintah/aws-lambda-model-evaluation"],"tags_count":9,"template":false,"template_full_name":"kwame-mintah/aws-lambda-function-template","purl":"pkg:github/kwame-mintah/aws-lambda-model-evaluation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwame-mintah%2Faws-lambda-model-evaluation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwame-mintah%2Faws-lambda-model-evaluation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwame-mintah%2Faws-lambda-model-evaluation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwame-mintah%2Faws-lambda-model-evaluation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kwame-mintah","download_url":"https://codeload.github.com/kwame-mintah/aws-lambda-model-evaluation/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwame-mintah%2Faws-lambda-model-evaluation/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32750490,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-07T02:14:30.463Z","status":"ssl_error","status_checked_at":"2026-05-07T02:14:29.405Z","response_time":62,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-lambda","model-evaluation","python","python312"],"created_at":"2024-12-05T20:15:02.246Z","updated_at":"2026-05-07T18:32:07.326Z","avatar_url":"https://github.com/kwame-mintah.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS Lambda Model Evaluation\n\n[![Python 3.12](https://img.shields.io/badge/python-3.12-blue.svg)](https://www.python.org/downloads/release/python-3121/)\n[![🚧 Bump version](https://github.com/kwame-mintah/aws-lambda-model-evaluation/actions/workflows/bump-repository-version.yml/badge.svg)](https://github.com/kwame-mintah/aws-lambda-model-evaluation/actions/workflows/bump-repository-version.yml)\n[![🚀 Push Docker image to AWS ECR](https://github.com/kwame-mintah/aws-lambda-model-evaluation/actions/workflows/push-docker-image-to-aws-ecr.yml/badge.svg)](https://github.com/kwame-mintah/aws-lambda-model-evaluation/actions/workflows/push-docker-image-to-aws-ecr.yml)\n[![🧹 Run linter](https://github.com/kwame-mintah/aws-lambda-model-evaluation/actions/workflows/run-python-linter.yml/badge.svg)](https://github.com/kwame-mintah/aws-lambda-model-evaluation/actions/workflows/run-python-linter.yml)\n\u003ca href=\"https://github.com/psf/black\"\u003e\u003cimg alt=\"Code style: black\" src=\"https://img.shields.io/badge/code%20style-black-000000.svg\"\u003e\u003c/a\u003e\n\nA lambda to invoke an AWS SageMaker model endpoint with test data stored within an S3 Bucket. For analyzing a dataset to\n[generate baseline constraints](https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_model_monitoring.html).\nAllowing for monitoring machine learning (ML) model quality to determine if new model predictions are accurate or not,\nadditionally detect changes in properties.\n\nThis repository does not create the AWS Simple Queue Service, this is created via Terraform found here [terraform-aws-machine-learning-pipeline](https://github.com/kwame-mintah/terraform-aws-machine-learning-pipeline).\nAdditionally, data uploaded for ML purposes can be found here [ml-data-copy-to-aws-s3](https://github.com/kwame-mintah/ml-data-copy-to-aws-s3).\nFor more details on the entire flow and how this lambda is deployed, see [aws-automlops-serverless-deployment](https://github.com/kwame-mintah/aws-automlops-serverless-deployment).\n\n# Flowchart\n\nThe [diagram below](https://mermaid.js.org/syntax/flowchart.html#flowcharts-basic-syntax) demonstrates what happens when\nthe lambda is trigger, when a new message is placed on the SQS.\n\n```mermaid\ngraph LR\n  S0(Start)\n  T1(Check endpoint is available)\n  T2(Configure Predictor for endpoint)\n  T3(Pull test dataset from S3 Bucket)\n  T4(Use test data as payload to invoke endpoint)\n  T5(Save confusion matrix to S3 Bucket)\n  E0(End)\n\n  S0--\u003eT1\n  T1--\u003eT2\n  T2--\u003eT3\n  T3--\u003eT4\n  T4--\u003eT5\n  T5--\u003eE0\n```\n\n# Notice\n\nAn assumption has been made that a serverless endpoint has been created for inference. And not able to make of\n[`DataCaptureConfig`](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-data-capture-endpoint.html), this is\nbecause the configuration is not supported for serverless endpoint(s) and will have to rely on CloudWatch logs as per\nofficial [documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints-monitoring.html).\n\nA workaround is to invoke the endpoint and create a confusion matrix with the predicated vs actuals, this is then uploaded\nto another bucket as in Markdown format.\n\n## Development\n\n### Dependencies\n\n- [Python](https://www.python.org/downloads/release/python-3121/)\n- [Docker for Desktop](https://www.docker.com/products/docker-desktop/)\n- [Amazon Web Services](https://aws.amazon.com/?nc2=h_lg)\n\n## Usage\n\n1. Build the docker image locally:\n\n   ```shell\n   docker build --no-cache -t aws_lambda:local .\n   ```\n\n2. Run the docker image built:\n\n   ```shell\n   docker run --platform linux/amd64 -p 9000:8080 aws_lambda:local\n   ```\n\n3. Send an event to the lambda via curl:\n   ```shell\n   curl \"http://localhost:9000/2015-03-31/functions/function/invocations\" -d '{\u003cEXPAND_BELOW_AND_REPLACE_WITH_JSON_BELOW\u003e}'\n   ```\n   \u003cdetails\u003e\n   \u003csummary\u003eExample AWS SQS event received\u003c/summary\u003e\n   ```json\n    {\n      \"Records\": [\n        {\n          \"messageId\": \"059f36b4-87a3-44ab-83d2-661975830a7d\",\n          \"receiptHandle\": \"AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a...\",\n          \"body\": \"{\\\"endpointName\\\": \\\"example\\\", \\\"testDataS3BucketName\\\": \\\"example-bucket\\\", \\\"testDataS3Key\\\" : \\\"test-data.csv\\\"}\",\n          \"attributes\": {\n            \"ApproximateReceiveCount\": \"1\",\n            \"SentTimestamp\": \"1545082649183\",\n            \"SenderId\": \"AIDAIENQZJOLO23YVJ4VO\",\n            \"ApproximateFirstReceiveTimestamp\": \"1545082649185\"\n          },\n          \"messageAttributes\": {},\n          \"md5OfBody\": \"098f6bcd4621d373cade4e832627b4f6\",\n          \"eventSource\": \"aws:sqs\",\n          \"eventSourceARN\": \"arn:aws:sqs:us-east-1:111122223333:my-queue\",\n          \"awsRegion\": \"us-east-1\"\n        }\n      ]\n    }\n   ```\n   \u003c/details\u003e\n\n## GitHub Action (CI/CD)\n\nThe GitHub Action \"🚀 Push Docker image to AWS ECR\" will check out the repository and push a docker image to the chosen AWS ECR using\n[configure-aws-credentials](https://github.com/aws-actions/configure-aws-credentials/tree/v4.0.1/) action. The following repository secrets need to be set:\n\n| Secret             | Description                  |\n|--------------------|------------------------------|\n| AWS_REGION         | The AWS Region.              |\n| AWS_ACCOUNT_ID     | The AWS account ID.          |\n| AWS_ECR_REPOSITORY | The AWS ECR repository name. |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkwame-mintah%2Faws-lambda-model-evaluation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkwame-mintah%2Faws-lambda-model-evaluation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkwame-mintah%2Faws-lambda-model-evaluation/lists"}