{"id":36734605,"url":"https://github.com/lambda-feedback/evaluation-function-boilerplate-python","last_synced_at":"2026-01-20T17:45:56.665Z","repository":{"id":312070344,"uuid":"859386346","full_name":"lambda-feedback/evaluation-function-boilerplate-python","owner":"lambda-feedback","description":"Boilerplate repository for evaluation functions written in Python","archived":false,"fork":false,"pushed_at":"2025-08-28T10:13:15.000Z","size":21,"stargazers_count":0,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-08-28T17:24:17.985Z","etag":null,"topics":["evaluation-function-boilerplate"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lambda-feedback.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-09-18T15:16:26.000Z","updated_at":"2025-08-28T10:13:18.000Z","dependencies_parsed_at":"2025-08-28T17:25:40.027Z","dependency_job_id":"a5394028-202b-4d9a-8221-a1996ff83877","html_url":"https://github.com/lambda-feedback/evaluation-function-boilerplate-python","commit_stats":null,"previous_names":["lambda-feedback/evaluation-function-boilerplate-python"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/lambda-feedback/evaluation-function-boilerplate-python","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lambda-feedback%2Fevaluation-function-boilerplate-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lambda-feedback%2Fevaluation-function-boilerplate-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lambda-feedback%2Fevaluation-function-boilerplate-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lambda-feedback%2Fevaluation-function-boilerplate-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lambda-feedback","download_url":"https://codeload.github.com/lambda-feedback/evaluation-function-boilerplate-python/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lambda-feedback%2Fevaluation-function-boilerplate-python/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28338975,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T10:58:46.209Z","status":"ssl_error","status_checked_at":"2026-01-12T10:58:42.742Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["evaluation-function-boilerplate"],"created_at":"2026-01-12T12:18:28.334Z","updated_at":"2026-01-12T12:18:28.420Z","avatar_url":"https://github.com/lambda-feedback.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Python Evaluation Function\n\nThis repository contains the boilerplate code needed to create a containerized evaluation function written in Python.\n\n## Quickstart\n\nThis chapter helps you to quickly set up a new Python evaluation function using this template repository.\n\n\u003e [!NOTE]\n\u003e After setting up the evaluation function, delete this chapter from the `README.md` file, and add your own documentation.\n\n#### 1. Create a new repository\n\n- In GitHub, choose `Use this template` \u003e `Create a new repository` in the repository toolbar.\n\n- Choose the owner, and pick a name for the new repository.\n\n  \u003e [!IMPORTANT]\n  \u003e If you want to deploy the evaluation function to Lambda Feedback, make sure to choose the Lambda Feedback organization as the owner.\n\n- Set the visibility to `Public` or `Private`.\n\n  \u003e [!IMPORTANT]\n  \u003e If you want to use GitHub [deployment protection rules](https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#deployment-protection-rules), make sure to set the visibility to `Public`.\n\n- Click on `Create repository`.\n\n#### 2. Clone the new repository\n\nClone the new repository to your local machine using the following command:\n\n```bash\ngit clone \u003crepository-url\u003e\n```\n\n#### 3. Configure the evaluation function\n\nWhen deploying to Lambda Feedback, set the evaluation function name in the `config.json` file. Read the [Deploy to Lambda Feedback](#deploy-to-lambda-feedback) section for more information.\n\n#### 4. Develop the evaluation function\n\nYou're ready to start developing your evaluation function. Head over to the [Development](#development) section to learn more.\n\n#### 5. Update the README\n\nIn the `README.md` file, change the title and description so it fits the purpose of your evaluation function.\n\nAlso, don't forget to delete the Quickstart chapter from the `README.md` file after you've completed these steps.\n\n## Usage\n\nYou can run the evaluation function either using [the pre-built Docker image](#run-the-docker-image) or build and run [the binary executable](#build-and-run-the-binary).\n\n### Run the Docker Image\n\nThe pre-built Docker image comes with [Shimmy](https://github.com/lambda-feedback/shimmy) installed.\n\n\u003e [!TIP]\n\u003e Shimmy is a small application that listens for incoming HTTP requests, validates the incoming data and forwards it to the underlying evaluation function. Learn more about Shimmy in the [Documentation](https://github.com/lambda-feedback/shimmy).\n\nThe pre-built Docker image is available on the GitHub Container Registry. You can run the image using the following command:\n\n```bash\ndocker run -p 8080:8080 ghcr.io/lambda-feedback/evaluation-function-boilerplate-python:latest\n```\n\n### Run the Script\n\nYou can choose between running the Python evaluation function itself, ore using Shimmy to run the function.\n\n**Raw Mode**\n\nUse the following command to run the evaluation function directly:\n\n```bash\npython -m evaluation_function.main\n```\n\nThis will run the evaluation function using the input data from `request.json` and write the output to `response.json`.\n\n**Shimmy**\n\nTo have a more user-friendly experience, you can use [Shimmy](https://github.com/lambda-feedback/shimmy) to run the evaluation function.\n\nTo run the evaluation function using Shimmy, use the following command:\n\n```bash\nshimmy -c \"python\" -a \"-m\" -a \"evaluation_function.main\" -i ipc\n```\n\n## Development\n\n### Prerequisites\n\n- [Docker](https://docs.docker.com/get-docker/)\n- [Python](https://www.python.org)\n\n### Repository Structure\n\n```bash\n.github/workflows/\n    build.yml                           # builds the public evaluation function image\n    deploy.yml                          # deploys the evaluation function to Lambda Feedback\n\nevaluation_function/main.py             # evaluation function entrypoint\nevaluation_function/evaluation.py       # evaluation function implementation\nevaluation_function/evaluation_test.py  # evaluation function tests\nevaluation_function/preview.py          # evaluation function preview\nevaluation_function/preview_test.py     # evaluation function preview tests\n\nconfig.json                             # evaluation function deployment configuration file\n```\n\n### Development Workflow\n\nIn its most basic form, the development workflow consists of writing the evaluation function in the `evaluation_function.wl` file and testing it locally. As long as the evaluation function adheres to the Evaluation Function API, a development workflow which incorporates using Shimmy is not necessary.\n\nTesting the evaluation function can be done by running the `dev.py` script using the Python interpreter like so:\n\n```bash\npython -m evaluation_function.dev \u003cresponse\u003e \u003canswer\u003e\n```\n\n\u003e [!NOTE]\n\u003e Specify the `response` and `answer` as command-line arguments.\n\n### Building the Docker Image\n\nTo build the Docker image, run the following command:\n\n```bash\ndocker build -t my-python-evaluation-function .\n```\n\n### Running the Docker Image\n\nTo run the Docker image, use the following command:\n\n```bash\ndocker run -it --rm -p 8080:8080 my-python-evaluation-function\n```\n\nThis will start the evaluation function and expose it on port `8080`.\n\n## Deployment\n\nThis section guides you through the deployment process of the evaluation function. If you want to deploy the evaluation function to Lambda Feedback, follow the steps in the [Lambda Feedback](#deploy-to-lambda-feedback) section. Otherwise, you can deploy the evaluation function to other platforms using the [Other Platforms](#deploy-to-other-platforms) section.\n\n### Deploy to Lambda Feedback\n\nDeploying the evaluation function to Lambda Feedback is simple and straightforward, as long as the repository is within the [Lambda Feedback organization](https://github.com/lambda-feedback).\n\nAfter configuring the repository, a [GitHub Actions workflow](.github/workflows/deploy.yml) will automatically build and deploy the evaluation function to Lambda Feedback as soon as changes are pushed to the main branch of the repository.\n\n**Configuration**\n\nThe deployment configuration is stored in the `config.json` file. Choose a unique name for the evaluation function and set the `EvaluationFunctionName` field in [`config.json`](config.json).\n\n\u003e [!IMPORTANT]\n\u003e The evaluation function name must be unique within the Lambda Feedback organization, and must be in `lowerCamelCase`. You can find a example configuration below:\n\n```json\n{\n  \"EvaluationFunctionName\": \"compareStringsWithPython\"\n}\n```\n\n### Deploy to other Platforms\n\nIf you want to deploy the evaluation function to other platforms, you can use the Docker image to deploy the evaluation function.\n\nPlease refer to the deployment documentation of the platform you want to deploy the evaluation function to.\n\nIf you need help with the deployment, feel free to reach out to the Lambda Feedback team by creating an issue in the template repository.\n\n## FAQ\n\n### Pull Changes from the Template Repository\n\nIf you want to pull changes from the template repository to your repository, follow these steps:\n\n1. Add the template repository as a remote:\n\n```bash\ngit remote add template https://github.com/lambda-feedback/evaluation-function-boilerplate-python.git\n```\n\n2. Fetch changes from all remotes:\n\n```bash\ngit fetch --all\n```\n\n3. Merge changes from the template repository:\n\n```bash\ngit merge template/main --allow-unrelated-histories\n```\n\n\u003e [!WARNING]\n\u003e Make sure to resolve any conflicts and keep the changes you want to keep.\n\n## Troubleshooting\n\n### Containerized Evaluation Function Fails to Start\n\nIf your evaluation function is working fine when run locally, but not when containerized, there is much more to consider. Here are some common issues and solution approaches:\n\n**Run-time dependencies**\n\nMake sure that all run-time dependencies are installed in the Docker image.\n\n- Python packages: Make sure to add the dependency to the `pyproject.toml` file, and run `poetry install` in the Dockerfile.\n- System packages: If you need to install system packages, add the installation command to the Dockerfile.\n- ML models: If your evaluation function depends on ML models, make sure to include them in the Docker image.\n- Data files: If your evaluation function depends on data files, make sure to include them in the Docker image.\n\n**Architecture**\n\nSome package may not be compatible with the architecture of the Docker image. Make sure to use the correct platform when building and running the Docker image.\n\nE.g. to build a Docker image for the `linux/x86_64` platform, use the following command:\n\n```bash\ndocker build --platform=linux/x86_64 .\n```\n\n**Verify Standalone Execution**\n\nIf requests are timing out, it might be due to the evaluation function not being able to run. Make sure that the evaluation function can be run as a standalone script. This will help you to identify issues that are specific to the containerized environment.\n\nTo run just the evaluation function as a standalone script, without using Shimmy, use the following command:\n\n```bash\ndocker run -it --rm my-python-evaluation-function python -m evaluation_function.main\n```\n\nIf the command starts without any errors, the evaluation function is working correctly. If not, you will see the error message in the console.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flambda-feedback%2Fevaluation-function-boilerplate-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flambda-feedback%2Fevaluation-function-boilerplate-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flambda-feedback%2Fevaluation-function-boilerplate-python/lists"}