{"id":36456326,"url":"https://github.com/lambda-feedback/shimmy","last_synced_at":"2026-01-11T23:04:00.338Z","repository":{"id":232956938,"uuid":"775115006","full_name":"lambda-feedback/shimmy","owner":"lambda-feedback","description":"A language-agnostic evaluation function executor, written in go","archived":false,"fork":false,"pushed_at":"2025-08-14T14:42:34.000Z","size":306,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-08-14T16:31:12.109Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lambda-feedback.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-03-20T19:38:01.000Z","updated_at":"2025-08-14T14:42:34.000Z","dependencies_parsed_at":"2024-05-29T08:48:48.898Z","dependency_job_id":"d34479a0-1539-4d7b-9315-b249392829ce","html_url":"https://github.com/lambda-feedback/shimmy","commit_stats":null,"previous_names":["lambda-feedback/shimmy"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lambda-feedback/shimmy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lambda-feedback%2Fshimmy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lambda-feedback%2Fshimmy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lambda-feedback%2Fshimmy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lambda-feedback%2Fshimmy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lambda-feedback","download_url":"https://codeload.github.com/lambda-feedback/shimmy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lambda-feedback%2Fshimmy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28326179,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-11T22:11:01.104Z","status":"ssl_error","status_checked_at":"2026-01-11T22:10:58.990Z","response_time":60,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-11T23:03:59.611Z","updated_at":"2026-01-11T23:04:00.318Z","avatar_url":"https://github.com/lambda-feedback.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Lambda Feedback Evaluation Function Shim\n\nShimmy is a shim written in Go that interacts with language-agnostic evaluation functions as part of the lambda feedback platform, and exposes them as a RESTful API.\n\n## Overview\n\nShimmy listens for incoming HTTP requests / events from feedback clients, validates the incoming data, and forwards it to the underlying evaluation function. The shim is responsible for managing the lifecycle of the evaluation function, and ensures that it is available to process incoming requests. The evaluation function is expected to be a executable application that implements the evaluation runtime interface.\n\nThis abstraction allows the evaluation function to be written in any language, and provides a consistent interface for interacting with the lambda feedback platform. Moreover, the shim provides common functionality such as logging, error handling, and request validation, which simplifies the development of evaluation functions and allows developers to focus on the core logic.\n\n### Architecture\n\nShimmy is designed to be a lightweight, stateless, OS- and architecture-agnostic binary that is intended to be run alongside other, containerized applications. The shim handles incoming evaluation requests, and forwards them to the evaluation function. The evaluation function is expected to be a standalone application that implements the evaluation runtime interface, and is managed by the shim. The following diagram illustrates the architecture of the shim:\n\n![Component Diagram](./docs/img/evaluation-function-shim-component-diagram.svg)\n\nAs shown in the diagram, the shim allows the evaluation function to be deployed in three different execution environments, all supported out of the box:\n\n1. **AWS Lambda (managed)**: The evaluation function image is deployed as an AWS Lambda function. The shim implements the AWS Lambda runtime interface, and forwards incoming events to the evaluation function. This allows the evaluation function to be executed in a serverless environment.\n\n2. **AWS Lambda (self-hosted)**: The evaluation function image contains the [AWS Lambda Runtime Interface Emulator](https://github.com/aws/aws-lambda-runtime-interface-emulator). The shim implements the AWS Lambda runtime interface, and forwards incoming events to the evaluation function. This allows the evaluation function to be executed in a local or self-hosted environment, while maintaining compatibility with the AWS Lambda runtime interface.\n\n3. **Standalone (self-hosted)**: The shim includes a standalone HTTP server that listens for incoming evaluation requests. As with the other environments, the shim forwards incoming requests to the evaluation function. This allows for maximum deployment flexibility, without being restricted to a specific runtime environment.\n\n## Usage\n\n`shimmy --help` displays the available command-line options:\n\n```shell\nNAME:\n   shimmy - A shim for running arbitrary, language-agnostic evaluation\n            functions on arbitrary, serverless platforms.\n\nUSAGE:\n   shimmy [global options] command [command options] [arguments...]\n\nVERSION:\n   local\n\nCOMMANDS:\n   lambda  Run the AWS Lambda handler.\n   run     Detect execution environment and start shim.\n   serve   Start a http server and listen for events.\n\nGLOBAL OPTIONS:\n   --help, -h          show help\n   --log-format value  set the log format. Options: production, development. [$LOG_FORMAT]\n   --log-level value   set the log level. Options: debug, info, warn, error, panic, fatal. [$LOG_LEVEL]\n   --version           print the version\n\n   auth\n\n   --auth-key value, -k value  the authentication key to use for incoming requests. [$AUTH_KEY]\n\n   function\n\n   --arg value, -a value [ --arg value, -a value ]  additional arguments for to the worker process. [$FUNCTION_ARGS]\n   --command value, -c value                        the command to invoke to start the worker process. [$FUNCTION_COMMAND]\n   --cwd value, -d value                            the working directory for the worker process. [$FUNCTION_WORKING_DIR]\n   --env value, -e value [ --env value, -e value ]  additional environment variables for the worker process. [$FUNCTION_ENV]\n   --interface value, -i value                      the interface to use for worker process communication. Options: rpc, file. (default: \"rpc\") [$FUNCTION_INTERFACE]\n   --max-workers value, -n value                    the maximum number of worker processes to run concurrently. (default: number of CPU cores) [$FUNCTION_MAX_PROCS]\n\n   rpc\n\n   --rpc-transport value, -t value     the transport to use for the RPC interface. Options: stdio, ipc, http, tcp, ws. (default: \"stdio\") [$FUNCTION_RPC_TRANSPORT]\n   --rpc-transport-http-url value      the url to use for the HTTP transport. Default: http://127.0.0.1:7321 (default: \"http://127.0.0.1:7321\") [$FUNCTION_RPC_TRANSPORT_HTTP_URL]\n   --rpc-transport-ipc-endpoint value  the IPC endpoint to use for the IPC transport. Default: /tmp/eval.sock [$FUNCTION_RPC_TRANSPORT_IPC_ENDPOINT]\n   --rpc-transport-tcp-address value   the address to use for the TCP transport. Default: 127.0.0.1:7321 (default: \"127.0.0.1:7321\") [$FUNCTION_RPC_TRANSPORT_TCP_ADDRESS]\n   --rpc-transport-ws-url value        the url to use for the WebSocket transport. Default: ws://127.0.0.1:7321 (default: \"ws://127.0.0.1:7321\") [$FUNCTION_RPC_TRANSPORT_WS_URL]\n\n   worker\n\n   --worker-send-timeout value  the timeout for a single message send operation. (default: 30s) [$FUNCTION_WORKER_SEND_TIMEOUT]\n   --worker-stop-timeout value  the duration to wait for a worker process to stop. (default: 5s) [$FUNCTION_WORKER_STOP_TIMEOUT]\n```\n\n## Evaluation Runtime Interface\n\nThe evaluation function is expected to be a standalone application or script that implements the evaluation runtime interface. The evaluation runtime interface is a simple, language-agnostic, JSON-based protocol that defines how the shim communicates with the evaluation function.\n\nThe evaluation function is responsible for parsing the input JSON message, performing the evaluation, and responding with the output JSON message. The evaluation function should exit with a status code of `0` if the evaluation was successful, and a non-zero status code if an error occurred.\n\n### Messages\n\nThe interface consists of input and output messages, which are exchanged between the shim and the evaluation function. Both the input and output to the evaluation function are expected to be valid JSON messages, conforming to the schema defined in the following.\n\n#### Input\n\nThe input message is a JSON-encoded object that contains the following fields:\n\n- `$id` (int, optional): A unique identifier for the evaluation request.\n- `command` (string): The command to be executed by the evaluation function.\n- `*` (object): The input data for the evaluation function.\n\n\u003e The `$id` field is not used for file-based communication.\n\nThe object should follow one of the following schemas, depending on the command:\n\n- `eval`: [Evaluation Schema](./runtime/schema/request-eval.json)\n- `preview`: [Preview Schema](./runtime/schema/request-preview.json)\n\nAn example input message for the `evaluate` command is shown below:\n\n```json\n{\n  \"$id\": 1,\n  \"command\": \"eval\",\n  \"response\": \"...\",\n  \"answer\": \"...\",\n  \"params\": {\n    \"param1\": \"...\"\n  }\n}\n```\n\n#### Output\n\nThe output message is expected to be a JSON-encoded string that contains the following fields:\n\n- `$id` (int, optional): The unique identifier of the evaluation request.\n- `*` (object): The output data from the evaluation function.\n\n\u003e The `$id` field is not used for file-based communication.\n\nThe object should follow one of the following schemas, depending on the command:\n\n- `eval`: [Evaluation Schema](./runtime/schema/response-eval.json)\n- `preview`: [Preview Schema](./runtime/schema/response-preview.json)\n\nAn example output message for the `evaluate` command is shown below:\n\n```json\n{\n  \"$id\": 1,\n  \"command\": \"eval\",\n  \"result\": {\n    \"is_correct\": \"...\"\n  }\n}\n```\n\n### Communication Channels\n\nThe shim is capable of communicating with the evaluation function using two different channels:\n\n1. **Standard I/O (stdio)**: The shim communicates with the evaluation function using standard input and output. The evaluation function reads the input JSON object from standard input, and writes the output JSON object to standard output.\n\n2. **File System (file)**: The shim communicates with the evaluation function using the file system. The evaluation function is expected to read the input JSON object from a file, and write the output JSON object to a file.\n\n   The file paths are always the last two command-line arguments passed to the evaluation function.\n   \n   NOTE: Using the file system is good for large request, such as base64 images.\n\n   For example, a wolframscript evaluation function in `evaluation.wl`, that reads the input JSON object from a file named `input.json` and writes the output JSON object to a file named `output.json` would be invoked as follows:\n\n   ```shell\n   wolframscript -file evaluation.wl input.json output.json\n   ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flambda-feedback%2Fshimmy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flambda-feedback%2Fshimmy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flambda-feedback%2Fshimmy/lists"}