{"id":13524704,"url":"https://github.com/ldclabs/idempotent-proxy","last_synced_at":"2025-04-01T03:32:25.413Z","repository":{"id":232057464,"uuid":"783228915","full_name":"ldclabs/idempotent-proxy","owner":"ldclabs","description":"💈 Reverse proxy server with built-in idempotency support, written in Rust \u0026 Cloudflare Worker.","archived":false,"fork":false,"pushed_at":"2024-10-21T02:58:50.000Z","size":1141,"stargazers_count":12,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-10-23T01:05:03.757Z","etag":null,"topics":["canister","cloudflare-workers","idempotency","reverse-proxy","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ldclabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-07T09:50:24.000Z","updated_at":"2024-10-23T00:33:57.000Z","dependencies_parsed_at":"2024-07-07T15:27:26.926Z","dependency_job_id":"fc5a82d9-b4ec-46ee-9f20-c88f68c4fb69","html_url":"https://github.com/ldclabs/idempotent-proxy","commit_stats":null,"previous_names":["ldclabs/idempotency-proxy","ldclabs/idempotent-proxy"],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ldclabs%2Fidempotent-proxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ldclabs%2Fidempotent-proxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ldclabs%2Fidempotent-proxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ldclabs%2Fidempotent-proxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ldclabs","download_url":"https://codeload.github.com/ldclabs/idempotent-proxy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222698186,"owners_count":17024877,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["canister","cloudflare-workers","idempotency","reverse-proxy","rust"],"created_at":"2024-08-01T06:01:12.637Z","updated_at":"2024-11-02T09:30:22.906Z","avatar_url":"https://github.com/ldclabs.png","language":"Rust","funding_links":[],"categories":["Infrastructure"],"sub_categories":["Candid implementations"],"readme":"# Idempotent Proxy\n💈 Reverse proxy server with built-in idempotency support, written in Rust \u0026 Cloudflare Worker.\n\n💝 This project received a [**$5k Developer Grant**](https://forum.dfinity.org/t/idempotent-proxy-proxy-https-outcalls-to-any-web2-service/30624) from the [DFINITY Foundation](https://dfinity.org/grants).\n\n## Overview\n\nThe idempotent-proxy is a reverse proxy service written in Rust with built-in idempotency support.\n\nWhen multiple requests with the same idempotency key arrive within a specific timeframe, only the first request is forwarded to the target service. The response is cached in Redis (or DurableObject in Cloudflare Worker), and subsequent requests retrieve the cached response, ensuring consistent results.\n\nThis service can be used to proxy [HTTPS outcalls](https://internetcomputer.org/docs/current/references/https-outcalls-how-it-works) for [ICP canisters](https://internetcomputer.org/docs/current/developer-docs/smart-contracts/overview/introduction), enabling integration with any Web2 http service.\n\n![Idempotent Proxy](./idempotent-proxy.webp)\n\n## Features\n- Reverse proxy with built-in idempotency support\n- Confidential information masking\n- JSON and CBOR response filtering\n- Response headers filtering\n- Access control using Secp256k1 and Ed25519\n- Deployable with Docker or Cloudflare Worker\n- On-chain Idempotent Proxy service on the ICP\n\n## Libraries\n\n| Library                                                                                                            | Description                                                                             |\n| :----------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------- |\n| [idempotent-proxy-server](https://github.com/ldclabs/idempotent-proxy/tree/main/src/idempotent-proxy-server)       | Idempotent Proxy implemented in Rust.                                                   |\n| [idempotent-proxy-cf-worker](https://github.com/ldclabs/idempotent-proxy/tree/main/src/idempotent-proxy-cf-worker) | Idempotent Proxy implemented as Cloudflare Worker.                                      |\n| [idempotent-proxy-canister](https://github.com/ldclabs/idempotent-proxy/tree/main/src/idempotent-proxy-canister)   | A ICP canister Make Idempotent Proxy service on-chain.                                  |\n| [idempotent-proxy-types](https://github.com/ldclabs/idempotent-proxy/tree/main/src/idempotent-proxy-types)         | Idempotent Proxy types in Rust. Should not be used in ICP canister!                     |\n| [examples/eth-canister](https://github.com/ldclabs/idempotent-proxy/tree/main/examples/eth-canister)               | A ICP canister integration with Ethereum JSON-RPC API.                                  |\n| [examples/eth-canister-lite](https://github.com/ldclabs/idempotent-proxy/tree/main/examples/eth-canister-lite)     | A ICP canister integration with Ethereum JSON-RPC API through idempotent-proxy-canister |\n\n## Who's using?\n\n- [CK-Doge](https://github.com/ldclabs/ck-doge): An on-chain integration with the Dogecoin network on the Internet Computer.\n\nIf you plan to use this project and have any questions, feel free to open an issue. I will address it as soon as possible.\n\n## Usage\n\n### On-chain Idempotent Proxy\n\nThe `idempotent-proxy-canister` is an ICP smart contract that can connect to 1 to N Idempotent Proxy services deployed by `idempotent-proxy-server` or `idempotent-proxy-cf-worker`. It provides on-chain HTTPS outcalls with idempotency for other smart contracts.\n\n![Idempotent Proxy Canister](./idempotent-proxy-canister.webp)\n\nGo to the [idempotent-proxy-canister](./src/idempotent-proxy-canister) directory for more information.\n\n**Online Demo**: https://a4gq6-oaaaa-aaaab-qaa4q-cai.raw.icp0.io/?id=hpudd-yqaaa-aaaap-ahnbq-cai\n\n### ICP Canister Integration Examples\n\n- [examples/eth-canister-lite](./examples/eth-canister-lite): A ICP canister integration with Ethereum JSON-RPC API through idempotent-proxy-canister.\n- [examples/eth-canister](./examples/eth-canister): A ICP canister integration with Ethereum JSON-RPC API.\n\n### Run proxy in development mode\n\nRun proxy:\n```bash\ncargo run -p idempotent-proxy-server\n```\n\nMake a request:\n```bash\ncurl -v -X POST \\\n  --url http://YOUR_HOST/eth \\\n  --header 'content-type: application/json' \\\n  --header 'x-forwarded-host: rpc.ankr.com' \\\n  --header 'idempotency-key: key_001' \\\n  --data '{\n\t  \"id\": 1,\n    \"jsonrpc\": \"2.0\",\n    \"method\": \"eth_getBlockByNumber\",\n    \"params\": [\"latest\", false]\n}'\n```\n\n### Building enclave image for Marlin Oyster\n\nhttps://docs.marlin.org/user-guides/oyster/instances/quickstart/build\n\n```bash\nsudo docker run --rm --privileged --name nitro-cli -v `pwd`:/mnt/my-server marlinorg/nitro-cli\n```\n\nIn a new terminal, run:\n```bash\ncd /mnt/my-server\nsudo docker exec -it nitro-cli sh\nnitro-cli build-enclave --docker-uri ghcr.io/ldclabs/idempotent-proxy_enclave_amd64:latest --output-file idempotent-proxy_enclave_amd64.eif\n```\n\nThe image URL to deploy on Marlin Oyster:\n```text\nhttps://pub-eea759c16b114748bd3b170eadbb2c30.r2.dev/idempotent-proxy_enclave_amd64.eif\n```\n\nGo to the [idempotent-proxy-server](./src/idempotent-proxy-server) directory for more information.\n\n### Running as Cloudflare Worker\n\nIdempotent Proxy can be running as a Cloudflare Worker. In order to use Durable Objects, you must switch to a paid plan.\n\n```bash\ncd src/idempotent-proxy-cf-worker\nnpm i\nnpx wrangler deploy\n```\n\nA online version for testing is available at:\n\nhttps://idempotent-proxy-cf-worker.zensh.workers.dev\n\nTry it out:\n```\ncurl -v -X GET 'https://idempotent-proxy-cf-worker.zensh.workers.dev/URL_HTTPBIN' \\\n  -H 'idempotency-key: idempotency_key_001' \\\n  -H 'content-type: application/json'\n```\n\nMore `URL_` constants:\n- URL_CF_ETH: https://cloudflare-eth.com\n- URL_ANKR_ETH: https://rpc.ankr.com/eth\n\n`idempotent-proxy-cf-worker` does not enable `proxy-authorization`, so it can be accessed.\n\nGo to the [idempotent-proxy-cf-worker](./src/idempotent-proxy-cf-worker) directory for more information.\n\n### Run proxy with Docker\n\nfiles in `/mnt/idempotent-proxy` directory:\n```\n/mnt/idempotent-proxy/.env\n/mnt/idempotent-proxy/keys/doge-test-rpc.panda.fans.key\n/mnt/idempotent-proxy/keys/doge-test-rpc.panda.fans.pem\n```\n\n`.env` file:\n```text\nSERVER_ADDR=0.0.0.0:443\nREDIS_URL=172.16.32.1:6379\nPOLL_INTERVAL=100 # in milliseconds\nREQUEST_TIMEOUT=10000 # in milliseconds\nLOG_LEVEL=info # debug, info, warn, error\n# cert file path to enable https, for example: /etc/https/mydomain.crt\nTLS_CERT_FILE = \"keys/doge-test-rpc.panda.fans.pem\"\n# key file path to enable https, for example: /etc/https/mydomain.key\nTLS_KEY_FILE = \"keys/doge-test-rpc.panda.fans.key\"\n\nECDSA_PUB_KEY_1=\"A44DZpzDwDvq9HwW3_dynOfDgkMJHKgOxUyCOrv5Pl3O\"\n\n# ECDSA_PUB_KEY_2=\"xxxxxx\"\n\nALLOW_AGENTS=\"ICPanda\"\n\nURL_DOGE_TEST=\"http://172.16.32.1:44555/\"\nURL_DOGE=\"http://172.16.32.1:22555/\"\n# URL_XXX=...\n\nHEADER_API_TOKEN=\"Basic SUNQYW5kYTpJVEZDNlJjam56RkdEQnd0SzByYV9kS0swR29lSElqVUl3V2lEb3VrRWU0\"\n# HEADER_XXX=...\n```\n\nRun proxy with Docker:\n```bash\ndocker run --restart=always -v /mnt/idempotent-proxy/.env:/app/.env -v /mnt/idempotent-proxy/keys:/app/keys --name proxy -d -p 443:443 ghcr.io/ldclabs/idempotent-proxy:latest\n```\n\n## Request Examples\n\n### Regular Proxy Request Example\n\nMake a request:\n```bash\ncurl -v -X GET 'http://localhost:8080/get' \\\n  -H 'x-forwarded-host: httpbin.org' \\\n  -H 'idempotency-key: idempotency_key_001' \\\n  -H 'content-type: application/json'\n```\n\nResponse:\n```text\n\u003c HTTP/1.1 200 OK\n\u003c date: Wed, 22 May 2024 11:03:33 GMT\n\u003c content-type: application/json\n\u003c content-length: 375\n\u003c server: gunicorn/19.9.0\n\u003c access-control-allow-origin: *\n\u003c access-control-allow-credentials: true\n\u003c\n{\n  \"args\": {},\n  \"headers\": {\n    \"Accept\": \"*/*\",\n    \"Accept-Encoding\": \"gzip\",\n    \"Content-Type\": \"application/json\",\n    \"Host\": \"httpbin.org\",\n    \"Idempotency-Key\": \"idempotency_key_001\",\n    \"User-Agent\": \"curl/8.6.0\",\n    \"X-Amzn-Trace-Id\": \"Root=1-664dd105-7930bcc43ae6081a4508d114\"\n  },\n  \"origin\": \"120.204.60.218\",\n  \"url\": \"https://httpbin.org/get\"\n}\n```\n\nRequest again with the same idempotency key will return the same response.\n\n### Proxy Request Example with `URL_` Constant Defined\n\nSetting in .env file:\n```text\nURL_HTTPBIN=\"https://httpbin.org/get?api-key=abc123\"\n```\n\nMake a request with `URL_HTTPBIN` constant in url path:\n```bash\ncurl -v -X GET 'http://localhost:8080/URL_HTTPBIN' \\\n  -H 'idempotency-key: idempotency_key_001' \\\n  -H 'content-type: application/json'\n```\n\nResponse:\n```text\n\u003c HTTP/1.1 200 OK\n\u003c date: Wed, 22 May 2024 11:07:05 GMT\n\u003c content-type: application/json\n\u003c content-length: 417\n\u003c server: gunicorn/19.9.0\n\u003c access-control-allow-origin: *\n\u003c access-control-allow-credentials: true\n\u003c\n{\n  \"args\": {\n    \"api-key\": \"abc123\"\n  },\n  \"headers\": {\n    \"Accept\": \"*/*\",\n    \"Accept-Encoding\": \"gzip\",\n    \"Content-Type\": \"application/json\",\n    \"Host\": \"httpbin.org\",\n    \"Idempotency-Key\": \"idempotency_key_001\",\n    \"User-Agent\": \"curl/8.6.0\",\n    \"X-Amzn-Trace-Id\": \"Root=1-664dd1d9-6612bfd076e95b814dd9329d\"\n  },\n  \"origin\": \"120.204.60.218\",\n  \"url\": \"https://httpbin.org/get?api-key=abc123\"\n}\n```\n\n### Proxy Request Example with `HEADER_` Constant Defined\n\nSetting in .env file:\n```text\nURL_HTTPBIN=\"https://httpbin.org/get?api-key=abc123\"\nHEADER_TOKEN=\"Bearer xyz123456\"\n```\n\nMake a request with `HEADER_TOKEN` constant in header:\n```bash\ncurl -v -X GET 'http://localhost:8080/URL_HTTPBIN' \\\n  -H 'idempotency-key: idempotency_key_001' \\\n  -H 'authorization: HEADER_TOKEN' \\\n  -H 'content-type: application/json'\n```\n\nResponse:\n```text\n\u003c HTTP/1.1 200 OK\n\u003c date: Wed, 22 May 2024 11:11:17 GMT\n\u003c content-type: application/json\n\u003c content-length: 459\n\u003c server: gunicorn/19.9.0\n\u003c access-control-allow-origin: *\n\u003c access-control-allow-credentials: true\n\u003c\n{\n  \"args\": {\n    \"api-key\": \"abc123\"\n  },\n  \"headers\": {\n    \"Accept\": \"*/*\",\n    \"Accept-Encoding\": \"gzip\",\n    \"Authorization\": \"Bearer xyz123456\",\n    \"Content-Type\": \"application/json\",\n    \"Host\": \"httpbin.org\",\n    \"Idempotency-Key\": \"idempotency_key_001\",\n    \"User-Agent\": \"curl/8.6.0\",\n    \"X-Amzn-Trace-Id\": \"Root=1-664dd2d5-15b233f974a01ca34bd9a8ab\"\n  },\n  \"origin\": \"120.204.60.218\",\n  \"url\": \"https://httpbin.org/get?api-key=abc123\"\n}\n```\n\n### Proxy Request Example with Response Headers Filtered\n\nMake a request with `response-headers` header:\n```bash\ncurl -v -X GET 'http://localhost:8080/URL_HTTPBIN' \\\n  -H 'idempotency-key: idempotency_key_001' \\\n  -H 'authorization: HEADER_TOKEN' \\\n  -H 'response-headers: content-type,content-length' \\\n  -H 'content-type: application/json'\n```\n\nResponse:\n```text\n\u003c HTTP/1.1 200 OK\n\u003c content-type: application/json\n\u003c content-length: 515\n\u003c date: Wed, 22 May 2024 11:13:39 GMT\n\u003c\n{\n  \"args\": {\n    \"api-key\": \"abc123\"\n  },\n  \"headers\": {\n    \"Accept\": \"*/*\",\n    \"Accept-Encoding\": \"gzip\",\n    \"Authorization\": \"Bearer xyz123456\",\n    \"Content-Type\": \"application/json\",\n    \"Host\": \"httpbin.org\",\n    \"Idempotency-Key\": \"idempotency_key_001\",\n    \"Response-Headers\": \"content-type,content-length\",\n    \"User-Agent\": \"curl/8.6.0\",\n    \"X-Amzn-Trace-Id\": \"Root=1-664dd363-2bbae4420bf9add8512f5930\"\n  },\n  \"origin\": \"120.204.60.218\",\n  \"url\": \"https://httpbin.org/get?api-key=abc123\"\n}\n```\n\n### Proxy Request Example with JSON Response Filtered\n\nMake a request with `x-json-mask` header:\n```bash\ncurl -v -X GET 'http://localhost:8080/URL_HTTPBIN' \\\n  -H 'idempotency-key: idempotency_key_001' \\\n  -H 'authorization: HEADER_TOKEN' \\\n  -H 'response-headers: content-type,content-length' \\\n  -H 'x-json-mask: args,url' \\\n  -H 'content-type: application/json'\n```\n\nResponse:\n```text\n\u003c HTTP/1.1 200 OK\n\u003c content-type: application/json\n\u003c content-length: 76\n\u003c date: Wed, 22 May 2024 12:19:03 GMT\n\u003c\n* Connection #0 to host localhost left intact\n{\"args\":{\"api-key\":\"abc123\"},\"url\":\"https://httpbin.org/get?api-key=abc123\"}\n```\n\n### Proxy Request Example with Access Control Added\n\nSetting in .env file:\n```text\nECDSA_PUB_KEY_1=\"A6t1U8kc10AbLJ3-V1avU4rYvmAsYjXuzY0kPublttot\"\n```\n\nYou can add other public keys by adding `ECDSA_PUB_KEY_2`, `ECDSA_PUB_KEY_abc` for key rotation.\n\nMake a request with `proxy-authorization` header, the bearer token is signed with the private key:\n```bash\ncurl -v -X GET 'http://localhost:8080/URL_HTTPBIN' \\\n  -H 'idempotency-key: idempotency_key_001' \\\n  -H 'proxy-authorization: Bearer 6LduPbIpAAAAANSOUfb-8bU45eilZFSmlSguN5TO' \\\n  -H 'authorization: HEADER_TOKEN' \\\n  -H 'response-headers: content-type,content-length' \\\n  -H 'x-json-mask: args,url' \\\n  -H 'content-type: application/json'\n```\n\nA 407 response:\n```text\n\u003c HTTP/1.1 407 Proxy Authentication Required\n\u003c content-type: text/plain; charset=utf-8\n\u003c content-length: 34\n\u003c date: Wed, 22 May 2024 12:24:40 GMT\n\u003c\n* Connection #0 to host localhost left intact\nproxy authentication verify failed: failed to decode CBOR data\n```\n\n## License\nCopyright © 2024 [LDC Labs](https://github.com/ldclabs).\n\n`ldclabs/idempotent-proxy` is licensed under the MIT License. See [LICENSE](LICENSE-MIT) for the full license text.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fldclabs%2Fidempotent-proxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fldclabs%2Fidempotent-proxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fldclabs%2Fidempotent-proxy/lists"}