{"id":20812531,"url":"https://github.com/superstreamlabs/memphis-rest-gateway","last_synced_at":"2025-10-10T03:03:22.576Z","repository":{"id":103056171,"uuid":"571502217","full_name":"superstreamlabs/memphis-rest-gateway","owner":"superstreamlabs","description":"Produce messages using REST to Memphis.dev station","archived":false,"fork":false,"pushed_at":"2024-05-20T11:28:38.000Z","size":230,"stargazers_count":11,"open_issues_count":5,"forks_count":7,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-07T10:52:58.294Z","etag":null,"topics":["http","memphis","memphis-dev","rest-api"],"latest_commit_sha":null,"homepage":"https://docs.memphis.dev","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/superstreamlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-28T09:20:56.000Z","updated_at":"2024-08-19T16:27:23.000Z","dependencies_parsed_at":"2023-11-26T10:23:37.825Z","dependency_job_id":"728589c8-b367-4f52-8af1-248ae22bce0c","html_url":"https://github.com/superstreamlabs/memphis-rest-gateway","commit_stats":null,"previous_names":["superstreamlabs/memphis-rest-gateway","memphisdev/memphis-rest-gateway"],"tags_count":25,"template":false,"template_full_name":null,"purl":"pkg:github/superstreamlabs/memphis-rest-gateway","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/superstreamlabs%2Fmemphis-rest-gateway","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/superstreamlabs%2Fmemphis-rest-gateway/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/superstreamlabs%2Fmemphis-rest-gateway/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/superstreamlabs%2Fmemphis-rest-gateway/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/superstreamlabs","download_url":"https://codeload.github.com/superstreamlabs/memphis-rest-gateway/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/superstreamlabs%2Fmemphis-rest-gateway/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279002477,"owners_count":26083405,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-10T02:00:06.843Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["http","memphis","memphis-dev","rest-api"],"created_at":"2024-11-17T20:55:18.904Z","updated_at":"2025-10-10T03:03:22.556Z","avatar_url":"https://github.com/superstreamlabs.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ca href=\"![Github (2)](https://github.com/memphisdev/memphis.js/assets/107035359/731a59be-0f46-4a94-84c3-c0b2a07fe01c)\"\u003e![Github (2)](https://github.com/memphisdev/memphis.js/assets/107035359/281222f9-8f93-4a20-9de8-7c26541bded7)\u003c/a\u003e\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://memphis.dev/discord\"\u003e\u003cimg src=\"https://img.shields.io/discord/963333392844328961?color=6557ff\u0026label=discord\" alt=\"Discord\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/memphisdev/memphis/issues?q=is%3Aissue+is%3Aclosed\"\u003e\u003cimg src=\"https://img.shields.io/github/issues-closed/memphisdev/memphis?color=6557ff\"\u003e\u003c/a\u003e \n  \u003cimg src=\"https://img.shields.io/npm/dw/memphis-dev?color=ffc633\u0026label=installations\"\u003e\n\u003ca href=\"https://github.com/memphisdev/memphis/blob/master/CODE_OF_CONDUCT.md\"\u003e\u003cimg src=\"https://img.shields.io/badge/Code%20of%20Conduct-v1.0-ff69b4.svg?color=ffc633\" alt=\"Code Of Conduct\"\u003e\u003c/a\u003e \n\u003cimg alt=\"GitHub release (latest by date)\" src=\"https://img.shields.io/github/v/release/memphisdev/memphis?color=61dfc6\"\u003e\n\u003cimg src=\"https://img.shields.io/github/last-commit/memphisdev/memphis?color=61dfc6\u0026label=last%20commit\"\u003e\n\u003c/p\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \n  \u003cimg width=\"200\" alt=\"CNCF Silver Member\" src=\"https://github.com/cncf/artwork/raw/master/other/cncf-member/silver/color/cncf-member-silver-color.svg#gh-light-mode-only\"\u003e\n  \u003cimg width=\"200\" alt=\"CNCF Silver Member\" src=\"https://github.com/cncf/artwork/raw/master/other/cncf-member/silver/white/cncf-member-silver-white.svg#gh-dark-mode-only\"\u003e\n  \n\u003c/div\u003e\n \u003cb\u003e\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://memphis.dev/pricing/\"\u003eCloud\u003c/a\u003e - \u003ca href=\"https://memphis.dev/docs/\"\u003eDocs\u003c/a\u003e - \u003ca href=\"https://twitter.com/Memphis_Dev\"\u003eX\u003c/a\u003e - \u003ca href=\"https://www.youtube.com/channel/UCVdMDLCSxXOqtgrBaRUHKKg\"\u003eYouTube\u003c/a\u003e\n\u003c/p\u003e\u003c/b\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n  \u003ch4\u003e\n\n**[Memphis.dev](https://memphis.dev)** is a highly scalable, painless, and effortless data streaming platform.\u003cbr\u003e\nMade to enable developers and data teams to collaborate and build\u003cbr\u003e\nreal-time and streaming apps fast.\n\n  \u003c/h4\u003e\n  \n\u003c/div\u003e..\n\n# REST Gateway (HTTP Proxy)\n\n## Introduction\n\nTo enable message production via HTTP calls for various use cases and ease of use, Memphis added an HTTP gateway to receive REST-based requests (=messages) and produce those messages to the required station.\n\nCommon use cases for the REST Gateway are\u0026#x20;\n\n* Produce events directly from a frontend\n* Produce CDC events using the Debezium HTTP server\n* ArgoCD webhooks\n* Receive data from Fivetran/Rivery/Any ETL platform using HTTP calls\n\n## Architecture\n\n1. An endpoint creates an HTTP request toward the REST Gateway using **port 4444**\n2. The REST gateway receives the incoming request and produces it as a message to the station\n\n![REST gateway](https://user-images.githubusercontent.com/70286779/212469259-9f092921-63fa-4121-83cf-90f745d4b952.jpeg)\n\n\nFor scale requirements, the \"REST gateway\" component is separate from the brokers' pod and can scale out individually.\n\n## Security Mechanisms\n\n### JWT\n\nMemphis REST (HTTP) gateway makes use of JWT-type identification.\\\n[JSON Web Tokens](https://jwt.io/) are an open, industry-standard RFC 7519 method for representing claims securely between two parties.\n\n### API Token\n\nSoon.\n\n## Sequence diagram\n\n![Sequence diagram](https://user-images.githubusercontent.com/70286779/212469294-ebf2da3f-af30-46bc-bb42-ef860159356e.jpeg)\n\n\n## Getting started\n\nIf you are using Memphis **Open-Source** version, please make sure your 'REST gateway' component is exposed either through localhost or public IP.\u003cbr\u003e\u003cbr\u003e\nIf you are using Memphis **Cloud**, it is already in.\n\n### 1. Create a JWT token\n\nPlease create a JWT token, which will be part of each produce/consume request. For authentication purposes.\n\n* The generated JWT will encapsulate all the needed information for the broker to ensure the requester is authenticated to communicate with Memphis.\n* JWT token (by design) has an expiration time. Token refreshment can take place progrematically, but as it is often used to integrate memphis with other systems which are not supporting JWT refreshment, a workaround to overcome it would be to set a very high value in the `token_expiry_in_minutes`.\n* The default expiry time is 15 minutes.\n\n**Cloud (Using body params)**\u003cbr\u003e\n* Please replace the [Cloud], [Region], Username, Password, and Account ID with your parameters.\n```bash\ncurl --location --request POST 'https://[Cloud]-[Region].restgw.cloud.memphis.dev/auth/authenticate' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\n    \"username\": \"CLIENT_TYPE_USERNAME\",\n    \"password\": \"CLIENT_TYPE_PASSWORD\",\n    \"account_id\": 123456789,\n    \"token_expiry_in_minutes\": 6000000,\n    \"refresh_token_expiry_in_minutes\": 100000\n}'\n```\n\n**Cloud (Using query params)**\u003cbr\u003e\n* Please replace the [Cloud], [Region], Username, Password, and Account ID with your parameters.\n```bash\ncurl --location --request POST 'https://[Cloud]-[Region].restgw.cloud.memphis.dev/auth/authenticate?accountId=123456789' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\n    \"username\": \"CLIENT_TYPE_USERNAME\",\n    \"password\": \"CLIENT_TYPE_PASSWORD\",\n    \"token_expiry_in_minutes\": 6000000,\n    \"refresh_token_expiry_in_minutes\": 100000\n}'\n```\n\n**Open-source**\n```bash\ncurl --location --request POST 'https://REST_GW_URL:4444/auth/authenticate' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\n    \"username\": \"CLIENT_TYPE_USERNAME\",\n    \"password\": \"CLIENT_TYPE_PASSWORD,\n    \"token_expiry_in_minutes\": 6000000,\n    \"refresh_token_expiry_in_minutes\": 100000\n}'\n```\n\nExpected output:\u0026#x20;\n\n```JSON\n{\"expires_in\":3600000,\"jwt\":\"eyJhbGciO***************nR5cCI6IkpXVCJ9.eyJleHAiOjE2NzQ3MTg0MjV9._A************UFoWZjp21UYVcjXwGWiYtacYPZR8\",\"jwt_refresh_token\":\"eyJhbGciOiJIUzI1N***************kpXVCJ9.eyJleHAiOjIy*********************7csm-jmJv0J45YrD_slvlEOKu2rs7Q\",\"refresh_token_expires_in\":600005520000}\n```\n\u003chr\u003e\n\n**Refresh a token**\n\nBefore the JWT token expires or after an authentication failure, you must call the refresh procedure and get a new token. The refresh JWT token is valid by default for 5 hours.\n\n**Cloud**\u003cbr\u003e\n* Please replace the [Cloud], [Region], Username, and Password with your parameters.\n```bash\ncurl --location --request POST 'https://[Cloud]-[Region].restgw.cloud.memphis.dev/auth/refreshToken' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\n    \"jwt_refresh_token\": \"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjIyNz*******************VPSjtU4x02z_jbWhu5pIByhCRh6FU8\",\n    \"token_expiry_in_minutes\": 60000000,\n    \"refresh_token_expiry_in_minutes\": 10000000\n}'\n```\n\n**Open-source**\n```bash\ncurl --location --request POST 'https://REST_GW_URL:4444/auth/refreshToken' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\n    \"jwt_refresh_token\": \"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjIyNz*******************VPSjtU4x02z_jbWhu5pIByhCRh6FU8\",\n    \"token_expiry_in_minutes\": 60000000,\n    \"refresh_token_expiry_in_minutes\": 10000000\n}'\n```\n\nExpected output:\n\n```json\n{\"expires_in\":3600000,\"jwt\":\"eyJhb**************5cCI6IkpXVCJ9.eyJleHAiOjE2NzQ3MTg3N*******************F1-MmFGXRKn2sM8Yw\",\"jwt_refresh_token\":\"eyJhbGciOiJIUzI*****************IkpXVCJ9.eyJleHAiOjIyNz***********************grsqYMPApAPS8YDgkT8R-69-Y5E\",\"refresh_token_expires_in\":600005520000}\n```\n\u003chr\u003e\n\n### 2. Produce a single message\n\n**Supported content types:**\n\n* text\n* application/json\n* application/x-protobuf\n\n\n**Cloud (Using body params)**\n* Please replace the [Cloud], [Region], JWT token (right after `Bearer`) with your parameters.\n\n\n```bash\ncurl --location --request POST 'https://[Cloud]-[Region].restgw.cloud.memphis.dev/stations/STATION_NAME/produce/single' \\\n--header 'Authorization: Bearer eyJhbGciOiJIU**********.e30.4KOGR**************VrLKsvHo33u3UdJ0qYP0kI' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\"message\": \"New Message\"}'\n```\n\n**Cloud (Using query params)**\n* Please replace the [Cloud], [Region], JWT token with your parameters.\n\n```bash\ncurl --location --request POST 'https://[Cloud]-[Region].restgw.cloud.memphis.dev/stations/STATION_NAME/produce/single?authorization=eyJhbGciOiJIU**********.e30.4KOLKsvHo33u3UdJ0qYP0kI' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\"message\": \"New Message\"}'\n```\n\n**Open-source**\n* Please replace the JWT token (right after `Bearer`) with your parameter.\n\n```bash\ncurl --location --request POST 'rest_gateway:4444/stations/STATION_NAME/produce/single' \\\n--header 'Authorization: Bearer eyJhbGciOiJIU**********.e30.4KOGRhUaqvmUdJ0qYP0kI' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\"message\": \"New Message\"}'\n```\n\nExpected output:\n\n```json\n{\"error\":null,\"success\":true}\n```\n\nSchema error example:\n\n```json\n{\"error\":\"Schema validation has failed: jsonschema: '' does not validate with file:///Users/user/memphisdev/memphis-rest-gateway/123#/required: missing properties: 'field1', 'field2', 'field3'\",\"success\":false}\n```\n\n\u003chr\u003e\n\n### 3. Produce a batch of messages\n**Supported content types:**\n\n* application/json\n\n**Cloud (Using body params)**\n* Please replace the [Cloud], [Region], JWT token (right after `Bearer`) with your parameters.\n\n\n```bash\ncurl --location --request POST 'https://[Cloud]-[Region].restgw.cloud.memphis.dev/stations/STATION_NAME/produce/batch' \\\n--header 'Authorization: Bearer eyJhbGciOiJIU**********.e30.4KOGR**************VrLKsvHo33u3UdJ0qYP0kI' \\\n--header 'Content-Type: application/json' \\\n--data-raw '[\n    {\"message\": \"x\"},\n    {\"message\": \"y\"},\n    {\"message\": \"z\"}\n]'\n```\n\n**Cloud (Using query params)**\n* Please replace the [Cloud], [Region], JWT token with your parameters.\n\n```bash\ncurl --location --request POST 'https://[Cloud]-[Region].restgw.cloud.memphis.dev/stations/STATION_NAME/produce/batch?authorization=eyJhbGciOiJIU**********.e30.4KOLKsvHo33u3UdJ0qYP0kI' \\\n--header 'Content-Type: application/json' \\\n--data-raw '[\n    {\"message\": \"x\"},\n    {\"message\": \"y\"},\n    {\"message\": \"z\"}\n]'\n```\n\n**Open-source**\n* Please replace the JWT token (right after `Bearer`) with your parameter.\n\n```bash\ncurl --location --request POST 'rest_gateway:4444/stations/STATION_NAME/produce/batch' \\\n--header 'Authorization: Bearer eyJhbGciOiJIU**********.e30.4KOGRhUaqvmUdJ0qYP0kI' \\\n--header 'Content-Type: application/json' \\\n--data-raw '[\n    {\"message\": \"x\"},\n    {\"message\": \"y\"},\n    {\"message\": \"z\"}\n]'\n```\n\nExpected output:\n\n```json\n{\"error\":null,\"success\":true}\n```\n\nSchema error example:\n\n```json\n{\"errors\":[\"Schema validation has failed: jsonschema: '' does not validate with file:///Users/user/memphisdev/memphis-rest-gateway/123#/required: missing properties: 'field1'\",\"Schema validation has failed: jsonschema: '' does not validate with file:///Users/user/memphisdev/memphis-rest-gateway/123#/required: missing properties: 'field1'\"],\"fail\":2,\"sent\":1,\"success\":false}\n```\n\n\u003chr\u003e\n\n\n### 4. Consume a batch of messages\u0026#x20;\n\nTo avoid reading the same message twice and reduce network traffic for an ack per message, which is not scalable, messages are auto-acknowledged by the rest gateway.\n\n**Supported content types:**\n\n* application/json\n\n**Cloud (Using body params)**\n* Please replace the [Cloud], [Region], JWT token (right after `Bearer`) with your parameters.\n\n\n```bash\ncurl --location --request POST 'https://[Cloud]-[Region].restgw.cloud.memphis.dev/stations/STATION_NAME/consume/batch' \\\n--header 'Authorization: Bearer eyJ***************XVCJ9.e30.4KOGRhUaqvm-qSHnmMw****************' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\n    \"consumer_name\": \u003cconsumer_name\u003e,\n    \"consumer_group\": \u003cconsumer_group\u003e,\n    \"batch_size\": \u003cbatch_size\u003e,\n    \"batch_max_wait_time_ms\": \u003cbatch_max_wait_time\u003e,\n}'\n```\n\n**Open-source**\n* Please replace the JWT token (right after `Bearer`) with your parameter.\n\n```bash\ncurl --location --request POST 'rest_gateway:4444/stations/STATION_NAME/consume/batch' \\\n--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.e30.4KOGRhUaqvm-qSHnmMwX5VrLKsvHo33u3UdJ0qYP0kI' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\n    \"consumer_name\": \u003cconsumer_name\u003e,\n    \"consumer_group\": \u003cconsumer_group\u003e,\n    \"batch_size\": \u003cbatch_size\u003e,\n    \"batch_max_wait_time_ms\": \u003cbatch_max_wait_time\u003e\n}'\n```\n\nExpected output:\n\n```json\n[\n  {\n    \"message\": \"{\\n    \\\"message\\\": \\\"message x\\\"\\n}\",\n    \"headers\": {\n      \"Accept\": \"*/*\",\n      \"Accept-Encoding\": \"gzip, deflate, br\"\n    }\n  },\n  {\n    \"message\": \"{\\n    \\\"message\\\": \\\"message y\\\"\\n}\",\n    \"headers\": {\n      \"Content-Type\": \"application/json\",\n      \"Host\": \"localhost:4444\"\n    }\n  }\n]\n```\n\n#### Error Examples:\n\n```json\n{\n    \"error\": \"Consumer name is required\",\n    \"success\": false\n}\n```\n\n## Support 🙋‍♂️🤝\n\n### Ask a question ❓ about Memphis{dev} or something related to us:\n\nWe welcome you to our discord server with your questions, doubts and feedback.\n\n\u003ca href=\"https://memphis.dev/discord\"\u003e\u003cimg src=\"https://amplication.com/images/discord_banner_purple.svg\"/\u003e\u003c/a\u003e\n\n### Submit a feature 💡 request \n\nIf you have an idea, or you think that we're missing a capability that would make development easier and more robust, please [Submit feature request](https://github.com/memphisdev/memphis/issues/new?assignees=\u0026labels=type%3A%20feature%20request).\n\nIf an issue❗with similar feature request already exists, don't forget to leave a \"+1\".\nIf you add some more information such as your thoughts and vision about the feature, your comments will be embraced warmly :)\n\n## Contributing\n\nMemphis{dev} is an open-source project.\u003cbr\u003e\nWe are committed to a fully transparent development process and appreciate highly any contributions.\u003cbr\u003e\nWhether you are helping us fix bugs, proposing new features, improving our documentation or spreading the word - we would love to have you as part of the Memphis{dev} community.\n\nPlease refer to our [Contribution Guidelines](https://github.com/memphisdev/memphis/CONTRIBUTING.md) and [Code of Conduct](https://github.com/memphisdev/memphis/code_of_conduct.md).\n\n## Contributors ✨\n\nThanks goes to these wonderful people ❤:\u003cbr\u003e\u003cbr\u003e\n \u003ca href = \"https://github.com/memphisdev/memphis-broker/graphs/contributors\"\u003e\n   \u003cimg src = \"https://contrib.rocks/image?repo=memphisdev/memphis\"/\u003e\n \u003c/a\u003e\n\n## License 📃\nMemphis is open-sourced and operates under the \"Memphis Business Source License 1.0\" license\nBuilt out of Apache 2.0, the main difference between the licenses is:\n\"You may make use of the Licensed Work (i) only as part of your own product or service, provided it is not a message broker or a message queue product or service; and (ii) provided that you do not use, provide, distribute, or make available the Licensed Work as a Service. A “Service” is a commercial offering, product, hosted, or managed service, that allows third parties (other than your own employees and contractors acting on your behalf) to access and/or use the Licensed Work or a substantial set of the features or functionality of the Licensed Work to third parties as a software-as-a-service, platform-as-a-service, infrastructure-as-a-service or other similar services that compete with Licensor products or services.\"\nPlease check out [License](https://github.com/memphisdev/memphis/LICENSE) to read the full text.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuperstreamlabs%2Fmemphis-rest-gateway","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsuperstreamlabs%2Fmemphis-rest-gateway","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuperstreamlabs%2Fmemphis-rest-gateway/lists"}