{"id":17101741,"url":"https://github.com/nicholasjackson/pipe","last_synced_at":"2025-04-13T00:25:49.300Z","repository":{"id":64306550,"uuid":"112008950","full_name":"nicholasjackson/pipe","owner":"nicholasjackson","description":"Multi provider event grid written in go","archived":false,"fork":false,"pushed_at":"2018-07-30T17:18:54.000Z","size":9495,"stargazers_count":10,"open_issues_count":5,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-26T18:21:29.321Z","etag":null,"topics":["events","golang","nats","openfaas"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nicholasjackson.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-11-25T14:17:02.000Z","updated_at":"2020-02-16T14:59:46.000Z","dependencies_parsed_at":"2023-01-15T10:45:38.115Z","dependency_job_id":null,"html_url":"https://github.com/nicholasjackson/pipe","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicholasjackson%2Fpipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicholasjackson%2Fpipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicholasjackson%2Fpipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicholasjackson%2Fpipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nicholasjackson","download_url":"https://codeload.github.com/nicholasjackson/pipe/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248649838,"owners_count":21139563,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["events","golang","nats","openfaas"],"created_at":"2024-10-14T15:26:37.974Z","updated_at":"2025-04-13T00:25:49.278Z","avatar_url":"https://github.com/nicholasjackson.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pipe - Event Grid and Message Router\n\n[![Docker Repository on Quay](https://quay.io/repository/nicholasjackson/faas-nats/status \"Docker Repository on Quay\")](https://quay.io/repository/nicholasjackson/pipe)\n[![CircleCI](https://circleci.com/gh/nicholasjackson/pipe.svg?style=svg)](https://circleci.com/gh/nicholasjackson/pipe) \n[![Maintainability](https://api.codeclimate.com/v1/badges/a3c44667f431244a86ae/maintainability)](https://codeclimate.com/github/nicholasjackson/pipe/maintainability)\n\nThis project allows you to listen to a variety of message sources and perform an action when a message is received.  The documentation and the project is currently work in progress however curretly supported providers are:\n* Nats.io - read and write to nats streaming\n* HTTP - receive and send events over http\n\nThe project is built around a provider model where plugable elements can be added to the server to allow support for a variety of message sources.\n\nPlanned providers:\n* Log files - read and write to log files\n* SQS - AWS Simple Message Queue\n* PubSub - Google pub sub\n* Kafka\n* And more.\n\n## Configuration\nTo configure pipes HCL configuration file is used...\n\n```yaml\n# Input block, will listen for nats messages on defined queue\ninput \"nats_queue\" \"nq_in\" {\n  server = \"nats://nats.service.consul:4222\"\n  cluster_id = \"test-cluster\"\n  queue = \"testmessagequeue\"\n}\n\n# Output block, defines a http output\noutput \"http\" \"nq_out\" {\n  protocol = \"http\"\n  server = \"localhost\"\n  port = 8080\n  path = \"/message\"\n}\n\npipe \"accept_nats\" {\n  # Name of the input block\n  input = \"nq_in\"\n\n  # Do not handle messages older than\n  expiration = \"1h\"\n\n  # Action to perform when a new message is received\n  action {\n    # Name of the output\n    output = \"nq_out\"\n\n    # Transform the initial message\n    template = \u003c\u003cEOF\n      {\n        \"text\": \"Hey a picture from selfi drone\",\n        \"image\": \"{{ .JSON.Data }}\"\n      }\n    EOF\n\n  }\n\n  # Called when action succeedes\n  on_success {\n    output = \"success\"\n  }\n \n  on_success {\n    output = \"success\"\n  }\n\n  # Called when the action fails\n  on_fail {\n    output = \"fail\"\n  }\n}\n```\n\n## Template values\n### .Raw\nReturn raw binary data as an array of bytes from the message\n\n### .JSON\nIf the message type is application/json return an object which allows access to elements\ni.e.   \nGiven:  \n```json\n{\n  \"Pets\": [\n    {\"name\": \"fido\"}\n  ]\n}\n```\n\nThen:  \n```\n  {{ .JSON.Pets[0].name }} // fido\n```\n\nNote .JSON does not convert the output to JSON format, writing the direct output of .JSON.Pets would produce a go formatted\nobject.  To output json see the template function `tojson`.\n\n## Template functions\n### base64encode\nBase64 encode []byte\n\n```yaml\ninput_template: |\n{\n  \"image\": \"{{ base64encode .Raw }}\"\n}\n```\n\n### base64decode\nBase64 decode a string\n\n```yaml\ninput_template: |\n  {{ base64decode .JSON.Image }}\n```\n\n### tojson\nConvert to valid json\n```yaml\ninput_template: |\n  {{ tojson .JSON.Pets }}\n```\n\n## Metrics\nMetrics are exported using StatsD to import metrics into Prometheus please use the prometheus StatsD exporter [https://hub.docker.com/r/prom/statsd-exporter/](https://hub.docker.com/r/prom/statsd-exporter/)\n\n## Running the queue\nTo run the listener you can use the build docker container and provide a configuration file as a volume mount.\n\n```bash\ndocker run -it \\\n  -v $(shell pwd)/examples:/etc/config \\\n  quay.io/nicholasjackson/faas-nats:latest \\\n  -config /etc/config/examples\n```\n\n## Testing\nThere is a simple test harness in ./testharness/main.go which can be used to validate the subscription and transformations.\n\n## TODO\n[x] Implement monitoring and metrics with StatsD  \n[ ] Finish documentation  \n[ ] Write more examples  \n[ ] Finish basic provider implementation  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnicholasjackson%2Fpipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnicholasjackson%2Fpipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnicholasjackson%2Fpipe/lists"}