{"id":13826220,"url":"https://github.com/prdn/pigato","last_synced_at":"2025-04-13T05:07:43.904Z","repository":{"id":21709799,"uuid":"25031289","full_name":"prdn/pigato","owner":"prdn","description":"PIGATO - an high-performance  Node.js microservices framework","archived":false,"fork":false,"pushed_at":"2018-08-19T15:35:01.000Z","size":385,"stargazers_count":302,"open_issues_count":15,"forks_count":30,"subscribers_count":26,"default_branch":"master","last_synced_at":"2025-04-13T05:07:39.588Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://prdn.github.io/pigato/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/prdn.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-10-10T10:38:52.000Z","updated_at":"2024-12-07T18:14:09.000Z","dependencies_parsed_at":"2022-07-27T02:47:21.099Z","dependency_job_id":null,"html_url":"https://github.com/prdn/pigato","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prdn%2Fpigato","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prdn%2Fpigato/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prdn%2Fpigato/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prdn%2Fpigato/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/prdn","download_url":"https://codeload.github.com/prdn/pigato/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248665747,"owners_count":21142123,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T09:01:34.066Z","updated_at":"2025-04-13T05:07:43.877Z","avatar_url":"https://github.com/prdn.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"PIGATO\n========\n\nCheckout **[Grenache](https://github.com/bitfinexcom/grenache)** , a DHT based high-performance microservices framework for Node.js, Ruby and Go. Supports ZeroMQ and WebSocket transports.\n\n**PIGATO - an high-performance Node.js microservices framework based on ZeroMQ**\n\nPIGATO aims to offer an high-performance, reliable, scalable and extensible service-oriented framework supporting multiple programming languages: Node.js/Io.js and Ruby.\n\n[![Travis Build Status](https://travis-ci.org/prdn/pigato.svg?style=flat)](https://travis-ci.org/prdn/pigato)\n[![NPM version](http://img.shields.io/npm/v/pigato.svg?style=flat)](https://www.npmjs.com/package/pigato)\n\n**Supported Programming Languages**\n\n* [PIGATO](https://github.com/prdn/pigato) : PIGATO Client/Worker/Broker for Node.js / Io.js\n* [PIGATO-RUBY](https://github.com/prdn/pigato-ruby) : PIGATO Client/Worker for Ruby\n* [PIGATO-GO](https://github.com/prdn/pigato-go) : PIGATO Client for Go\n\n\n## Structure and Protocol\n\n### Actors\n* Worker : receives requests, does something and replies. A Worker offers a Service, should be a functionality as atomic as possible\n* Client : creates, pushes Requests and waits for results. A request always includes a service name and data for the Worker\n* Broker : handles Requests queueing and routing\n\n### Benefits\n* High-performance\n* Realiable, Distributed and Scalable\n* Load Balancing\n* No central point of failure\n* Multi-Worker : infinite Services and infinite Workers for each Service\n* Multi-Client : infinite Clients\n* Multi-Broker : infinite Brokers to avoid bottlenecks and improve network reliability\n\n### Features\n* Request / Reply protocol\n* Support for partial Replies\n* Client concurrent Requests\n* Client streaming Requests\n* Worker concurrent Requests\n* Worker dynamic load balancing\n* Client heartbeating for long running requests. Allows Workers to dected whenever Clients disconnect or lose interest in some request. This feature is very useful to stop long-running partial requests (i.e data streaming).\n\n## Examples\n\nStart a **Broker**\n```\nnode examples/broker\n```\n\n1) **echo** : simple echo request-reply\n```\nnode examples/echo/worker\nnode examples/echo/client\n```\n\n2) **stocks** : get stocks data from yahoo\n```\nnode examples/stocks/worker\nnode examples/stocks/client\n```\n\n\nMore examples\n\n[PIGATO-EXAMPLES](https://github.com/fincluster/pigato-examples) : a collection of multi-purpose useful examples.\n\n### Performance\n\n[PIGATO-PERF](https://github.com/prdn/pigato-perf) : a command-line tool to test PIGATO performances in different scenarios.\ni\n## API\n\n### Broker\n#### `PIGATO.Broker(addr, conf)`\n* `addr` - Broker address (string, i.e: 'tcp://*:12345') \n* `conf` - configuration override (type=object, i.e { concurrency: 20 })\n  * `onStart`: function to be called when the Broker start \n  * `onStop`: function to be called when the Broker stop\n\nSimply starts up a broker.\n\n```\nvar Broker = require('./../index').Broker;\n\nvar broker = new Broker(\"tcp://*:55555\");\nbroker.start(function() {\n  console.log(\"Broker started\");\n});\n```\n\n#### Events\n* `start` : on Client start\n* `stop` : on Client stop\n\n### Worker\n#### `PIGATO.Worker(addr, serviceName, conf)`\n* `addr` - Broker address (type=string, i.e: 'tcp://localhost:12345') \n* `serviceName` - service implemented by the Worker (type=string, i.e: 'echo')\n  * wildcards * are supported (i.e: 'ech*')\n* `conf` - configuration override (type=object, i.e { concurrency: 20 })\n  * `prefix` - sets the Worker identifier prefix\n  * `concurrency` - sets max number of concurrent requests (type=int, -1 = no limit)\n  * `onConnect`: function to be called when the Worker connects to the Broker\n  * `onDisconnnect`: function to be called when the Worker disconnects from the Broker\n\n#### Methods\n\n##### `on`\nWorker receives `request` events with 3 arguments:\n* `data` - data sent from the Client (type=string/object/array).\n* `reply` - extended writable stream (type=object)\n* `opts` - client request options (type=object)\n\n`reply` writable stream exposes also following methods and attributes:\n\n* `write()` - sends partial data to the Client \n* `end()` - sends last data to the Client and completes/closes current Request\n* `reject()` - rejects a Request.\n* `heartbeat()` - forces sending heartbeat to the Broker\n* `active()` - returns the status of the Request (type=boolean). A Request becomes inactive when the Worker disconnects from the Broker or it has been discarded by the Client or the Client disconnects from the Broker. This is useful for long running tasks so the Worker can monitor whether or not continue processing a Request.\n* `ended` - tells if the Request has been ended (type=boolean).\n\n**Example**\n\n```\nvar worker = new PIGATO.Worker('tcp://localhost:12345', 'my-service');\nworker.start();\n\nworker.on('request', function(data, reply, copts) {\n  for (var i = 0; i \u003c 1000; i++) {\n    reply.write('PARTIAL DATA ' + i);\n  }\n  reply.end('FINAL DATA');\n});\n\n// or\nworker.on('request', function(data, reply, copts) {\n  fs.createReadStream(data).pipe(reply);\n});\n```\n\nWorker can change concurrency level updating its configuration. This information is carried with the heartbeat message.\n\n**Example**\n\n```\nworker.conf.concurrency = 2;\n```\n\nTake note: due to the framing protocol of `zmq` only the data supplied to `response.end(data)` will be given to the client's final callback.\n\n#### Events\n* `start` : on Worker start\n* `stop` : on Worker stop\n* `connect` : on Worker connect\n* `disconnect` : on Worker disconnect\n\n\n### Client\n#### `PIGATO.Client(addr, conf)`\n* `addr` - Broker address (type=string, i.e: 'tcp://localhost:12345') \n* `conf`\n  * `prefix` - sets the Client identifier prefix\n  * `autostart`: automatically starts the Client (type=boolean, default=false)\n  * `onConnect`: function to be called when the Client connects to the Broker\n  * `onDisconnnect`: function to be called when the Client disconnects from the Broker\n\n#### Methods\n\n##### `start`\n\nStart the Client\n\n##### `request`\n\nSend a Request\n\n* `serviceName` - name of the Service we wish to connect to (type=string)\n* `data` - data to give to the Service (type=string/object/buffer)\n* `opts` - options for the Request (type=object)\n  * `timeout`: timeout in milliseconds (type=number, default=60000, -1 for infinite timeout)\n  * `retry`: if a Worker dies before replying, the Request is automatically requeued. (type=number, values=0|1, default=0)\n  * `nocache`: skip Broker's cache\n  * `workerId`: ID of the Worker that must handle the Request (type=string)\n\n\n**Example**\n\n```\nvar client = new PIGATO.Client('tcp://localhost:12345');\nclient.start()\n\nclient.request('my-service', { foo: 'bar' }, { timeout: 120000 })\n.on('data', function(data) {\n  console.log(\"DATA\", data);\t\n})\n.on('end', function() {\n  console.log(\"END\");\t  \n});\n\n// or\nclient.request('my-service', 'foo', { timeout: 120000 }).pipe(process.stdout);\n```\n\nClients may also make request with partial and final callbacks instead of using streams.\n\n* `serviceName`\n* `data`\n* `partialCallback(err, data)` - called whenever the request does not end but emits data\n* `finalCallback(err, data)` - called when the request will emit no more data\n* `opts`\n\n**Example**\n\n```\nclient.request('my-service', 'foo', function (err, data) {\n  // frames sent prior to final frame\n  console.log('PARTIAL', data);\n}, function (err, data) {\n  // this is the final frame sent\n  console.log('FINAL', data);\n}, { timeout: 30000 });\n\n```\n\n#### Events\n* `start` : on Client start\n* `stop` : on Client stop\n* `connect` : on Client connect\n* `disconnect` : on Client disconnect\n\n### Core Services\nCore services are a set of Services that interact with a Broker via a dedicated PUB/SUB channel to extend its core functionalities.\n\n#### Initialization\n\n```\nvar broker = new PIGATO.Broker(bhost);\nvar csrv = new PIGATO.services.ExampleCoreService(bhost, {\n  intch: broker.conf.intch // internal pub/sub channel                  \n});\nbroker.start();\ncsrv.start();\n```\n\n#### Directory\n##### `PIGATO.services.Directory`\n\nDirectory service ($dir) replies to Requests with the list of available Workers for a selected service. \n\n**Example**\n\n```\n// Broadcast a message to all Workers that offer 'echo' Service\nclient.request(\n  '$dir', 'echo', undefined, \n  function(err, workers) {\n    workers.forEach(function(wid) {\n      client.request('echo', 'foo', { workerId: wid });\n    });\n  }\n);\n```\n\n### Notes\n* when using a `inproc` socket the broker *must* become active before any queued messages.\n\n## Specification (good for RFC)\n* Worker \u003c-\u003e Broker heartbeating.\n* Broker tracks Worker/Client/Request relation.\n* Client MAY send heartbeat for active request. If the request is being processed by Worker, Broker forwards heartbeat to Worker. \n* Worker MAY decide to stop an inactive Request (tracks liveness for Request).\n* Client MAY assign a timeout to a Request.\n* Worker SHALL NOT send more W_REPLY (for a Request) after sending first W_REPLY message.\n* Broker SHALL force disconnect Worker if any error occurs.\n\n## Protocol\n\n#### Common\n* Frame 0: Side tag (MDP.CLIENT/MDP.WORKER)\n* Frame 1: Message type (MDP.W_REQUEST, MDP.W_REPLY, MDP.W_REPLY_REJECT, ...)\n* Frame 2: Service name\n* Frame 3: Request ID (uuid)\n\n#### Client request\n* Frame 4: JSON encode request data\n* Frame 5: JSON encode request options\n\n#### Worker reply\n* Frame 4: Numeric status (0=OK, -1=ERROR)\n* Frame 5: JSON encode request data / error\n* Frame 6: JSON encode request options\n\n## Changelog\n\n[CHANGELOG](CHANGELOG.md)\n\n## Roadmap\n* Add authentication support through [zmq-zap](https://github.com/msealand/zmq-zap.node) ZeroMQ ZAP to trust Clients and Workers.\n\n## Follow me\n\n* My personal blog : [ardoino.com](http://ardoino.com) / [@paoloardoino](https://twitter.com/paoloardoino)\n\n## Contributors\n* [bmeck](https://github.com/bmeck)\n* [maxired](https://github.com/maxired)\n* [leonpegg](https://github.com/leonpegg)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprdn%2Fpigato","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprdn%2Fpigato","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprdn%2Fpigato/lists"}