{"id":19024153,"url":"https://github.com/raykitajima/bucket-relay","last_synced_at":"2026-06-18T06:32:00.478Z","repository":{"id":57190863,"uuid":"145276945","full_name":"RayKitajima/Bucket-Relay","owner":"RayKitajima","description":"Bucket brigade architecture to build a flexible micro services network as your streaming processor for unstructured data.","archived":false,"fork":false,"pushed_at":"2020-04-20T07:58:39.000Z","size":15,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-02T02:04:17.055Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RayKitajima.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-19T05:49:11.000Z","updated_at":"2020-04-20T07:57:50.000Z","dependencies_parsed_at":"2022-09-15T15:50:31.928Z","dependency_job_id":null,"html_url":"https://github.com/RayKitajima/Bucket-Relay","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RayKitajima%2FBucket-Relay","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RayKitajima%2FBucket-Relay/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RayKitajima%2FBucket-Relay/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RayKitajima%2FBucket-Relay/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RayKitajima","download_url":"https://codeload.github.com/RayKitajima/Bucket-Relay/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240072063,"owners_count":19743527,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T20:35:21.948Z","updated_at":"2026-04-29T17:30:15.879Z","avatar_url":"https://github.com/RayKitajima.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# bucket-relay\n\nBuket-Relay is a bucket brigade architecture to build a flexible micro services network as your streaming processor for unstructured data.\nEspacially for prosessor like for IoT solution, order processing, transportation tracking.\nThis is a stream engine for event object file across your micro services pool.\n\n\n## Architecture\n\n``` \n                          (network server)\n                                  ^\n                                  :\n           +. . . . . . . . . . . + . . . . . . . . . . . +\n           :                      :                       :\n           v                      v                       v\n    +----------+         +-----------+            +----------+\n    | uploader |-(http)-\u003e| collector |            | uploader |.....\u003e    \u003cobject streaming layer\u003e\n    +----------+         +-----------+            +----------+\n      ^                         |                   ^\n      |                         v                   |\n +      +                     +       +          +      +\n | next |                     | spool |          | next |\n +---+--+                     +----+--+          +-+----+\n     ^                             :               ^\n     : output file        use file :               : out used file\n     :                             :               :\n (service)                         +...(service)...+                    \u003cmaicro-services layer\u003e\n``` \n\n1. A service output a `file`.\n2. **uploader** send the `file` to **collector**s who like to consume the `file`. Those consumers would be found by **network server**.\n3. **collector** get the `file`, and stack it to the `spool`.\n4. A service use the `file`. And move it to the `next` place. (It may modify the file.)\n5. **uploader** send the `file` to ......\n\n* uploader represented by **bucket-uploader** command\n* collector represented by **bucket-collector** command\n* network server represented by **bucket-network** command\n* A servcie would be implemented by you, to solve your problem\n\n\n## Demo\n\n![demo](https://user-images.githubusercontent.com/4404088/44306219-51bc9b80-a3c5-11e8-8ce2-ddbe1100333d.gif)\n\n```\n                                    (visualizer)\n                                         :\n                                         :\n+--------+         +---------+     +-----+-----+     +---------+\n| sensor |--------\u003e| gateway |----\u003e| visualize |----\u003e| archive |\n+--------+         +---------+     +-----------+     +---------+\n                                                          :\n                                                          v\n                                                      (warehouse/YYYY-MM-DD)\n                           +- - - - -+\n                           : network :\n                           +- - - - -+\n```\n\n* **sensor** reports your os.loadavg() every sec.\n* **gateway** recieves loadavg report. And send it to **visualize**.\n* **visualize** caches latest 10sec report for web client (**visualizer**). And send it to **archive**.\n* **archive** stacks reports into `warehouse/YYYY-MM-DD`.\n* **network** provides structure of above uploader/collector nodes, and get heatbeat from them to automatically regress network structure against unexpected or expected service stop. _(the **network** is optional. you can use statically defined network structure by using second argument for uploader/collector command)_\n\nTo run demo, it is easy to use [quick-pm](https://github.com/RayKitajima/Quick-PM) a simple process manager:\n\n``` \n$ git clone https://github.com/RayKitajima/Bucket-Relay.git\n$ cd Bucket-Relay\n$ npm install quick-pm\n\n$ quick-pm demo/dyanmic.hjson *1\n$ open http://127.0.0.1:10092/web/index.html\n\n$ quick-pm demo/static.hjson *2\n``` \n\n*1 use dynamic network configuration  \n*2 use statically defined network\n\n\u003cdetails\u003e\n\u003csummary\u003eotherwise you can run it manually:\u003c/summary\u003e\n\n``` \n$ git clone https://github.com/RayKitajima/Bucket-Relay.git\n$ cd Bucket-Relay\n\n// network server\n$ node lib/bucket-network.js demo/network/dynamic.js\n\n// gateway : collecting the loadavg report\n$ node demo/gateway/service.js\n$ node lib/bucket-uploader.js demo/gateway/conf/uploader.js\n\n// visualize : realtime chart\n$ node demo/visualize/service.js\n$ node lib/bucket-collector.js demo/visualize/conf/collector.js\n$ node lib/bucket-uploader.js demo/visualize/conf/uploader.js\n\n// archive : report log\n$ node demo/archive/service.js\n$ node lib/bucket-collector.js demo/archive/conf/collector.js\n\n// sensor : report loadavg of your machine\n$ node demo/sensor/loadavg.js\n``` \n\n\u003c/details\u003e\n\n\n## Usage\n\n``` \n$ npm install bucket-relay\n\n$ bucket-network dynamic_network_config.js *1\n\n$ bucket-uploader uploader_config.js *2\n$ bucket-collector collector_config.js static_network_config.js *3\n``` \n\n*1 booting network server  \n*2 uploader working with network  \n*3 collector working with static network definition\n\nWhether to use dynamic or static network should be consensus of your micro-services network.\n\n\n## Advanced\n\nWriting, reading and sending file is high cost operation.  \nTo reduce I/O cost, you can use [bucket-maker](https://github.com/RayKitajima/Bucket-Maker) a time-slicing utility.\n\n``` \n// get 5sec(msec) bucket maker\nlet bucket = require('bucket-maker').createBucket({\n\tunit: 5000\n});\n\n// put loadavg() every sec\nsetInterval(function(){\n\tbucket.put(JSON.stringify(os.loadavg()));\n},1000);\n\n// will write out\n$ ls \n17764.4761.bucket  17764.4762.bucket\n``` \n\nsee more about: [bucket-maker](https://github.com/RayKitajima/Bucket-Maker)\n\n\n## Configuration\n\nConfiguration is defined as JavaScript file, that should honor some module.exports contract.\n\n### bucket-uploader\n\n| name      | what is                                                                     |\n|:----------|:----------------------------------------------------------------------------|\n| spool     | where uploaded file is saved in                                             |\n| max_conn  | max http thread                                                             |\n| cron_time | frequency of spool directory check, as cron expression: \"\\* \\* \\* \\* \\* \\*\" |\n| filter    | how to filter files from spool directory                                    |\n| nexthop   | how to define name of nexthop cluster for a file                            |\n| log       | logging function called by core module with an argument of string message   |\n| network   | optional: defines network server and heatbeat interval (as cron expression) |\n\n\u003cdetails\u003e\n\u003csummary\u003eexample:\u003c/summary\u003e\n\n``` \nmodule.exports = {\n\tspool     : '/path/to/spool',\n\tmax_conn  : 30,\n\tcron_time : '*/10 * * * * *',\n\tfilter    : function(files){ files.filter( v =\u003e v.match(/\\.json|\\.log$/) ); },\n\tnexthop   : function(f){ if( f.match(/\\.json$/) ){ return 'cluster1' }else{ return 'cluster2' } },\n\tlog       : function(msg){ console.log(msg); },\n\tnetwork   : { server:'127.0.0.1:10088', heatbeat:'*/30 * * * * *' },\n};\n``` \n\n\u003c/details\u003e\n\n### bucket-collector\n\n| name    | what is                                                                     |\n|:--------|:----------------------------------------------------------------------------|\n| id      | optional: if you use static network, define your unique id                  |\n| address | listen address                                                              |\n| port    | listen port                                                                 |\n| spool   | where to write out collected files                                          |\n| log     | logging function called by core module with an argument of string message   |\n| network | optional: defines network server and my cluster name, and heatbeat interval |\n\n\u003cdetails\u003e\n\u003csummary\u003eexample:\u003c/summary\u003e\n\n``` \nmodule.exports = {\n\tid      : 'json_consumer_1'\n\taddress : '127.0.0.1',\n\tport    : 10100,\n\tspool   : /path/to/spool,\n\tlog     : function(msg){ console.log(msg); },\n\tnetwork : { server:\"http://127.0.0.1:10088/\", cluster:'cluster1', heatbeat:\"*/10 * * * * *\" }\n};\n``` \n\n\u003c/details\u003e\n\n\n### bucket-network\n\n| name     | what is                                                  |\n|:---------|:---------------------------------------------------------|\n| server   | defines listen address and port                          | \n| clusters | start point of publishing port to be used by collectors  |\n\nnetwork server publishes port to collectors. That is simply incremented from clusters.port.start.\n\n\u003cdetails\u003e\n\u003csummary\u003eexample:\u003c/summary\u003e\n\n```\nmodule.exports = {\n\tserver   : { address:'127.0.0.1', port:10088 },\n\tclusters : { port:{start:10100} }\n};\n```\n\n\u003c/details\u003e\n\n### static network definition\n\nIf you want to work with statically defined network structure, you can do it by passing second argument to the uploader/collector command.\nThe configuration file also should have some module.exports contract.\n\n| name      | what is           |\n|:----------|:------------------|\n| clusters  | hash with\u003cbr\u003ekey: \\\u003ccluster name\\\u003e\u003cbr\u003evalue: list of collectors binded by the name |\n\ncollector info:\n\n| name      | sub-property      |\n|:----------|:------------------|\n| id        | unique id         |\n| address   | collector address |\n| port      | collector port    |\n\nexample:\n\n``` \nmodule.exports = {\n\tclusters : {\n\t\tcluster1 : [ { id:'json_consumer_1', address:'127.0.0.1', port:10100 } ],\n\t\tcluster2 : [ { id:'log_consumer_1', address:'127.0.0.1', port:10101 } ]\n\t}\n};\n``` \n\n## License\n\nMIT\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraykitajima%2Fbucket-relay","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fraykitajima%2Fbucket-relay","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraykitajima%2Fbucket-relay/lists"}