{"id":17084357,"url":"https://github.com/xiaokai-wang/nginx-stream-upsync-module","last_synced_at":"2025-05-12T14:53:12.858Z","repository":{"id":74543866,"uuid":"55300560","full_name":"xiaokai-wang/nginx-stream-upsync-module","owner":"xiaokai-wang","description":"For stream protocol. syncing upstreams from etcd or consul and so on, dynamically updating backend servers attribute, not need to reload nginx for tcp protocol, stream configure.","archived":false,"fork":false,"pushed_at":"2020-01-02T18:05:45.000Z","size":159,"stargazers_count":176,"open_issues_count":8,"forks_count":49,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-03-31T23:29:20.620Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xiaokai-wang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-04-02T15:04:13.000Z","updated_at":"2025-03-11T02:09:15.000Z","dependencies_parsed_at":"2023-07-21T09:30:28.803Z","dependency_job_id":null,"html_url":"https://github.com/xiaokai-wang/nginx-stream-upsync-module","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiaokai-wang%2Fnginx-stream-upsync-module","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiaokai-wang%2Fnginx-stream-upsync-module/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiaokai-wang%2Fnginx-stream-upsync-module/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiaokai-wang%2Fnginx-stream-upsync-module/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xiaokai-wang","download_url":"https://codeload.github.com/xiaokai-wang/nginx-stream-upsync-module/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253759193,"owners_count":21959715,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-14T13:06:53.553Z","updated_at":"2025-05-12T14:53:12.816Z","avatar_url":"https://github.com/xiaokai-wang.png","language":"C","funding_links":[],"categories":["Upstreams, rate limiting and access control"],"sub_categories":[],"readme":"Name\n====\n\nnginx-stream-upsync-module - Nginx C module, sync upstreams from consul or others, dynamically modify backend-servers attribute(weight, max_fails,...), needn't reload nginx.\n\nIt may not always be convenient to modify configuration files and restart NGINX. For example, if you are experiencing large amounts of traffic and high load, restarting NGINX and reloading the configuration at that point further increases load on the system and can temporarily degrade performance.\n\nThe module can be more smoothly expansion and constriction, and will not influence the performance.\n\nAnother module, [nginx-upsync-module](https://github.com/weibocom/nginx-upsync-module) supports nginx http module(HTTP protocol), please be noticed.\n\nIf you want to use [nginx-upsync-module](https://github.com/weibocom/nginx-upsync-module) and [nginx-stream-upsync-module](https://github.com/xiaokai-wang/nginx-stream-upsync-module) both, please refer to [nginx-upsync](https://github.com/CallMeFoxie/nginx-upsync).\n\nTable of Contents\n=================\n\n* [Name](#name)\n* [Status](#status)\n* [Synopsis](#synopsis)\n* [Description](#description)\n* [Directives](#directives)\n    * [upsync](#upsync)\n        * [upsync_interval](#upsync_interval)\n        * [upsync_timeout](#upsync_timeout)\n        * [upsync_type](#upsync_type)\n        * [strong_dependency](#strong_dependency)\n    * [upsync_dump_path](#upsync_dump_path)\n    * [upsync_lb](#upsync_lb)\n    * [upstream_show](#upstream_show)\n* [Consul_interface](#consul_interface)\n* [Etcd_interface](#etcd_interface)\n* [TODO](#todo)\n* [Compatibility](#compatibility)\n* [Installation](#installation)\n* [Code style](#code-style)\n* [Author](#author)\n* [Copyright and License](#copyright-and-license)\n* [See Also](#see-also)\n* [Source Dependency](#source-dependency)\n\nStatus\n======\n\nThis module is still under active development and is considered production ready.\n\nSynopsis\n========\n\nnginx-consul:\n```nginx-consul\nstream {\n    upstream test {\n        upsync 127.0.0.1:8500/v1/kv/upstreams/test/ upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;\n        upsync_dump_path /usr/local/nginx/conf/servers/servers_test.conf;\n\n        include /usr/local/nginx/conf/servers/servers_test.conf;\n    }\n\n    upstream bar {\n        server 127.0.0.1:8090 weight=1 fail_timeout=10 max_fails=3;\n    }\n\n    server {\n        listen 12345;\n\n        proxy_connect_timeout 1s;\n        proxy_timeout 3s;\n        proxy_pass test;\n    }\n\n    server {\n        listen 2345;\n\n        upstream_show\n    }\n\n    server {\n        listen 127.0.0.1:9091;\n\n        proxy_responses 1;\n        proxy_timeout 20s;\n        proxy_pass bar;\n    }\n}\n```\nnginx-etcd:\n```nginx-etcd\nstream {\n    upstream test {\n        upsync 127.0.0.1:2379/v2/keys/upstreams/test upsync_timeout=6m upsync_interval=500ms upsync_type=etcd strong_dependency=off;\n        upsync_dump_path /usr/local/nginx/conf/servers/servers_test.conf;\n\n        include /usr/local/nginx/conf/servers/servers_test.conf;\n    }\n\n    upstream bar {\n        server 127.0.0.1:8090 weight=1 fail_timeout=10 max_fails=3;\n    }\n\n    server {\n        listen 12345;\n\n        proxy_connect_timeout 1s;\n        proxy_timeout 3s;\n        proxy_pass test;\n    }\n\n    server {\n        listen 2345;\n\n        upstream_show\n    }\n\n    server {\n        listen 127.0.0.1:9091;\n\n        proxy_responses 1;\n        proxy_timeout 20s;\n        proxy_pass bar;\n    }\n}\n```\nupsync_lb:\n```upsync_lb\nstream {\n    upstream test {\n        least_conn; //hash $uri consistent;\n\n        upsync 127.0.0.1:8500/v1/kv/upstreams/test/ upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;\n        upsync_dump_path /usr/local/nginx/conf/servers/servers_test.conf;\n        upsync_lb least_conn; //hash_ketama;\n\n        include /usr/local/nginx/conf/servers/servers_test.conf;\n    }\n\n    upstream bar {\n        server 127.0.0.1:8090 weight=1 fail_timeout=10 max_fails=3;\n    }\n\n    server {\n        listen 12345;\n\n        proxy_connect_timeout 1s;\n        proxy_timeout 3s;\n        proxy_pass test;\n    }\n\n    server {\n        listen 2345;\n\n        upstream_show\n    }\n\n    server {\n        listen 127.0.0.1:9091;\n\n        proxy_responses 1;\n        proxy_timeout 20s;\n        proxy_pass bar;\n    }\n}\n```\n\nNOTE: upstream: include command is neccesary, first time the dumped file should include all the servers.\n\n[Back to TOC](#table-of-contents)       \n\nDescription\n======\n\nThis module provides a method to discover backend servers. Supporting dynamicly adding or deleting backend server through consul/etcd and dynamicly adjusting backend servers weight, module will timely pull new backend server list from consul/etcd to upsync nginx ip router. Nginx needn't reload. Having some advantages than others:\n\n* timely\n\n      module send key to consul/etcd with index, consul/etcd will compare it with its index, if index doesn't change connection will hang five minutes, in the period any operation to the key-value, will feed back rightaway.\n\n* performance\n\n      Pulling from consul/etcd equal a request to nginx, updating ip router nginx needn't reload, so affecting nginx performance is little.\n\n* stability\n\n      Even if one pulling failed, it will pull next upsync_interval, so guaranteing backend server stably provides service. And support dumping the latest config to location, so even if consul/etcd hung up, and nginx can be reload anytime. \n\n[Back to TOC](#table-of-contents)       \n\nDirectives\n======\n\nupsync\n-----------\n```\nsyntax: upsync $consul/etcd.api.com:$port/v1/kv/upstreams/$upstream_name/ [upsync_type=consul/etcd] [upsync_interval=second/minutes] [upsync_timeout=second/minutes] [strong_dependency=off/on]\n```\ndefault: none, if parameters omitted, default parameters are upsync_interval=5s upsync_timeout=6m strong_dependency=off\n\ncontext: upstream\n\ndescription: Pull upstream servers from consul/etcd... .\n\nThe parameters' meanings are:\n\n* upsync_interval\n\n    pulling servers from consul/etcd interval time.\n\n* upsync_timeout\n\n    pulling servers from consul/etcd request timeout.\n\n* upsync_type\n\n    pulling servers from conf server type.\n\n* strong_dependency\n\n    when nginx start up if strong_dependency is on that means servers will be depended on consul/etcd and will pull servers from consul/etcd.\n\n\nupsync_dump_path\n-----------\n`syntax: upsync_dump_path $path`\n\ndefault: /tmp/servers_$host.conf\n\ncontext: upstream\n\ndescription: dump the upstream backends to the $path.\n\n\nupsync_lb\n-----------\n`syntax: upsync_lb $load_balance`\n\ndefault: round_robin/ip_hash/hash modula\n\ncontext: upstream\n\ndescription: mainly for least_conn and hash consistent, when using one of them, you must point out using upsync_lb.\n\n\nupsync_show\n-----------\n`syntax: upsync_show`\n\ndefault: none\n\ncontext: server\n\ndescription: show all upstreams.\n\n```request\ncurl http://localhost:2345/upstream_show\n\nshow all upstreams\n```\n\n[Back to TOC](#table-of-contents)       \n\nConsul_interface\n======\n\nData can be taken from key/value store or service catalog. In the first case parameter upsync_type of directive must be *consul*. For example\n \n```nginx-consul\n    upsync 127.0.0.1:8500/v1/kv/upstreams/test upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;\n```\n\nIn the second case it must be *consul_services*.\n\n```nginx-consul\n    upsync 127.0.0.1:8500/v1/catalog/service/test upsync_timeout=6m upsync_interval=500ms upsync_type=consul_services strong_dependency=off;\n```\n\nyou can add or delete backend server through consul_ui or http_interface. Below are examples for key/value store.\n\nhttp_interface example:\n\n* add\n```\n    curl -X PUT http://$consul_ip:$port/v1/kv/upstreams/$upstream_name/$backend_ip:$backend_port\n```\n    default: weight=1 max_fails=2 fail_timeout=10 down=0 backup=0;\n\n```\n    curl -X PUT -d \"{\\\"weight\\\":1, \\\"max_fails\\\":2, \\\"fail_timeout\\\":10}\" http://$consul_ip:$port/v1/kv/$dir1/$upstream_name/$backend_ip:$backend_port\nor\n    curl -X PUT -d '{\"weight\":1, \"max_fails\":2, \"fail_timeout\":10}' http://$consul_ip:$port/v1/kv/$dir1/$upstream_name/$backend_ip:$backend_port\n```\n    value support json format.\n\n* delete\n```\n    curl -X DELETE http://$consul_ip:$port/v1/kv/upstreams/$upstream_name/$backend_ip:$backend_port\n```\n\n* adjust-weight\n```\n    curl -X PUT -d \"{\\\"weight\\\":2, \\\"max_fails\\\":2, \\\"fail_timeout\\\":10}\" http://$consul_ip:$port/v1/kv/$dir1/$upstream_name/$backend_ip:$backend_port\nor\n    curl -X PUT -d '{\"weight\":2, \"max_fails\":2, \"fail_timeout\":10}' http://$consul_ip:$port/v1/kv/$dir1/$upstream_name/$backend_ip:$backend_port\n```\n\n* mark server-down\n```\n    curl -X PUT -d \"{\\\"weight\\\":2, \\\"max_fails\\\":2, \\\"fail_timeout\\\":10, \\\"down\\\":1}\" http://$consul_ip:$port/v1/kv/$dir1/$upstream_name/$backend_ip:$backend_port\nor\n    curl -X PUT -d '{\"weight\":2, \"max_fails\":2, \"fail_timeout\":10, \"down\":1}' http://$consul_ip:$port/v1/kv/$dir1/$upstream_name/$backend_ip:$backend_port\n```\n\n* check\n```\n    curl http://$consul_ip:$port/v1/kv/upstreams/$upstream_name?recurse\n```\n\n[Back to TOC](#table-of-contents)       \n\nEtcd_interface\n======\n\nyou can add or delete backend server through http_interface.\n\nmainly like etcd, http_interface example:\n\n* add\n```\n    curl -X PUT http://$etcd_ip:$port/v2/keys/upstreams/$upstream_name/$backend_ip:$backend_port\n```\n    default: weight=1 max_fails=2 fail_timeout=10 down=0 backup=0;\n\n```\n    curl -X PUT -d value=\"{\\\"weight\\\":1, \\\"max_fails\\\":2, \\\"fail_timeout\\\":10}\" http://$etcd_ip:$port/v2/keys/$dir1/$upstream_name/$backend_ip:$backend_port\n```\n    value support json format.\n\n* delete\n```\n    curl -X DELETE http://$etcd_ip:$port/v2/keys/upstreams/$upstream_name/$backend_ip:$backend_port\n```\n\n* adjust-weight\n```\n    curl -X PUT -d \"{\\\"weight\\\":2, \\\"max_fails\\\":2, \\\"fail_timeout\\\":10}\" http://$etcd_ip:$port/v2/keys/$dir1/$upstream_name/$backend_ip:$backend_port\n```\n\n* mark server-down\n```\n    curl -X PUT -d value=\"{\\\"weight\\\":2, \\\"max_fails\\\":2, \\\"fail_timeout\\\":10, \\\"down\\\":1}\" http://$etcd_ip:$port/v2/keys/$dir1/$upstream_name/$backend_ip:$backend_port\n```\n\n* check\n```\n    curl http://$etcd_ip:$port/v2/keys/upstreams/$upstream_name\n```\n\n[Back to TOC](#table-of-contents)       \n\nTODO\n====\n\n* support zookeeper and so on\n\n[Back to TOC](#table-of-contents)\n\nCompatibility\n=============\n\nThe module was developed base on nginx-1.9.10.\n\nMaster branch compatible with nginx-1.11.0+.\n\nNginx-1.10.3- branch compatible with nginx-1.9.10 ~ nginx-1.11.0+.\n\n[Back to TOC](#table-of-contents)\n\nInstallation\n============\n\nThis module can be used independently, can be download[Github](https://github.com/xiaokai-wang/nginx-stream-upsync-module.git).\n\nGrab the nginx source code from [nginx.org](http://nginx.org/), for example, the version 1.8.0 (see nginx compatibility), and then build the source with this module:\n\n```bash\nwget 'http://nginx.org/download/nginx-1.8.0.tar.gz'\ntar -xzvf nginx-1.8.0.tar.gz\ncd nginx-1.8.0/\n```\n\n```bash\n./configure --add-module=/path/to/nginx-stream_upsync-module\nmake\nmake install\n```\n\nif you support nginx-upstream-check-module\n```bash\n./configure --add-module=/path/to/nginx-upstream-check-module --add-module=/path/to/nginx-stream_upsync-module\nmake\nmake install\n```\n\n[Back to TOC](#table-of-contents)\n\nCode style\n======\n\nCode style is mainly based on [style](http://tengine.taobao.org/book/appendix_a.html)\n\n[Back to TOC](#table-of-contents)\n\nAuthor\n======\n\nXiaokai Wang (王晓开) \u003cxiaokai.wang@live.com\u003e, Weibo Inc.\n\n[Back to TOC](#table-of-contents)\n\nCopyright and License\n=====================\n\nThis README template copy from agentzh.\n\nThis module is licensed under the BSD license.\n\nCopyright (C) 2014 by Xiaokai Wang \u003cxiaokai.wang@live.com\u003e\u003c/xiaokai.wang@live.com\u003e\n\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:\n\n* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.\n\n* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n[Back to TOC](#table-of-contents)\n\nsee also\n========\n* the nginx_upstream_check_module: https://github.com/alibaba/tengine/blob/master/src/http/ngx_http_upstream_check_module.c\n* the nginx_upstream_check_module patch: https://github.com/yaoweibin/nginx_upstream_check_module\n* or based on https://github.com/xiaokai-wang/nginx_upstream_check_module\n\n[back to toc](#table-of-contents)\n\nsource dependency\n========\n* Cjson: https://github.com/kbranigan/cJSON\n* http-parser: https://github.com/nodejs/http-parser\n\n[back to toc](#table-of-contents)\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxiaokai-wang%2Fnginx-stream-upsync-module","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxiaokai-wang%2Fnginx-stream-upsync-module","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxiaokai-wang%2Fnginx-stream-upsync-module/lists"}