{"id":13582384,"url":"https://github.com/youzan/nsq","last_synced_at":"2025-04-06T14:30:55.943Z","repository":{"id":24835129,"uuid":"102559161","full_name":"youzan/nsq","owner":"youzan","description":"A realtime distributed messaging platform (forked from https://github.com/nsqio/nsq)","archived":false,"fork":false,"pushed_at":"2024-09-09T03:41:38.000Z","size":27200,"stargazers_count":636,"open_issues_count":35,"forks_count":106,"subscribers_count":48,"default_branch":"master","last_synced_at":"2024-11-05T22:41:11.320Z","etag":null,"topics":["distributed-systems","message-queue","messaging","queue"],"latest_commit_sha":null,"homepage":"http://nsq.io/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/youzan.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-09-06T03:39:25.000Z","updated_at":"2024-10-30T06:09:08.000Z","dependencies_parsed_at":"2024-01-14T08:02:44.690Z","dependency_job_id":"5f535552-7b95-428c-a8b8-0d4d631ddc5d","html_url":"https://github.com/youzan/nsq","commit_stats":{"total_commits":2964,"total_committers":79,"mean_commits":37.51898734177215,"dds":0.5398110661268556,"last_synced_commit":"2ecb3f9fda8b4bef4ddbcb7976704a8a29d91d05"},"previous_names":[],"tags_count":74,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/youzan%2Fnsq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/youzan%2Fnsq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/youzan%2Fnsq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/youzan%2Fnsq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/youzan","download_url":"https://codeload.github.com/youzan/nsq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247495733,"owners_count":20948101,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-systems","message-queue","messaging","queue"],"created_at":"2024-08-01T15:02:39.914Z","updated_at":"2025-04-06T14:30:54.920Z","avatar_url":"https://github.com/youzan.png","language":"Go","funding_links":[],"categories":["Go","Message Queue Middlewares"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n\u003cimg align=\"left\" width=\"175\" src=\"http://nsq.io/static/img/nsq_blue.png\"\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eSource\u003c/strong\u003e: https://github.com/youzan/nsq\n\u003cli\u003e\u003cstrong\u003eIssues\u003c/strong\u003e: https://github.com/youzan/nsq/issues\n\u003cli\u003e\u003cstrong\u003eMailing List\u003c/strong\u003e: \u003ca href=\"https://groups.google.com/d/forum/nsq-users\"\u003ensq-users@googlegroups.com\u003c/a\u003e\n\u003cli\u003e\u003cstrong\u003eIRC\u003c/strong\u003e: #nsq on freenode\n\u003cli\u003e\u003cstrong\u003eDocs\u003c/strong\u003e: http://nsq.io\n\u003cli\u003e\u003cstrong\u003eTwitter\u003c/strong\u003e: \u003ca href=\"https://twitter.com/nsqio\"\u003e@nsqio\u003c/a\u003e, \u003ca href=\"https://twitter.com/absolute8511\"\u003e@absolute8511\u003c/a\u003e\n\u003c/ul\u003e\n\u003c/p\u003e\n\n![Go](https://github.com/youzan/nsq/workflows/Go/badge.svg)\n[![Build Status](https://travis-ci.com/youzan/nsq.svg?branch=master)](https://travis-ci.com/youzan/nsq) [![GitHub release](https://img.shields.io/github/release/youzan/nsq.svg)](https://github.com/youzan/nsq/releases/latest)\n[![Go Report Card](https://goreportcard.com/badge/github.com/youzan/nsq)](https://goreportcard.com/report/github.com/youzan/nsq)\n[![codecov](https://codecov.io/gh/youzan/nsq/branch/master/graph/badge.svg)](https://codecov.io/gh/youzan/nsq)\n\n**NSQ** is a realtime distributed messaging platform designed to operate at scale, handling\nbillions of messages per day.\n\nIt promotes *distributed* and *decentralized* topologies without single points of failure,\nenabling fault tolerance and high availability coupled with a reliable message delivery\nguarantee.  See [features \u0026 guarantees][features_guarantees].\n\nOperationally, **NSQ** is easy to configure and deploy (all parameters are specified on the command\nline and compiled binaries have no runtime dependencies). For maximum flexibility, it is agnostic to\ndata format (messages can be JSON, MsgPack, Protocol Buffers, or anything else). Official Go and\nPython libraries are available out of the box (as well as many other [client\nlibraries][client_libraries]) and, if you're interested in building your own, there's a [protocol\nspec][protocol].\n\nWe publish [binary releases][installing] for linux and darwin.\n\nNOTE: master is our *development* branch and may not be stable at all times.\n\n## About this fork\n\n**This fork has been moved to https://github.com/youzan/nsq**\n\nThis fork add many features missing in the original and has been used in Youzan.com.\nFeatures:\n* Replication\n* HA\n* Auto balance and migrate\n* Partition\n* Delivery in Order\n* Consume history data\n* Trace the message life-cycle\n* Performance improvement\n\n## Deploy Dependency\nThis fork need the etcd cluster, so to deploy we need set the configure for the nsqd and nsqlookupd. \nFor details please refer to the sample configure files under contrib directory.\n\nOther deploy can be the same as the Official.\n\n## 有赞NSQ文档索引\n\n### 用户指南\n\n[有赞NSQ用户指南](doc/NSQ-user-guide.md)\n\n[有赞NSQ部署运维指南](doc/nsq-operation-reference.md)\n\n[JavaSDK使用指南](https://github.com/youzan/nsqJavaSDK/wiki/Get-started)\n\n### 内部实现文档\n\n[重构概览](doc/redesign-overview.md)\n\n[重构实现细节](doc/The-details-of-the-new-arch.md)\n\n[客户端SDK实现](doc/how-we-redesign-the-nsq-smart-client.md)\n\n[扩展消息设计](doc/Message-Extend-Header-Design.md)\n\n## FAQ for this Fork\n\n### Can I use the Official client SDK?\n\nMost client SDK can be used to communication with the new server without using the new features and for new features in this \nfork the new client SDK should be used.\n\nGolang: \u003ca href=\"https://github.com/youzan/go-nsq\" \u003ehttps://github.com/youzan/go-nsq\u003c/a\u003e\n\nJava: \u003ca href=\"https://github.com/youzan/nsqJavaSDK\"\u003ehttps://github.com/youzan/nsqJavaSDK\u003c/a\u003e\n\nPHP: \u003ca href=\"https://github.com/youzan/php-nsq-client\"\u003ehttps://github.com/youzan/php-nsq-client\u003c/a\u003e\n\n### Auto topic create has been disabled.\n\nIn order to manager the topics by nsqlookupd, we removed the create api on the nsqd. So in order to create a new topic,\nyou need create the topic in the nsqadmin or\nsend the create api to the nsqlookupd as below:\n\u003cpre\u003e\ncurl -X POST \"http://127.0.0.1:4161/topic/create?topic=xxx\u0026partition_num=2\u0026replicator=3\u0026syncdisk=1000\"\n\nExplain:\npartition_num=2 : topic partition number\nreplicator=3: topic replica number\nsyncdisk=1000: fsync disk every syncdisk messages\nfor ordered topic the query param \"orderedmulti=true\" is needed.\n\u003c/pre\u003e\nThe new server introduced the partitions but in order to keep compatible with the old clients, each node can hold\nonly one partition of the topic. So the partition_num*replicator should not be great than the nsqd nodes.\nFor the ordered topic, we can create as much as we need partitions. (replicator \u003c= number of nsqd node)\nAlso, the first channel for this topic should be created after the topic is created, otherwise this topic is not \nallowed to be written.\n\n### The log can be adjust dynamically.\n\nThis is useful to debug on the fly. \n\u003cpre\u003e\nnsqd: curl -X POST \"http://127.0.0.1:4151/loglevel/set?loglevel=3\"\nnsqlookupd: curl -X POST \"http://127.0.0.1:4161/loglevel/set?loglevel=3\"\n\u003c/pre\u003e\nThe log is much more detail when the log level increased.\n\n### Change the consume position to allow consume history or skip some messages.\n\n\u003cpre\u003e\ncurl -X POST -d \"timestamp:xxxxxxxxx\" \"http://127.0.0.1:4151/channel/setoffset?topic=xxx\u0026partition=xx\u0026channel=xxx\"\n\nPost body explain:\ntimestamp:xxxx (the seconds since 1970-1-1)\nvirtual_queue:xxx (the queue offset position from the begin)\nmsgcount:xxx (the message count from the begin)\n\u003c/pre\u003e\nFor multi nsqd nodes, each node for each partition should be done using the api above.\n\n### For more documents\n\nplease see the other documents under the doc \n\n## In Production\n\n\u003ca href=\"http://bitly.com\"\u003e\u003cimg src=\"http://nsq.io/static/img/bitly_logo.png\" width=\"84\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"https://www.life360.com/\"\u003e\u003cimg src=\"http://nsq.io/static/img/life360_logo.png\" width=\"100\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"https://www.hailoapp.com/\"\u003e\u003cimg src=\"http://nsq.io/static/img/hailo_logo.png\" width=\"62\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"http://www.simplereach.com/\"\u003e\u003cimg src=\"http://nsq.io/static/img/simplereach_logo.png\" width=\"136\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"https://moz.com/\"\u003e\u003cimg src=\"http://nsq.io/static/img/moz_logo.png\" width=\"108\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"https://path.com/\"\u003e\u003cimg src=\"http://nsq.io/static/img/path_logo.png\" width=\"84\" align=\"middle\"/\u003e\u003c/a\u003e\u003cbr/\u003e\n\n\u003ca href=\"https://segment.com/\"\u003e\u003cimg src=\"http://nsq.io/static/img/segmentio_logo.png\" width=\"70\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"http://eventful.com/events\"\u003e\u003cimg src=\"http://nsq.io/static/img/eventful_logo.png\" width=\"95\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"http://www.energyhub.com\"\u003e\u003cimg src=\"http://nsq.io/static/img/energyhub_logo.png\" width=\"99\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"https://project-fifo.net\"\u003e\u003cimg src=\"http://nsq.io/static/img/project_fifo.png\" width=\"97\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"http://trendrr.com\"\u003e\u003cimg src=\"http://nsq.io/static/img/trendrr_logo.png\" width=\"97\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"https://reonomy.com/\"\u003e\u003cimg src=\"http://nsq.io/static/img/reonomy_logo.png\" width=\"100\" align=\"middle\"/\u003e\u003c/a\u003e\u003cbr/\u003e\n\n\u003ca href=\"http://hw-ops.com\"\u003e\u003cimg src=\"http://nsq.io/static/img/heavy_water.png\" width=\"50\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"http://www.getlytics.com/\"\u003e\u003cimg src=\"http://nsq.io/static/img/lytics.png\" width=\"100\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"http://mediaforge.com\"\u003e\u003cimg src=\"http://nsq.io/static/img/rakuten.png\" width=\"100\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"http://socialradar.com\"\u003e\u003cimg src=\"http://nsq.io/static/img/socialradar_logo.png\" width=\"100\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"http://wistia.com\"\u003e\u003cimg src=\"http://nsq.io/static/img/wistia_logo.png\" width=\"140\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"https://stripe.com/\"\u003e\u003cimg src=\"http://nsq.io/static/img/stripe_logo.png\" width=\"96\" align=\"middle\"/\u003e\u003c/a\u003e\u003cbr/\u003e\n\n\u003ca href=\"https://www.soundest.com/\"\u003e\u003cimg src=\"http://nsq.io/static/img/soundest_logo.png\" width=\"96\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"https://www.docker.com/\"\u003e\u003cimg src=\"http://nsq.io/static/img/docker_logo.png\" width=\"100\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"http://www.getweave.com/\"\u003e\u003cimg src=\"http://nsq.io/static/img/weave_logo.png\" width=\"94\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"http://www.shipwire.com\"\u003e\u003cimg src=\"http://nsq.io/static/img/shipwire_logo.png\" width=\"97\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"http://digg.com\"\u003e\u003cimg src=\"http://nsq.io/static/img/digg_logo.png\" width=\"97\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"http://www.scalabull.com/\"\u003e\u003cimg src=\"http://nsq.io/static/img/scalabull_logo.png\" width=\"97\" align=\"middle\"/\u003e\u003c/a\u003e\u003cbr/\u003e\n\n\u003ca href=\"http://www.augury.com/\"\u003e\u003cimg src=\"http://nsq.io/static/img/augury_logo.png\" width=\"97\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"http://www.buzzfeed.com/\"\u003e\u003cimg src=\"http://nsq.io/static/img/buzzfeed_logo.png\" width=\"97\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"http://eztable.com\"\u003e\u003cimg src=\"http://nsq.io/static/img/eztable_logo.png\" width=\"97\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"http://www.dotabuff.com/\"\u003e\u003cimg src=\"http://nsq.io/static/img/dotabuff_logo.png\" width=\"97\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"https://www.fastly.com/\"\u003e\u003cimg src=\"http://nsq.io/static/img/fastly_logo.png\" width=\"97\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\u003ca href=\"https://talky.io\"\u003e\u003cimg src=\"http://nsq.io/static/img/talky_logo.png\" width=\"97\" align=\"middle\"/\u003e\u003c/a\u003e\u003cbr/\u003e\n\n\u003ca href=\"https://groupme.com\"\u003e\u003cimg src=\"http://nsq.io/static/img/groupme_logo.png\" width=\"97\" align=\"middle\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n\n## Code of Conduct\n\nHelp us keep NSQ open and inclusive. Please read and follow our [Code of Conduct](CODE_OF_CONDUCT.md).\n\n## Authors\n\nNSQ was designed and developed by Matt Reiferson ([@imsnakes][snakes_twitter]) and Jehiah Czebotar\n([@jehiah][jehiah_twitter]) but wouldn't have been possible without the support of\n[bitly][bitly] and all our [contributors][contributors].\n\nLogo created by Wolasi Konu ([@kisalow][wolasi_twitter]).\n\n[protocol]: http://nsq.io/clients/tcp_protocol_spec.html\n[installing]: https://github.com/youzan/nsq/releases/latest\n[snakes_twitter]: https://twitter.com/imsnakes\n[jehiah_twitter]: https://twitter.com/jehiah\n[bitly]: https://bitly.com\n[features_guarantees]: http://nsq.io/overview/features_and_guarantees.html\n[contributors]: https://github.com/nsqio/nsq/graphs/contributors\n[client_libraries]: http://nsq.io/clients/client_libraries.html\n[wolasi_twitter]: https://twitter.com/kisalow\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyouzan%2Fnsq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyouzan%2Fnsq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyouzan%2Fnsq/lists"}