Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/youzan/nsq

A realtime distributed messaging platform (forked from https://github.com/nsqio/nsq)
https://github.com/youzan/nsq

distributed-systems message-queue messaging queue

Last synced: 3 months ago
JSON representation

A realtime distributed messaging platform (forked from https://github.com/nsqio/nsq)

Awesome Lists containing this project

README 

        




    

  • 

    Source: https://github.com/youzan/nsq

    • 

  • 

    Issues: https://github.com/youzan/nsq/issues

    • 

  • 

    Mailing List: [email protected]

    • 

  • 

    IRC: #nsq on freenode

    • 

  • 

    Docs: http://nsq.io

    • 

  • 

    Twitter: @nsqio, @absolute8511

    • 




![Go](https://github.com/youzan/nsq/workflows/Go/badge.svg)

[![Build Status](https://travis-ci.com/youzan/nsq.svg?branch=master)](https://travis-ci.com/youzan/nsq) [![GitHub release](https://img.shields.io/github/release/youzan/nsq.svg)](https://github.com/youzan/nsq/releases/latest)

[![Go Report Card](https://goreportcard.com/badge/github.com/youzan/nsq)](https://goreportcard.com/report/github.com/youzan/nsq)

[![codecov](https://codecov.io/gh/youzan/nsq/branch/master/graph/badge.svg)](https://codecov.io/gh/youzan/nsq)



**NSQ** is a realtime distributed messaging platform designed to operate at scale, handling

billions of messages per day.



It promotes *distributed* and *decentralized* topologies without single points of failure,

enabling fault tolerance and high availability coupled with a reliable message delivery

guarantee.  See [features & guarantees][features_guarantees].



Operationally, **NSQ** is easy to configure and deploy (all parameters are specified on the command

line and compiled binaries have no runtime dependencies). For maximum flexibility, it is agnostic to

data format (messages can be JSON, MsgPack, Protocol Buffers, or anything else). Official Go and

Python libraries are available out of the box (as well as many other [client

libraries][client_libraries]) and, if you're interested in building your own, there's a [protocol

spec][protocol].



We publish [binary releases][installing] for linux and darwin.



NOTE: master is our *development* branch and may not be stable at all times.



## About this fork



**This fork has been moved to https://github.com/youzan/nsq**



This fork add many features missing in the original and has been used in Youzan.com.

Features:

* Replication

* HA

* Auto balance and migrate

* Partition

* Delivery in Order

* Consume history data

* Trace the message life-cycle

* Performance improvement



## Deploy Dependency

This fork need the etcd cluster, so to deploy we need set the configure for the nsqd and nsqlookupd. 

For details please refer to the sample configure files under contrib directory.



Other deploy can be the same as the Official.



## 有赞NSQ文档索引



### 用户指南



[有赞NSQ用户指南](doc/NSQ-user-guide.md)



[有赞NSQ部署运维指南](doc/nsq-operation-reference.md)



[JavaSDK使用指南](https://github.com/youzan/nsqJavaSDK/wiki/Get-started)



### 内部实现文档



[重构概览](doc/redesign-overview.md)



[重构实现细节](doc/The-details-of-the-new-arch.md)



[客户端SDK实现](doc/how-we-redesign-the-nsq-smart-client.md)



[扩展消息设计](doc/Message-Extend-Header-Design.md)



## FAQ for this Fork



### Can I use the Official client SDK?



Most client SDK can be used to communication with the new server without using the new features and for new features in this 

fork the new client SDK should be used.



Golang: https://github.com/youzan/go-nsq



Java: https://github.com/youzan/nsqJavaSDK



PHP: https://github.com/youzan/php-nsq-client



### Auto topic create has been disabled.



In order to manager the topics by nsqlookupd, we removed the create api on the nsqd. So in order to create a new topic,

you need create the topic in the nsqadmin or

send the create api to the nsqlookupd as below:


curl -X POST "http://127.0.0.1:4161/topic/create?topic=xxx&partition_num=2&replicator=3&syncdisk=1000"



Explain:

partition_num=2 : topic partition number

replicator=3: topic replica number

syncdisk=1000: fsync disk every syncdisk messages

for ordered topic the query param "orderedmulti=true" is needed.



The new server introduced the partitions but in order to keep compatible with the old clients, each node can hold

only one partition of the topic. So the partition_num*replicator should not be great than the nsqd nodes.

For the ordered topic, we can create as much as we need partitions. (replicator <= number of nsqd node)

Also, the first channel for this topic should be created after the topic is created, otherwise this topic is not 

allowed to be written.



### The log can be adjust dynamically.



This is useful to debug on the fly. 


nsqd: curl -X POST "http://127.0.0.1:4151/loglevel/set?loglevel=3"

nsqlookupd: curl -X POST "http://127.0.0.1:4161/loglevel/set?loglevel=3"



The log is much more detail when the log level increased.



### Change the consume position to allow consume history or skip some messages.




curl -X POST -d "timestamp:xxxxxxxxx" "http://127.0.0.1:4151/channel/setoffset?topic=xxx&partition=xx&channel=xxx"



Post body explain:

timestamp:xxxx (the seconds since 1970-1-1)

virtual_queue:xxx (the queue offset position from the begin)

msgcount:xxx (the message count from the begin)



For multi nsqd nodes, each node for each partition should be done using the api above.



### For more documents



please see the other documents under the doc 



## In Production



  

  

  

  

  





  

  

  

  

  





  

  

  

  

  





  

  

  

  

  





  

  

  

  

  





  



## Code of Conduct



Help us keep NSQ open and inclusive. Please read and follow our [Code of Conduct](CODE_OF_CONDUCT.md).



## Authors



NSQ was designed and developed by Matt Reiferson ([@imsnakes][snakes_twitter]) and Jehiah Czebotar

([@jehiah][jehiah_twitter]) but wouldn't have been possible without the support of

[bitly][bitly] and all our [contributors][contributors].



Logo created by Wolasi Konu ([@kisalow][wolasi_twitter]).



[protocol]: http://nsq.io/clients/tcp_protocol_spec.html

[installing]: https://github.com/youzan/nsq/releases/latest

[snakes_twitter]: https://twitter.com/imsnakes

[jehiah_twitter]: https://twitter.com/jehiah

[bitly]: https://bitly.com

[features_guarantees]: http://nsq.io/overview/features_and_guarantees.html

[contributors]: https://github.com/nsqio/nsq/graphs/contributors

[client_libraries]: http://nsq.io/clients/client_libraries.html

[wolasi_twitter]: https://twitter.com/kisalow