{"id":15348808,"url":"https://github.com/zuston/atcalmq","last_synced_at":"2026-06-20T08:31:41.179Z","repository":{"id":93116625,"uuid":"126816744","full_name":"zuston/AtcalMq","owner":"zuston","description":"the message queue for ane trace big data, which serves for the maching learning prediction","archived":false,"fork":false,"pushed_at":"2018-08-18T09:45:27.000Z","size":2123,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-08-01T03:52:50.901Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zuston.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-03-26T11:10:39.000Z","updated_at":"2018-08-18T09:45:28.000Z","dependencies_parsed_at":"2023-06-04T23:45:27.258Z","dependency_job_id":null,"html_url":"https://github.com/zuston/AtcalMq","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zuston/AtcalMq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zuston%2FAtcalMq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zuston%2FAtcalMq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zuston%2FAtcalMq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zuston%2FAtcalMq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zuston","download_url":"https://codeload.github.com/zuston/AtcalMq/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zuston%2FAtcalMq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34563535,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-20T02:00:06.407Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-01T11:52:32.045Z","updated_at":"2026-06-20T08:31:41.174Z","avatar_url":"https://github.com/zuston.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AtcalMq\nthe message queue for ane trace big data, which serves for the maching learning prediction\n\n# Introduction\n此为ane消息队列处理框架，所以主要分为consumer和producer两个守护进程，当然两者是分开执行的。编译而成的主要为 main 和 push 二进制程序。\n\n# Install\n1. __如果对代码进行修改，需要配置go的开发环境__\n2. __也可以下载编译好的运行文件，直接运行二进制程序。__\n\n# Compile\n编译linux下的可执行程序执行命令：make  \n### 编译脚本\n执行 make 进行编译，生成的 push, pull, console 二进制可执行程序在target下\n\n# Deploy\n1. 其实部署脚本已写好，涉及到的登录信息就没有开源\n2. 目前采用 supervisor来进行守护 pull 和 push 两个程序，配置均在集群 master 机器上 /etc/supervisor/conf.d下可见\n\n# Usage\n1. pull 二进制文件为 ane 消息队列的消费者，主要处理接收的消息的存储和备份操作。执行 ./pull -h 可以查看到多种参数配置。主要有两个参数需要注意。   \n__执行 ./pull -show true 时候，只是处理给 console 程序提供 rpc 接口，不进行消息的消费操作.__  \n__执行 ./pull -queueName \"{queueName}\" 的时候，是给消费者预定一定消费队列名，默认是全选。__  \n__执行 ./pull 的时候，是一切默认，全部消费队列注册，开始消费。__  \n__执行 ./pull -backup true 执行备份队列，从自建消息队列中读取消息，存储为二元组模型。__\n2. 队列控制台 ./console 程序只是提供给服务器端的消息队列的积压情况的可视化展示，展示分钟级别的处理能力。必须在 pull 程序执行之后执行，因为需要接受 pull 程序的 rpc 消息。\n3. push 程序为推送程序，是一个定时执行的守护程序。根据参数，可以设定定时的间隔时间和model映射的文件位置，还有时间参数（用以重发在某个时间点之后的消息）\n4. backup 程序为从日志文件中读取消息，推送到自建的消息队列中去。可以通过 backup -h 查看参数配置等。\n\n# Config\n本框架主要有两个地方的配置文件，且自带默认地址。\n1. 关于消息队列的配置问题，生产环境位置为 /opt/mq.ini, 测试环境位置即为当前代码的根目录。在此代码中不含有，已被 .gitignore 了。具体配置请参见服务器的 /opt 下的位置。\n2. 关于 push 信息的 map 配置问题。环境位置均为 /opt 下，默认名字均可在push代码中自定义。在此代码中可见，格式类似于 optional.model文件中。\n3. 关于 pullcore handler 中的 hbase 多集群备份的配置默认为 /opt/hbase.ini 文件中写明，参见 cluster 下的配置。只需要提供 zookeeper 的节点字符串。\n\n# Architecture\n`golang`新手，对`golang`的编程范式还不够。所以设计方面还需要提升。主要讲一个`consumer`这一侧的`rabbitmq`的消费调用。  \n\n```go\nfunc main(){\n  // 开启一个rabbitmq的消费者实例\n  cf, err := rabbitmq.NewConsumerFactory(mq_uri,exchange,exchange_type,true)\n  // 向实例注册某个消息队列，且写入处理方法的回调\n  cf.Register(\"ane_test\",testHandler)\n  // 注册多个消息队列，且用同一个回调方法\n  cf.RegisterAll([]string{\"ane_1\",\"ane_2\"},testHandler)\n  \n  func testHandler(queueName string, msgChan \u003c-chan amqp.Delivery){\n    for msg := range msgChan{}\n  }\n}\n```\n__就是这么简单和清晰__  \n```go\n// 生产者\nfunc main(){\n  pf, _ := rabbitmq.NewProducerFactory(mq_uri,exchange,exchange_type,false)\n  pf.Publish(\"ane_push\",\"hello world\")\n}\n```\n\n# Console UI\n![image](https://raw.githubusercontent.com/zuston/AtcalMq/master/snapshot/console.jpg)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzuston%2Fatcalmq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzuston%2Fatcalmq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzuston%2Fatcalmq/lists"}