{"id":20482107,"url":"https://github.com/uk0/vladimir","last_synced_at":"2025-04-13T14:13:46.112Z","repository":{"id":103060939,"uuid":"196355426","full_name":"uk0/Vladimir","owner":"uk0","description":"数据采集","archived":false,"fork":false,"pushed_at":"2021-02-19T11:33:55.000Z","size":24186,"stargazers_count":5,"open_issues_count":1,"forks_count":2,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-04-13T14:13:35.086Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uk0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-07-11T08:44:04.000Z","updated_at":"2024-12-12T14:31:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"aa789086-e45a-4bb2-9f30-d00cee364e5b","html_url":"https://github.com/uk0/Vladimir","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uk0%2FVladimir","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uk0%2FVladimir/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uk0%2FVladimir/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uk0%2FVladimir/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uk0","download_url":"https://codeload.github.com/uk0/Vladimir/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248724628,"owners_count":21151561,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T16:11:31.242Z","updated_at":"2025-04-13T14:13:46.107Z","avatar_url":"https://github.com/uk0.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"### Octopoda采集系统\n\n\n## TODO\n\n* 将日志文件一行一行读取出来发送到自定义的Sink | 100%\n* 支持自动重试（fib） \n* 心跳检测发送到自定义的Topic[health] | 100%\n* 日志启动点(创建CheckPoint) | 100%\n* Offset 同步到Etcd | 100%\n* Web监控心跳以及创建新的采集任务 | 100%\n* 热启动插件 | 10%\n\n## Logs - 采集工具\n```bash\n\n./Logs_Pulse  install   -- 安装服务\n./Logs_Pulse  start     -- 启动服务\n./Logs_Pulse  stop      -- 停止服务\n./Logs_Pulse  restart   -- 重启服务\n\n```\n\n\n\n\n## Owner \n* 邮箱`echo emhhbmdqaWFueGlubmV0QGdtYWlsLmNvbQo= | base64 --decode`\n```bash\n\n```\n\n\n### 命令行客户端使用教程\n\n```bash\n\n(base) ➜  cli ./cli         \nNAME:\n   pulse client - a tool for driving pulse cli\n\nUSAGE:\n   cli [global options] command [command options] [arguments...]\n\nVERSION:\n   v1 darwin/amd64 go1.11.4\n\nAUTHOR:\n   Zhangjianxin \u003czhangjianxinnet@gmail.com\u003e\n\nCOMMANDS:\n     list      pulse on etcd config\n     services  pulse on services\n     task      addtask on services\n     clear     delete root path / data\n     host      host list\n     help, h   Shows a list of commands or help for one command\n\nGLOBAL OPTIONS:\n   --help, -h     show help\n   --version, -v  print the version\n\n\n\n(base) ➜  cli ./cli host\n|---------------|\n| Host          |\n|---------------|\n| 192.168.2.150 |\n|---------------|\n| 192.168.2.150 |\n|---------------|\n| 192.168.2.150 |\n|---------------|\n| 192.168.2.150 |\n|---------------|\n\n\n(base) ➜  cli ./cli list\n|-----------------------------------------------------------------------------------------------------------------------------------------------------|\n| Topic | Log_path                                                                          | Service             | Send_rate | IP            | Type  |\n|-----------------------------------------------------------------------------------------------------------------------------------------------------|\n| test  | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est.log  | sentry-test-service | 1000      | 192.168.2.150 | kafka |\n|-----------------------------------------------------------------------------------------------------------------------------------------------------|\n| test  | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est.log  | sentry-test-service | 1000      | 192.168.2.150 | http  |\n|-----------------------------------------------------------------------------------------------------------------------------------------------------|\n| test  | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est2.log | sentry-test-service | 1000      | 192.168.2.150 | http  |\n|-----------------------------------------------------------------------------------------------------------------------------------------------------|\n\n\n(base) ➜  cli ./cli services\n|------------------------------------------------------------------------------------------------------------------------------------------------------|\n| Offset | FilePatch                                                                         | Time                | Topic | TPS | QPS | IP            |\n|------------------------------------------------------------------------------------------------------------------------------------------------------|\n| 39     | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est.log  | 2019-04-27 03:38:32 | test  | 0   | 0   | 192.168.2.150 |\n|------------------------------------------------------------------------------------------------------------------------------------------------------|\n| 63     | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est.log  | 2019-04-27 03:45:17 | test  | 0   | 0   | 192.168.2.150 |\n|------------------------------------------------------------------------------------------------------------------------------------------------------|\n| 87     | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est2.log | 2019-04-27 03:49:32 | test  | 0   | 0   | 192.168.2.150 |\n|------------------------------------------------------------------------------------------------------------------------------------------------------|\n\n\n\n\n(base) ➜  cli \n    ./cli task  -host 192.168.43.223  -t buried-point-mall -f /var/log/system.log -s buried-point-mall-service-test -p http\n    ./cli task   -re \"\" -host 192.168.2.150  -t buried-point-mall -f /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/ -s buried-point-mall-service-test -p http\n    \n    ./cli task   -re \"\" -host 172.17.193.23  -t sentry-test -f /var/log/messages -s service-test -p http\n    \n    ./cli task  -re \"\" -host  172.19.184.193 -t sentry-test -f /home/soft/Octopoda/release/logs/pulse.log -s system-pulse -p http\n    ./cli task  -re \"\" -host  172.19.184.193 -t buried-point-mall -f /data/sa/ -s sa-pulse -p http\n    \n    \n    ./cli task  -re \"\" -host 172.17.193.23  -t sentry-test -f /home/Octopoda/release/logs/ -s service-test2 -p http\n    ./cli task  -re \"\" -host 192.168.131.249  -t buried-point-sma-r2p10 -f  /home/admin/app/logs/ -s service-test4 -p http\n    ./cli task  -re \"\" -host 192.168.131.249  -t buried-point-sma-r2p10 -f  /home/admin/app/logs/behaviorSave.log -s service-test4 -p http\n    \n    \n    \n    ./cli task  -re \"\" -host 192.168.131.249  -t sentry-test -f  /home/Octopoda/release/logs/pulse.log -s system-pulse -p http\n\n```\n\n### 提交作业\n\n```bash\n\n./cli task \n         -host 192.168.2.150  #采集某个节点\n         -t test  #发送到topic\n         -f /var/log/system.log # 采集日志\n         -s sentry-test-service # 服务名字定义\n         -p http #发送方式\n\n\n# create topic\nkafka-topics --create --zookeeper cdh-m1.sxkj.online:2181 --replication-factor 1 --partitions 1 --topic kdc_pdp\n\n```\n\n### kerberos KDC Proxy\n```bash\nhttps://medium.com/@Santii/how-to-setup-kerberos-behind-a-reverse-proxy-nginx-d692f609191c\n```\n\n\n\n![请添加描述](http://172.17.0.200:3000/look_look/3a9f023e-f995-4345-89ef-81a210903387.png)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuk0%2Fvladimir","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fuk0%2Fvladimir","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuk0%2Fvladimir/lists"}