{"id":16779683,"url":"https://github.com/simfg/etcd-analysis","last_synced_at":"2025-08-20T15:03:49.349Z","repository":{"id":49782743,"uuid":"518001865","full_name":"SimFG/etcd-analysis","owner":"SimFG","description":"🔦 Etcd Data Analysis Tool","archived":false,"fork":false,"pushed_at":"2023-11-08T08:00:54.000Z","size":73475,"stargazers_count":22,"open_issues_count":2,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-04T02:03:04.567Z","etag":null,"topics":["data-analysis","etcd","go","raft"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SimFG.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-26T09:46:34.000Z","updated_at":"2025-02-25T08:54:58.000Z","dependencies_parsed_at":"2023-11-08T09:25:11.662Z","dependency_job_id":null,"html_url":"https://github.com/SimFG/etcd-analysis","commit_stats":{"total_commits":30,"total_committers":3,"mean_commits":10.0,"dds":0.1333333333333333,"last_synced_commit":"b3c7f9bb0def4a532e55609021d897f0fa2d953f"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/SimFG/etcd-analysis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SimFG%2Fetcd-analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SimFG%2Fetcd-analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SimFG%2Fetcd-analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SimFG%2Fetcd-analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SimFG","download_url":"https://codeload.github.com/SimFG/etcd-analysis/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SimFG%2Fetcd-analysis/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271337906,"owners_count":24742057,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-20T02:00:09.606Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-analysis","etcd","go","raft"],"created_at":"2024-10-13T07:31:39.277Z","updated_at":"2025-08-20T15:03:49.313Z","avatar_url":"https://github.com/SimFG.png","language":"Go","readme":"# etcd-analysis\netcd is generally used to store system metadata or service discovery, and is suitable for storing small key-value pairs. At the same time, etcd is sensitive to the size of key-value pairs. When storing large key-value pairs, if the number is too large, it will bring many adverse effects, such as the stability of the watch function is reduced, and a large amount of memory is occupied.\n\nWhen testing the stability of the system, it may be necessary to pay attention to the size distribution of the data currently stored in etcd by the system. That's why this project came about.\n\n# 🌟 New Function: unmarshal\n\n**implement proto.Unmarshal byte array through proto source file**\n\nGenerally speaking, we store some system meta formation in etcd, and the stored pseudocode is:\n\n```go\nsegBytes, _ := proto.Marshal(*foopb.SystemInfo{})\netcdclient.put(\"foo/system\", segBytes)\n```\n\nWhen we query, we cannot clearly see the value in the struct. Of course, it can be easily implemented through code. You only need to import the relevant pb file and then call the proto.Unmarshal method to see the clear value. But many times we may not have the environment, or writing this part of the code will waste a little time.\n\nThe unmarshal instruction will solve this trouble. You only need to copy the proto source file to quickly view the value in the struct. The example:\n\n```\n➜ etcdctl+ unmarshal --key by-dev/meta/channelwatch/4/by-dev-rootcoord-dml_0_445337303926193462v0 --import-path ../birdwatcher/proto/v2.2 --proto ../birdwatcher/proto/v2.2/data_coord.proto --full-message-name milvus.protov2.data.ChannelWatchInfo\n\nvchan: collectionID:445337303926193462 channelName:\"by-dev-rootcoord-dml_0_445337303926193462v0\" seek_position:\u003cchannel_name:\"by-dev-rootcoord-dml_0\" msgID:\"\\010?\\020\\223\\261\\002\\030\\000 \\000\" msgGroup:\"datanode-3-by-dev-rootcoord-dml_0_445337303926193462v0-true\" timestamp:445342561987461121\u003e flushedSegmentIds:445337303926393472\nstartTs: 1698912217\nstate: 3\ntimeoutTs: 0\nschema: name:\"foo\" description:\"hello_milvus is the simplest demo to introduce the APIs\" fields:\u003cfieldID:100 name:\"pk\" is_primary_key:true data_type:VarChar type_params:\u003ckey:\"max_length\" value:\"100\"\u003e\u003e fields:\u003cfieldID:101 name:\"random\" data_type:Double\u003e fields:\u003cfieldID:102 name:\"embeddings\" data_type:FloatVector type_params:\u003ckey:\"dim\" value:\"8\"\u003e\u003e fields:\u003cname:\"RowID\" description:\"row id\" data_type:Int64\u003e fields:\u003cfieldID:1 name:\"Timestamp\" description:\"time stamp\" data_type:Int64\u003e\nprogress: 0\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003ch1\u003eGetting started\u003c/h1\u003e\u003c/summary\u003e\n\n## Getting the source code\n\nClone this code repository\n\n```shell\n$ git clone https://github.com/SimFG/etcd-analysis.git\n```\n\n## Build\n\nCompile code into executable\n\n```shell\n$ go build -o etcdctl+\n```\n\n## Usage\n\nGet help with functions\n\n```shell\n$ etcdctl+ distribute -h\n```\n\n\u003c/details\u003e\n\n\u003cdetails open\u003e\n\u003csummary\u003e\u003ch1\u003eFunction List\u003c/h1\u003e\u003c/summary\u003e\n\n\n1. **distribute** View data distribution according to data size\n2. **look** Show or export all the etcd data, and be used with terminal or loki\n3. **find** Get key based on certain characters\n4. **leader** Get the leader node info\n5. **clear** Clear all the etcd data\n6. **decode** Base64Decode the etcd value that is encoded \n7. **rename** Rename the etcd data key\n8. **unmarshal** Implement proto.Unmarshal byte array through proto source file\n\n## distribute\n\nView data distribution in etcd according to the `key` size , `value` size or `key + value` size by setting the `type` command param.\n\n![distribute.gif](pic/20230225-150850.gif)\n\n```shell\n$ etcdctl+ distribute\n\nSummary:\n  Count:        116.\n  Total:        7.3 KB.\n  Smallest:     22.0 B.\n  Largest:      85.0 B.\n  Average:      64.0 B.\n\nSize histogram:\n  22.0 B [1]    |\n  34.0 B [6]    |∎∎∎\n  46.0 B [29]   |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎\n  58.0 B [13]   |∎∎∎∎∎∎∎\n  70.0 B [1]    |\n  85.0 B [66]   |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎\n\nSize distribution:\n  10% in 38.0 B.\n  25% in 39.0 B.\n  50% in 76.0 B.\n  75% in 83.0 B.\n  90% in 85.0 B.\n```\n\n## look\nGet all data in etcd, and you can use system tools to search.\n```shell\n$ etcdctl+ look | more\n\nCurrent Stage\n  cluster_id:2037210783374497686 member_id:13195394291058371180 revision:254946 raft_term:9 \nKv List\n| Key | Value | CreateRevision | ModRevision | Version | Lease |\n| by-dev/kv/gid/idTimestamp | - | 253775 | 254802 | 12 | 0 |\n```\n- Case 1:  Get all data continuously and display it on the console\n```shell\n$ etcdctl+ look --write-out=file --hang=true\n\n# New Terminal\n$ vim analysis.txt\n# update the file in vim, using `:e`\n```\n- Case 2:  Get the kv data of the specified size range.\n```shell\n$ etcdctl+ look --filter=key --filter-min=74 --filter-max=100\n\nCurrent Stage\n  cluster_id:2037210783374497686 member_id:13195394291058371180 revision:326021 raft_term:14 \nKv List\n| Key | Value | CreateRevision | ModRevision | Version | Lease |\n| by-dev/meta/channelwatch/-9223372036854775808/by-dev-rootcoord-dml_4_435191634150817793v0 | - | 326013 | 326013 | 1 | 0 |\n```\n\n## find\n\nGet key based on certain characters\n\n```shell\n$ ./etcdctl+ find --key=index\nKv List\n| Key | Value |\n| by-dev/meta/field-index/438660758500016136/438660903999573339 |  |\n| by-dev/meta/segment-index/438660758500016136/438660758500016137/438660758500216145/438660903999573340 |  |\n| by-dev/meta/segment-index/438660758500016136/438660758500016137/438660758500216146/438660903999573341 |  |\n| by-dev/meta/segment-index/438660758500016136/438660758500016137/438672571137461581/438672571137461597 |  |\n| by-dev/meta/segment-index/438660758500016136/438660758500016137/438672571137461582/438672571137461598 |  |\n```\n\n## leader\n\nGet the leader node info\n\n```shell\n$ etcdctl+ leader\n\nName: default\nClientUrls: [http://127.0.0.1:2379]\n```\n\n## clear\n\nClear all etcd data\n\n```shell\n$ etcdctl+ clear\n\nClear All Data, (Y/n):\n```\n\n## unmarshal\n\nImplement proto.Unmarshal byte array through proto source file.\n\n```\n$ etcdctl+ unmarshal --key by-dev/meta/channelwatch/4/by-dev-rootcoord-dml_0_445337303926193462v0 --import-path ../birdwatcher/proto/v2.2 --proto ../birdwatcher/proto/v2.2/data_coord.proto --full-message-name milvus.protov2.data.ChannelWatchInfo\n```\n\n- key: the etcd full key\n- import-path: all proto directory\n- proto: the proto file path where the message is located\n- full-message-name: the full message name, usually a combination of proto package name and message\n\n\u003c/details\u003e\n\n# All contributors\n\n\u003ca href=\"https://github.com/SimFG/etcd-analysis/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=SimFG/etcd-analysis\" /\u003e\n\u003c/a\u003e\n\n# Stargazers over time\n\n[![Stargazers over time](https://starchart.cc/SimFG/etcd-analysis.svg)](https://starchart.cc/SimFG/etcd-analysis)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimfg%2Fetcd-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimfg%2Fetcd-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimfg%2Fetcd-analysis/lists"}