{"id":21170051,"url":"https://github.com/covenantsql/hashstablepack","last_synced_at":"2025-07-09T19:32:19.438Z","repository":{"id":57483760,"uuid":"145376848","full_name":"CovenantSQL/HashStablePack","owner":"CovenantSQL","description":"Serialization code generator for QUICK struct content comparison","archived":false,"fork":false,"pushed_at":"2019-03-27T09:55:22.000Z","size":5298,"stargazers_count":95,"open_issues_count":0,"forks_count":5,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-06-20T14:25:49.460Z","etag":null,"topics":["deterministic","hash","messagepack","serialization","stable"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CovenantSQL.png","metadata":{"files":{"readme":"README-zh.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-20T06:41:33.000Z","updated_at":"2023-12-15T09:57:08.000Z","dependencies_parsed_at":"2022-08-28T17:02:05.671Z","dependency_job_id":null,"html_url":"https://github.com/CovenantSQL/HashStablePack","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CovenantSQL%2FHashStablePack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CovenantSQL%2FHashStablePack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CovenantSQL%2FHashStablePack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CovenantSQL%2FHashStablePack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CovenantSQL","download_url":"https://codeload.github.com/CovenantSQL/HashStablePack/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225587778,"owners_count":17492632,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deterministic","hash","messagepack","serialization","stable"],"created_at":"2024-11-20T15:55:51.372Z","updated_at":"2024-11-20T15:55:52.083Z","avatar_url":"https://github.com/CovenantSQL.png","language":"Go","readme":"##  HSP 为什么比 golang 的 reflect.DeepEqual 快10倍\n\n\n\n使用 Golang 的时候遇到一个棘手的问题：\n\n假设我们现在有一个比较复杂的结构体 ComplexStruct\n\n```go\ntype SimpleStruct struct {\n    Str string\n}\ntype ComplexStruct struct {\n    Simple map[string]*SimpleStruct\n    Other []byte   \n    Nums  [8]float64 \n}\n```\n\n\n\n我们需要在 10000 个 ComplexStruct 类型的变量 Slice 里找到和已知变量内容相同的。\n\n首先，最笨的办法当然是手写:\n\n```go\nvar target = ComplexStruct {\n    ...\n}\nfor _, cs := range csSlice {\n    for k, v := range cs.Simple {\n        if v1, ok := target.Simple[k]; ok {\n             // 算了老子不干了\n        } else {\n            \n        }\n    }\n}\n```\n\n\n\n## DeepEqual\n\n聪明的朋友都会用 `reflect.DeepEqual` 来比较：\n\n```go\nreflect.DeepEqual(v1, v2)\n```\n\n但我们知道，无论是哪种语言，基于反射（reflect）的各种方法都会比普通的函数调用至少慢上 1 ~ 2 个数量级。\n\n有没有又懒又快的方法呢？我反正是没找到，所以自己基于[一个 MsgPack 的库](https://github.com/tinylib/msgp)写了一个 [HashStatePack](https://github.com/CovenantSQL/HashStablePack)，主要功能如下：\n\n1. 根据已经定义好的 golang 结构体自动生成代码，避免搬砖\n\n2. 比`reflect.DeepEqual`快大概10 ~ 20 倍，[测试在此](https://github.com/CovenantSQL/HashStablePack/blob/master/test/hashstable_test.go#L106)\n\n   ```go\n   BenchmarkCompare/benchmark_reflect-8          20074 ns/op //reflect.DeepEqual\n   BenchmarkCompare/benchmark_hsp-8               2322 ns/op\n   BenchmarkCompare/benchmark_hsp_1_cached-8      1101 ns/op\n   BenchmarkCompare/benchmark_hsp_both_cached-8   11.2 ns/op\n   ```\n\n## 为什么不用……\n\n为什么不用 Protobuf 或者 MsgPack，甚至 JSON 序列化之后再比较？\n\n- JSON: 内存使用效率太低，特别是遇到 Binary 类型。\n- 大部分 JSON、MsgPack 的库也是基于反射的实现，也很慢。\n- Prorobuf: struct must defined in proto language, and other limitations discussed [here](https://gist.github.com/kchristidis/39c8b310fd9da43d515c4394c3cd9510)\n- 最后，也是最棘手的问题：Golang 的设计者为了避免大家错误的依赖 map 的顺序，在迭代 map 的时候故意加入了一定的洗牌算法。这就导致几乎针对同样一个 map 的 range 每次的结果都不一样。\n\n### 原理\n\n1. 读取 .go 源代码文件，生成 AST（抽象语法树）\n2. 找到所有的类型，排序\n3. 对每个 Struct 内部成员按照 tag （`hsp:\"xxx\"`）进行排序，没有 tag 的用名字\n4. 根据不同的类型生成不同的序列化代码\n5. 生成测试代码\n\n例如，我们开头例子中的 ComplexStruct，生成的核心代码如下：\n\n```go\n// MarshalHash marshals for hash\nfunc (z *ComplexStruct) MarshalHash() (o []byte, err error) {\n   var b []byte\n   o = hsp.Require(b, z.Msgsize())\n   // map header, size 3\n   o = append(o, 0x83)\n   o = hsp.AppendArrayHeader(o, uint32(8))\n   for za0003 := range z.Nums {\n      o = hsp.AppendFloat64(o, z.Nums[za0003])\n   }\n   o = hsp.AppendBytes(o, z.Other)\n   o = hsp.AppendMapHeader(o, uint32(len(z.Simple)))\n   za0001Slice := make([]string, 0, len(z.Simple))\n   for i := range z.Simple {\n      za0001Slice = append(za0001Slice, i)\n   }\n   sort.Strings(za0001Slice)\n   for _, za0001 := range za0001Slice {\n      za0002 := z.Simple[za0001]\n      o = hsp.AppendString(o, za0001)\n      if za0002 == nil {\n         o = hsp.AppendNil(o)\n      } else {\n         // map header, size 1\n         o = append(o, 0x81)\n         o = hsp.AppendString(o, za0002.Str)\n      }\n   }\n   return\n}\n```\n\n剩下的代码就可以这么写了：\n\n```\nbts1, _ := v1.MarshalHash()\nbts2, _ := v2.MarshalHash()\nif bytes.Equal(bts1, bts2) {\n    ...\n}\n```\n\n针对我们遇到的在大量 Slice 中 寻找相同内容的问题，如果我们对生成的`[]byte`，进行一次哈希。然后用哈希只作为 key，对象作为 Value，效率将会非常的高。\n\nHashStablePack 目前主要被 [CovenantSQL](https://github.com/CovenantSQL/CovenantSQL) 用来做签名、校验，以及区块哈希计算上，希望可以帮到你:-)\n\n\n\n## 怎么使用\n\n```go\ngo get -u github.com/CovenantSQL/HashStablePack/hsp\n```\n\n在你需要生成的源文件头部加上\n\n```go\n//go:generate hsp\n```\n\n运行\n\n```go\ngo generate ./...\n```\n\n代码、测试代码，就统统生成好了","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcovenantsql%2Fhashstablepack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcovenantsql%2Fhashstablepack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcovenantsql%2Fhashstablepack/lists"}