{"id":13695611,"url":"https://github.com/ClickHouse/ch-go","last_synced_at":"2025-05-03T13:32:41.934Z","repository":{"id":36963037,"uuid":"428047243","full_name":"ClickHouse/ch-go","owner":"ClickHouse","description":"Low-level Go Client for ClickHouse","archived":false,"fork":false,"pushed_at":"2024-10-21T12:21:27.000Z","size":2156,"stargazers_count":322,"open_issues_count":18,"forks_count":50,"subscribers_count":11,"default_branch":"main","last_synced_at":"2024-10-21T18:13:30.313Z","etag":null,"topics":["binary","clickhouse","driver","go","golang","protocol"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ClickHouse.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-14T21:37:40.000Z","updated_at":"2024-10-21T12:22:38.000Z","dependencies_parsed_at":"2023-07-18T11:28:09.424Z","dependency_job_id":"f4e23b01-ee64-452d-ae68-b27f28016231","html_url":"https://github.com/ClickHouse/ch-go","commit_stats":{"total_commits":1005,"total_committers":21,"mean_commits":"47.857142857142854","dds":"0.24776119402985075","last_synced_commit":"d4dca7ad654447012744d05769c9bc87a28d2627"},"previous_names":["go-faster/ch","go-faster/ch-go"],"tags_count":100,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fch-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fch-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fch-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fch-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ClickHouse","download_url":"https://codeload.github.com/ClickHouse/ch-go/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224364337,"owners_count":17299052,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binary","clickhouse","driver","go","golang","protocol"],"created_at":"2024-08-02T18:00:30.900Z","updated_at":"2025-05-03T13:32:41.924Z","avatar_url":"https://github.com/ClickHouse.png","language":"Go","readme":"# ch [![](https://img.shields.io/badge/go-pkg-00ADD8)](https://pkg.go.dev/github.com/ClickHouse/ch-go#section-documentation)\nLow level TCP [ClickHouse](https://clickhouse.com/) client and protocol implementation in Go. Designed for very fast data block streaming with low network, cpu and memory overhead.\n\nNB: **No pooling, reconnects** and **not** goroutine-safe by default, only single connection.\nUse [clickhouse-go](https://github.com/ClickHouse/clickhouse-go) for high-level `database/sql`-compatible client,\npooling for ch-go is available as [chpool](https://pkg.go.dev/github.com/ClickHouse/ch-go/chpool) package.\n\n* [Feedback](https://github.com/ClickHouse/ch-go/discussions/6)\n* [Benchmarks](https://github.com/go-faster/ch-bench#benchmarks)\n* [Protocol reference](https://go-faster.org/docs/clickhouse)\n\n*[ClickHouse](https://clickhouse.com/) is an open-source, high performance columnar OLAP database management system for real-time analytics using SQL.*\n\n```console\ngo get github.com/ClickHouse/ch-go@latest\n```\n\n## Example\n```go\npackage main\n\nimport (\n  \"context\"\n  \"fmt\"\n\n  \"github.com/ClickHouse/ch-go\"\n  \"github.com/ClickHouse/ch-go/proto\"\n)\n\nfunc main() {\n  ctx := context.Background()\n  c, err := ch.Dial(ctx, ch.Options{Address: \"localhost:9000\"})\n  if err != nil {\n    panic(err)\n  }\n  var (\n    numbers int\n    data    proto.ColUInt64\n  )\n  if err := c.Do(ctx, ch.Query{\n    Body: \"SELECT number FROM system.numbers LIMIT 500000000\",\n    Result: proto.Results{\n      {Name: \"number\", Data: \u0026data},\n    },\n    // OnResult will be called on next received data block.\n    OnResult: func(ctx context.Context, b proto.Block) error {\n      numbers += len(data)\n      return nil\n    },\n  }); err != nil {\n    panic(err)\n  }\n  fmt.Println(\"numbers:\", numbers)\n}\n```\n\n```\n393ms 0.5B rows  4GB  10GB/s 1 job\n874ms 2.0B rows 16GB  18GB/s 4 jobs\n```\n\n### Results\n\nTo stream query results, set `Result` and `OnResult` fields of [Query](https://pkg.go.dev/github.com/ClickHouse/ch-go#Query).\nThe `OnResult` will be called after `Result` is filled with received data block.\n\nThe `OnResult` is optional, but query will fail if more than single block is received, so it is ok to solely set the `Result`\nif only one row is expected.\n\n#### Automatic result inference\n```go\nvar result proto.Results\nq := ch.Query{\n  Body:   \"SELECT * FROM table\",\n  Result: result.Auto(),\n}\n```\n\n#### Single result with column name inference\n```go\nvar res proto.ColBool\nq := ch.Query{\n  Body:   \"SELECT v FROM test_table\",\n  Result: proto.ResultColumn{Data: \u0026res},\n}\n```\n\n### Writing data\n\nSee [examples/insert](./examples/insert).\n\nFor table\n```sql\nCREATE TABLE test_table_insert\n(\n    ts                DateTime64(9),\n    severity_text     Enum8('INFO'=1, 'DEBUG'=2),\n    severity_number   UInt8,\n    service_name      LowCardinality(String),\n    body              String,\n    name              String,\n    arr               Array(String)\n) ENGINE = Memory\n```\n\nWe prepare data block for insertion as follows:\n\n```go\nvar (\n\tbody      proto.ColStr\n\tname      proto.ColStr\n\tsevText   proto.ColEnum\n\tsevNumber proto.ColUInt8\n\n    // or new(proto.ColStr).LowCardinality()\n    serviceName = proto.NewLowCardinality(new(proto.ColStr))\n    ts          = new(proto.ColDateTime64).WithPrecision(proto.PrecisionNano) // DateTime64(9)\n    arr         = new(proto.ColStr).Array()                                   // Array(String)\n    now         = time.Date(2010, 1, 1, 10, 22, 33, 345678, time.UTC)\n)\n\n// Append 10 rows to initial data block.\nfor i := 0; i \u003c 10; i++ {\n\tbody.AppendBytes([]byte(\"Hello\"))\n\tts.Append(now)\n\tname.Append(\"name\")\n\tsevText.Append(\"INFO\")\n\tsevNumber.Append(10)\n\tarr.Append([]string{\"foo\", \"bar\", \"baz\"})\n    serviceName.Append(\"service\")\n}\n\ninput := proto.Input{\n\t{Name: \"ts\", Data: ts},\n\t{Name: \"severity_text\", Data: \u0026sevText},\n\t{Name: \"severity_number\", Data: sevNumber},\n\t{Name: \"body\", Data: body},\n\t{Name: \"name\", Data: name},\n\t{Name: \"arr\", Data: arr},\n}\n```\n\n#### Single data block\n```go\nif err := conn.Do(ctx, ch.Query{\n\t// Or \"INSERT INTO test_table_insert (ts, severity_text, severity_number, body, name, arr) VALUES\"\n\t// Or input.Into(\"test_table_insert\")\n\tBody: \"INSERT INTO test_table_insert VALUES\",\n\tInput: input,\n}); err != nil {\n\tpanic(err)\n}\n```\n\n### Stream data\n```go\n// Stream data to ClickHouse server in multiple data blocks.\nvar blocks int\nif err := conn.Do(ctx, ch.Query{\n\tBody:  input.Into(\"test_table_insert\"), // helper that generates INSERT INTO query with all columns\n\tInput: input,\n\n\t// OnInput is called to prepare Input data before encoding and sending\n\t// to ClickHouse server.\n\tOnInput: func(ctx context.Context) error {\n\t\t// On OnInput call, you should fill the input data.\n\t\t//\n\t\t// NB: You should reset the input columns, they are\n\t\t// not reset automatically.\n\t\t//\n\t\t// That is, we are re-using the same input columns and\n\t\t// if we will return nil without doing anything, data will be\n\t\t// just duplicated.\n\n\t\tinput.Reset() // calls \"Reset\" on each column\n\n\t\tif blocks \u003e= 10 {\n\t\t\t// Stop streaming.\n\t\t\t//\n\t\t\t// This will also write tailing input data if any,\n\t\t\t// but we just reset the input, so it is currently blank.\n\t\t\treturn io.EOF\n\t\t}\n\n\t\t// Append new values:\n\t\tfor i := 0; i \u003c 10; i++ {\n\t\t\tbody.AppendBytes([]byte(\"Hello\"))\n\t\t\tts.Append(now)\n\t\t\tname.Append(\"name\")\n\t\t\tsevText.Append(\"DEBUG\")\n\t\t\tsevNumber.Append(10)\n\t\t\tarr.Append([]string{\"foo\", \"bar\", \"baz\"})\n            serviceName.Append(\"service\")\n\t\t}\n\n\t\t// Data will be encoded and sent to ClickHouse server after returning nil.\n\t\t// The Do method will return error if any.\n\t\tblocks++\n\t\treturn nil\n\t},\n}); err != nil {\n\tpanic(err)\n}\n```\n\n### Writing dumps in Native format\n\nYou can use `ch-go` to write ClickHouse dumps in [Native][native] format:\n\n\u003e The most efficient format. Data is written and read by blocks in binary format. For each block, the number of rows,\n\u003e number of columns, column names and types, and parts of columns in this block are recorded one after another.\n\u003e In other words, this format is “columnar” – it does not convert columns to rows.\n\u003e This is the format used in the native interface for interaction between servers,\n\u003e for using the command-line client, and for C++ clients.\n\n[native]: https://clickhouse.com/docs/en/interfaces/formats/#native\n\nSee [./internal/cmd/ch-native-dump](./internal/cmd/ch-native-dump/main.go) for more sophisticated example.\n\nExample:\n```go\nvar (\n    colK proto.ColInt64\n    colV proto.ColInt64\n)\n// Generate some data.\nfor i := 0; i \u003c 100; i++ {\n    colK.Append(int64(i))\n    colV.Append(int64(i) + 1000)\n}\n// Write data to buffer.\nvar buf proto.Buffer\ninput := proto.Input{\n    {\"k\", colK},\n    {\"v\", colV},\n}\nb := proto.Block{\n    Rows:    colK.Rows(),\n    Columns: len(input),\n}\n// Note that we are using version 54451, proto.Version will fail.\nif err := b.EncodeRawBlock(\u0026buf, 54451, input); err != nil {\n    panic(err)\n}\n\n// You can write buf.Buf to io.Writer, e.g. os.Stdout or file.\nvar out bytes.Buffer\n_, _ = out.Write(buf.Buf)\n\n// You can encode multiple buffers in sequence.\n//\n// To do this, reset buf and all columns, append new values\n// to columns and call EncodeRawBlock again.\nbuf.Reset()\ncolK.Reset()\ncolV.Reset()\n```\n\n## Features\n* OpenTelemetry support\n* No reflection or `interface{}`\n* Generics (go1.18) for `Array[T]`, `LowCardinaliy[T]`, `Map[K, V]`, `Nullable[T]`\n* [Reading or writing](#dumps) ClickHouse dumps in `Native` format\n* **Column**-oriented design that operates directly with **blocks** of data\n  * [Dramatically more efficient](https://github.com/ClickHouse/ch-bench)\n  * Up to 100x faster than row-first design around `sql`\n  * Up to 700x faster than HTTP API\n  * Low memory overhead (data blocks are slices, i.e. continuous memory)\n  * Highly efficient input and output block streaming\n  * As close to ClickHouse as possible\n* Structured query execution telemetry streaming\n  * Query progress\n  * Profiles\n  * Logs\n  * [Profile events](https://github.com/ClickHouse/ClickHouse/issues/26177)\n* LZ4, ZSTD or *None* (just checksums for integrity check) compression\n* [External data](https://clickhouse.com/docs/en/engines/table-engines/special/external-data/) support\n* Rigorously tested\n  * Windows, Mac, Linux (also x86)\n  * Unit tests for encoding and decoding\n    * ClickHouse **Server** in **Go** for faster tests\n    * Golden files for all packets, columns\n    * Both server and client structures\n    * Ensuring that partial read leads to failure\n  * End-to-end [tests](.github/workflows/e2e.yml) on multiple LTS and stable versions\n  * Fuzzing\n\n## Supported types\n* UInt8, UInt16, UInt32, UInt64, UInt128, UInt256\n* Int8, Int16, Int32, Int64, Int128, Int256\n* Date, Date32, DateTime, DateTime64\n* Decimal32, Decimal64, Decimal128, Decimal256 (only low-level raw values)\n* IPv4, IPv6\n* String, FixedString(N)\n* UUID\n* Array(T)\n* Enum8, Enum16\n* LowCardinality(T)\n* Map(K, V)\n* Bool\n* Tuple(T1, T2, ..., Tn)\n* Nullable(T)\n* Point\n* Nothing, Interval\n\n## Enums\n\nYou can use automatic enum inference in `proto.ColEnum`, this will come with some performance penalty.\n\nTo use `proto.ColEnum8` and `proto.ColEnum16`, you need to explicitly provide DDL for them via `proto.Wrap`:\n\n```go\nvar v proto.ColEnum8\n\nconst ddl = `'Foo'=1, 'Bar'=2, 'Baz'=3`\ninput := []proto.InputColumn{\n  {Name: \"v\", Data: proto.Wrap(\u0026v, ddl)},\n}\n```\n\n## Generics\n\nMost columns implement [proto.ColumnOf\\[T\\]](https://pkg.go.dev/github.com/ClickHouse/ch-go/proto#ColumnOf) generic constraint:\n```go\ntype ColumnOf[T any] interface {\n\tColumn\n\tAppend(v T)\n\tAppendArr(vs []T)\n\tRow(i int) T\n}\n```\n\nFor example, [ColStr](https://pkg.go.dev/github.com/ClickHouse/ch-go/proto#ColStr) (and [ColStr.LowCardinality](https://pkg.go.dev/github.com/ClickHouse/ch-go/proto#ColStr.LowCardinality)) implements `ColumnOf[string]`.\nSame for arrays: `new(proto.ColStr).Array()` implements `ColumnOf[[]string]`, column of `[]string` values.\n\n### Array\n\nGeneric for `Array(T)`\n\n```go\n// Array(String)\narr := proto.NewArray[string](new(proto.ColStr))\n// Or\narr := new(proto.ColStr).Array()\nq := ch.Query{\n  Body:   \"SELECT ['foo', 'bar', 'baz']::Array(String) as v\",\n  Result: arr.Results(\"v\"),\n}\n// Do ...\narr.Row(0) // [\"foo\", \"bar\", \"baz\"]\n```\n\n## Dumps\n\n### Reading\n\nUse `proto.Block.DecodeRawBlock` on `proto.NewReader`:\n\n```go\nfunc TestDump(t *testing.T) {\n\t// Testing decoding of Native format dump.\n\t//\n\t// CREATE TABLE test_dump (id Int8, v String)\n\t//   ENGINE = MergeTree()\n\t// ORDER BY id;\n\t//\n\t// SELECT * FROM test_dump\n\t//   ORDER BY id\n\t// INTO OUTFILE 'test_dump_native.raw' FORMAT Native;\n\tdata, err := os.ReadFile(filepath.Join(\"_testdata\", \"test_dump_native.raw\"))\n\trequire.NoError(t, err)\n\tvar (\n\t\tdec    proto.Block\n\t\tids    proto.ColInt8\n\t\tvalues proto.ColStr\n\t)\n\trequire.NoError(t, dec.DecodeRawBlock(\n\t\tproto.NewReader(bytes.NewReader(data)),\n\t\tproto.Results{\n\t\t\t{Name: \"id\", Data: \u0026ids},\n\t\t\t{Name: \"v\", Data: \u0026values},\n\t\t}),\n\t)\n}\n```\n\n### Writing\n\nUse `proto.Block.EncodeRawBlock` with version `54451` on `proto.Buffer` with `Rows` and `Columns` set:\n\n```go\nfunc TestLocalNativeDump(t *testing.T) {\n\tctx := context.Background()\n\t// Testing clickhouse-local.\n\tvar v proto.ColStr\n\tfor _, s := range data {\n\t\tv.Append(s)\n\t}\n\tbuf := new(proto.Buffer)\n\tb := proto.Block{Rows: 2, Columns: 2}\n\trequire.NoError(t, b.EncodeRawBlock(buf, 54451, []proto.InputColumn{\n\t\t{Name: \"title\", Data: v},\n\t\t{Name: \"data\", Data: proto.ColInt64{1, 2}},\n\t}), \"encode\")\n\n\tdir := t.TempDir()\n\tinFile := filepath.Join(dir, \"data.native\")\n\trequire.NoError(t, os.WriteFile(inFile, buf.Buf, 0600), \"write file\")\n\n\tcmd := exec.Command(\"clickhouse-local\", \"local\",\n\t\t\"--logger.console\",\n\t\t\"--log-level\", \"trace\",\n\t\t\"--file\", inFile,\n\t\t\"--input-format\", \"Native\",\n\t\t\"--output-format\", \"JSON\",\n\t\t\"--query\", \"SELECT * FROM table\",\n\t)\n\tout := new(bytes.Buffer)\n\terrOut := new(bytes.Buffer)\n\tcmd.Stdout = out\n\tcmd.Stderr = errOut\n\n\tt.Log(cmd.Args)\n\trequire.NoError(t, cmd.Run(), \"run: %s\", errOut)\n\tt.Log(errOut)\n\n\tv := struct {\n\t\tRows int `json:\"rows\"`\n\t\tData []struct {\n\t\t\tTitle string `json:\"title\"`\n\t\t\tData  int    `json:\"data,string\"`\n\t\t}\n\t}{}\n\trequire.NoError(t, json.Unmarshal(out.Bytes(), \u0026v), \"json\")\n\tassert.Equal(t, 2, v.Rows)\n\tif assert.Len(t, v.Data, 2) {\n\t\tfor i, r := range []struct {\n\t\t\tTitle string `json:\"title\"`\n\t\t\tData  int    `json:\"data,string\"`\n\t\t}{\n\t\t\t{\"Foo\", 1},\n\t\t\t{\"Bar\", 2},\n\t\t} {\n\t\t\tassert.Equal(t, r, v.Data[i])\n\t\t}\n\t}\n}\n```\n\n## TODO\n- [ ] Types\n  - [ ] [Decimal(P, S)](https://clickhouse.com/docs/en/sql-reference/data-types/decimal/) API\n  - [ ] JSON\n  - [ ] SimpleAggregateFunction\n  - [ ] AggregateFunction\n  - [x] Nothing\n  - [x] Interval\n  - [ ] Nested\n  - [ ] [Geo types](https://clickhouse.com/docs/en/sql-reference/data-types/geo/)\n    - [x] Point\n    - [ ] Ring\n    - [ ] Polygon\n    - [ ] MultiPolygon\n- [ ] Improved i/o timeout handling for reading packets from server\n  - [ ] Close connection on context cancellation in all cases\n  - [ ] Ensure that reads can't block forever\n\n## Reference\n* [clickhouse-cpp](https://github.com/ClickHouse/clickhouse-cpp)\n* [clickhouse-go](https://github.com/ClickHouse/clickhouse-go)\n* [python driver](https://github.com/mymarilyn/clickhouse-driver)\n\n## License\nApache License 2.0\n","funding_links":[],"categories":["Go","Language bindings"],"sub_categories":["Golang"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FClickHouse%2Fch-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FClickHouse%2Fch-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FClickHouse%2Fch-go/lists"}