Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/vahid-sohrabloo/chconn
Low-level ClickHouse database driver for Golang
https://github.com/vahid-sohrabloo/chconn
clickhouse fast go golang tcp
Last synced: 3 months ago
JSON representation
Low-level ClickHouse database driver for Golang
- Host: GitHub
- URL: https://github.com/vahid-sohrabloo/chconn
- Owner: vahid-sohrabloo
- License: mit
- Created: 2020-03-04T07:48:52.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2024-04-06T20:24:38.000Z (3 months ago)
- Last Synced: 2024-04-06T21:27:21.091Z (3 months ago)
- Topics: clickhouse, fast, go, golang, tcp
- Language: Go
- Homepage:
- Size: 1.62 MB
- Stars: 147
- Watchers: 6
- Forks: 11
- Open Issues: 8
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Lists
- awesome-clickhouse - vahid-sohrabloo/chconn - chconn is a low-level ClickHouse database driver for Golang with a connection pool. (Language bindings / Golang)
README
[](https://pkg.go.dev/github.com/vahid-sohrabloo/chconn/v2)
[](https://codecov.io/gh/vahid-sohrabloo/chconn)
[](https://goreportcard.com/report/github.com/vahid-sohrabloo/chconn/v2)
[](https://github.com/vahid-sohrabloo/chconn/actions)
[](https://app.fossa.com/projects/git%2Bgithub.com%2Fvahid-sohrabloo%2Fchconn?ref=badge_shield)
# chconn - ClickHouse low level Driver
chconn is a pure generic Go (1.18+) driver for [ClickHouse](https://clickhouse.com/) that use Native protocol
chconn aims to be low-level, fast, and performant.
For comparison with other libraries, please see https://github.com/vahid-sohrabloo/go-ch-benchmark and https://github.com/go-faster/ch-bench#benchmarks
If you have any suggestion or comment, please feel free to open an issue
## Example Usage
```go
package main
import (
"context"
"fmt"
"os"
"time"
"github.com/vahid-sohrabloo/chconn/v2/chpool"
"github.com/vahid-sohrabloo/chconn/v2/column"
)
func main() {
conn, err := chpool.New(os.Getenv("DATABASE_URL"))
if err != nil {
panic(err)
}
defer conn.Close()
// to check if the connection is alive
err = conn.Ping(context.Background())
if err != nil {
panic(err)
}
err = conn.Exec(context.Background(), `DROP TABLE IF EXISTS example_table`)
if err != nil {
panic(err)
}
err = conn.Exec(context.Background(), `CREATE TABLE example_table (
uint64 UInt64,
uint64_nullable Nullable(UInt64)
) Engine=Memory`)
if err != nil {
panic(err)
}
col1 := column.New[uint64]()
col2 := column.New[uint64]().Nullable()
rows := 1_000_0000 // Ten million rows - insert in 10 times
numInsert := 10
col1.SetWriteBufferSize(rows)
col2.SetWriteBufferSize(rows)
startInsert := time.Now()
for i := 0; i < numInsert; i++ {
for y := 0; y < rows; y++ {
col1.Append(uint64(i))
if i%2 == 0 {
col2.Append(uint64(i))
} else {
col2.AppendNil()
}
}
ctxInsert, cancelInsert := context.WithTimeout(context.Background(), time.Second*30)
// insert data
err = conn.Insert(ctxInsert, "INSERT INTO example_table (uint64,uint64_nullable) VALUES", col1, col2)
if err != nil {
cancelInsert()
panic(err)
}
cancelInsert()
}
fmt.Println("inserted 10M rows in ", time.Since(startInsert))
// select data
col1Read := column.New[uint64]()
col2Read := column.New[uint64]().Nullable()
ctxSelect, cancelSelect := context.WithTimeout(context.Background(), time.Second*30)
defer cancelSelect()
startSelect := time.Now()
selectStmt, err := conn.Select(ctxSelect, "SELECT uint64,uint64_nullable FROM example_table", col1Read, col2Read)
if err != nil {
panic(err)
}
// make sure the stmt close after select. but it's not necessary
defer selectStmt.Close()
var col1Data []uint64
var col2DataNil []bool
var col2Data []uint64
// read data block by block
// for more information about block, see: https://clickhouse.com/docs/en/development/architecture/#block
for selectStmt.Next() {
col1Data = col1Data[:0]
col1Data = col1Read.Read(col1Data)
col2DataNil = col2DataNil[:0]
col2DataNil = col2Read.ReadNil(col2DataNil)
col2Data = col2Data[:0]
col2Data = col2Read.Read(col2Data)
}
// check errors
if selectStmt.Err() != nil {
panic(selectStmt.Err())
}
fmt.Println("selected 10M rows in ", time.Since(startSelect))
}
```
```
inserted 10M rows in 1.206666188s
selected 10M rows in 880.505004ms
```
**For moe information**, please see the [documentation](https://github.com/vahid-sohrabloo/chconn/wiki)
## Features
* Generics (go1.18) for column types
* Connection pool with after-connect hook for arbitrary connection setup similar to pgx (thanks @jackc)
* Support DSN and Query connection string (thanks @jackc)
* Support All ClickHouse data types
* Read and write data in column-oriented (like ClickHouse)
* Do not use `interface{}` , `reflect`
* Batch select and insert
* Full TLS connection control
* Read raw binary data
* Supports profile and progress
* database url connection very like pgx (thanks @jackc)
* Code generator for Insert
* Support LZ4 and ZSTD compression protocol
* Support execution telemetry streaming profiles and progress
## Supported types
* UInt8, UInt16, UInt32, UInt64, UInt128, UInt256
* Int8, Int16, Int32, Int64, Int128, Int256
* Date, Date32, DateTime, DateTime64
* Decimal32, Decimal64, Decimal128, Decimal256
* IPv4, IPv6
* String, FixedString(N)
* UUID
* Array(T)
* Enums
* LowCardinality(T)
* Map(K, V)
* Tuple(T1, T2, ..., Tn)
* Nullable(T)
* Point, Ring, Polygon, MultiPolygon
# Benchmarks
the source code of this benchmark here
https://github.com/vahid-sohrabloo/go-ch-benchmark
```
name \ time/op chconn chgo go-clickhouse uptrace
TestSelect100MUint64-16 150ms 154ms 8019ms 3045ms
TestSelect10MString-16 271ms 447ms 969ms 822ms
TestInsert10M-16 198ms 514ms 561ms 304ms
name \ alloc/op chconn chgo go-clickhouse uptrace
TestSelect100MUint64-16 111kB 262kB 3202443kB 800941kB
TestSelect10MString-16 1.63MB 1.79MB 1626.51MB 241.03MB
TestInsert10M-16 26.0MB 283.7MB 1680.4MB 240.2MB
name \ allocs/op chconn chgo go-clickhouse uptrace
TestSelect100MUint64-16 35.0 6683.0 200030937.0 100006069.0
TestSelect10MString-16 49.0 1748.0 30011991.0 20001120.0
TestInsert10M-16 26.0 80.0 224.0 50.0
```
## License
[](https://app.fossa.com/projects/git%2Bgithub.com%2Fvahid-sohrabloo%2Fchconn?ref=badge_large)