{"id":19779388,"url":"https://github.com/ranjanrak/clickhouse-tickstore","last_synced_at":"2025-07-07T05:07:15.168Z","repository":{"id":42175952,"uuid":"486092774","full_name":"ranjanrak/clickhouse-tickstore","owner":"ranjanrak","description":"Go package to store real time streaming websocket data in clickhouse using queuing and bulk insert.","archived":false,"fork":false,"pushed_at":"2023-03-04T06:48:25.000Z","size":24,"stargazers_count":23,"open_issues_count":1,"forks_count":9,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-30T21:39:22.097Z","etag":null,"topics":["clickhouse","go-channel","go-routine","stream-processing"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ranjanrak.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-27T07:37:13.000Z","updated_at":"2025-03-25T21:00:10.000Z","dependencies_parsed_at":"2024-06-19T01:56:28.453Z","dependency_job_id":null,"html_url":"https://github.com/ranjanrak/clickhouse-tickstore","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/ranjanrak/clickhouse-tickstore","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ranjanrak%2Fclickhouse-tickstore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ranjanrak%2Fclickhouse-tickstore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ranjanrak%2Fclickhouse-tickstore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ranjanrak%2Fclickhouse-tickstore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ranjanrak","download_url":"https://codeload.github.com/ranjanrak/clickhouse-tickstore/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ranjanrak%2Fclickhouse-tickstore/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264016742,"owners_count":23544627,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clickhouse","go-channel","go-routine","stream-processing"],"created_at":"2024-11-12T05:34:34.755Z","updated_at":"2025-07-07T05:07:15.142Z","avatar_url":"https://github.com/ranjanrak.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# clickhouse-tickstore\n\n[![Run Tests](https://github.com/ranjanrak/clickhouse-tickstore/actions/workflows/go-test.yml/badge.svg)](https://github.com/ranjanrak/clickhouse-tickstore/actions/workflows/go-test.yml)\n[![Go Reference](https://pkg.go.dev/badge/github.com/ranjanrak/clickhouse-tickstore.svg)](https://pkg.go.dev/github.com/ranjanrak/clickhouse-tickstore)\n\nGo package to store real time [streaming websocket data](https://kite.trade/docs/connect/v3/websocket/) in [clickhouse](https://clickhouse.tech/) using queuing and bulk insert based on go-routine and channels.\n\n## Installation\n\n```\ngo get -u github.com/ranjanrak/clickhouse-tickstore\n```\n\n## Usage\n\n```go\npackage main\n\nimport (\n\ttickstore \"github.com/ranjanrak/clickhouse-tickstore\"\n)\nfunc main() {\n    // Create new ticker instance\n    tickClient := tickstore.New(tickstore.ClientParam{\n        // Send DSN as per your clickhouse DB setup.\n        // visit https://github.com/ClickHouse/clickhouse-go#dsn to know more\n        DBSource:    \"\",\n        ApiKey:      \"your_api_key\",\n        AccessToken: \"your_access_token\",\n        TokenList:   []uint32{633601, 895745, 1723649, 3050241, 975873, 969473, 3721473, 738561, 969473},\n        DumpSize:    5000,\n\t})\n    // Start the ticker instance\n    // Nothing will run after this\n    tickClient.StartTicker()\n\n    // Fetch minute candle OHLC data\n    timeStart := time.Date(2022, 5, 11, 9, 51, 0, 0, time.Local)\n    timeEnd := time.Date(2022, 5, 11, 10, 02, 0, 0, time.Local)\n    candles, err := tickClient.FetchCandle(633601, timeStart, timeEnd)\n    if err != nil {\n        log.Fatalf(\"Error fetching candle data: %v\", err)\n    }\n    fmt.Printf(\"%+v\\n\", candles)\n}\n\n```\n\n## Response\n\n\u003e FetchCandle(633601, timeStart, timeEnd)\n\n```\n[{InstrumentToken:633601 TimeStamp:2022-05-11 09:51:00 +0530 IST Open:156.65 High:156.75 Low:156.45 Close:156.65}\n{InstrumentToken:633601 TimeStamp:2022-05-11 09:52:00 +0530 IST Open:156.75 High:156.95 Low:156.7 Close:156.75}\n{InstrumentToken:633601 TimeStamp:2022-05-11 09:53:00 +0530 IST Open:156.75 High:156.75 Low:156.2 Close:156.3}\n{InstrumentToken:633601 TimeStamp:2022-05-11 09:54:00 +0530 IST Open:156.3 High:156.3 Low:156 Close:156.1}\n......]\n```\n\n## Example\n\n```sql\nSELECT *\nFROM tickdata\nFINAL\nWHERE (instrument_token = 633601) AND\n(timestamp \u003e= toDateTime('2022-04-22 13:23:00', 'Asia/Calcutta')) AND\n(timestamp \u003c= toDateTime('2022-04-22 13:25:00', 'Asia/Calcutta'))\nORDER BY timestamp ASC\n```\n\n```sql\n\nQuery id: 8e356516-107c-4012-948b-df90e49e9906\n\n┌─instrument_token─┬───────────timestamp─┬──price─┐\n│           633601 │ 2022-04-22 13:23:00 │ 174.65 │\n│           633601 │ 2022-04-22 13:23:01 │  174.7 │\n│           633601 │ 2022-04-22 13:23:02 │ 174.65 │\n│           633601 │ 2022-04-22 13:23:04 │  174.7 │\n│           633601 │ 2022-04-22 13:23:05 │ 174.65 │\n│           633601 │ 2022-04-22 13:23:06 │  174.7 │\n│           633601 │ 2022-04-22 13:23:08 │  174.7 │\n│           633601 │ 2022-04-22 13:23:09 │  174.7 │\n│           633601 │ 2022-04-22 13:23:10 │  174.7 │\n│           633601 │ 2022-04-22 13:23:13 │  174.7 │\n│           633601 │ 2022-04-22 13:23:14 │ 174.65 │\n│           633601 │ 2022-04-22 13:23:15 │ 174.65 │\n│           633601 │ 2022-04-22 13:23:16 │  174.7 │\n│           633601 │ 2022-04-22 13:23:17 │ 174.65 │\n│           633601 │ 2022-04-22 13:23:19 │  174.7 │\n│           633601 │ 2022-04-22 13:23:21 │ 174.65 │\n│           633601 │ 2022-04-22 13:23:24 │ 174.65 │\n│           633601 │ 2022-04-22 13:23:25 │  174.7 │\n│           633601 │ 2022-04-22 13:23:26 │  174.7 │\n│           633601 │ 2022-04-22 13:23:27 │ 174.65 │\n│           633601 │ 2022-04-22 13:23:28 │ 174.65 │\n│           633601 │ 2022-04-22 13:23:29 │  174.7 │\n│           633601 │ 2022-04-22 13:23:31 │  174.7 │\n│           633601 │ 2022-04-22 13:23:32 │  174.7 │\n│           633601 │ 2022-04-22 13:23:33 │  174.7 │\n│           633601 │ 2022-04-22 13:23:35 │  174.7 │\n│           633601 │ 2022-04-22 13:23:36 │  174.7 │\n|           ...... | ..................  | ...... |\n\n84 rows in set. Elapsed: 0.006 sec. Processed 8.19 thousand rows, 98.30 KB (1.28 million rows/s., 15.37 MB/s.)\n```\n\n### Fetch OHLC candle data for any given interval using [CTE](https://clickhouse.com/docs/en/sql-reference/statements/select/with/) and [aggregate function](https://clickhouse.com/docs/en/sql-reference/aggregate-functions/reference/grouparray/)\n\n```sql\nWITH price_select AS (SELECT price\nFROM tickdata\nFINAL\nWHERE (instrument_token = 975873) AND\n(timestamp \u003e= toDateTime('2022-05-02 14:47:00')) AND\n(timestamp \u003c= toDateTime('2022-05-02 14:47:59'))\nORDER BY timestamp ASC)\nSELECT groupArray(price)[1] AS open,\nmax(price) AS high,\nmin(price) AS low,\ngroupArray(price)[-1] AS close FROM price_select;\n```\n\n```sql\nQuery id: 98d92c26-e054-4f0a-8448-064bc0d939a0\n┌───open─┬───high─┬───low─┬─close─┐\n│ 252.25 │ 252.35 │ 252.1 │ 252.2 │\n└────────┴────────┴───────┴───────┘\n```\n\n### Create base minute candle OHLC\n\n```sql\nSELECT\n    instrument_token,\n    time_minute,\n    groupArray(price)[1] AS open,\n    max(price) AS high,\n    min(price) AS low,\n    groupArray(price)[-1] AS close\nFROM\n(\n    SELECT\n        instrument_token,\n        toStartOfMinute(timestamp) AS time_minute,\n        price\n    FROM tickdata\n    WHERE (instrument_token = 975873) AND\n    (timestamp \u003e= toDateTime('2022-05-02 14:47:00')) AND\n    (timestamp \u003c= toDateTime('2022-05-02 14:59:59'))\n)\nGROUP BY (instrument_token, time_minute)\nORDER BY time_minute ASC\n```\n\n```sql\nQuery id: 2ba74fd2-6047-42c9-9436-be8987a5d3a9\n\n┌─instrument_token─┬─────────time_minute─┬───open─┬───high─┬────low─┬──close─┐\n│           975873 │ 2022-05-02 14:47:00 │ 252.25 │ 252.35 │  252.1 │  252.2 │\n│           975873 │ 2022-05-02 14:48:00 │  252.2 │  252.3 │  251.9 │ 252.25 │\n│           975873 │ 2022-05-02 14:49:00 │  252.3 │  252.3 │ 252.05 │  252.1 │\n│           975873 │ 2022-05-02 14:50:00 │  252.1 │ 252.45 │ 252.05 │ 252.35 │\n│           975873 │ 2022-05-02 14:51:00 │  252.2 │ 252.45 │  252.2 │ 252.35 │\n│           975873 │ 2022-05-02 14:52:00 │ 252.35 │ 252.35 │    252 │    252 │\n│           975873 │ 2022-05-02 14:53:00 │    252 │ 253.15 │    252 │  252.8 │\n│           975873 │ 2022-05-02 14:54:00 │  252.8 │  253.2 │  252.7 │  252.8 │\n│           975873 │ 2022-05-02 14:55:00 │  252.8 │  253.4 │ 252.75 │  253.3 │\n│           975873 │ 2022-05-02 14:56:00 │ 253.25 │  253.4 │    253 │  253.1 │\n│           975873 │ 2022-05-02 14:57:00 │  253.1 │  253.1 │ 252.85 │ 252.85 │\n│           975873 │ 2022-05-02 14:58:00 │  252.8 │ 253.05 │  252.5 │  252.6 │\n│           975873 │ 2022-05-02 14:59:00 │  252.6 │  253.4 │  252.6 │  253.4 │\n└──────────────────┴─────────────────────┴────────┴────────┴────────┴────────┘\n```\n\n### Create candle_data materialized views to store minute OHLC\n\n```sql\nCREATE MATERIALIZED VIEW candle_data\nENGINE = ReplacingMergeTree\nORDER BY (instrument_token, time_minute)\nPRIMARY KEY (instrument_token, time_minute) POPULATE AS\nSELECT\n    instrument_token,\n    time_minute,\n    groupArray(price)[1] AS open,\n    max(price) AS high,\n    min(price) AS low,\n    groupArray(price)[-1] AS close\nFROM\n(\n    SELECT\n        instrument_token,\n        toStartOfMinute(timestamp) AS time_minute,\n        price\n    FROM tickdata\n)\nGROUP BY (instrument_token, time_minute)\n```\n\n### Create n-minute candle OHLC\n\n```sql\nWITH w AS (\n    SELECT open, high, low, close,\n    time, intDiv(toUnixTimestamp(toDateTime('2022-05-10 14:59:00')) - toUnixTimestamp(time), 60*n(in minutes of interval)) as grp\n    FROM tickdata\n    WHERE (instrument_token = 975873) AND\n    (timestamp \u003e= toDateTime('2022-05-02 14:47:00')) AND\n    (timestamp \u003c= toDateTime('2022-05-10 14:59:00')) order by time asc\n)\nSELECT\n    first_value(time) as time, first_value(open) as open,\n    max(high) as high, min(low) as low,\n    last_value(close) as close\nFROM\n    w\nGROUP BY grp ORDER BY time asc;\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Franjanrak%2Fclickhouse-tickstore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Franjanrak%2Fclickhouse-tickstore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Franjanrak%2Fclickhouse-tickstore/lists"}