{"id":20834959,"url":"https://github.com/hstreamdb/hstreamdb-go","last_synced_at":"2025-05-08T02:23:03.211Z","repository":{"id":37099359,"uuid":"459002869","full_name":"hstreamdb/hstreamdb-go","owner":"hstreamdb","description":"Go Client for HStreamDB","archived":false,"fork":false,"pushed_at":"2025-03-13T00:55:14.000Z","size":596,"stargazers_count":2,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-15T00:51:12.731Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hstreamdb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-02-14T03:29:01.000Z","updated_at":"2024-07-10T01:15:17.000Z","dependencies_parsed_at":"2023-01-31T11:00:43.771Z","dependency_job_id":"1d0b8a02-175a-4211-a607-6fcf959f5ed9","html_url":"https://github.com/hstreamdb/hstreamdb-go","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hstreamdb%2Fhstreamdb-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hstreamdb%2Fhstreamdb-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hstreamdb%2Fhstreamdb-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hstreamdb%2Fhstreamdb-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hstreamdb","download_url":"https://codeload.github.com/hstreamdb/hstreamdb-go/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252984974,"owners_count":21835891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-18T00:22:06.245Z","updated_at":"2025-05-08T02:23:03.167Z","avatar_url":"https://github.com/hstreamdb.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# hstreamdb-go\nGo Client for HStreamDB\n\n**Note that the current release is not suitable for production use - APIs are not yet stable and the package has not been thoroughly tested in real-life use.**\n\n## Content\n\n- [Installation](#installation)\n- [Example Usage](#example-usage)\n  - [Connect to HServer](#connect-to-hserver)\n  - [Work with Streams](#work-with-streams)\n  - [Write to Stream](#write-to-stream)\n  - [Work with Subscriptions](#work-with-subscriptions)\n  - [Consume from Subscription](#consume-from-subscription)\n  - [Work with ShardReader](#Work_with_ShardReader)\n\n## Installation\n\n**Go 1.19** or later is required.\n\nAdd the package to your project dependencies (go.mod).\n\n```shell\ngo get github.com/hstreamdb/hstreamdb-go\n```\n\n## Example Usage\n### Connect to HServer\n\n```go\nimport (\n    \"log\"\n    \"github.com/hstreamdb/hstreamdb-go/hstream\"\n)\n\nfunc main() {\n\tserverUrl := \"hstream://localhost:6580,localhost:6581,localhost:6582\"\n\tclient, err := hstream.NewHStreamClient(serverUrl)\n\tif err != nil {\n\t\tlog.Fatalf(\"Creating client error: %s\", err)\n\t}\n\t// do sth.\n\tclient.Close()\n}\n```\n\n### Work with Streams\n\n```go\nimport (\n    \"log\"\n    \"github.com/hstreamdb/hstreamdb-go/hstream\"\n)\n\nfunc main() {\n\t// -------------- connect to server first --------------------\n\t// Create a new stream with 1 replica, 5 shards, set the data retention to 1800s.\n\terr := client.CreateStream(\"testStream\",\n             hstream.WithReplicationFactor(1),\n             hstream.WithShardCount(5),\n             hstream.EnableBacklog(1800))\n\tif err != nil {\n\t\tlog.Fatalf(\"Creating stream error: %s\", err)\n\t}\n\n\t// List all streams\n\tstreams, err := client.ListStreams()\n\tif err != nil {\n\t\tlog.Fatalf(\"Listing streams error: %s\", err)\n\t}\n\tfor _, stream := range streams {\n\t\tlog.Printf(\"Stream: %+v\\n\", stream)\n\t}\n\t\n\t// Delete stream\n\terr := client.DeleteStream(\"testStream\", \n\t\thstream.EnableForceDelete, \n\t\thstream.EnableIgnoreNoneExist)\n\tif err != nil {\n\t\tlog.Fatalf(\"Delete stream error: %s\", err)\n\t}\n}\n```\n\n### Write to Stream\n\n#### Write RawRecord\n\n```go\nimport (\n    \"log\"\n    \"strconv\"\n    \"github.com/hstreamdb/hstreamdb-go/hstream\"\n    \"github.com/hstreamdb/hstreamdb-go/hstream/Record\"\n)\n\nfunc main() {\n\t//------------- connect to server and create related stream first --------------------\n\tproducer, err := client.NewProducer(\"testStream\")\n\tif err != nil {\n\t\tlog.Fatalf(\"Create producer error: %s\", err)\n\t}\n\tdefer producer.Stop()\n\n\tres := make([]hstream.AppendResult, 0, 100)\n\tfor i := 0; i \u003c 100; i++ {\n\t\trawRecord, err := Record.NewHStreamRawRecord(\"key-1\", []byte(\"test-value\"+strconv.Itoa(i)))\n\t\tif err != nil {\n\t\t\tlog.Fatalf(\"Creating rawRecord error: %s\", err)\n\t\t}\n\t\tr := producer.Append(rawRecord)\n\t\tres = append(res, r)\n\t}\n\n\tfor _, r := range res {\n\t\tresp, err := r.Ready()\n\t\tif err != nil {\n\t\t\tlog.Printf(\"Append error: %s\", err)\n\t\t} else {\n\t\t\tlog.Printf(\"Append response: %s\", resp)\n\t\t}\n\t}\n}\n```\n\n#### Write HRecord\n\n```go\nimport (\n    \"log\"\n    \"strconv\"\n    \"github.com/hstreamdb/hstreamdb-go/hstream\"\n    \"github.com/hstreamdb/hstreamdb-go/hstream/Record\"\n)\n\nfunc main() {\n\t//------------- connect to server and create related stream first --------------------\n\tproducer, err := client.NewProducer(\"testStream\")\n\tif err != nil {\n\t\tlog.Fatalf(\"Create producer error: %s\", err)\n\t}\n\tdefer producer.Stop()\n\n\tpayload := map[string]interface{}{\n\t\t\"key1\": \"value1\",\n\t\t\"key2\": 123,\n\t\t\"key3\": struct {\n\t\t\tname string\n\t\t\tage  int\n\t\t}{\n\t\t\tname: \"John\",\n\t\t\tage:  30,\n\t\t},\n\t}\n\n\thRecord, err := Record.NewHStreamHRecord(\"testStream\", payload)\n\tif err != nil {\n\t\tlog.Fatalf(\"Creating hRecord error: %s\", err)\n\t}\n\tvalue := producer.Append(hRecord)\n\tif resp, err := value.Ready(); err != nil {\n\t\tlog.Printf(\"Append error: %s\", err)\n\t} else {\n\t\tlog.Printf(\"Append response: %s\", resp)\n\t}\n}\n```\n\n#### Batch Writter\n\n```go\nimport (\n    \"fmt\"\n    \"log\"\n    \"sync\"\n    \"github.com/hstreamdb/hstreamdb-go/hstream\"\n    \"github.com/hstreamdb/hstreamdb-go/hstream/Record\"\n)\n\nfunc main() {\n\t//------------- connect to server and create related stream first --------------------\n\tproducer, err := client.NewBatchProducer(\"testStream\",\n\t\t// optional: set batch size and max batch bytes trigger\n\t\thstream.WithBatch(10, 1000),\n\t\t// optional: set timeout trigger\n\t\thstream.TimeOut(-1), \n\t\t// optional: set client compression\n\t\thstream.WithCompression(compression.Zstd), \n\t\t// optional: set flow control\n\t\thstream.WithFlowControl(80 * 1024 * 1024)) \n\tdefer producer.Stop()\n\n\tkeys := []string{\"test-key1\", \"test-key2\", \"test-key3\"}\n\trids := sync.Map{}\n\twg := sync.WaitGroup{}\n\twg.Add(3)\n\tfor _, key := range keys {\n\t\tgo func(key string) {\n\t\t\tresult := make([]hstream.AppendResult, 0, 100)\n\t\t\tfor i := 0; i \u003c 100; i++ {\n\t\t\t\trawRecord, _ := Record.NewHStreamRawRecord(\"key-1\", []byte(fmt.Sprintf(\"test-value-%s-%d\", key, i)))\n\t\t\t\tr := producer.Append(rawRecord)\n\t\t\t\tresult = append(result, r)\n\t\t\t}\n\t\t\trids.Store(key, result)\n\t\t\twg.Done()\n\t\t}(key)\n\t}\n\n\twg.Wait()\n\trids.Range(func(key, value interface{}) bool {\n\t\tk := key.(string)\n\t\tres := value.([]hstream.AppendResult)\n\t\tfor idx, r := range res {\n\t\t\tresp, err := r.Ready()\n\t\t\tif err != nil {\n\t\t\t\tlog.Printf(\"write error: %s\\n\", err.Error())\n\t\t\t}\n\t\t\tlog.Printf(\"[key: %s]: record[%d]=%s\\n\", k, idx, resp.String())\n\t\t}\n\t\treturn true\n\t})\n}\n```\n\n### Work with Subscriptions\n\n```go\nimport (\n    \"log\"\n    \"github.com/hstreamdb/hstreamdb-go/hstream\"\n)\n\nfunc main() {\n\t// -------------- connect to server and create related stream first --------------------\n\t// Create a new subscription\n\tstreamName := \"testStream\"\n\tsubId := \"SubscriptionId\"\n\terr := client.CreateSubscription(subId, streamName, \n\t\thstream.WithAckTimeout(60), \n\t\thstream.WithOffset(hstream.LATEST))\n\n\t// List all subscriptions\n\tsubs, err := client.ListSubscriptions()\n\tif err != nil {\n\t\tlog.Fatalf(\"Listing subscriptions error: %s\", err)\n\t}\n\tfor _, sub := range subs {\n\t\tlog.Printf(\"Subscription: %+v\", sub)\n\t}\n}\n```\n\n### Consume from Subscription\n\n```go\nimport (\n    \"log\"\n    \"github.com/hstreamdb/hstreamdb-go/hstream\"\n    \"github.com/hstreamdb/hstreamdb-go/hstream/Record\"\n)\n\nfunc main() {\n\t// ------- connect to server and create related stream and subscription first -------\n\tstreamName := \"testStream\"\n\tsubId := \"SubscriptionId\"\n\tconsumer := client.NewConsumer(\"consumer-1\", subId)\n\tdefer consumer.Stop()\n\n\tdataCh := consumer.StartFetch()\n\tfetchRes := make([]Record.RecordId, 0, 100)\n\tfor res := range dataCh {\n\t\tif res.Err != nil {\n\t\t\tlog.Printf(\"Fetch error: %s\\n\", res.Err)\n\t\t\tcontinue\n\t\t}\n\n\t\tfor _, record := range res.Result {\n\t\t\trid := record.GetRecordId()\n\t\t\tlog.Printf(\"receive recordId: %s\\n\", rid.String())\n\t\t\tfetchRes = append(fetchRes, rid)\n\t\t\trecord.Ack()\n\t\t}\n\t\tif len(fetchRes) == 100 {\n\t\t\tbreak\n\t\t}\n\t}\n}\n```\n\n### Work with ShardReader\n\n```go\nimport (\n    \"log\"\n    \"github.com/hstreamdb/hstreamdb-go/hstream\"\n    \"github.com/hstreamdb/hstreamdb-go/hstream/Record\"\n)\n\nfunc main() {\n    // ------- connect to server and create related stream first -------\n    shards, err := client.ListShards(streamName)\n    if err != nil {\n        log.Fatalf(\"List Shards error: %s\", err)\n    }\n    \n    readerId := \"reader\"\n    reader, err := client.NewShardReader(streamName, readerId, shards[0].ShardId, \n\t\thstream.WithShardOffset(hstream.EarliestShardOffset), \n\t\thstream.WithReaderTimeout(100),hstream.WithMaxRecords(10))\n    if err != nil {\n        log.Fatalf(\"Create shard reader error: %s\", err)\n    }\n    defer client.DeleteShardReader(shards[0].ShardId, readerId)\n    defer reader.Close()\n    \n    // ------- make sure that data has been written to the target shard ------\n    readRecords := make([]Record.ReceivedRecord, 0, totalRecords)\n    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)\n    defer cancel()\n    for {\n        res, err := reader.Read(ctx)\n        s.NoError(err)\n        readRecords = append(readRecords, res...)\n        if len(readRecords) \u003e= totalRecords {\n            break\n        }\n    }\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhstreamdb%2Fhstreamdb-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhstreamdb%2Fhstreamdb-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhstreamdb%2Fhstreamdb-go/lists"}