{"id":26826718,"url":"https://github.com/hslam/writer","last_synced_at":"2025-03-30T11:30:45.117Z","repository":{"id":57545303,"uuid":"273935810","full_name":"hslam/writer","owner":"hslam","description":"Package writer implements batch writing.","archived":false,"fork":false,"pushed_at":"2023-05-17T13:45:27.000Z","size":77,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-07-27T18:04:45.486Z","etag":null,"topics":["auto","automatic","batch","batch-writing","concurrency","go","golang","shared-buffer","write"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hslam.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-21T15:49:56.000Z","updated_at":"2022-01-08T10:53:51.000Z","dependencies_parsed_at":"2022-09-16T23:20:40.344Z","dependency_job_id":null,"html_url":"https://github.com/hslam/writer","commit_stats":null,"previous_names":[],"tags_count":1,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hslam%2Fwriter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hslam%2Fwriter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hslam%2Fwriter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hslam%2Fwriter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hslam","download_url":"https://codeload.github.com/hslam/writer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246313665,"owners_count":20757446,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["auto","automatic","batch","batch-writing","concurrency","go","golang","shared-buffer","write"],"created_at":"2025-03-30T11:30:44.281Z","updated_at":"2025-03-30T11:30:45.068Z","avatar_url":"https://github.com/hslam.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# writer\n[![PkgGoDev](https://pkg.go.dev/badge/github.com/hslam/writer)](https://pkg.go.dev/github.com/hslam/writer)\n[![Build Status](https://github.com/hslam/writer/workflows/build/badge.svg)](https://github.com/hslam/writer/actions)\n[![codecov](https://codecov.io/gh/hslam/writer/branch/master/graph/badge.svg)](https://codecov.io/gh/hslam/writer)\n[![Go Report Card](https://goreportcard.com/badge/github.com/hslam/writer?v=7e100)](https://goreportcard.com/report/github.com/hslam/writer)\n[![LICENSE](https://img.shields.io/github/license/hslam/writer.svg?style=flat-square)](https://github.com/hslam/writer/blob/master/LICENSE)\n\nPackage writer implements batch writing.\n\n## Feature\n* Shared Buffer\n* Automatic Batch Writing\n\n##### Write Throughputs\n\n\u003cimg src=\"https://raw.githubusercontent.com/hslam/writer/master/write-throughputs.png\" width = \"480\" height = \"360\" alt=\"write-throughputs\" align=center\u003e\n\n## Get started\n\n### Install\n```\ngo get github.com/hslam/writer\n```\n### Import\n```\nimport \"github.com/hslam/writer\"\n```\n### Usage\n#### Example\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/hslam/writer\"\n\t\"io\"\n\t\"sync\"\n\t\"sync/atomic\"\n\t\"time\"\n)\n\nfunc main() {\n\tRunParallel(1, false)\n\tRunParallel(16, false)\n\tRunParallel(32, false)\n\tRunParallel(1, true)\n\tRunParallel(16, true)\n\tRunParallel(32, true)\n}\n\nfunc RunParallel(num int, batch bool) {\n\tstart := time.Now()\n\tr, w := io.Pipe()\n\tsize := 0\n\tdone := make(chan struct{})\n\tgo func() {\n\t\tbuf := make([]byte, 65536)\n\t\tfor {\n\t\t\tn, err := r.Read(buf)\n\t\t\tif err != nil {\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tsize += n\n\t\t}\n\t\tclose(done)\n\t}()\n\tworker := \u0026Worker{msg: make([]byte, 512)}\n\tif batch {\n\t\tworker.w = writer.NewWriter(w, worker.concurrency, 512*64, false)\n\t} else {\n\t\tworker.w = w\n\t}\n\twg := sync.WaitGroup{}\n\tfor i := 0; i \u003c num; i++ {\n\t\twg.Add(1)\n\t\tgo worker.run(\u0026wg)\n\t}\n\twg.Wait()\n\tworker.w.Close()\n\tw.Close()\n\t\u003c-done\n\tfmt.Printf(\"time - %.1fs,\\tbatch - %t,\\tparallel - %d,\\twrite - %dMByte/s\\n\",\n\t\tfloat64(time.Now().Sub(start))/1E9, batch, num, size/1E6)\n}\n\ntype Worker struct {\n\tcount int64\n\tw     io.WriteCloser\n\tmsg   []byte\n}\n\nfunc (w *Worker) concurrency() int {\n\treturn int(atomic.LoadInt64(\u0026w.count))\n}\n\nfunc (w *Worker) run(wg *sync.WaitGroup) {\n\tdefer wg.Done()\n\tt := time.NewTimer(time.Second)\n\tfor {\n\t\tatomic.AddInt64(\u0026w.count, 1)\n\t\tw.w.Write(w.msg)\n\t\tatomic.AddInt64(\u0026w.count, -1)\n\t\tselect {\n\t\tcase \u003c-t.C:\n\t\t\treturn\n\t\tdefault:\n\t\t}\n\t}\n}\n```\n\n#### Output\n```\ntime - 1.0s,\tbatch - false,\tparallel - 1,\twrite - 554MByte/s\ntime - 1.0s,\tbatch - false,\tparallel - 16,\twrite - 384MByte/s\ntime - 1.0s,\tbatch - false,\tparallel - 32,\twrite - 384MByte/s\ntime - 1.0s,\tbatch - true,\tparallel - 1,\twrite - 485MByte/s\ntime - 1.0s,\tbatch - true,\tparallel - 16,\twrite - 962MByte/s\ntime - 1.0s,\tbatch - true,\tparallel - 32,\twrite - 2013MByte/s\n```\n\n### License\nThis package is licensed under a MIT license (Copyright (c) 2020 Meng Huang)\n\n\n### Author\nwriter was written by Meng Huang.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhslam%2Fwriter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhslam%2Fwriter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhslam%2Fwriter/lists"}