{"id":17550900,"url":"https://github.com/shoshinnikita/go-disk-buffer","last_synced_at":"2025-04-24T02:10:08.010Z","repository":{"id":57492789,"uuid":"203040496","full_name":"ShoshinNikita/go-disk-buffer","owner":"ShoshinNikita","description":"This package helps to work with huge amount of data, which cannot be stored in RAM","archived":false,"fork":false,"pushed_at":"2022-11-22T12:30:50.000Z","size":229,"stargazers_count":43,"open_issues_count":3,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-24T02:09:53.909Z","etag":null,"topics":["buffer","disk","go","golang","ram"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ShoshinNikita.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-18T18:15:27.000Z","updated_at":"2024-10-30T19:33:33.000Z","dependencies_parsed_at":"2023-01-22T01:57:44.297Z","dependency_job_id":null,"html_url":"https://github.com/ShoshinNikita/go-disk-buffer","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShoshinNikita%2Fgo-disk-buffer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShoshinNikita%2Fgo-disk-buffer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShoshinNikita%2Fgo-disk-buffer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShoshinNikita%2Fgo-disk-buffer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ShoshinNikita","download_url":"https://codeload.github.com/ShoshinNikita/go-disk-buffer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250546083,"owners_count":21448260,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["buffer","disk","go","golang","ram"],"created_at":"2024-10-21T04:44:11.843Z","updated_at":"2025-04-24T02:10:07.970Z","avatar_url":"https://github.com/ShoshinNikita.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Go Disk Buffer\n\nPackage `buffer` helps to work with huge amount of data, which cannot be stored in RAM. Instead of keeping all data in RAM `buffer.Buffer` can store the data on a disk in a temporary file.\n\n**Features:**\n\n- `buffer.Buffer` is compatible with `io.Reader` and `io.Writer` interfaces\n- `buffer.Buffer` can replace `bytes.Buffer` (except some methods – check [Unavailable methods](#unavailable-methods))\n- You can encrypt data on a disk. Just use `Buffer.EnableEncryption` method\n\n**Notes:**\n\n- It is **not** recommended to use zero value of `buffer.Buffer`. Use `buffer.NewBuffer()` or `buffer.NewBufferWithMaxMemorySize()` instead\n- `buffer.Buffer` is **not** thread-safe!\n- `buffer.Buffer` uses a directory returned by `os.TempDir()` to store temp files. You can change the directory with `Buffer.ChangeTempDir` method\n\n##\n\n- [Example](#example)\n- [Benchmark](#benchmark)\n- [Available methods](#available-methods)\n  - [Read](#read)\n  - [Write](#write)\n  - [Other](#other)\n- [Unavailable methods](#unavailable-methods)\n  - [Can be added](#can-be-added)\n  - [Won't be adeed](#wont-be-adeed)\n\n## Example\n\nWith `bytes.Buffer`\n\n```go\npackage main\n\nimport (\n    \"bytes\"\n    \"fmt\"\n)\n\nfunc main() {\n    b := bytes.Buffer{}\n    // b := bytes.NewBuffer(nil)\n    // b := bytes.NewBufferString(\"\")\n\n    b.Write([]byte(\"Hello,\"))\n    b.WriteByte(' ')\n    b.Write([]byte(\"World!\"))\n\n    data := b.Next(13)\n    fmt.Println(string(data)) // \"Hello, World!\"\n}\n```\n\nWith `github.com/ShoshinNikita/go-disk-buffer.Buffer`\n\n```go\npackage main\n\nimport (\n    \"fmt\"\n\n    buffer \"github.com/ShoshinNikita/go-disk-buffer\"\n)\n\nfunc main() {\n    b := buffer.NewBufferWithMaxMemorySize(7) // store only 7 bytes in RAM\n    // b := buffer.NewBuffer(nil)\n    // b := buffer.NewBufferString(\"\")\n\n    b.Write([]byte(\"Hello,\"))\n    b.WriteByte(' ')\n    b.Write([]byte(\"World!\")) // will be stored on a disk in a temporary file\n\n    data := b.Next(13)\n    fmt.Println(string(data)) // \"Hello, World!\"\n}\n```\n\n## Benchmark\n\n**CPU:** Intel Core i7-3630QM  \n**RAM:** 8 GB  \n**Disk:** HDD, 5400 rpm\n\n```\nBuffer_size_is_greater_than_data/bytes.Buffer-8     1000       1591091 ns/op      10043209 B/op     36 allocs/op\nBuffer_size_is_greater_than_data/utils.Buffer-8     1000       1346077 ns/op       6901679 B/op     26 allocs/op\n\nBuffer_size_is_equal_to_data/bytes.Buffer-8         1000       1760100 ns/op      10043195 B/op     36 allocs/op\nBuffer_size_is_equal_to_data/utils.Buffer-8         2000       1357077 ns/op       7434159 B/op     27 allocs/op\n\nBuffer_size_is_less_than_data/bytes.Buffer-8          50      36522090 ns/op     177848123 B/op     53 allocs/op\nBuffer_size_is_less_than_data/utils.Buffer-8          10     110406320 ns/op     112327659 B/op     62 allocs/op\n```\n\n## Available methods\n\n### Read\n\n- `Read(p []byte) (n int, err error)`\n- `ReadByte() (byte, error)`\n- `Next(n int) []byte`\n- `WriteTo(w io.Writer) (n int64, err error)`\n\n### Write\n\n- `Write(p []byte) (n int, err error)`\n- `WriteByte(c byte) error`\n- `WriteRune(r rune) (n int, err error)`\n- `WriteString(s string) (n int, err error)`\n- `ReadFrom(r io.Reader) (n int64, err error)`\n\n### Other\n\n- `Len() int`\n- `Cap() int` – equal to `Len()` method\n- `Reset()`\n\n## Unavailable methods\n\n### Can be added\n\n- `ReadBytes(delim byte) (line []byte, err error)`\n- `ReadString(delim byte) (line string, err error)`\n- `ReadRune() (r rune, size int, err error)` – **help wanted** (check `Buffer.readRune()` method)\n\n### Won't be adeed\n\n- `Bytes() []byte`\n\n  **Reason:** `go-disk-buffer` was created to store a huge amount of data. If your data can fit in RAM, you should use `bytes.Buffer`\n\n- `String() string`\n\n  **Reason:** see the previous reason\n\n- `Grow(n int)`\n\n  **Reason:** we can allocate the memory only in RAM. It doesn't make sense to allocate space on a disk\n\n- `Truncate(n int)`\n- `UnreadByte() error`\n- `UnreadRune() error`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshoshinnikita%2Fgo-disk-buffer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshoshinnikita%2Fgo-disk-buffer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshoshinnikita%2Fgo-disk-buffer/lists"}