{"id":16092340,"url":"https://github.com/alecthomas/vheap","last_synced_at":"2025-03-18T06:30:59.557Z","repository":{"id":3105878,"uuid":"4131910","full_name":"alecthomas/vheap","owner":"alecthomas","description":"Fast, persistent, mmapped, virtual heap.","archived":false,"fork":false,"pushed_at":"2023-12-02T11:40:46.000Z","size":6,"stargazers_count":8,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-28T07:51:14.841Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alecthomas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2012-04-25T02:06:16.000Z","updated_at":"2022-02-10T18:49:45.000Z","dependencies_parsed_at":"2024-10-27T17:23:50.742Z","dependency_job_id":"1771d843-e614-499b-98d2-0a740fa8b7df","html_url":"https://github.com/alecthomas/vheap","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alecthomas%2Fvheap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alecthomas%2Fvheap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alecthomas%2Fvheap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alecthomas%2Fvheap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alecthomas","download_url":"https://codeload.github.com/alecthomas/vheap/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243910275,"owners_count":20367536,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-09T16:07:09.030Z","updated_at":"2025-03-18T06:30:59.021Z","avatar_url":"https://github.com/alecthomas.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Virtual Heap\n\nThis [Go](http://golang.org) package provides fast, mmapped access to a virtual heap. Each block allocated from the heap is assigned a unique, persistent reference ID that can be used to address the block across application runs.\n\nThe implementation is currently very basic, but sufficient for its intended purpose.\n\n## Use-cases\n\nWhy would you want to use this?\n\nIt is useful for situations where you want very low overhead object creation and tight control over the layout of and access to, data.\n\nIts initial use will be in a search index, storing posting lists, document names, scoring information, and so on.\n\n## Example\n\nCreating a heap and adding some data:\n\n    // Create a heap with an initial size of 32MB.\n    h, err := vheap.OpenForUpdate(\"hello.vheap\", 32)\n    defer h.Close()\n    \n    // Allocate a 32 byte block.\n    b, err := h.Allocate(32)\n    \n    // Print the block's persistent ID (the first block is guaranteed to have an ID of 0,\n    // which is useful for storing references to other blocks)\n    println(b.Id)\n    \n    // Copy some text into the block\n    copy(b.Bytes, []byte(\"hello world\"))\n\nOpen the heap and retrieve the created block:\n\n    h, err := vheap.Open(\"hello.vheap\")\n    defer h.Close()\n    b := h.GetBlock(0)\n    println(string(b.Bytes))\n\n## Details\n\nThe storage consists of a single file partitioned into multiple regions. Each\nregion is mmapped. When a region fills up, a new region is appended. Blocks are\nallocated from the first region with sufficient space.\n\n*Currently, deallocated blocks are not reclaimed. This will lead to heavy\nfragmentation if blocks are frequently freed.*\n\nA region size is specified at index creation, typically in the order of 32-128MB. Total index size is increased by the region size when an allocation does not fit in any existing region. Additionally, if the new allocation exceeds the region size, the new region will be expanded to fit the allocation.\n\n## On-disk Structure\n\nThe structure at the start of each region is:\n\n    signature [8]byte     // Region signature\n    freeListPointer int64 // Pointer to start of free memory\n    regionSize int64      // Size, in bytes, of each region, including header and block list.\n    regionId int64        // Region ID, starting at 0.\n    data []byte           // Data\n\nThe block list is located at the end of each region, growing down:\n\n    id int64                             // Next free block ID in this region\n    entries []struct{offset, size int64} // Array of offset and raw size of allocated blocks.\n\nWhen the top of the heap reaches the bottom of the index, a new region is\nallocated.\n\n## Concurrent Access\n\nAllocations are (almost) atomic on 64-bit machines. *Almost atomic* in that free space is allocated first, then the list of block IDs is updated. If termination occurs in between these two operations, free space will be lost, though the heap will remain consistent.\n\nThe implication of this consistency is that a storage block can be read from concurrently while being written to by another process, though currently there is no support for automatically opening appended regions in readers. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falecthomas%2Fvheap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falecthomas%2Fvheap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falecthomas%2Fvheap/lists"}