{"id":21549128,"url":"https://github.com/pschichtel/qewqew","last_synced_at":"2025-03-18T02:18:55.353Z","repository":{"id":57743268,"uuid":"155654649","full_name":"pschichtel/QewQew","owner":"pschichtel","description":"A simple durable queue backed by binary files.","archived":false,"fork":false,"pushed_at":"2024-12-09T00:06:48.000Z","size":126,"stargazers_count":1,"open_issues_count":4,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-24T09:41:29.367Z","etag":null,"topics":["durable","java","queue"],"latest_commit_sha":null,"homepage":"https://schich.tel","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pschichtel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"pschichtel"}},"created_at":"2018-11-01T03:06:25.000Z","updated_at":"2024-12-08T00:12:36.000Z","dependencies_parsed_at":"2024-12-03T02:38:11.728Z","dependency_job_id":null,"html_url":"https://github.com/pschichtel/QewQew","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pschichtel%2FQewQew","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pschichtel%2FQewQew/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pschichtel%2FQewQew/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pschichtel%2FQewQew/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pschichtel","download_url":"https://codeload.github.com/pschichtel/QewQew/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244141479,"owners_count":20404837,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["durable","java","queue"],"created_at":"2024-11-24T06:21:16.940Z","updated_at":"2025-03-18T02:18:55.336Z","avatar_url":"https://github.com/pschichtel.png","language":"Java","funding_links":["https://github.com/sponsors/pschichtel"],"categories":[],"sub_categories":[],"readme":"QewQew [![Maven Central](https://img.shields.io/maven-central/v/tel.schich/qewqew.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:%22tel.schich%22%20AND%20a:%22qewqew%22)\n======\n\nA simple durable queue backed by binary files.\n\nThis project is inspired by [Tape](https://github.com/square/tape/), but uses a completely different approach.\n\nUnder the hood QewQew implements a linked list of chunk files of a certain maximum capacity with a special file that stores the first chunk's index. Upon opening the queue all chunks are locked exclusively and mapped into memory for efficient random access. The chunk format is defined below. Consumed chunks are removed from disk, an empty queue will remove all files upon closure, new files are pre-allocated to the given chunk size.\n\nFile formats\n------------\n\n### Types\n\n1. `chunk-ref := 16 bit unsigned integer`  \n    a value of `0` is the `NULL_REF` and indicates the absense of a reference\n2. `pointer := 32 bit signed integer`\n3. `data-length := 16 bit unsigned integer`\n4. `data := 0 to 2^16-1 bytes`\n\n### Queue Header\n\n```\nfirst-chunk := chunk-ref\n```\n\nThe queue header file consists only of a reference to the first chunk. If it is the `NULL_REF` it indicates an empty queue.\n\n### Chunk\n\nChunk files are named like the the queue header file with their chunk index appended and separated by a dot (e.g. `queue.dat.1` if `queue.dat` is the header file).\n\nThe file is structured as follows:\n\n```\nchunk    := head-ptr tail-ptr next-ref payload*\npayload  := data-length data\nhead-ptr := pointer\ntail-ptr := pointer\nnext-ref := chunk-ref\n```\n\n\n```\n+-----------------+\n|                 |\n| Queue Header    |\n|                 |\n|                 |\n| |first-chunk|    |\n|  |              |\n|  |              |\n+-----------------+\n   |\n   |  +------------------------------------------------------------------+\n   |  |                                                                  |\n   +-\u003e|  Chunk                                                           |\n      |                                                                  |\n      |            +---------------------------------------------------+ |\n      |            |                                                   | |\n      |            |                                                   v |\n      |  |head-ptr|tail-ptr|next-ref|data-length|data|data-length|data|  |\n      |   |                 |        ^                                   |\n      |   |                 |        |                                   |\n      |   +--------------------------+                                   |\n      |                     |                                            |\n      |                     |                                            |\n      +------------------------------------------------------------------+\n                            |\n         +------------------+\n         |\n         v\n      +------------------------------------------------------------------+\n      |                                                                  |\n      |  Chunk                                                           |\n      |                                                                  |\n      |            +-----...                                             |\n      |            |                                                     |\n      |            |                                                     |\n      |  |........|........|........|......                              |\n      |   |                 |                                            |\n      |   |                 |                                            |\n      |   +-----...         |                                            |\n      |                     |                                            |\n      |                     |                                            |\n      +------------------------------------------------------------------+\n                            |\n                    ...-----+\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpschichtel%2Fqewqew","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpschichtel%2Fqewqew","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpschichtel%2Fqewqew/lists"}