{"id":13485650,"url":"https://github.com/bulldog2011/bigqueue","last_synced_at":"2025-10-21T23:07:09.850Z","repository":{"id":6507198,"uuid":"7747823","full_name":"bulldog2011/bigqueue","owner":"bulldog2011","description":"A big, fast and persistent queue based on memory mapped file.","archived":false,"fork":false,"pushed_at":"2022-07-26T04:07:24.000Z","size":2993,"stargazers_count":567,"open_issues_count":27,"forks_count":221,"subscribers_count":38,"default_branch":"master","last_synced_at":"2025-10-21T23:02:19.692Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://bulldog2011.github.com/bigqueue","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bulldog2011.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-01-22T07:15:11.000Z","updated_at":"2025-10-10T03:34:44.000Z","dependencies_parsed_at":"2022-08-06T19:15:40.576Z","dependency_job_id":null,"html_url":"https://github.com/bulldog2011/bigqueue","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/bulldog2011/bigqueue","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bulldog2011%2Fbigqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bulldog2011%2Fbigqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bulldog2011%2Fbigqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bulldog2011%2Fbigqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bulldog2011","download_url":"https://codeload.github.com/bulldog2011/bigqueue/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bulldog2011%2Fbigqueue/sbom","scorecard":{"id":258049,"data":{"date":"2025-08-11","repo":{"name":"github.com/bulldog2011/bigqueue","commit":"60f6954604cfa671aa0d04d90f375fd6f25e8771"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.7,"checks":[{"name":"Code-Review","score":4,"reason":"Found 6/14 approved changesets -- score normalized to 4","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":6,"reason":"binaries present in source code","details":["Warn: binary detected: samples/thriftqueue/lib/libthrift-0.6.1.jar:1","Warn: binary detected: samples/thriftqueue/lib/slf4j-api-1.5.8.jar:1","Warn: binary detected: samples/thriftqueue/lib/slf4j-log4j12-1.5.8.jar:1","Warn: binary detected: samples/thriftqueue/thrift/thrift-0.6.1.exe:1"],"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":7,"reason":"3 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-5mg8-w23w-74h3","Warn: Project is vulnerable to: GHSA-7g45-4rm6-3mm3","Warn: Project is vulnerable to: GHSA-mvr2-9pj6-7w5j"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 22 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-17T10:07:56.465Z","repository_id":6507198,"created_at":"2025-08-17T10:07:56.465Z","updated_at":"2025-08-17T10:07:56.465Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280348062,"owners_count":26315368,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-21T02:00:06.614Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T18:00:28.786Z","updated_at":"2025-10-21T23:07:09.834Z","avatar_url":"https://github.com/bulldog2011.png","language":"Java","funding_links":[],"categories":["Projects","Java","项目"],"sub_categories":["Data Structures","数据结构"],"readme":"# Big Queue\n\n\nA big, fast and persistent queue based on memory mapped file.\n\n***Notice***, bigqueue is just a standalone library, for a high-throughput, persistent, distributed, publish-subscrible messaging system, please refer to [Luxun](https://github.com/bulldog2011/luxun), Luxun messaging system uses bigqueue internally as fast and persistent queue. \n\n##Feature Highlight:  \n1. **Fast**: close to the speed of direct memory access, both enqueue and dequeue are close to O(1) memory access.  \n2. **Big**: the total size of the queue is only limited by the available disk space.  \n3. **Persistent**: all data in the queue is persisted on disk, and is crash resistant.\n4. **Reliable**: OS will be responsible to presist the produced messages even your process crashes.  \n5. **Realtime**: messages produced by producer threads will be immediately visible to consumer threads.\n6. **Memory-efficient**: automatic paging \u0026 swapping algorithm, only most-recently accessed data is kept in memory.  \n7. **Thread-safe**: multiple threads can concurrently enqueue and dequeue without data corruption.  \n8. **Simple\u0026Light-weight**: current number of source files is 12 and the library jar is less than 30K.\n\n\n## The Big Picture\n\n###Memory Mapped Sliding Window\n\n![design](http://bulldog2011.github.com/images/luxun/sliding_window.png)\n\n\n##Performance Highlight:\n* In concurrent producing and consuming case, the average throughput is around ***166M bytes per second***.\n* In sequential producing then consuming case, the average throughput is around ***333M bytes per second***.\n\nSuppose the average message size is 1KB, then big queue can concurrently producing and consuming  \n166K message per second. Basically, the throughput is only limited by disk IO bandwidth.\n\n[here is a detailed performance report](https://github.com/bulldog2011/bigqueue/wiki/Performance-Test-Report)\n\n##How to Use\n1. Direct jar or source reference  \nDownload jar from repository mentioned in version history section below, latest stable release is [0.7.0](https://github.com/bulldog2011/bulldog-repo/tree/master/repo/releases/com/leansoft/bigqueue/0.7.0).   \n***Note*** : bigqueue depends on log4j, please also added log4j jar reference if you use bigqueue.\n\n2. Maven reference  \n\n\t\t\u003cdependency\u003e\n\t\t  \u003cgroupId\u003ecom.leansoft\u003c/groupId\u003e\n\t\t  \u003cartifactId\u003ebigqueue\u003c/artifactId\u003e\n          \u003cversion\u003e0.7.0\u003c/version\u003e\n\t\t\u003c/dependency\u003e\n\t\t\n\t\t\u003crepository\u003e\n\t\t  \u003cid\u003egithub.release.repo\u003c/id\u003e\n\t\t  \u003curl\u003ehttps://raw.github.com/bulldog2011/bulldog-repo/master/repo/releases/\u003c/url\u003e\n\t\t\u003c/repository\u003e\n\n\n##Docs\n\n1. [a simple design doc](http://bulldog2011.github.com/blog/2013/01/23/big-queue-design/)\n2. [big queue tutorial](http://bulldog2011.github.com/blog/2013/01/24/big-queue-tutorial/)\n3. [fanout queue tutorial](http://bulldog2011.github.com/blog/2013/03/25/fanout-queue-tutorial/)\n4. [big array tutorial](http://bulldog2011.github.com/blog/2013/01/24/big-array-tutorial/)\n5. [how to turn big queue into a thrift based queue service](http://bulldog2011.github.com/blog/2013/01/27/thrift-queue/)\n6. [use case : producing and consuming 4TB log daily on one commodity machine](http://bulldog2011.github.com/blog/2013/01/28/log-collecting/)\n7. [use case : sort and search 100GB data on a single commodity machine](http://bulldog2011.github.com/blog/2013/01/25/merge-sort-using-big-queue/)\n8. [the architecture and design of a pub-sub messaging system tailored for big data collecting and analytics](http://bulldog2011.github.com/blog/2013/03/27/the-architecture-and-design-of-a-pub-sub-messaging-system/)\n9. [a big, fast and persistent queue[ppt]](http://www.slideshare.net/yang75108/a-big-fast-and-persistent-queue)\n\n## Version History\n\n#### 0.7.0 - *March 24, 2013* : [repository](https://github.com/bulldog2011/bulldog-repo/tree/master/repo/releases/com/leansoft/bigqueue/0.7.0)\n  * Feature: support fanout queue semantics\n  * Enhancement: make data file size configurable\n\n#### 0.6.1 — *January 29, 2013* : [repository](https://github.com/bulldog2011/bulldog-repo/tree/master/repo/releases/com/leansoft/bigqueue/0.6.1)\n\n  * Initial version:)\n\n\n##Copyright and License\nCopyright 2012 Leansoft Technology \u003c51startup@sina.com\u003e\n\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use this work except in compliance with the License. You may obtain a copy of the License in the LICENSE file, or at:\n\n[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)\n\nUnless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbulldog2011%2Fbigqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbulldog2011%2Fbigqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbulldog2011%2Fbigqueue/lists"}