{"id":19337276,"url":"https://github.com/infobip/popout","last_synced_at":"2025-04-23T01:30:57.267Z","repository":{"id":37336178,"uuid":"117979235","full_name":"infobip/popout","owner":"infobip","description":"Java file-based extremely fast and reliable FIFO queue.","archived":false,"fork":false,"pushed_at":"2022-05-19T15:31:38.000Z","size":153,"stargazers_count":58,"open_issues_count":6,"forks_count":12,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-02T06:12:22.222Z","etag":null,"topics":["collections","custom-serialization","files","infobip","java","popout","queue"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/infobip.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-01-18T12:20:59.000Z","updated_at":"2024-09-19T04:17:05.000Z","dependencies_parsed_at":"2022-08-18T02:10:52.838Z","dependency_job_id":null,"html_url":"https://github.com/infobip/popout","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infobip%2Fpopout","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infobip%2Fpopout/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infobip%2Fpopout/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infobip%2Fpopout/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/infobip","download_url":"https://codeload.github.com/infobip/popout/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250352197,"owners_count":21416454,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["collections","custom-serialization","files","infobip","java","popout","queue"],"created_at":"2024-11-10T03:13:54.519Z","updated_at":"2025-04-23T01:30:56.775Z","avatar_url":"https://github.com/infobip.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Popout\n\n[![build_status](https://travis-ci.org/infobip/popout.svg?branch=master)](https://travis-ci.org/infobip/popout)\n[![maven_central](https://maven-badges.herokuapp.com/maven-central/org.infobip.lib/popout/badge.svg)](https://maven-badges.herokuapp.com/maven-central/org.infobip.lib/popout)\n[![License](http://img.shields.io/:license-apache-brightgreen.svg)](http://www.apache.org/licenses/LICENSE-2.0.html)\n[![JavaDoc](http://www.javadoc.io/badge/org.infobip.lib/popout.svg)](http://www.javadoc.io/doc/org.infobip.lib/popout)\n\n`Popout` is a file-based queue for Java.\n\nDon't forget to take a look at our [benchmarks](./benchmarks/README.md).\n\n## Contents\n\n- [Requirements](#requirements)\n- [Usage](#usage)\n  - [Add dependency](#add-dependency)\n  - [Create queue](#create-queue)\n  - [Basic operations](#basic-operations)\n  - [Custom serialization and deserialization](#custom-serialization-and-deserialization)\n- [Development](#development)\n  - [Prerequisites](#prerequisites)\n  - [Building](#building)\n  - [Running the tests](#running-the-tests)\n- [Built With](#built-with)\n- [Changelog](#changelog)\n- [Contributing](#contributing)\n- [Versioning](#versioning)\n- [Authors](#authors)\n- [License](#license)\n\n## Requirements\n\n- [Java](http://www.oracle.com/technetwork/java/javase) (minimal required version is 8);\n- [Maven](https://maven.apache.org)\n\n## Usage\n\n### Add dependency\n\nInclude the dependency to your project's pom.xml file:\n\n```xml\n\u003cdependencies\u003e\n    ...\n    \u003cdependency\u003e\n        \u003cgroupId\u003eorg.infobip.lib\u003c/groupId\u003e\n        \u003cartifactId\u003epopout\u003c/artifactId\u003e\n        \u003cversion\u003e2.1.0\u003c/version\u003e\n    \u003c/dependency\u003e\n    ...\n\u003c/dependencies\u003e\n```\n\nor Gradle:\n\n```groovy\ncompile 'org.infobip.lib:popout:2.1.0'\n```\n\n### Create a queue\n\nLet's create a minimal config `FileQueue` instance:\n\n```java\nFileQueue\u003cString\u003e queue = FileQueue.\u003cString\u003esynced().build();\n```\n\nThe code above creates **synced** `FileQueue` implementation with default config. The queue data writes on the disk in a small files, named `WAL`-files. When the amount of that files is sufficiently (specified in the config, see below) that files merges to a big `compressed` file, the next portions of `WAL`-files to the next `compressed`-file and etc.\n\nThe differences between the `FileQueue` and the Java-default `java.util.Queue` interfaces are the following:\n\n- `FileQueue`.`longSize` - returns the number of elements in this queue with wide range, than `int`;\n- `FileQueue`.`diskSize` - tells the amount of bytes, which the `queue` takes on the disk;\n- `FileQueue`.`flush` - flushes all this queue's data to the disk;\n- `FileQueue`.`compress` - manually compress all WAL-files into a compressed file;\n- `FileQueue`.`close` - flushes and closes the files descriptors of the queue.\n\nThere are two main `FileQueue` implementations:\n\n- **synced** - every `add` operation is flushes on disk immediately and every `poll` reads the items from the disk directly. There is no buffers or something in-memory. It suits for cases, when you don't want to lose your data at all and you don't care about performance. It is the most reliable kind of the `FileQueue`;\n\n- **batched** - a concept of `tail` and `head` buffers is present here. You can specify a `batchSize` option, which tells to the queue builder how many elements could be store in memory, before writing to the disk. Writes and reads to/from the disk operations are batched and it boosts the queue's performance, but you always should remember that in case of unexpected crash you could lose your *head* or *tail* data. This kind of queue suits well when your need more performant queue and you don't afraid to lose some amount of data, or you are ready to control it your self by periodically invoking the `flush` method.\n\n\u003e **NOTICE:** you also could instantiate WAL `maxCount` option and `batchSize` to `Integer.MAX_VALUE` and use `flush` and `compress` by yourself in fully manual manner.\n\nMore advanced `FileQueue` usage:\n\n```java\nQueue\u003cInteger\u003e queue = FileQueue.\u003cInteger\u003ebatched()\n        // the name of the queue, used in file patterns\n        .name(\"popa\")\n        // the default folder for all queue's files\n        .folder(\"/folder/where/store/queue/files\")\n        // sets custom serializer\n        .serializer(Serializer.INTEGER)\n        // sets custom deserializer\n        .deserializer(Deserializer.INTEGER)\n        // set up the queue's limits settings\n        .limit(QueueLimit.queueLength()\n                .length(1_000_000)\n                .handler(myQueueLimitExceededHandler))\n        // restores from disk or not, during startup. If 'false' - the previous files will be removed\n        .restoreFromDisk(false)\n        // handler for corrupted data from disk\n        .corruptionHandler(new MyCorruptionHandler())\n        // WAL files configuration\n        .wal(WalFilesConfig.builder()\n            // the place where WAL files stores. Default is a queue's folder above\n            .folder(\"some/wal/files/folder\")\n            // the maximum allowed amount of WAL files before compression\n            .maxCount(1000)\n            .build())\n        // compressed files config\n        .compressed(CompressedFilesConfig.builder()\n            // the place where compressed files stores. Default is a queue's folder above\n            .folder(\"some/compressed/files/folder\")\n            // the maximum allowed compressed file's size\n            .maxSizeBytes(SizeUnit.MEGABYTES.toBytes(256))\n            .build())\n        // the amount of elements in one WAL file. only batched queue option\n        .batchSize(10_000)\n        .build();\n```\n\n### Basic operations\n\nAdd some data to the queue to the end of the queue. `FileQueue` accepts a generic type of arbitrary length:\n\n```java\nqueue.add(\"popa\");\n```\n\nRead data at the head of the queue (without removing it):\n\n```java\nString record = queue.peek();\n```\n\nIn short, `FileQueue` implements `Queue`, `Collection` and `Iterable` interfaces and you are able to use all theirs methods:\n\n```java\n// Queue's size\nint queueSize = queue.size();\n\n// Add many items\nqueue.addAll(asList(\"one\", \"two\", \"three\"));\n\n// Retrieves and removes the head of this queue,\nString head = queue.poll();\n\n// Remove all elements.\nqueue.clear();\n\n// Use queue's iterator\nIterator\u003cString\u003e iterator = queue.iterator();\n```\n\n### Custom serialization and deserialization\n\nBy default, queue uses standard [Java's serialization/deserialization mechanism](https://docs.oracle.com/javase/8/docs/technotes/guides/serialization/index.html), but you could override it by implementing [Serializer](https://github.com/infobip/popout/blob/master/popout/src/main/java/org/infobip/lib/popout/Serializer.java) and [Deserializer](https://github.com/infobip/popout/blob/master/popout/src/main/java/org/infobip/lib/popout/Deserializer.java):\n\n```java\nQueue\u003cString\u003e queue = FileQueue.\u003cString\u003esynced()\n        .serializer(\u003cyour_serializaer_impl\u003e)\n        .deserializer(\u003cyour_deserializaer_impl\u003e)\n        .build();\n```\n\n## Development\n\nThese instructions will get you a copy of the project up and running on your local machine for development and testing purposes.\n\n### Prerequisites\n\nFor building the project you need only a [Java compiler](http://www.oracle.com/technetwork/java/javase/downloads/index.html).\n\n\u003e **IMPORTANT:** Popout requires Java version starting from **8**\n\nAnd, of course, you need to clone Popout from GitHub:\n\n```bash\n$\u003e git clone https://github.com/infobip/popout\n$\u003e cd popout\n```\n\n### Building\n\nFor building routine automation, I am using [maven](https://maven.apache.org).\n\nTo build the Popout project, do the following:\n\n```bash\n$\u003e mvn clean package\n...\n[INFO] ------------------------------------------------------------------------\n[INFO] BUILD SUCCESS\n[INFO] ------------------------------------------------------------------------\n[INFO] Total time: 35.491 s\n[INFO] Finished at: 2019-01-18T23:25:12+03:00\n[INFO] Final Memory: 50M/548M\n[INFO] ------------------------------------------------------------------------\n```\n\n### Running the tests\n\nTo run the project's test, do the following:\n\n```bash\n$\u003e mvn clean test\n...\n[INFO] -------------------------------------------------------\n[INFO]  T E S T S\n[INFO] -------------------------------------------------------\n...\n[INFO]\n[INFO] Results:\n[INFO]\n[INFO] Tests run: 32, Failures: 0, Errors: 0, Skipped: 0\n[INFO]\n...\n```\n\nAlso, if you do `package` or `install` goals, the tests launch automatically.\n\n## Built With\n\n- [Java](http://www.oracle.com/technetwork/java/javase) - is a systems and applications programming language\n- [Lombok](https://projectlombok.org) - is a java library that spicing up your java\n- [Junit](http://junit.org/junit4/) - is a simple framework to write repeatable tests\n- [AssertJ](http://joel-costigliola.github.io/assertj/) - AssertJ provides a rich set of assertions, truly helpful error messages, improves test code readability\n- [Maven](https://maven.apache.org) - is a software project management and comprehension tool\n\n## Changelog\n\nTo see what has changed in recent versions of Popout, see the [changelog](./CHANGELOG.md) file.\n\n## Contributing\n\nPlease read [contributing](./CONTRIBUTING.md) file for details on my code of conduct, and the process for submitting pull requests to me.\n\n## Versioning\n\nWe use [SemVer](http://semver.org/) for versioning. For the versions available, see the [tags on this repository](https://github.com/infobip/popout/tags).\n\n## Authors\n\n- **[Artem Labazin](https://github.com/xxlabaza)** - creator and the main developer\n\n## License\n\nThis project is licensed under the Apache License 2.0 License - see the [license](./LICENSE) file for details\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finfobip%2Fpopout","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finfobip%2Fpopout","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finfobip%2Fpopout/lists"}