{"id":16782604,"url":"https://github.com/davidmoten/rxjava2-extras","last_synced_at":"2025-04-12T18:40:31.867Z","repository":{"id":12717095,"uuid":"72622361","full_name":"davidmoten/rxjava2-extras","owner":"davidmoten","description":"Utilities for use with RxJava 2","archived":false,"fork":false,"pushed_at":"2024-10-29T12:04:33.000Z","size":2268,"stargazers_count":171,"open_issues_count":12,"forks_count":16,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-10-29T14:34:49.276Z","etag":null,"topics":["concurrency","java","reactive","reactive-streams","reactivex","rxjava","stream"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davidmoten.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-11-02T09:06:25.000Z","updated_at":"2024-10-29T12:04:30.000Z","dependencies_parsed_at":"2023-02-10T18:15:51.250Z","dependency_job_id":"d67e78ae-41ed-49f8-ba3b-fbc01fafbeef","html_url":"https://github.com/davidmoten/rxjava2-extras","commit_stats":null,"previous_names":[],"tags_count":41,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidmoten%2Frxjava2-extras","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidmoten%2Frxjava2-extras/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidmoten%2Frxjava2-extras/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidmoten%2Frxjava2-extras/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davidmoten","download_url":"https://codeload.github.com/davidmoten/rxjava2-extras/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248616409,"owners_count":21134076,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","java","reactive","reactive-streams","reactivex","rxjava","stream"],"created_at":"2024-10-13T07:46:34.131Z","updated_at":"2025-04-12T18:40:31.834Z","avatar_url":"https://github.com/davidmoten.png","language":"Java","readme":"# rxjava2-extras\n\u003ca href=\"https://github.com/davidmoten/rxjava2-extras/actions/workflows/ci.yml\"\u003e\u003cimg src=\"https://github.com/davidmoten/rxjava2-extras/actions/workflows/ci.yml/badge.svg\"/\u003e\u003c/a\u003e\u003cbr/\u003e\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.github.davidmoten/rxjava2-extras/badge.svg?style=flat)](https://maven-badges.herokuapp.com/maven-central/com.github.davidmoten/rxjava2-extras)\u003cbr/\u003e\n[![codecov](https://codecov.io/gh/davidmoten/rxjava2-extras/branch/master/graph/badge.svg)](https://codecov.io/gh/davidmoten/rxjava2-extras)\u003cbr/\u003e\n\nUtilities for use with RxJava 2\n\nFeatures\n----------\n* [`Strings`](#strings) - create/manipulate streams of `String`, conversions to and from\n* [`Bytes`](#bytes) - create/manipulate streams of `byte[]`\n* [`StateMachine`](#statemachine) - a more expressive form of `scan` that can emit multiple events for each source event\n* [`onBackpressureBufferToFile`](#onbackpressurebuffertofile) - high throughput with memory-mapped files\n* [`Flowable Transformers`](#transformers-flowable)\n* [`Observable Transformers`](#transformers-observable)\n* [`Serialized`](#serialized)\n* tests pass on Linux, Windows 10, Solaris 10\n* supports Java 1.8+\n* requires rxjava 2.0.7+\n\nStatus: *released to Maven Central*\n\nMaven site reports are [here](https://davidmoten.github.io/rxjava2-extras/index.html) including [javadoc](https://davidmoten.github.io/rxjava2-extras/apidocs/index.html).\n\nGetting started\n-----------------\nAdd this to your pom.xml:\n\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.github.davidmoten\u003c/groupId\u003e\n  \u003cartifactId\u003erxjava2-extras\u003c/artifactId\u003e\n  \u003cversion\u003eVERSION_HERE\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nOr add this to your build.gradle:\n```groovy\nrepositories {\n    mavenCentral()\n}\n\ndependencies {\n    compile 'com.github.davidmoten:rxjava2-extras:VERSION_HERE'\n}\n```\n\nAndroid\n--------------\nTo use *rxjava2-extras* on Android you need these [proguard rules](src/main/proguard/proguard-rules.txt).\n\nMigration\n------------\n* Primary target type is `Flowable` (the backpressure supporting stream)\n* Operators will be implemented initially without fusion support (later)  \n* Where applicable `Single`, `Maybe` and `Completable` will be used\n* To cross types (say from `Flowable` to `Maybe`) it is necessary to use `to` rather than `compose`\n* Transformers (for use with `compose` and `to`) are clustered within the primary owning class rather than bunched together in the `Transformers` class. For example, `Strings.join`:\n\n```java\n//produces a stream of \"ab\"\nMaybe\u003cString\u003e o = Flowable\n  .just(\"a\",\"b\")\n  .to(Strings.join()); \n```\n\n[Strings](https://davidmoten.github.io/rxjava2-extras/apidocs/com/github/davidmoten/rx2/Strings.html)\n----------\n`concat`, `join`\n\n`decode`\n\n`from(Reader)`,`from(InputStream)`,`from(File)`, ..\n\n`fromClasspath(String, Charset)`, ..\n\n`split(String)`, `split(Pattern)`\n\n`splitSimple(String)`\n\n`toInputStream`\n\n`trim`\n\n`strings`\n\n`splitLinesSkipComments`\n\n[Bytes](https://davidmoten.github.io/rxjava2-extras/apidocs/com/github/davidmoten/rx2/Bytes.html)\n--------------\n`collect`\n\n`from(InputStream)`, `from(File)`\n\n`unzip(File)`, `unzip(InputStream)`\n\n[RetryWhen](https://davidmoten.github.io/rxjava2-extras/apidocs/com/github/davidmoten/rx2/RetryWhen.html)\n------------\n[Builder](#retrywhen-1) for `.retryWhen()`\n\n[IO](https://davidmoten.github.io/rxjava2-extras/apidocs/com/github/davidmoten/rx2/IO.html)\n-------------\n`serverSocket(port)` \n\n[Flowables](https://davidmoten.github.io/rxjava2-extras/apidocs/com/github/davidmoten/rx2/Flowables.html)\n---------------\n[`fetchPagesByRequest`](#fetchpagesbyrequest)\n\n[`match`](#match-matchwith)\n\n`repeat`\n\n[`mergeInterleaved`](#mergeinterleaved)\n\n[Transformers (Flowable)](https://davidmoten.github.io/rxjava2-extras/apidocs/com/github/davidmoten/rx2/flowable/Transformers.html)\n---------------------------\n[`buffer`](#buffer) with size and timeout\n\n[`collectStats`](#collectstats)\n\n[`doOnEmpty`](#doonempty)\n\n[`collectWhile`](#collectwhile)\n\n[`flatMapInterleaved`](#mergeinterleaved)\n\n[`insert`](#insert)\n\n[`mapLast`](#maplast)\n\n[`match`, `matchWith`](#match-matchwith)\n\n[`maxRequest`](#maxrequest)\n\n[`minRequest`](#minrequest)\n\n[`onBackpressureBufferToFile`](#onbackpressurebuffertofile)\n\n[`rebatchRequests`](#rebatchrequests)\n\n[`reverse`](#reverse)\n\n[`stateMachine`](#statemachine)\n\n[`toListWhile`](#tolistwhile)\n\n[`windowMin`](#windowminmax)\n\n[`windowMax`](#windowminmax)\n\n\n[Transformers (Observable)](https://davidmoten.github.io/rxjava2-extras/apidocs/com/github/davidmoten/rx2/observable/Transformers.html)\n---------------------------\n[`onBackpressureBufferToFile`](#onbackpressurebuffertofile)\n\nSchedulerHelper\n----------------\n`blockUntilWorkFinished`\n\n`withThreadId`\n\n`withThreadIdFromCallSite`\n\n[Maybes](https://davidmoten.github.io/rxjava2-extras/apidocs/com/github/davidmoten/rx2/Maybes.html)\n-------------\n`fromNullable`\n\n[Actions](https://davidmoten.github.io/rxjava2-extras/apidocs/com/github/davidmoten/rx2/Actions.html)\n--------------------\n`doNothing`\n`setToTrue`\n`throwing`\n\n[BiFunctions](https://davidmoten.github.io/rxjava2-extras/apidocs/com/github/davidmoten/rx2/BiFunctions.html)\n-------------\n`constant`\n`throwing`\n\n[BiPredicates](https://davidmoten.github.io/rxjava2-extras/apidocs/com/github/davidmoten/rx2/BiPredicates.html)\n----------------\n`alwaysTrue`\n`alwaysFalse`\n`throwing`\n\n[Callables](https://davidmoten.github.io/rxjava2-extras/apidocs/com/github/davidmoten/rx2/Callables.html)\n---------------\n`constant`\n`throwing`\n\n[Consumers](https://davidmoten.github.io/rxjava2-extras/apidocs/com/github/davidmoten/rx2/Consumers.html)\n--------------\n`addLongTo`\n`addTo`\n`assertBytesEquals`\n`close`\n`decrement`\n`doNothing`\n`increment`\n`printStackTrace`\n`println`\n`set`\n`setToTrue`\n\n[Functions](https://davidmoten.github.io/rxjava2-extras/apidocs/com/github/davidmoten/rx2/Functions.html)\n------------\n`constant`\n`identity`\n`throwing`\n\n[Predicates](https://davidmoten.github.io/rxjava2-extras/apidocs/com/github/davidmoten/rx2/Predicates.html)\n-------------\n`alwaysFalse`\n`alwaysTrue`\n\n[Serialized](https://davidmoten.github.io/rxjava2-extras/apidocs/com/github/davidmoten/rx2/Serialized.html)\n---------------\n[`read`](#serialized)\n\n[`write`](#serialized)\n\n[`kryo().read`](#serialized)\n\n[`kryo().write`](#serialized)\n\n# Documentation\n\nbuffer\n--------------------------\nTo buffer on maximum size and time since last source emission:\n\n```java\nflowable.compose(\n  Transformers.buffer(maxSize, 10, TimeUnit.SECONDS));\n```\n You can also make the timeout dependent on the last emission:\n\n```java\nflowable.compose(\n  Transformers.buffer(maxSize, x -\u003e timeout(x), TimeUnit.SECONDS));\n```\n\ncollectStats\n---------------------------\nAccumulate statistics, emitting the accumulated results with each item.\n\n\u003cimg src=\"src/docs/collectStats.png?raw=true\" /\u003e\n\n\ncollectWhile\n-------------------------\nBehaves as per `toListWhile` but allows control over the data structure used. \n\n\u003cimg src=\"src/docs/collectWhile.png?raw=true\" /\u003e\n\nThis operator supports [request-one micro-fusion](http://akarnokd.blogspot.com.au/2016/03/operator-fusion-part-1.html).\n\ndoOnEmpty\n-------------------------\nPerforms an action only if a stream completes without emitting an item.\n\n\u003cimg src=\"src/docs/doOnEmpty.png?raw=true\" /\u003e\n\n```java\nflowable.compose(\n    Transformers.doOnEmpty(action));\n```\n\nfetchPagesByRequest\n-----------------------\nThis is a `Flowable` creation method that is aimed at supporting calls to a service that provides data in pages where the page sizes are determined by requests from downstream (*requests* are a part of the backpressure machinery of RxJava).\n\n\u003cimg src=\"src/docs/fetchPagesByRequest.png?raw=true\" /\u003e\n\nHere's an example.\n\nSuppose you have a stateless web service, say a rest service that returns JSON/XML and supplies you with\n\n* the most popular movies of the last 24 hours sorted by descending popularity\n\nThe service supports paging in that you can pass it a start number and a page size and it will return just that slice from the list.\n\nNow I want to give a library with a `Flowable` definition of this service to my colleagues that they can call in their applications whatever they may be. For example, \n\n* Fred may just want to know the most popular movie each day, \n* Greta wants to get the top 20 and then have the ability to keep scrolling down the list in her UI.\n\nLet's see how we can efficiently support those use cases. I'm going to assume that the movie data returned by the service are mapped conveniently to objects by whatever framework I'm using (JAXB, Jersey, etc.). The fetch method looks like this:\n\n```java\n// note that start is 0-based\nList\u003cMovie\u003e mostPopularMovies(int start, int size);\n\n```\nNow I'm going to wrap this synchronous call as a `Flowable` to give to my colleagues:\n\n```java\nFlowable\u003cMovie\u003e mostPopularMovies(int start) {\n    return Flowables.fetchPagesByRequest(\n          (position, n) -\u003e Flowable.fromIterable(mostPopular(position, n)),\n          start)\n        // rebatch requests so that they are always between \n        // 5 and 100 except for the first request\n      .compose(Transformers.rebatchRequests(5, 100, false));\n}\n\nFlowable\u003cMovie\u003e mostPopularMovies() {\n    return mostPopularMovies(0);\n}\n```\nNote particularly that the method above uses a variant of [`rebatchRequests`](#rebatchrequests) to limit both minimum and maximum requests. We particularly don't want to allow a single call requesting the top 100,000 popular movies because of the memory and network pressures that arise from that call.\n\nRighto, Fred now uses the new API like this:\n\n```java\nMovie top = mostPopularMovies()\n    .compose(Transformers.maxRequest(1))\n    .first()\n    .blockingFirst();\n```\nThe use of `maxRequest` above may seem unnecessary but strangely enough the `first` operator requests `Long.MAX_VALUE` of upstream and cancels as soon as one arrives. The `take`, `elemnentAt` and `firstXXX` operators all have this counter-intuitive characteristic.\n\nGreta uses the new API like this:\n\n```java\nmostPopularMovies()\n    .rebatchRequests(20)\n    .doOnNext(movie -\u003e addToUI(movie))\n    .subscribe(subscriber);\n```\n\nA bit more detail about `fetchPagesByRequest`:\n* if the `fetch` function returns a `Flowable` that delivers fewer than the requested number of items then the overall stream completes.\n\ninsert\n-------------------------\nInserts zero or one items into a stream if the given Maybe succeeds before the next source emission.\n\nExample:\n```java\nFlowable\n  .interval(1, TimeUnit.SECONDS)\n  .compose(Transformers.insert(\n     Maybe.just(-1L).delay(500, TimeUnit.MILLISECONDS))) \n  .forEach(System.out::println);\n```\nproduces (with 500ms intervals between every emission):\n```\n0\n-1\n1\n-1\n2\n-1\n3\n-1\n4\n-1\n...\n```\n    \nmapLast\n-------------------------\nModifies the last element of the stream via a defined Function.\n\n\u003cimg src=\"src/docs/mapLast.png?raw=true\" /\u003e\n\nExample:\n\n```java\nFlowable\n    .just(1, 2, 3)\n    .compose(Transformers.mapLast(x -\u003e x + 1))\n    .forEach(System.out::println);\n```\nproduces\n```\n1\n2\n4\n```\n\nmatch, matchWith\n-------------------------\nFinds out-of-order matches in two streams.\n\n\u003cimg src=\"src/docs/match.png?raw=true\" /\u003e\n\n[javadoc](http://davidmoten.github.io/rxjava-extras/apidocs/com/github/davidmoten/rx/Transformers.html#matchWith--)\n\nYou can use `FlowableTranformers.matchWith` or `Flowables.match`:\n\n```java\nFlowable\u003cInteger\u003e a = Flowable.just(1, 2, 4, 3);\nFlowable\u003cInteger\u003e b = Flowable.just(1, 2, 3, 5, 6, 4);\nFlowables.match(a, b,\n     x -\u003e x, // key to match on for a\n     x -\u003e x, // key to match on for b\n     (x, y) -\u003e x // combiner\n    )\n   .forEach(System.out::println);\n```\ngives\n```\n1\n2\n3\n4\n```\nDon't rely on the output order!\n\nUnder the covers elements are requested from `a` and `b` in alternating batches of 128 by default. The batch size is configurable in another overload.\n\nmaxRequest\n-------------\nLimits upstream requests. \n\n\u003cimg src=\"src/docs/maxRequest.png?raw=true\" /\u003e\n\n* may allow requests less than the maximum \n* serializes requests\n* does not buffer items\n* requests at start and just before emission of last item in current batch\n\n```java\nflowable\n  .compose(Transformers.maxRequest(100));\n\n```\nTo constrain some requests then no constraint:\n```java\nflowable\n  .compose(Transformers.maxRequest(100, 256, 256, 256, Long.MAX_VALUE);\n```\n\nSee also: [`minRequest`](#minrequest), [`rebatchRequests`](#rebatchrequests)\n\nmergeInterleaved\n-------------------\n\u003cimg src=\"/src/docs/mergeInterleaved.png?raw=true\"/\u003e\n\nWhen you use `Flowable.merge` on synchronous sources the sources are completely consumed serially. As a consequence if you want to merge two infinite sources synchronously then you only get the first source and the second source is never read.\n\n`Flowables.mergeInterleaved` round-robins the requests to the window of current publishers (up to `maxConcurrent`) so that two or more infinite streams can be merged without resorting to apply asynchronicity via an asynchronous scheduler.\n\nThere is a sacrifice made in that the operator is less performant than merging asynchronous streams because only one stream is requested from at a time. In addition the operator does not include micro- and macro-fusion optimisations.\n\nminRequest\n-------------\nEnsures requests are at least the given value(s).\n\n\u003cimg src=\"src/docs/minRequest.png?raw=true\" /\u003e\n\n* serializes requests\n* may buffer items\n* requests at start and just after emission of last item in current batch\n\n```java\nflowable\n  .compose(Transformers.minRequest(10));\n```\nTo allow the first request through unconstrained:\n```java\nflowable\n  .compose(Transformers.minRequest(1, 10));\n```\n\nSee also: [`maxRequest`](#maxrequest), [`rebatchRequests`](#rebatchrequests)\n\n## onBackpressureBufferToFile\nWith this operator you can offload a stream's emissions to disk to reduce memory pressure when you have a fast producer + slow consumer (or just to minimize memory usage).\n\n\u003cimg src=\"src/docs/onBackpressureBufferToFile.png\" /\u003e\n\nIf you have used the `onBackpressureBuffer` operator you'll know that when a stream is producing faster than the downstream operators can process (perhaps the producer cannot respond meaningfully to a *slow down* request from downstream) then `onBackpressureBuffer` buffers the items to an in-memory queue until they can be processed. Of course if memory is limited then some streams might eventually cause an `OutOfMemoryError`. One solution to this problem is to increase the effectively available memory for buffering by using off-heap memory and disk instead. That's why `onBackpressureBufferToFile` was created. \n\n*rxjava-extras* uses standard file io to buffer serialized stream items. This operator can still be used with RxJava2 using the [RxJava2Interop](https://github.com/akarnokd/RxJava2Interop) library. \n\n*rxjava2-extras* uses fixed size memory-mapped files to perform the same operation but with much greater throughput. \n\nNote that new files for a file buffered observable are created for each subscription and those files are in normal circumstances deleted on cancellation (triggered by `onCompleted`/`onError` termination or manual cancellation). \n\nHere's an example:\n\n```java\n// write the source strings to a \n// disk-backed queue on the subscription\n// thread and emit the items read from \n// the queue on the io() scheduler.\nFlowable\u003cString\u003e flowable = \n  Flowable\n    .just(\"a\", \"b\", \"c\")\n    .compose(\n      Transformers\n          .onBackpressureBufferToFile()\n          .serializerUtf8())\n```\n\nYou can also use an `Observable` source (without converting to `Flowable` with `toFlowable`):\n```java\nFlowable\u003cString\u003e flowable = \n  Observable\n    .just(\"a\", \"b\", \"c\")\n    .to(\n      ObservableTransformers\n          .onBackpressureBufferToFile()\n          .serializerUtf8())\n```\nNote that `to` is used above to cross types (`Observable` to `Flowable`).\n\nThis example does the same as above but more concisely and uses standard java IO serialization (normally it will be more efficient to write your own `DataSerializer`):\n\n```java\nFlowable\u003cString\u003e flowable = \n  Flowable\n    .just(\"a\", \"b\", \"c\")\n    .compose(Transformers\n        .\u003cString\u003eonBackpressureBufferToFile());\n```\n\nAn example with a custom serializer:\n\n```java\n// define how the items in the source stream would be serialized\nDataSerializer\u003cString\u003e serializer = new DataSerializer\u003cString\u003e() {\n\n    @Override\n    public void serialize(String s, DataOutput out) throws IOException {\n        output.writeUTF(s);\n    }\n\n    @Override\n    public String deserialize(DataInput in) throws IOException {\n        return input.readUTF();\n    }\n    \n    @Override\n    public int sizeHint() {\n        // exact size unknown\n        return 0;\n    }\n};\nFlowable\n  .just(\"a\", \"b\", \"c\")\n  .compose(\n    Transformers\n        .onBackpressureBufferToFile()\n        .serializer(serializer));\n  ...\n```\nYou can configure various options:\n\n```java\nFlowable\n  .just(\"a\", \"b\", \"c\")\n  .compose(\n    Transformers\n        .onBackpressureBufferToFile()\n        .scheduler(Schedulers.computation()) \n        .fileFactory(fileFactory)\n        .pageSizeBytes(1024)\n        .serializer(serializer)); \n  ...\n```\n`.fileFactory(Func0\u003cFile\u003e)` specifies the method used to create the temporary files used by the queue storage mechanism. The default is a factory that calls `Files.createTempFile(\"bufferToFile\", \".obj\")`.\n\nThere are some inbuilt `DataSerializer` implementations:\n\n* `DataSerializers.utf8()`\n* `DataSerializers.string(Charset)`\n* `DataSerializers.bytes()`\n* `DataSerializers.javaIO()` - uses standard java serialization (`ObjectOutputStream` and such)\n\nUsing default java serialization you can buffer array lists of integers to a file like so:\n\n```java\nFlowable.just(1, 2, 3, 4)\n    //accumulate into sublists of length 2\n    .buffer(2)\n    .compose(Transformers\n        .\u003cList\u003cInteger\u003e\u003eonBackpressureBufferToFile()\n        .serializerJavaIO())\n```\n\nIn the above example it's fortunate that `.buffer` emits `ArrayList\u003cInteger\u003e` instances which are serializable. To be strict you might want to `.map` the returned list to a data type you know is serializable:\n\n```java\nFlowable.just(1, 2, 3, 4)\n    .buffer(2)\n    .map(list -\u003e new ArrayList\u003cInteger\u003e(list))\n    .compose(\n      Transformers\n          .\u003cList\u003cInteger\u003e\u003eonBackpressureBufferToFile()\n          .serializerJavaIO())\n```\n### Algorithm\nUsual queue drain practices are in place but the queue this time is based on memory-mapped file storage. The memory-mapped queue borrows tricks used by [Aeron](https://github.com/real-logic/Aeron). In particular:\n\n* every byte array message is preceded by a header \n\n```\nmessage length in bytes (int, 4 bytes)\nmessage type (1 byte) (0 or 1 for FULL_MESSAGE or FRAGMENT)\npadding length in bytes (1 byte)\nzeroes according to padding length\n```\n\n* high read/write throughput is achieved via minimal use of `Unsafe` volatile puts and gets.\n\nWhen a message is placed on the memory-mapped file based queue:\n\n* The header and message bytes above are appended at the current write position in the file but the message length is written as zero (or in our case not written at all because the value defaults to zero). \n* Only once all the message bytes are written is message length given its actual value (and this is done using `Unsafe.putOrderedInt`). \n* When a read is attempted the length field is read using `Unsafe.getIntVolatile` and if zero we do nothing (until the next read attempt). \n\nCancellation complicates things somewhat because pulling the plug suddenly on `Unsafe` memory mapped files means crashing the JVM with a sigsev fault. To reduce contention with cancellation checks, resource disposal is processed by the queue drain method (reads) and writes to the queue are serialized with cancellation via CAS semantics. \n\nTODO \nDescribe fragmentation handling.\n\n### Performance\nThroughput is increased dramatically by using memory-mapped files. \n\n*rxjava2-extras* can push through 800MB/s using 1K messages compared to *rxjava-extras* 43MB/s (2011 i7-920 @2.67GHz). My 2016 2 core i5 HP Spectre laptop with SSD pushes through up to 1.5GB/s for 1K messages.\n\nSmaller messages mean more contention but still on my laptop I am seeing 6 million 40B messages per second.\n\nTo do long-running perf tests (haven't set up jmh for this one yet) do this:\n\n```bash\nmvn test -Dn=500000000\n```\n\nrebatchRequests\n------------------\nConstrains requests to a range of values (rebatches).\n\n\u003cimg src=\"src/docs/rebatchRequests.png?raw=true\" /\u003e \n\n`rebatchRequests` is the composition of the operators [`minRequest`](#minrequest) and [`maxRequest`](#maxrequest).\n\nExample:\n\n```java\nflowable\n  .compose(Transformers.rebatchRequests(5, 100));\n```\nAllow the first request to be unconstrained by the minimum:\n```java\nflowable\n  .compose(Transformers.rebatchRequests(5, 100, false));\n```\n\nSee also: [`minRequest`](#minrequest), [`maxRequest`](#maxrequest)\n\nrepeatLast\n------------------------\nIf a stream has elements and completes then the last element is repeated.\n\n\u003cimg src=\"src/docs/repeatLast.png?raw=true\" /\u003e \n\n```java\nflowable.compose(\n    Transformers.repeatLast());\n```\n\nRetryWhen\n----------------------\nA common use case for `.retry()` is some sequence of actions that are attempted and then after a delay a retry is attempted. RxJava does not provide \nfirst class support for this use case but the building blocks are there with the `.retryWhen()` method. `RetryWhen` offers static methods that build a `Function` for use with `Flowable.retryWhen()`.\n\n\u003cimg src=\"http://reactivex.io/documentation/operators/images/retry.C.png\" width=\"500\"/\u003e\n\n### Retry after a constant delay\n\n```java\nflowable.retryWhen(\n    RetryWhen.delay(10, TimeUnit.SECONDS).build());\n```\n\n### Retry after a constant delay with a maximum number of retries\n\n```java\nflowable.retryWhen(\n    RetryWhen.delay(10, TimeUnit.SECONDS)\n        .maxRetries(10).build());\n```\n\n### Retry after custom delays\n\n```java\n//the length of waits determines number of retries\nFlowable\u003cLong\u003e delays = Flowable.just(10L,20L,30L,30L,30L);\nflowable.retryWhen(\n    RetryWhen.delays(delays, TimeUnit.SECONDS).build());\n```\n\n### Retry only for a particular exception\n\n```java\nflowable.retryWhen(\n    RetryWhen.retryWhenInstanceOf(IOException.class)\n        .build());\n```\n\nreverse\n----------------\nReverses the order of emissions of a stream. Does not emit till source completes.\n\n\u003cimg src=\"src/docs/reverse.png?raw=true\" /\u003e\n\n```java\nflowable.compose(\n    Transformers.reverse());\n```\n\nstateMachine\n--------------------------\nCustom operators are difficult things to get right in RxJava mainly because of the complexity of supporting backpressure efficiently. `Transformers.stateMachine` enables a custom operator implementation when:\n\n* each source emission is mapped to 0 to many emissions (of a different type perhaps) to downstream but those emissions are calculated based on accumulated state\n\n\u003cimg src=\"src/docs/stateMachine.png?raw=true\" /\u003e\n\n[javadoc](http://davidmoten.github.io/rxjava-extras/apidocs/com/github/davidmoten/rx/Transformers.html#stateMachine-rx.functions.Func0-rx.functions.Func3-rx.functions.Action2-)\n\nAn example of such a transformation might be from a list of temperatures you only want to emit sequential values that are less than zero but are part of a sub-zero sequence at least 1 hour in duration. You could use `toListWhile` above (when migrated!) but `Transformers.stateMachine` offers the additional efficiency that it will immediately emit temperatures as soon as the duration criterion is met. \n\nTo implement this example, suppose the source is half-hourly temperature measurements:\n\n```java\nstatic class State {\n     final List\u003cDouble\u003e list;\n     final boolean reachedThreshold;\n     State(List\u003cDouble\u003e list, boolean reachedThreshold) {\n         this.list = list; \n         this.reachedThreshold = reachedThreshold;\n     }\n}\n\nint MIN_SEQUENCE_LENGTH = 2;\n\nFlowableTransformer\u003cDouble, Double\u003e trans = Transformers \n    .stateMachine() \n    .initialStateFactory(() -\u003e new State(new ArrayList\u003c\u003e(), false))\n    .\u003cDouble, Double\u003e transition((state, t, subscriber) -\u003e {\n        if (t \u003c 0) {\n            if (state.reachedThreshold) {\n                if (subscriber.isUnsubscribed()) {\n                    return null;\n                }\n                subscriber.onNext(t);\n                return state;\n            } else if (state.list.size() == MIN_SEQUENCE_LENGTH - 1) {\n                for (Double temperature : state.list) {\n                    if (subscriber.isUnsubscribed()) {\n                        return null;\n                    }\n                    subscriber.onNext(temperature);\n                }\n                return new State(null, true);\n            } else {\n                List\u003cDouble\u003e list = new ArrayList\u003c\u003e(state.list);\n                list.add(t);\n                return new State(list, false);\n            }\n        } else {\n            return new State(new ArrayList\u003c\u003e(), false);\n        }\n    }).build();\nFlowable\n    .just(10.4, 5.0, 2.0, -1.0, -2.0, -5.0, -1.0, 2.0, 5.0, 6.0)\n    .compose(trans)\n    .forEach(System.out::println);\n```\n\nSerialized\n------------------\nTo read serialized objects from a file:\n\n```java\nFlowable\u003cItem\u003e items = Serialized.read(file);\n```\n\nTo write a `Flowable` to a file:\n\n```java\nSerialized.write(flowable, file).subscribe();\n```\n\n### Kryo\n`Serialized` also has support for the very fast serialization library [kryo](https://github.com/EsotericSoftware/kryo). Unlike standard Java serialization *Kryo* can also serialize/deserialize objects that don't implement `Serializable`. \n\nAdd this to your pom.xml:\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.esotericsoftware\u003c/groupId\u003e\n    \u003cartifactId\u003ekryo\u003c/artifactId\u003e\n    \u003cversion\u003e4.0.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nFor example,\n\nTo read:\n```java\nFlowable\u003cItem\u003e items = Serialized.kryo().read(file);\n```\n\nTo write:\n```java\nFlowable.write(flowable, file).subscribe();\n```\n\nYou can also call `Serialized.kryo(kryo)` to use an instance of `Kryo` that you have configured specially. \n\ntoListWhile\n---------------------------\nYou may want to group emissions from a `Flowable` into lists of variable size. This can be achieved safely using `toListWhile`.\n\n\u003cimg src=\"src/docs/toListWhile.png?raw=true\" /\u003e\n\nAs an example from a sequence of temperatures lets group the sub-zero and zero or above temperatures into contiguous lists:\n\n```java\nFlowable.just(10, 5, 2, -1, -2, -5, -1, 2, 5, 6)\n    .compose(Transformers.toListWhile( \n        (list, t) -\u003e list.isEmpty() \n            || Math.signum(list.get(0)) \u003c 0 \u0026\u0026 Math.signum(t) \u003c 0\n            || Math.signum(list.get(0)) \u003e= 0 \u0026\u0026 Math.signum(t) \u003e= 0)\n    .forEach(System.out::println);\n```\nproduces\n```\n[10, 5, 2]\n[-1, -2, -5, -1]\n[2, 5, 6]\n```\n\nSee also [`collectWhile`](#collectwhile). This operator supports [request-one micro-fusion](http://akarnokd.blogspot.com.au/2016/03/operator-fusion-part-1.html).\n\nwindowMin/Max\n----------------------------\nSliding window minimum/maximum:\n\n\u003cimg src=\"src/docs/windowMinMax.png?raw=true\" /\u003e\n\n```java\nFlowable.just(3, 2, 5, 1, 6, 4)\n    .compose(Transformers.\u003cInteger\u003ewindowMin(3))\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidmoten%2Frxjava2-extras","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidmoten%2Frxjava2-extras","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidmoten%2Frxjava2-extras/lists"}