{"id":16782560,"url":"https://github.com/davidmoten/big-sorter","last_synced_at":"2025-04-05T18:07:49.953Z","repository":{"id":38377251,"uuid":"186568917","full_name":"davidmoten/big-sorter","owner":"davidmoten","description":"Java library that sorts very large files of records by splitting into smaller sorted files and merging","archived":false,"fork":false,"pushed_at":"2025-03-20T01:19:24.000Z","size":721,"stargazers_count":86,"open_issues_count":1,"forks_count":19,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-29T17:08:20.684Z","etag":null,"topics":["big-data","java","sorting"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davidmoten.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-14T07:32:57.000Z","updated_at":"2025-03-20T01:19:21.000Z","dependencies_parsed_at":"2023-02-10T17:55:29.001Z","dependency_job_id":"267d7707-3499-45c9-bb73-8c08fe04190f","html_url":"https://github.com/davidmoten/big-sorter","commit_stats":null,"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidmoten%2Fbig-sorter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidmoten%2Fbig-sorter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidmoten%2Fbig-sorter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidmoten%2Fbig-sorter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davidmoten","download_url":"https://codeload.github.com/davidmoten/big-sorter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247378142,"owners_count":20929296,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","java","sorting"],"created_at":"2024-10-13T07:45:52.633Z","updated_at":"2025-04-05T18:07:49.924Z","avatar_url":"https://github.com/davidmoten.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# big-sorter\n\u003ca href=\"https://github.com/davidmoten/big-sorter/actions/workflows/ci.yml\"\u003e\u003cimg src=\"https://github.com/davidmoten/big-sorter/actions/workflows/ci.yml/badge.svg\"/\u003e\u003c/a\u003e\u003cbr/\u003e\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.github.davidmoten/big-sorter/badge.svg?style=flat)](https://maven-badges.herokuapp.com/maven-central/com.github.davidmoten/big-sorter)\u003cbr/\u003e\n[![codecov](https://codecov.io/gh/davidmoten/big-sorter/branch/master/graph/badge.svg)](https://codecov.io/gh/davidmoten/big-sorter)\u003cbr/\u003e\n\nSorts very large files (or `InputStream`s) by splitting to many intermediate small sorted files and merging.\n\nStatus: *deployed to Maven Central*\n\n\u003cimg src=\"src/docs/big-sorter-annotated.png\" style=\"width:100%\"/\u003e\n\nNote that the merge step in the diagram above will happen repeatedly till one file remains.\n\n## Features\n\n* Easy to use builder\n* Single threaded\n* Sorts one billion integers from a file to a new file in 444s \n* Serialization helpers for \n  * lines of strings\n  * Java IO Serialization\n  * DataInputStream base\n  * fixed length binary records \n  * CSV\n  * JSON arrays\n* Serialization is customizable\n* Functional style transforms of input data (`filter`, `map`, `flatMap`, `transform`, `transformStream`), includes java.util.Stream support\n* Compare sorted files (`findSame`, `findDifferent`, `findComplement`)\n* Runtime complexity is O(n log(n))\n* 100% test coverage\n\n## Algorithm\n\nOne or more large files or `InputStream`s of records are sorted to one output file by:\n* splitting the whole files into smaller segments according to `maxItemsPerFile`\n* each segment is sorted in memory and then written to a file\n* the segment files are then merged in groups according to `maxFilesPerMerge`\n* the merged files are repeatedly merged in groups until only one file remains (with all of the sorted entries)\n* Note that, where possible, files are merged with similarly sized files to ensure that we don't start approaching insertion sort computational complexity (O(n\u003csup\u003e2\u003c/sup\u003e).\n* the merge step uses a Min Heap (`PriorityQueue`) for efficiency\n\n## Getting started\nAdd this dependency to your maven pom.xml:\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.github.davidmoten\u003c/groupId\u003e\n    \u003cartifactId\u003ebig-sorter\u003c/artifactId\u003e\n    \u003cversion\u003eVERSION_HERE\u003c/version\u003e\n\u003c/dependency\u003e\n```\nIf you want to sort csv add this extra dependency:\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003eorg.apache.commons\u003c/groupId\u003e\n    \u003cartifactId\u003ecommons-csv\u003c/artifactId\u003e\n    \u003cversion\u003e1.9_OR_LATER\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nIf you want to sort JSON arrays add this extra dependency:\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.fasterxml.jackson.core\u003c/groupId\u003e\n    \u003cartifactId\u003ejackson-databind\u003c/artifactId\u003e\n    \u003cversion\u003eLATEST_VERSION_HERE\u003c/version\u003e\n\u003c/dependency\u003e\n```\nIf you are new to Java or Maven, go to [big-sorter-example](https://github.com/davidmoten/big-sorter-example).\n\n## Serialization\nTo read records from files or InputStreams and to write records to files we need to specify the *serialization* method to use.\n\n### Example for sorting text lines\n\nMake special note of the ability to do functional style transforms of the input data (`filter`, `map`).\n\n```java\nFile in1 = ...\nFile in2 = ...\nFile out = ...\nSorter\n  // set both serializer and natural comparator\n  .linesUtf8()\n  .input(in1, in2)\n  .filter(line -\u003e !line.isEmpty())\n  .filter(line -\u003e !line.startsWith(\"#\"))\n  .map(line -\u003e line.toLowerCase())\n  .output(out)\n  .maxFilesPerMerge(100) // default is 100\n  .maxItemsPerFile(100000) // default is 100,000\n  .initialSortInParallel() // may want to use a large maxItemsPerFile for this to be effective\n  .bufferSize(8192) // default is 8192\n  .sort();\n```\n\nor for a different character set with \"\\r\\n\" line delimiters and in reverse order:\n\n```java\nSorter\n  // set both serializer and natural comparator\n  .serializerLines(charset, LineDelimiter.CARRIAGE_RETURN_LINE_FEED)\n  .comparator(Comparator.reverseOrder())\n  .input(in)\n  .output(out)\n  .sort();\n```\n\n### Example for sorting integers from a text file\nThis approach will work but there is a lot of overhead from `Integer.parseInt` and writing int values as strings:\n\n```java\nSorter\n  .serializerLinesUtf8()\n  .comparator((a, b) -\u003e Integer.compare(Integer.parseInt(a), Integer.parseInt(b)))\n  .input(new File(\"src/test/resources/numbers.txt\"))\n  .filter(line -\u003e !line.isEmpty())\n  .output(new File(\"target/numbers-sorted.txt\"))\n  .sort();\n```\nA more efficient approach (if you need better runtime) is to use an `inputMapper` (you can also use an `outputMapper` at the end):\n\n```java\nFile textInts = new File(\"src/test/resources/numbers.txt\");\n\n// It's much more efficient to be dealing in 4 bytes of integer\n// than strings\nSerializer\u003cInteger\u003e intSerializer = Serializer.dataSerializer( //\n                dis -\u003e (Integer) dis.readInt(), //\n                (dos, v) -\u003e dos.writeInt(v));\n        \nSorter \n  .serializer(intSerializer)\n  .inputMapper(Serializer.linesUtf8(), line -\u003e Integer.parseInt(line))\n  .naturalOrder()\n  .input(textInts)\n  .outputAsStream()\n  .sort()        \n  .peek(System.out::println)\n  .count();\n```\n\nA test was made sorting 100m random integers in a text file (one per line). \n\n* Using the first method the runtime was 456s \n* With the second more efficient method the runtime was 81s\n \n### Example for sorting CSV\nNote that for sorting CSV you need to add the *commons-csv* dependency (see [Gettting started](#getting-started)).\n\nGiven the CSV file below, we will sort on the second column (the \"number\" column):\n```\nname,number,cost\nWIPER BLADE,35,12.55\nALLEN KEY 5MM,27,3.80\n```\n\n```java\nSerializer\u003cCSVRecord\u003e serializer = \n  Serializer.csv(\n    CSVFormat\n      .DEFAULT\n      .withFirstRecordAsHeader()\n      .withRecordSeparator(\"\\n\"),\n    StandardCharsets.UTF_8);\nComparator\u003cCSVRecord\u003e comparator = (x, y) -\u003e {\n    int a = Integer.parseInt(x.get(\"number\"));\n    int b = Integer.parseInt(y.get(\"number\"));\n    return Integer.compare(a, b);\n};\nSorter \n  .serializer(serializer) \n  .comparator(comparator) \n  .input(inputFile) \n  .output(outputFile)\n  .sort();\n```\nThe result is:\n```\nname,number,cost\nALLEN KEY 5MM,27,3.80\nWIPER BLADE,35,12.55\n```\n### Example for sorting fixed length binary\nThis example uses a comparator based on byte arrays of length 32. You can also use [`DataSerializer`](#example-using-the-dataserializer-helper) to do more fine grained extraction from the byte arrays (or to handle non-fixed length records).\n\n```java\nSerializer\u003cbyte[]\u003e serializer = Serializer.fixedSizeRecord(32);\nSorter //\n  .serializer(serializer) \n  .comparator((x, y) -\u003e compare(x, y)) \n  .input(new File(\"input.bin\")) \n  .output(new File(\"sorted.bin\")) \n  .sort();\n```\nYou would of course have to implement the `compare(byte[], byte[])` function yourself ( returns -1 if x \u003c y, 1 if x \u003e y, 0 if x == y).\n\n### Example for sorting a JSON array\nNote that for sorting JSON you need to add the *jackson-databind* dependency (see [Gettting started](#getting-started)).\n\nGiven a JSON array like:\n\n```json\n[\n{ \"name\": \"fred\", \"age\": 23 },\n{ \"name\": \"anne\", \"age\": 31 }\n]\n```\nWe can sort the elements by the \"name\" field like this:\n\n```java\nSorter //\n  .serializer(Serializer.jsonArray()) \n  .comparator((x, y) -\u003e x.get(\"name\").asText().compareTo(y.get(\"name\").asText())) \n  .input(new File(\"input.json\")) \n  .output(new File(\"sorted.json\")) \n  .sort();\n```\n\nand we get:\n```json\n[\n{ \"name\": \"anne\", \"age\": 31 },\n{ \"name\": \"fred\", \"age\": 23 }\n]\n```\nIf your structure is more complex than this (for example the array might not be top-level) then copy and customize the class [JsonArraySerializer.java](src/main/java/com/github/davidmoten/bigsorter/JsonArraySerializer.java).\n\n### Example using Java IO Serialization\nIf each record has been written to the input file using `ObjectOutputStream` then we specify the *java()* Serializer:\n\n```java\nSorter \n  .serializer(Serializer.\u003cLong\u003ejava()) \n  .comparator(Comparator.naturalOrder()) \n  .input(in) \n  .output(out) \n  .sort();\n\n```\n### Example using the DataSerializer helper\nIf you would like to serializer/deserialize your objects using `DataOutputStream`/`DataInputStream` then extend the `DataSerializer` class as below. This is a good option for many binary formats. \n\nLet's use a binary format with a person's name and a height in cm. We'll keep it unrealistically simple with a short field for the length of the persons name, the bytes of the name, and an integer for the height in cm:\n\n```java\npublic static final class Person {\n    final String name;\n    final int heightCm;\n    ...\n}\n\nSerializer\u003cInteger\u003e serializer = new DataSerializer\u003cInteger\u003e() {\n\n    @Override\n    public Person read(DataInputStream dis) throws IOException {\n        short length;\n        // only check for EOF on first item. If it happens after then we have a corrupt file \n        // with incompletely written records\n        try {\n            length= dis.readShort();\n        } catch (EOFException e) {\n            return null;\n        }\n        byte[] bytes = new byte[length];\n        dis.readFully(bytes);\n        String name = new String(bytes, StandardCharsets.UTF_8);\n        int heightCm = dis.readInt();\n        return new Person(name, heightCm);\n    }\n\n    @Override\n    public void write(DataOutputStream dos, Person p) throws IOException {\n        dos.writeShort((short) p.name.length());\n        dos.write(p.name.getBytes(StandardCharsets.UTF_8));\n        dos.writeInt(p.heightCm);\n    }\n\n};\n      \nSorter \n  .serializer(serializer) \n  .comparator((x, y) -\u003e Integer.compare(x.heightCm, y.heightCm)) \n  .input(in) \n  .output(out) \n  .sort();\n``` \n### But my binary file has a header record!\n\nIn that case make a type T that can be header or an item and have your serializer return that T object. In your comparator ensure that the header is always sorted to the top and you are done.\n\n### Custom serialization\nTo fully do your own thing you need to implement the `Serializer` interface.\n\n## Filtering, transforming input\nYour large input file might have a lot of irrelevant stuff in it that you want to ignore or you might want to select only the columns from a csv line that you are interested in. You can use the java.util.Stream API to modify the input or use direct methods `filter`, `map`, `flatMap`:\n\n```java\nSorter\n  // set both serializer and natural comparator\n  .linesUtf8()\n  .input(in)\n  .filter(line -\u003e !line.isEmpty())\n  .filter(line -\u003e !line.startsWith(\"#\"))\n  .map(line -\u003e line.toLowerCase())\n  .output(out)\n  .sort();\n```\nor\n\n```java\nSorter\n  // set both serializer and natural comparator\n  .linesUtf8()\n  .input(in)\n  .transformStream(stream -\u003e \n      stream.filter(line -\u003e !line.isEmpty())\n            .filter(line -\u003e !line.startsWith(\"#\"))\n            .map(line -\u003e line.toLowerCase()))\n  .output(out)\n  .sort();\n```  \n### Converting input files\nBear in mind that the filter, map and transform methods don't change the Serializer and you'll notice that the map method only maps to the same type (so you can modify a string for instance but it stays a string). To transform input into a different format that is more efficient for sorting (like text integer to binary integer) you can use this utility method:\n\n```java\nimport com.github.davidmoten.bigsorter.Util;\n\nstatic \u003cS, T\u003e void convert(File in, Serializer\u003cS\u003e inSerializer, File out,  Serializer\u003cT\u003e outSerializer, Function\u003c? super S, ? extends T\u003e mapper);\n```\n\nFor example, let's convert a file of integers, one per text line to binary integers (4 bytes each):\n\n```java\nSerializer\u003cInteger\u003e intSerializer = Serializer.dataSerializer(\n   dis -\u003e (Integer) dis.readInt(),\n   (dos, v) -\u003e dos.writeInt(v));\n        \n// convert input from text integers to 4 byte binary integers\nFile textInts = new File(\"src/test/resources/numbers.txt\");\nFile ints = new File(\"target/numbers-integers\");\nUtil.convert(textInts, Serializer.linesUtf8(), ints, intSerializer, line -\u003e Integer.parseInt(line));\n```\n\nConverting the input to a more efficient format can make a big difference to the sort runtime. Sorting 100m integers was 6 times faster when the input was converted first to 4 byte binary ints (and that includes the conversion time).\n\n## How to read the output file\nHaving sorted to a file `f`, you can read from that file like so (`Reader` is `Iterable`):\n\n```java\nReader\u003cT\u003e reader = serializer.createReader(f);\nfor (T t: reader) {\n    System.out.println(t);\n}\n```\nAn example with String lines:\n\n```java\nReader\u003cString\u003e reader = Serializer.linesUtf8().createReader(f);\nreader.forEach(System.out::println);\n```\n\nIf you want to stream records from a file do this:\n\n```java\n// ensure reader is closed after handling stream\ntry (Reader\u003cT\u003e reader = serializer.createReader(output)) {\n    Stream\u003cT\u003e stream = reader.stream();\n    ...\n}\n```\n\n## Returning the result as a Stream\u003cT\u003e\n\nYou might want to deal with the results of the sort immediately and be prepared to throw away the output file once read by a stream:\n\n```java\ntry (Stream\u003cString\u003e stream = Sorter\n  .linesUtf8()\n  .input(in)\n  .outputAsStream()\n  .sort()) {\n    stream.forEach(System.out::println);\n}\n``` \nThe interaction is a little bit clumsy because you need the stream to be auto-closed by the try-catch-with-resources block.\nNote especially that a terminal operation (like `.collect(...)` or `count()`) does **not** close the stream. When called,  \nthe close action of the stream deletes the file used as output. If you don't close the stream then you will accumulate final output files in the temp directory and possibly run out of disk.\n\nThe fact that java.util.Stream has poor support for closing resources tempts the author to switch to a more appropriate functional library like [kool](https://github.com/davidmoten/kool). We'll see.\n\nSee [here](#how-to-read-the-output-file) to stream records from a file.\n\n## Comparing sorted files\nOnce you've got multiple sorted files you may want to perform some comparisons. Common comparisons include:\n\n* find common records (use `Util.findSame`)\n* find different records (use `Util.findDifferent`)\n* find records that are not present in the other file (use `Util.findComplement`)\n\nHere's an example of using `Util.findSame` (`findDifferent` and `findComplement` use the same approach):\n\n```java\n// already sorted\nFile a = ...\n// already sorted\nFile b = ...\n// result of the operation\nFile out = ...\nUtil.findSame(a, b, Serializer.linesUtf8(), Comparator.naturalOrder(), out);\n```\n\n## Logging\nIf you want some insight into the progress of the sort then set a logger in the builder:\n\n```java\nSorter\n  .linesUtf8()\n  .input(in)\n  .output(out)\n  .logger(x -\u003e log.info(x))\n  .sort();\n```\nYou can use the `.loggerStdOut()` method in the builder and you will get timestamped output written to the console:\n\n```\n2019-05-25 09:12:59.4+1000 starting sort\n2019-05-25 09:13:03.2+1000 total=100000, sorted 100000 records to file big-sorter2118475291065234969 in 1.787s\n2019-05-25 09:13:05.9+1000 total=200000, sorted 100000 records to file big-sorter2566930097970845433 in 2.240s\n2019-05-25 09:13:08.9+1000 total=300000, sorted 100000 records to file big-sorter6018566301838556627 in 2.243s\n2019-05-25 09:13:11.9+1000 total=400000, sorted 100000 records to file big-sorter4803313760593338955 in 0.975s\n2019-05-25 09:13:14.3+1000 total=500000, sorted 100000 records to file big-sorter9199236260699264566 in 0.962s\n2019-05-25 09:13:16.7+1000 total=600000, sorted 100000 records to file big-sorter2064358954108583653 in 0.989s\n2019-05-25 09:13:19.1+1000 total=700000, sorted 100000 records to file big-sorter6934618230625335397 in 0.964s\n2019-05-25 09:13:21.5+1000 total=800000, sorted 100000 records to file big-sorter5759615033643361667 in 0.975s\n2019-05-25 09:13:24.1+1000 total=900000, sorted 100000 records to file big-sorter6808081723248409045 in 0.948s\n2019-05-25 09:13:25.8+1000 total=1000000, sorted 100000 records to file big-sorter2456434677554311136 in 0.983s\n2019-05-25 09:13:25.8+1000 completed inital split and sort, starting merge\n2019-05-25 09:13:25.8+1000 merging 10 files\n2019-05-25 09:13:36.8+1000 sort of 1000000 records completed in 37.456s\n```\n\n## Memory usage\nMemory usage is directly linked to the value of the `maxItemsPerFile` parameter which you can set in the builder. Its default is 100000. If too much memory is being used reduce that number and test.\n\n\n## Benchmarks\n\n```\n10^3 integers sorted in 0.004s\n10^4 integers sorted in 0.013s\n10^5 integers sorted in 0.064s\n10^6 integers sorted in 0.605s\n10^7 integers sorted in 3.166s\n10^8 integers sorted in 35.978s\n10^9 integers sorted in 444.549s\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidmoten%2Fbig-sorter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidmoten%2Fbig-sorter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidmoten%2Fbig-sorter/lists"}