{"id":22918165,"url":"https://github.com/shridhad/externalsorting","last_synced_at":"2025-04-01T13:16:53.187Z","repository":{"id":147371351,"uuid":"193765547","full_name":"Shridhad/externalsorting","owner":"Shridhad","description":"External merge sort in Java","archived":false,"fork":false,"pushed_at":"2019-06-26T05:13:10.000Z","size":66,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-07T08:13:45.810Z","etag":null,"topics":["external-sort","java"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Shridhad.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-25T18:50:26.000Z","updated_at":"2023-03-05T05:36:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"b9796bd3-6cff-4685-b1e2-93f39cbe22b6","html_url":"https://github.com/Shridhad/externalsorting","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shridhad%2Fexternalsorting","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shridhad%2Fexternalsorting/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shridhad%2Fexternalsorting/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shridhad%2Fexternalsorting/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Shridhad","download_url":"https://codeload.github.com/Shridhad/externalsorting/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246644098,"owners_count":20810687,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["external-sort","java"],"created_at":"2024-12-14T06:29:14.443Z","updated_at":"2025-04-01T13:16:53.166Z","avatar_url":"https://github.com/Shridhad.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"## External Sort\n\nThis is demo application for external merge sort. The application reads `N` numbers from input file, \nsorts and stores in temporary file. Once the input file is read completely, all temp files are merged\ninto one single file.  \n\n#### Problem\n\n\u003e Given with a line separated text file of integers ranging anywhere from Integer.MIN to\nInteger.MAX, the program should be able to produce line separated text\nfile which has the sorted content of the input file.\n\n\n### Execution\n\n* Clone the repository and run `gradle build` to install dependencies\n* Import the project into IDE and run `Application`\n\n\n#### Generate big data\n\n* Running `sh generate.sh` should generate `src/main/resources/input-big.txt` file with 50000 numbers.\n* You can run the script multiple times to add more numbers in same file.\n* To sort this file you need to make small change in `Application.java:14`, to change the input file name.\n\n\n#### Performance Optimization \n\n- Implement K-Way merge using `PriorityQueue` to merge the tempFiles. All tempFiles will merged at once.\n   - Right now the merging process is really slow taking more than 90% of overall time and it hardly utilizes 25MB of memory \n- Parallelize the split and sorting of tempFiles\n    - Read `N` numbers from input file, create an async task to sort and save those numbers in tempFile\n- Faster read operation\n    - Read all `N` numbers from input file at once, instead of using `readLine` \n- Dynamically decide the size of tempFiles based of available memory\n\n\u003e Note: Ran with 1.6GB of data set and it took around 20 minutes to sort the file. \n\n### Improvements\n\n- Take input file and output files as arguments  \n- Log the progress on console, at each stage\n- Summary - Total time taken, total temp files created, count of numbers\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshridhad%2Fexternalsorting","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshridhad%2Fexternalsorting","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshridhad%2Fexternalsorting/lists"}