{"id":21635262,"url":"https://github.com/stefangt44/concurrent-word-distribution-tool","last_synced_at":"2026-04-29T08:33:11.593Z","repository":{"id":174292577,"uuid":"292297789","full_name":"stefanGT44/Concurrent-Word-Distribution-Tool","owner":"stefanGT44","description":"A JavaFX desktop application for concurrently computing the distribution of specified words in large files/directories and plotting the results on a graph.","archived":false,"fork":false,"pushed_at":"2020-09-15T23:09:54.000Z","size":22368,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-04T21:53:42.564Z","etag":null,"topics":["component-based","concurrent-programming","desktop-application","graph","javafx"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stefanGT44.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-09-02T13:55:48.000Z","updated_at":"2020-09-15T23:09:56.000Z","dependencies_parsed_at":null,"dependency_job_id":"81578af5-9e87-434c-ad16-22245a0d4641","html_url":"https://github.com/stefanGT44/Concurrent-Word-Distribution-Tool","commit_stats":null,"previous_names":["stefangt44/concurrent-word-distribution-tool"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/stefanGT44/Concurrent-Word-Distribution-Tool","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanGT44%2FConcurrent-Word-Distribution-Tool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanGT44%2FConcurrent-Word-Distribution-Tool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanGT44%2FConcurrent-Word-Distribution-Tool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanGT44%2FConcurrent-Word-Distribution-Tool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stefanGT44","download_url":"https://codeload.github.com/stefanGT44/Concurrent-Word-Distribution-Tool/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanGT44%2FConcurrent-Word-Distribution-Tool/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32417655,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-29T06:29:02.080Z","status":"ssl_error","status_checked_at":"2026-04-29T06:29:00.631Z","response_time":110,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["component-based","concurrent-programming","desktop-application","graph","javafx"],"created_at":"2024-11-25T03:20:32.788Z","updated_at":"2026-04-29T08:33:11.576Z","avatar_url":"https://github.com/stefanGT44.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Concurrent-Word-Distribution-Tool\nA JavaFX desktop application for concurrently computing the distribution of specified words in large directories/files and plotting the results on a graph. (Distribution - the number of times each word appears in a text file)\n\n## Overview\nThe application functions as a pipeline that consists of several types of components that are working concurrently in conjunction.\u003cbr\u003e\nThere are three types of components:\n1. Input component - data entry point\n2. Cruncher component - data processing\n3. Output component - storing and visualizing results\n\nThe user can make multiple instances of each component and link (connect) them in a way he sees fit.\u003cbr\u003e\nEvery component instance runs in its own thread and every component type has a dedicated \u003cb\u003ethread pool.\u003c/b\u003e\u003cbr\u003e\nInput components provide input to cruncher components, which provide input to the output components.\u003cbr\u003e\nComponent communication (data flow) is based on shared \u003cb\u003eblocking queues\u003c/b\u003e. \u003cbr\u003e\nThe architecture of the system makes it easy to integrate new types of components.\u003cbr\u003e\nThe application is optimized to use as little RAM as possible. \u003cbr\u003e\nComponents and the main app follow the \u003cb\u003eMVC\u003c/b\u003e design pattern.\u003cbr\u003e\u003cbr\u003e\n\n#### Component pipeline example:\n![Alt text](images/wdt.png?raw=true \"\")\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\n\n## Usage example\n\n![Alt text](images/de4.png?raw=true \"\")\u003cbr\u003e\u003cbr\u003e\nInput 0 is linked to Cruncher 0 which is automatically linked to the default output component.\u003cbr\u003e\nInput 0 is active and currently reading one text file (see the bottom blue label).\u003cbr\u003e\nCruncher 0 is currently computing the distribution in three files that Input 0 has provided.\u003cbr\u003e\nCruncher progress can also be monitored in the output component, if an item in the list has a prefix \\*, the results are not ready yet (cruncher is still working on that file).\n\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\n\n![Alt text](images/de5.png?raw=true \"\")\u003cbr\u003e\u003cbr\u003e\nIn this image the Input and Cruncher components have finished their work from the previous image.\u003cbr\u003e\nThe output component is showing the distribution of words in the file wiki-7.txt.\u003cbr\u003e\nIt is also currently computing the sum distribution that the user specified.\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\n\n![Alt text](images/de6.png?raw=true \"\")\u003cbr\u003e\u003cbr\u003e\nIn this example the output component is computing the specified distribution sum (aggregation) and is waiting for the final file results to become available, in order to finish the computation.\n\n## Component details:\nEvery component instance runs in its own thread and every component type has a dedicated thread pool for completing its main tasks.\u003cbr\u003e\nComponents communicate among each other using blocking queues. Every component has a blocking queue that its predecessors can write to.\u003cbr\u003e\n\n### Input components:\nEvery input component can be linked to one or more cruncher components.\u003cbr\u003e\nThe main objective of input components is to scan directories for text files which are then read and supplied to linked crunchers.\u003cbr\u003e\nThe reading of text files is done in a separate task within the input thread pool.\u003cbr\u003e\nInput components are tied to a disk (drive) that the user specifies when creating a new instance. \u003cbr\u003e\nOnly directories on the specified disk can be scanned, and only one reading task can be active in the thread pool per disk. \u003cbr\u003e\nAfter one scan cycle is finished, the component pauses for a certain duration before the next cycle (specified in the config file). \u003cbr\u003e\nThe user can manually pause and resume input components. \u003cbr\u003e\nThe last modified value of scanned directories is tracked, so if a directory has been modified, it is scanned again (the text files are read again). \u003cbr\u003e\n\n### Cruncher components:\nIn the current implementation, cruncher components are automatically linked to one default output component, but the code supports multiple output components.\nThe main objective of cruncher components is to count the word distribution in text objects that linked input components provided, and supply linked output components with the results. \u003cbr\u003e\nUpon receiving input text, a new \u003cb\u003eRecursiveTask\u003c/b\u003e is created within the cruncher \u003cb\u003ethread pool\u003c/b\u003e and a \u003cb\u003eFuture\u003c/b\u003e object is forwarded to all linked output components.\u003cbr\u003e\nThe task recursively creates new tasks and splits the job (text) into smaller chunks (chunk size specified in the config file), after which the distribution computation is done, and finally the results are combined.\nEvery cruncher instance has a specified arity number.\u003cbr\u003e\nIf arity = \u003cb\u003e1\u003c/b\u003e the cruncher counts the number of times every \u003cb\u003esingle\u003c/b\u003e word appears in a text,\u003cbr\u003e\nif arity = \u003cb\u003e3\u003c/b\u003e the cruncher counts the number of times every \u003cb\u003ethree\u003c/b\u003e consecutive words, in exactly that order, appear in a text , etc.\u003cbr\u003e\n\n### Output components:\nOutput components store results provided by the linked crunchers.\u003cbr\u003e\nThe results can be aggregated (this is done within the output component thread pool), sorted and plotted on the graph.\nOutput components are aware of all created jobs, even unfinished ones (active jobs have * as a prefix). \u003cbr\u003e\nThe component offers get (blocking) and poll (not blocking) methods for retrieving results.\nSingle result plotting uses the poll method and notifies the user if results are not ready yet. \u003cbr\u003e\nThe aggregation task uses the get method and waits (is blocked) if some results are not ready yet. \u003cbr\u003e\nAll types of results (single or aggregated) are sorted before plotting. \u003cbr\u003e\n\n## System quality:\nThe application is optimized to use as little RAM as possible. But in the events that RAM runs out, the user is notified and the application shut down. \u003cbr\u003e\nGUI buttons, lists, labels are always refreshed and enabled only when that makes sense. \u003cbr\u003e\nThe user is notified when errors occur with an error message alert. \u003cbr\u003e\nWhen exiting the application, new jobs cannot be started, and all unfinished jobs must finish (reading a text file, cruncher working on a file, output aggregating, sorting or plotting results). If unfinished jobs exist, the user is shown a modal dialog with a message that the application is in the process of exiting. \n\n## Configuration file (app.properties):\nParameters are read during app start and cannot be changed during app operation. \u003cbr\u003e\u003cbr\u003e\nFile structure: \u003cbr\u003e\u003cbr\u003e\nfile_input_sleep_time=5000 - pause duration for the input component \u003cbr\u003e\ndisks=data/disk1/;data/disk2 - list of disks for the input component \u003cbr\u003e\ncounter_data_limit=10000000 - job limit for counting tasks given in characters \u003cbr\u003e\nsort_progress_limit=10000 - number of comparisons after which progress bar is updated during sorting \u003cbr\u003e\u003cbr\u003e\n\n## Sidenote\nThis project was an assignment as a part of the course - Concurrent and Distributed Systems during the 8th semester at the Faculty of Computer Science in Belgrade. All system functionalities were defined in the assignment specifications.\n\n## Download\nYou can download the .jar files [here](download/Concurrent-Distribution-Tool.zip).\u003cbr\u003e\n\n## Contributors\n- Stefan Ginic - \u003cstefangwars@gmail.com\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefangt44%2Fconcurrent-word-distribution-tool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstefangt44%2Fconcurrent-word-distribution-tool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefangt44%2Fconcurrent-word-distribution-tool/lists"}