{"id":21565258,"url":"https://github.com/sandeepkv93/distributed-kth-frequent","last_synced_at":"2025-03-18T05:17:46.844Z","repository":{"id":262697475,"uuid":"888067845","full_name":"sandeepkv93/distributed-kth-frequent","owner":"sandeepkv93","description":null,"archived":false,"fork":false,"pushed_at":"2024-11-13T19:11:03.000Z","size":19,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-24T11:44:48.599Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sandeepkv93.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-13T18:56:51.000Z","updated_at":"2024-11-13T19:11:07.000Z","dependencies_parsed_at":"2024-11-13T20:18:39.515Z","dependency_job_id":"48f18fa0-da7d-4c51-a756-8b8125ffe15f","html_url":"https://github.com/sandeepkv93/distributed-kth-frequent","commit_stats":null,"previous_names":["sandeepkv93/distributed-kth-frequent"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sandeepkv93%2Fdistributed-kth-frequent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sandeepkv93%2Fdistributed-kth-frequent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sandeepkv93%2Fdistributed-kth-frequent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sandeepkv93%2Fdistributed-kth-frequent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sandeepkv93","download_url":"https://codeload.github.com/sandeepkv93/distributed-kth-frequent/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244160055,"owners_count":20408021,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-24T10:19:01.355Z","updated_at":"2025-03-18T05:17:46.822Z","avatar_url":"https://github.com/sandeepkv93.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🎯 Distributed Kth Frequent Element Finder\n\nA high-performance, distributed system for finding the kth most frequent element in a dataset. Built with Java 21 and designed for scalability and reliability.\n\n## 🚀 Features\n\n- **Distributed Processing**: Efficiently processes large datasets across multiple nodes\n- **Memory-Aware**: Intelligent memory management with configurable thresholds per node\n- **Fault Tolerant**: Handles node failures and timeouts gracefully\n- **Highly Scalable**: Supports dynamic node configuration and load balancing\n- **Thread-Safe**: Concurrent processing with proper synchronization\n- **Comprehensive Testing**: Extensive test coverage including unit, integration, and stress tests\n\n## 🛠️ Technical Stack\n\n- Java 21 (with preview features)\n- Maven for build management\n- Lombok for reducing boilerplate\n- SLF4J + Logback for logging\n- JUnit 5 for testing\n- Google Java Format for consistent code style\n- Apache Commons \u0026 Google Guava utilities\n\n## 📋 Prerequisites\n\n- Java 21 or higher\n- Maven 3.8+\n- At least 1GB of RAM (configurable based on dataset size)\n\n## 🏗️ Building the Project\n\n```bash\nmvn clean install\n```\n\nThis will:\n\n1. Compile the source code\n2. Run the tests\n3. Create an executable JAR with dependencies\n4. Format the code using Google Java Format\n\n## 🎮 Usage\n\n### Basic Example\n\n```java\n// Create a coordinator with 3 nodes and 1MB memory per node\nCoordinator coordinator = new Coordinator(3, 1024 * 1024);\n\n// Find the 3rd most frequent element\nList\u003cInteger\u003e data = Arrays.asList(9, 9, 6, 9, 8, 6, 8, 6, 4);\nint result = coordinator.findKthFrequent(data, 3);\n\n```\n\n### Advanced Configuration\n\n```java\n// Configure for large datasets with more nodes\nint numNodes = 8;\nlong memoryPerNode = 2 * 1024 * 1024;// 2MB per node\nCoordinator coordinator = new Coordinator(numNodes, memoryPerNode);\n\n// Process millions of records\nList\u003cInteger\u003e largeDataset = generateLargeDataset(1_000_000);\nint result = coordinator.findKthFrequent(largeDataset, 5);\n\n```\n\n## 🏛️ Architecture\n\nThe system consists of three main components:\n\n### 1. Coordinator (Coordinator.java)\n\n- Manages distributed processing workflow\n- Handles data partitioning and distribution\n- Aggregates results from nodes\n- Implements fault tolerance mechanisms\n\n### 2. Processing Nodes (ProcessingNode.java)\n\n- Processes data partitions independently\n- Maintains memory usage within thresholds\n- Reports processing status and results\n- Handles local error recovery\n\n### 3. Data Models\n\n- `DataPartition.java`: Encapsulates data chunks for processing\n- `FrequencyPair.java`: Represents element-frequency pairs\n- `ProcessingResult.java`: Contains node processing results\n\n```mermaid\nsequenceDiagram\n    participant C as Coordinator\n    participant N1 as Node 1\n    participant N2 as Node 2\n    participant N3 as Node 3\n\n    Note over C,N3: Phase 1: Data Distribution\n    C-\u003e\u003eN1: Chunk 1 of data\n    C-\u003e\u003eN2: Chunk 2 of data\n    C-\u003e\u003eN3: Chunk 3 of data\n\n    Note over C,N3: Phase 2: Local Processing\n    activate N1\n    N1-\u003e\u003eN1: Count frequencies\n    activate N2\n    N2-\u003e\u003eN2: Count frequencies\n    activate N3\n    N3-\u003e\u003eN3: Count frequencies\n\n    Note over C,N3: Phase 3: Send Local Results\n    N1--\u003e\u003eC: Local frequency dict 1\n    deactivate N1\n    N2--\u003e\u003eC: Local frequency dict 2\n    deactivate N2\n    N3--\u003e\u003eC: Local frequency dict 3\n    deactivate N3\n\n    Note over C: Phase 4: Global Processing\n    activate C\n    C-\u003e\u003eC: Merge frequency dictionaries\n    C-\u003e\u003eC: Create max heap\n    C-\u003e\u003eC: Extract Kth frequent element\n    deactivate C\n\n    Note over C: Phase 5: Memory Management\n    activate C\n    C-\u003e\u003eC: Check memory usage\n    alt Memory fits\n        C-\u003e\u003eC: Direct processing\n    else Memory exceeded\n        C-\u003e\u003eC: Use Count-Min Sketch\n        C-\u003e\u003eC: Process in batches\n    end\n    deactivate C\n```\n\n```mermaid\nflowchart TD\n    A[Input Large Dataset] --\u003e B[Distribute Data]\n    \n    subgraph Local_Processing[\"Local Processing (Map Phase)\"]\n        B --\u003e C1[Node 1]\n        B --\u003e C2[Node 2]\n        B --\u003e C3[Node 3]\n        \n        C1 --\u003e D1[Count Local Frequencies]\n        C2 --\u003e D2[Count Local Frequencies]\n        C3 --\u003e D3[Count Local Frequencies]\n        \n        D1 --\u003e E1[Local Dict 1]\n        D2 --\u003e E2[Local Dict 2]\n        D3 --\u003e E3[Local Dict 3]\n    end\n    \n    subgraph Global_Processing[\"Global Processing (Reduce Phase)\"]\n        E1 --\u003e F[Merge Dictionaries]\n        E2 --\u003e F\n        E3 --\u003e F\n        F --\u003e G[Global Frequency Dict]\n    end\n    \n    subgraph Memory_Management[\"Memory Management\"]\n        G --\u003e H{Memory Check}\n        H --\u003e|Fits| I[Direct Processing]\n        H --\u003e|Exceeded| J[Batch Processing]\n        J --\u003e K[Count-Min Sketch]\n    end\n    \n    subgraph Final_Processing[\"Final Processing\"]\n        I --\u003e L[Create Max Heap]\n        K --\u003e L\n        L --\u003e M[Extract K Elements]\n        M --\u003e N[Return Kth Element]\n    end\n    \n    classDef default fill:#f9f9f9,stroke:#333,stroke-width:2px\n    classDef phase fill:#e1f5fe,stroke:#01579b,stroke-width:2px\n    \n    class Local_Processing,Global_Processing,Memory_Management,Final_Processing phase\n```\n\n## 🧪 Testing\n\n```bash\n# Run all tests\nmvn test\n\n# Run only unit tests\nmvn test -Dtest=*Test\n\n# Run integration tests\nmvn verify -P integration-tests\n\n```\n\nTest categories:\n\n- 🎯 Unit Tests: Individual component testing\n- 🔄 Integration Tests: End-to-end workflow testing\n- 💪 Stress Tests: Performance and stability testing\n- 🐛 Edge Cases: Boundary condition testing\n- 🔀 Concurrent Tests: Multi-threading scenarios\n\n## 📊 Performance Characteristics\n\nThe system is optimized for:\n\n- 📈 Large datasets (millions of records)\n- 🔄 Multiple concurrent requests\n- 💾 Memory-constrained environments\n- 📊 Various data distributions:\n    - Normal distribution\n    - Skewed data\n    - Sparse datasets\n    - Power law distribution\n\n## 🚀 Getting Started\n\n1. Clone the repository:\n\n```bash\ngit clone https://github.com/yourusername/distributed-kth-frequent.git\n\n```\n\n2. Navigate to project directory:\n\n```bash\ncd distributed-kth-frequent\n\n```\n\n3. Build the project:\n\n```bash\nmvn clean install\n\n```\n\n4. Run the example:\n\n```bash\njava -jar target/kth-frequent-1.0-SNAPSHOT-jar-with-dependencies.jar\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsandeepkv93%2Fdistributed-kth-frequent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsandeepkv93%2Fdistributed-kth-frequent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsandeepkv93%2Fdistributed-kth-frequent/lists"}