{"id":25135031,"url":"https://github.com/headless-start/bias-aware-kmeans","last_synced_at":"2026-04-17T00:02:17.202Z","repository":{"id":275238087,"uuid":"925507048","full_name":"headless-start/bias-aware-kmeans","owner":"headless-start","description":"This repository includes Greedy Preprocessing and K-Means Integration for Large-Scale Biased Data.","archived":false,"fork":false,"pushed_at":"2025-02-01T07:53:35.000Z","size":5404,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-19T05:22:47.949Z","etag":null,"topics":["bias","distance-calculation","greedy-algorithm","kmeans-clustering","multiprocessing","python3","silhouette-score","subsampling"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/headless-start.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-01T03:06:30.000Z","updated_at":"2025-02-08T08:42:39.000Z","dependencies_parsed_at":null,"dependency_job_id":"3db62b6c-5a34-46fc-8c96-003bb78bb328","html_url":"https://github.com/headless-start/bias-aware-kmeans","commit_stats":null,"previous_names":["headless-start/bias_aware_k_means","headless-start/bias-aware-kmeans"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/headless-start/bias-aware-kmeans","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/headless-start%2Fbias-aware-kmeans","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/headless-start%2Fbias-aware-kmeans/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/headless-start%2Fbias-aware-kmeans/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/headless-start%2Fbias-aware-kmeans/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/headless-start","download_url":"https://codeload.github.com/headless-start/bias-aware-kmeans/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/headless-start%2Fbias-aware-kmeans/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31909235,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-16T18:22:33.417Z","status":"ssl_error","status_checked_at":"2026-04-16T18:21:47.142Z","response_time":69,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bias","distance-calculation","greedy-algorithm","kmeans-clustering","multiprocessing","python3","silhouette-score","subsampling"],"created_at":"2025-02-08T16:17:44.556Z","updated_at":"2026-04-17T00:02:17.180Z","avatar_url":"https://github.com/headless-start.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Biased Dataset Clustering: Greedy Preprocessing \u0026 k-Means Integration on Large Scale Data\n\n## 📌 Project Overview  \nThis project addresses challenges in applying **k-means clustering** to biased datasets by implementing a **parallelized greedy clustering algorithm** for preprocessing. The algorithm reduces dataset size by selecting representatives based on a tunable distance threshold (τ), enabling efficient clustering. A comparative analysis with **random subsampling** evaluates computational efficiency and clustering quality.\n\n**Dataset**: 118,821 data points with inherent biases (e.g., age, wealth).  \n**Goal**: Mitigate bias effects by preprocessing data into representative clusters (1%, 10%, 25% sizes) and compare methods.\n\n---\n\n## 🚀 Key Features  \n1. **Parallelized Greedy Clustering**:  \n   - Reduces dataset to target cluster ratios (1%, 10%, 25%) via adaptive τ tuning.  \n   - Optimized for cache behavior and minimal memory usage.  \n2. **k-Means Integration**:  \n   - Clusters representatives from preprocessing step.  \n   - Post-processing assigns original data points to clusters.  \n3. **Random Subsampling Baseline**:  \n   - Generates comparison dataset by randomly sampling equivalent proportions.  \n4. **Performance Analysis**:  \n   - Metrics: Runtime, memory usage, Silhouette Score, intra/inter-cluster distances.  \n\n---\n\n## 🔍 Findings  \n1. **Greedy Clustering**:  \n   - Achieved target cluster sizes (1%, 10%, 25%) with τ=100.  \n   - **Runtime**: 43.19s | **Memory**: 1.03MB.  \n   - **Clustering Quality**: Silhouette Score (-0.0029), Intra-cluster distance (15,289.29).  \n2. **Random Subsampling**:  \n   - **Runtime**: 37.14s | **Memory**: Negligible.  \n   - **Clustering Quality**: Silhouette Score (0.1127), Intra-cluster distance (12,397.45).  \n3. **Conclusion**:  \n   - Subsampling outperformed in speed and clustering quality for this dataset.  \n   - Greedy clustering offers structured preprocessing for bias mitigation but requires tuning.  \n\n---\n\n## 🛠 System Requirements \n### Dependencies  \n- Python 3.8+  \n- Libraries: `numpy`, `pandas`, `scikit-learn`, `multiprocessing`\n\n---\n\n## 📄 License  \nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fheadless-start%2Fbias-aware-kmeans","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fheadless-start%2Fbias-aware-kmeans","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fheadless-start%2Fbias-aware-kmeans/lists"}