{"id":22955136,"url":"https://github.com/rajshrestha86/kmeans-clusterize","last_synced_at":"2025-10-18T03:20:58.800Z","repository":{"id":110777069,"uuid":"439930876","full_name":"rajshrestha86/kmeans-clusterize","owner":"rajshrestha86","description":"Implementation of K-Means clustering algorithm from scratch. Parallelization of clustering algorithm across multiple nodes using OpenMP and MPI to reduce clustering time on a huge dataset. Also performance analysis of multiple approaches used and their comparison.","archived":false,"fork":false,"pushed_at":"2021-12-19T19:02:13.000Z","size":3885,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-07T16:16:37.894Z","etag":null,"topics":["c","kmeans-clustering","mpi","openmp","unsupervised-learning"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rajshrestha86.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-12-19T17:51:26.000Z","updated_at":"2022-05-18T14:05:58.000Z","dependencies_parsed_at":"2023-03-30T19:52:51.497Z","dependency_job_id":null,"html_url":"https://github.com/rajshrestha86/kmeans-clusterize","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rajshrestha86%2Fkmeans-clusterize","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rajshrestha86%2Fkmeans-clusterize/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rajshrestha86%2Fkmeans-clusterize/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rajshrestha86%2Fkmeans-clusterize/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rajshrestha86","download_url":"https://codeload.github.com/rajshrestha86/kmeans-clusterize/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246735343,"owners_count":20825223,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["c","kmeans-clustering","mpi","openmp","unsupervised-learning"],"created_at":"2024-12-14T16:27:58.201Z","updated_at":"2025-10-18T03:20:53.766Z","avatar_url":"https://github.com/rajshrestha86.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# KMeans Clustering\nKMeans clustering is a exploratory data-analysis technique which uses vector quantization to partition data into sub-groups. KMeans clustering groups data into groups which are called clusters. The data inside the cluster are similar to each other and the data between the clusters are different and are far from each other. Each cluster has a centroid. A datapoint belongs to a cluster if the distance between it and the centroid is minimum when compared to other clusters. \nThis approach groups similar data to each other and differentiates the data between the clusters. \nThe algorithm starts with the random data points assigned to clusters as initial centroids. Then for each datapoint the algorithm determines the nearest cluster according to centroids. After all the data points are assigned to clusters then a new centroid is calculated by computing the average of all the data points in a cluster. The algorithm then repeats until a certain number of iterations or until the centroid doesn’t change on further iteration. \n\n## Preparation of Dataset\nWe used a dataset of songs in spotify found from kaggle.\nThe dataset contains 42305 rows and 22 different columns. In order to decide the columns to be used for the clustering and the number of clusters to be determined we created an elbow diagram (Fig 1) with sum of squared distances for different numbers of  clusters as shown in Fig 1. From fig we can see that the elbow lies in between 6 and 7. \nBased on that the number of clusters was decided to be 6. Similarly, there were 22 different columns which include 10 numerical columns. To decide the columns to be used in clustering we created a correlation heatmap. From the correlation heatmap, 'danceability', 'energy', 'loudness', 'speechiness', 'liveness', and 'valence' was decided to be used for the clustering purpose.\n\u003e The dataset is obtained from \u003ca href=\"https://www.kaggle.com/mrmorj/dataset-of-songs-in-spotify\"\u003ekaggle\u003c/a\u003e.  It's also stored in data dir.\n\n### Correlation heatmap and Elbow Diagram\n\u003cimg src=\"img/elbow.png\" width=\"480\"/\u003e\n\u003cimg src=\"img/corr-map.png\" width=\"480\"/\u003e\n\n## Project Structure\nThe repository  includes the following directories.\n\n- “data”     -\u003e Contains a dataset used for clustering.\n- “lib” -\u003e\n    - csvReader.h: Code used to read dataset in data folder.\n    - kmeans.c, kmeans.h: Serial implementation of KMeans clustering.\n- “mpi” -\u003e MPI implementation of KMeans clustering and driver code   “main_mpi.c”. It also re-uses some codes from serial implementation in lib.\n- “openmp” -\u003e\nOpenmp implementation of KMeans clustering and driver code   “main_omp.c”. It also re-uses some codes from serial implementation in lib.\n- “main.c” -\u003e Serial driver code of KMeans clustering.\n\n## Serial Implementation\n#### Build Instructions:\n```bash\n$ gcc main.c lib/csvReader.h lib/kmeans.c -o main -lm -std=c11\n$ ./main\n```\n\nThe serial implementation of KMeans clustering algorithm is done in lib/kmeans.c. The file consists of KMeans algorithm and helper functions used by the fit function. The fit function takes the “struct KMeans'' model and runs a clustering algorithm on that model. The algorithm runs for 300 iterations. Before running a fit function the driver code initializes the model with random data points as centroids. For each iteration, the algorithm loops over all the data rows, assigns rows to clusters based on the nearest euclidean distance and re-compute the centroids by determining the average of all the data points inside the cluster. The centroid update algorithm is done using the “update_centroids'' method. The algorithm runs for 300 iterations. The driver code displays the results of the cluster counts for each cluster after the completion of fit method.\n\n***The program took 4.0000 seconds to execute on the “kingspeak” cluster.***\n\n\n## OpenMP Implementation\n#### Build Instructions:\n```bash\n$ cd openmp\n$ gcc -Wall main_omp.c kmeans_omp.c ../lib/kmeans.c -fopenmp -lm -o main_omp\n$ ./main \u003cno_threads\u003e    \n```\n\nThe OpenMP Implementation of KMeans algorithm is straightforward. The serial implementation iterates through each data point and assigns clusters to them. After assigning clusters to each data point it then recomputes the centroid for each cluster by computing an average. We can parallelize this algorithm by parallelizing the for loop used to iterate each data point and assign the cluster. Also we can parallelize the operation of updating centroids after each iteration. Since a centroid update is a calculation to compute mean data points in a centroid we can parallelize this operation too. From the following graphs, we can see that the time taken to complete the algorithm decreases as we increase the number of processor cores. Similarly, the speedup increases as we increase the number of cores. However, at 64 cores, the time is increased and speed up is decreased when compared to 32 cores. The test was run on a kingspeak clusters and as the number of cores reaches to the maximum numbers that system can provide and when there are other processes running as well, the operating system context switches between\nthe threads, thus decreasing the total time when the number of threads is 64. Similarly, the results obtained from this implementation are consistent with other implementations.\n\n\u003cimg src=\"img/openmp-res.png\"\u003e\n\n### Clustering Results\n\u003cimg src=\"img/kmean-res.png\"\u003e\n\n\n## CUDA Implementation\nThe repository for CUDA Implementation of KMeans can be found here.\n#### Build Instructions:\n```bash\n$ nvcc main.cu ./lib/kmeans.cu -o main \n$ ./main\n```\nThe CUDA Implementation of KMeans algorithm parallelizes the fit function and centroid update method. The iteration of data rows to assign data points to clusters based on the euclidean distance can be parallelized among multiple CUDA cores. Each core can work on a partition of data and assign them to respective clusters. After computation of data classification, all the threads in the block call the update_centroid method, to compute the average of the data points. Update centroid method uses the shared memory to store TPB (Threads Per Block) number of data items in a shared memory. This is used later to compute the average and reduce the global memory access time. After respective data points have been loaded, the first thread of each block uses the ReductionSum operation to sum up the total centroids and number of data items in the cluster. After each iteration, the variables in device code are then copied back to the host using cudaMemCpy and a new centroid is calculated and updated. \n\nHere are the timing performances and speedup of the KMeans algorithm done on notchpeak cluster. We can see that, as the number of CUDA cores increases, the time taken to complete a k-means algorithm decreases. Similarly, the speedup increases as the number of CUDA cores increases. The results of the clustering are also consistent with other implementation methods.\n\n\u003cimg src=\"img/cuda-res.png\"\u003e\n\n## MPI Implementation\n#### Build Instructions:\n```bash\n$ cd mpi\n$ mpicc -Wall main_mpi.c kmeans_mpi.c ../lib/kmeans.c -lm -o main_mpi -std=c11\n$ mpiexec -n \u003cno_processes\u003e ./main_mpi\n```\n\nThe MPI Implementation of KMeans clustering problem makes use of multiple processes to compute the centroids and the number of data points in the cluster. The root process, rank = 0, is responsible for loading the data from a dataset and scattering the data among other mpi processes. SInce the data may not be divisible by the number of processes, scatterv is used to send the data among the processes. Before scattering the data, some meta information is broadcasted from the root process to other processes so they will be able to receive the data. After the data is scattered, each process computes a new centroid and a reduction operation is done on a global centroid. After all the centroids have been collected from other processes, computed the average between the processes and broadcasted a new centroid to all processes. This operation is repeated for 300 iterations and then results are displayed.\n\u003cimg src=\"img/mpi-res.png\" /\u003e\n\nSimilar to other implementation approaches, the time taken to complete the algorithm decreases as we increase the number of processes. But as the number of processes is increased the total time to compute the clusters decreases as the work is divided among the processes. The reason behind this is, the total time also includes the time taken to scatter the data among the processes. And as the processes increase the overhead time of communication between the processes also increases thus decreasing speedup of the program. The results obtained from the implementation are also consistent with other implementations.\n\n### Results\nLooking at the results obtained from each implementation we can conclude that, MPI implementation of KMeans Clustering is by far the efficient and fast approach to implement the Clustering Algorithm. On fewer cores, CUDA implementation is the worst performing implementation. The reason behind this is the access of global memory for the data point on fewer cores and overhead of the “Memcpy” function on the execution of the program.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frajshrestha86%2Fkmeans-clusterize","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frajshrestha86%2Fkmeans-clusterize","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frajshrestha86%2Fkmeans-clusterize/lists"}