{"id":16841703,"url":"https://github.com/redtachyon/arxiv-clustering","last_synced_at":"2025-03-18T05:23:00.349Z","repository":{"id":163051361,"uuid":"638035700","full_name":"RedTachyon/arxiv-clustering","owner":"RedTachyon","description":"A simple clustering of arXiv articles","archived":false,"fork":false,"pushed_at":"2023-05-09T00:42:49.000Z","size":77,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-01-24T12:13:10.816Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RedTachyon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-09T00:27:16.000Z","updated_at":"2023-05-09T12:46:41.000Z","dependencies_parsed_at":"2023-06-02T13:45:31.999Z","dependency_job_id":null,"html_url":"https://github.com/RedTachyon/arxiv-clustering","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedTachyon%2Farxiv-clustering","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedTachyon%2Farxiv-clustering/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedTachyon%2Farxiv-clustering/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedTachyon%2Farxiv-clustering/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RedTachyon","download_url":"https://codeload.github.com/RedTachyon/arxiv-clustering/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244161051,"owners_count":20408241,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T12:42:47.048Z","updated_at":"2025-03-18T05:23:00.324Z","avatar_url":"https://github.com/RedTachyon.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# arXiv abstract clustering\n\n## Important note\n\nThis implementation is meant as a proof of concept. \nDue to the time constraints, both in terms of the deadline and my other obligations; \nas well as the computational cost of embedding all the abstracts, I restricted my analysis to an MVP-like state.\n\n## Literature review and Methodology\n\n(Note: I take some liberties in the formatting of this report by combining the literature review with the methods, considering it's not an official publication.)\n\nMy approach consists of three main parts:\n\n- Feature extraction via BERT\n- Clustering via K-Means\n- Clustering evaluation via Adjusted Rand Index and the Within-Cluster Sum of Squares\n\n\nBERT (Devlin et al., 2018) is a transformer-based model commonly used for natural language processing tasks. \nIt is pre-trained on the task of filling in missing tokens, can be fine-tuned for a wide variety of new tasks. \n\nIn this case, I used the pre-trained model to extract features from the abstracts. \nSpecifically, I used the `prajjwal1/bert-small` model from the Hugging Face hub, so that the embeddings are faster to compute, and take up less memory (512-dimensional instead of the default 768).\n\nK-Means (MacQueen, 1967) is a simple clustering algorithm which starts with a random set of cluster centers, and iteratively updates them to minimize the within-cluster sum of squares.\nHere, I used the implementation from scikit-learn, with the minibatch variant for computational efficiency.\n\nFinally, I evaluate the resulting clusterings using the Adjusted Rand Index (ARI) (Hubert and Arabie, 1985) and the Within-Cluster Sum of Squares (WCSS).\nWith ARI, I use some additional information, namely the top-level arXiv categories of the papers. \nThe ARI value indicates how well a given clustering matches the categories, with 1 being a perfect match, and 0 being a random clustering.\nWCSS on the other hand does not require any additional information, and is simply a measure of how well the clustering fits the data. \n\nWhile I did not expect a full match due to the unsupervised nature of the clustering, there should be at least some correlation between categories and clusters.\n\n### References\nHubert, L., \u0026 Arabie, P. (1985). Comparing partitions. Journal of Classification, 2(1), 193–218. https://doi.org/10.1007/BF01908075\n\nMacQueen, J. (1965). Some methods for classification and analysis of multivariate observations. In Proc. 5th Berkeley Symposium on Math., Stat., and Prob (p. 281).\n\nDevlin, J., Chang, M.-W., Lee, K., \u0026 Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186. https://doi.org/10.18653/v1/N19-1423\n\n\n\n## Running the code\n\n1. After initializing a virtual environment, install the dependencies with `pip install -r requirements.txt`\n2. Run `bash download_data.sh` to download the data from kaggle\n3. Run `python process_data.py` to extract the abstracts from the downloaded data, and save them in a separate file\n4. Run `python generate_embeddings.py` to generate the BERT-based embeddings for the abstracts. This will likely take a long time and require a decent GPU. I can provide the embeddings on request.\n5. Go through the `clustering.ipynb` notebook to see the results of the clustering and the evaluation.\n\n\n## Results\n\nFrom the analysis in the notebook, we can see that the clustering does not match the categories very well. \nNevertheless, the ARI index peaks at 0.267, which is not too bad considering that the clustering is unsupervised. \nConsidering this, as well as the shape of the WCSS curve, I propose that the **optimal number of clusters is 9**.\n\n\n## Limitations\n\nHere I want to address a few limitations, as well as comments on my interpretation of the original task.\n\n1. For brevity, I integrated the literature review with the description of my method, since both are very closely tied together.\n2. Exploratory data analysis happened mostly in a few ad-hoc notebooks, which I did not include in the repository.\n3. I did not exactly \"train an NLP\" model -- I am not sure if this was a mistake in the task description, or if I misunderstood it. Training a classification algorithm from scratch would be unsuitable for the unsupervised task at hand. Following the suggestion of a BERT model, it would also be relatively computationally expensive.\n4. I did not perform a full hyperparameter search except for the number of clusters, which is the core of this study. From my experience, K-Means tends to be relatively robust to hyperparameter changes, and the number of clusters is the most important one.\n5. The model I used is based on PyTorch, as that is what I exclusively use in my own research at the moment. Using a similar TF-based model would be straight-forward using the transformers library, but it would also be additional time spent on a task that is not the focus of this study. \n6. There are many visualizations and analyses that could be done to better understand the data, and the clustering results. Examples include -- a t-SNE visualization of the abstracts, their categories and clustering; a matrix similar to a confusion matrix, correlating categories with clusters; other metrics of clustering quality. However, at some point my regular life had to take priority, and I had to stop somewhere.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredtachyon%2Farxiv-clustering","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fredtachyon%2Farxiv-clustering","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredtachyon%2Farxiv-clustering/lists"}