{"id":19168383,"url":"https://github.com/bloomberg/fast-noise-aware-topic-clustering","last_synced_at":"2025-10-30T14:13:59.080Z","repository":{"id":66125168,"uuid":"412491389","full_name":"bloomberg/fast-noise-aware-topic-clustering","owner":"bloomberg","description":"Research code and scripts used in the Silburt et al. (2021) EMNLP 2021 paper 'FANATIC: FAst Noise-Aware TopIc Clustering'","archived":false,"fork":false,"pushed_at":"2023-07-06T22:49:30.000Z","size":116,"stargazers_count":10,"open_issues_count":3,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-19T23:31:57.816Z","etag":null,"topics":["clustering","emnlp","emnlp2021","machine-learning","nlp","python","topic-noise"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bloomberg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-01T14:07:26.000Z","updated_at":"2025-03-07T01:29:51.000Z","dependencies_parsed_at":"2024-11-09T09:42:33.202Z","dependency_job_id":"06fbecb8-34d0-4eb4-bde9-8b300a0eaac8","html_url":"https://github.com/bloomberg/fast-noise-aware-topic-clustering","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bloomberg%2Ffast-noise-aware-topic-clustering","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bloomberg%2Ffast-noise-aware-topic-clustering/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bloomberg%2Ffast-noise-aware-topic-clustering/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bloomberg%2Ffast-noise-aware-topic-clustering/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bloomberg","download_url":"https://codeload.github.com/bloomberg/fast-noise-aware-topic-clustering/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252897315,"owners_count":21821420,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clustering","emnlp","emnlp2021","machine-learning","nlp","python","topic-noise"],"created_at":"2024-11-09T09:42:29.284Z","updated_at":"2025-10-30T14:13:58.971Z","avatar_url":"https://github.com/bloomberg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FANATIC: FAst Noise-Aware TopIc Clustering\n\nAuthors: Ari Silburt, Anja Subasic, Evan Thompson, Carmeline Dsilva, Tarec Fares\n\n## General\nThis repo contains the research code and scripts used in the Silburt et al. (2021) paper \"FANATIC: FAst Noise-Aware TopIc Clustering\", and provides a basic overview of the code structure and its major components. For more questions, please directly contact the authors.\n\nIn particular, this repo allows a user to:\n- Download the reddit data\n- Train a word2vec embedding model\n- Use `FANATIC` to cluster the reddit data and dump results for downstream analysis. \n\nNote that the original paper results used an in-house preprocessor, but a very similar open-source one has been provided (see `fanatic/preprocess/nltk_preprocessor.py`).\n\n## License\nPlease read the `LICENSE`.\n\n## How-to\n### Setup\nIt is recommended to create a fresh python virual environment and run the following commands from the base repo:\n```\npip install --upgrade pip\npip install -r requirements.txt\n```\nAnd then in a python shell do:\n```python\nimport nltk\nnltk.download('stopwords')\n```\n\nThis repo has been tested against `python3.7`.\n\n### Download the Reddit Data\nData can be downloaded from [pushshift](https://files.pushshift.io/reddit/submissions/) using `wget`, e.g. `wget https://files.pushshift.io/reddit/submissions/RS_2017-11.zst`. If data files are downloaded to the `data/` directory, subsequent scripts are already set up to look there. \n\n### Training a Word2vec Embedding\nA new word2vec model can be trained using `embedding_driver.py`, and it is recommended to carefully inspect the arguments before running. In particular, a Reddit data file(s) must first be downloaded and specified in the `--data-files` argument. \n\n### Cluster via FANATIC\nOnce the data has been downloaded and word2vec model trained, a clustering run can be performed using the `clustering_driver.py` script. All input arguments are specified in the `parse_args` function of `fanatic/arguments.py` including data, label, preprocessing, clustering algorithm and output arguments. \n\nSee the Silburt et al. (2021) paper for a detailed explanation of FANATIC's hyperparameters.\n\n#### Clustering labels\nBy default, the data is clustered against the `data/subreddit_labels.json` labels file, which indicates:\n- what subreddits are considered for clustering (all other subreddits are discarded).\n- whether the subreddit is a \"coherent\" or \"noise\" topic, where all noise topics are assigned the same `NOISE` label. \n\n The `--subreddit-noise-percentage` argument sets the fraction of documents that come from noise subreddits. In the case where `--num-docs-read` and `--subreddit-noise-percentage` are incompatible with each other, honouring `--subreddit-noise-percentage` is prioritized. If `--subreddit-noise-percentage` is set to `None`, the noise percentage is set by the natural data distribution. \n\nThe `data/subreddit_labels.json` labels file can be substituted for a different one, or ignored entirely by setting `--subreddit-labels-file None`. When `--subreddit-labels-file` is set to `None` all encountered subreddits are used, each subreddit becomes its own coherent topic, and the concept of \"topic noise\" disappears. Thus, the `--subreddit-noise-percentage` argument becomes irrelevant. \n\n#### Clustering Outputs\nAfter a successful clustering run, the files that are output are:\n- `fanatic_\u003cdataset-id\u003e_\u003cseed-run\u003e_labels_and_assignments.json`: this file is generated for each seed run and contains, for each document-id, the *assignment* (what cluster the document ended up in) and *label* (the label associated with the document). Therefore, the full clustering result is contained within this file for downstream analysis. Document-ids can be mapped back to the original datafile should additional metadata be desired. \n- `fanatic_\u003cdataset-id\u003e_\u003cseed-run\u003e_sample_clusters.txt` - this file is generated for each seed run and contains the first 10 documents from each cluster and associated label. Format is `\u003ctext\u003e -\u003e \u003clabel\u003e`. This gives the user a qualitative sense of what each cluster contains.\n- `fanatic_\u003cdataset-id\u003e_\u003cseed-run\u003e_summary.txt` - this file is generated for each seed run and contains all input parameters and clustering stats/metrics. It is effectively a summary of the entire clustering run, allowing you to quickly parse results and/or recreate the job if needed. It can be consumed by [configparser](https://docs.python.org/3.7/library/configparser.html).\n- `fanatic_\u003cdataset-id\u003e_summary_averaged.txt` - this file is generated once for a dataset-id and contains the input arguments and *averaged* clustering stats/metrics across the seed runs. \n\nIn addition, the full clustering model can be dumped via pickle for deeper investigation by adding `--flag-save-clusteringmodel`. *Warning*: this file can become large, especially for big datasets.\n\n### Custom Preprocessor / Featurizer\nResults from the paper were generated using an in-house preprocessor that is not available to the public. Using nltk we created a very similar preprocessor, located at `fanatic/preprocess/nltk_preprocessor.py`, and inherits from `fanatic/preprocess/generic_preprocessor.py`. Users are free to create their own custom preprocessors that also inherit from `generic_preprocessor.py` and experiment with more sophisticated features (e.g. BERT embeddings). See `generic_preprocessor.py` for additional documentation and requirements.\n\n### Non-Reddit Datasets\nUsers are encouraged to substitute or modify `fanatic/preprocess/read_data.py` to read in different kinds of data. In particular, `DATASET_INPUT_FIELD`, `DATASET_LABEL_FIELD` and `DATASET_ID_FIELD` must be changed to extract the relevant content from the new dataset for downstream preprocessing and clustering. \n\n## Tests\nSome basic unit tests can be run from the home directory with `python3.7 -m pytest tests/unit/`","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbloomberg%2Ffast-noise-aware-topic-clustering","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbloomberg%2Ffast-noise-aware-topic-clustering","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbloomberg%2Ffast-noise-aware-topic-clustering/lists"}