{"id":19948919,"url":"https://github.com/brannondorsey/glove-experiments","last_synced_at":"2025-05-03T18:32:14.024Z","repository":{"id":16247215,"uuid":"79618492","full_name":"brannondorsey/GloVe-experiments","owner":"brannondorsey","description":"GloVe word vector embedding experiments (similar to Word2Vec)","archived":false,"fork":false,"pushed_at":"2023-07-06T21:33:19.000Z","size":22,"stargazers_count":64,"open_issues_count":2,"forks_count":19,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-04-14T18:50:09.037Z","etag":null,"topics":["embeddings","glove","glove-embeddings","glove-vectors","k-means","k-nearest-neighbors","machine-learning","nlp","word-game","word2vec"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brannondorsey.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-01-21T02:37:15.000Z","updated_at":"2024-02-23T07:26:59.000Z","dependencies_parsed_at":"2022-08-07T08:15:17.303Z","dependency_job_id":null,"html_url":"https://github.com/brannondorsey/GloVe-experiments","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brannondorsey%2FGloVe-experiments","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brannondorsey%2FGloVe-experiments/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brannondorsey%2FGloVe-experiments/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brannondorsey%2FGloVe-experiments/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brannondorsey","download_url":"https://codeload.github.com/brannondorsey/GloVe-experiments/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224370116,"owners_count":17299968,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["embeddings","glove","glove-embeddings","glove-vectors","k-means","k-nearest-neighbors","machine-learning","nlp","word-game","word2vec"],"created_at":"2024-11-13T00:43:13.711Z","updated_at":"2024-11-13T00:43:14.344Z","avatar_url":"https://github.com/brannondorsey.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GloVe Experiments\n\nThis repository contains a few brief experiments with [Stanford NLP's GloVe](https://nlp.stanford.edu/projects/glove/), an unsupervised learning algorithm for obtaining vector representations for words. Similar to Word2Vec, GloVe creates a continuous N-dimensional representation of a word that is learned from its surrounding context words in a training corpus. Trained on a large corpus of text, these co-occurance statistics (an N-dimensional vector embedding) cause semantically similar words to appear near each-other in their resulting N-dimensional embedding space (e.g. \"dog\" and \"cat\" may appear nearby a region of other pet related words in the embedding space because the context words that surround both \"dog\" and \"cat\" in the training corpus are similar).\n\nI've created three small python programs for exploring GloVe embeddings:\n\n- `word_arithmetic.py`: Create word analogy searches using basic arithmetic operations (e.g. `king - man + women = queen`).\n- `word_game.py`: A small terminal-based multiplayer text game for creating word analogies.\n- `word_clustering.py`: Create [K-Means clusters](https://en.wikipedia.org/wiki/K-means_clustering) using GloVe embeddings. Saves results to JSON.\n\nAll three scripts use the GloVe.6B pre-trained word embeddings created from the combined Wikipedia 2014 and Gigaword 5 datasets. They were trained using 6 billion tokens and contains 400,000 unique lowercase words. Trained embeddings are provided in 50, 100, 200, and 300 dimensions (822 MB download).\n\n## Getting Started\n\nThese small experiments can be run in MacOS or Linux environments (sorry ~~not sorry~~ Windoze users). If you'd prefer to run these experiments via Docker jump down to the [Running with Docker](#running-with-docker) section.\n\n```bash\n# clone this repo\ngit clone https://github.com/brannondorsey/GloVe-experiments.git\ncd GloVe-experiments\n\n# install python dependencies\npip3 install -r requirements.txt\n\n# dowload the pre-trained embeddings. This might take a while...\n./download_data.sh\n```\n\n## Word Arithmetic\n\n`word_arithmetic.py` allows you to write simple +/- arithmetic operations using words to find the closest approximated resulting word from the given word expression. Math operations are applied in the embedding space and a K-nearest-neighbor search is used to display the `K` words closest to the result of the algebraic transformation.\n\n```bash\npython3 word_arithmetic.py\n\u003e king - man + woman\n\nqueen                0.22\n```\n\n`word - word + word` is the traditional word analogy format, however `word_arithmetic.py` supports any number of `+` or `-` operations provided all words are in the database. The meaning of less traditional expressions, `word + word + word...` is more ambiguous but can lead to interesting results nonetheless. Specifying an order of operations is not supported at this time (e.g. `(word - word) + word`).\n\nBy default, `word_arithmetic.py` loads the 10,000 most frequently used words from the dataset and uses a 100-dimensional embedding vector. It also prints only the single nearest word to the resulting vector point from the expression (the \"nearest neighbor\"). You can specify your own values for each of these parameters if you would like:\n\n```bash\npython3 word_arithmetic.py --num_words 100000 --vector_dim 300 --num_output 10\n\u003e king - man + woman\n\nqueen                0.31\nmonarch              0.44\nthrone               0.44\nprincess             0.45\nmother               0.49\ndaughter             0.49\nkingdom              0.50\nprince               0.50\nelizabeth            0.51\nwife                 0.52\n```\n\nIncreasing `--num_words` and `--vector_dim` increases the number of usable words in the dictionary and accuracy of the resulting word expressions respectively. Increasing either will increase the processing time for each expression as well as the memory requirements needed to run the program.\n\n```\nusage: word_arithmetic.py [-h] [--vector_dim {50,100,200,300}]\n                          [--num_words NUM_WORDS] [--num_output NUM_OUTPUT]\n                          [--glove_path GLOVE_PATH]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --vector_dim {50,100,200,300}, -d {50,100,200,300}\n                        What vector GloVe vector depth to use (default: 100).\n  --num_words NUM_WORDS, -n NUM_WORDS\n                        The number of lines to read from the GloVe vector file\n                        (default: 10000).\n  --num_output NUM_OUTPUT, -o NUM_OUTPUT\n                        The number of result words to display (default: 1)\n  --glove_path GLOVE_PATH, -i GLOVE_PATH\n                        GloVe vector file path\n```\n\n## Word Game\n\n`word_game.py` is a small text-based multiplayer game where players take turns creating and answering `word_arithmetic.py`-style word expressions. Players win points when they propose a solution word to a word expression that is nearest to the answer word out of all players guesses.\n\n```\nEnter the name of each player, seperated by commas.\n\u003e bob, alice\nThere are 2 players correct? [yes]: yes\nWhat score would you like to play to? [10]: 10\nalice, please enter a word expression:\n\u003e home - earth + space\nalice, please enter your answer: rocket\nbob, please enter your answer: moon\nComputer says home - earth + space = office\nbob wins this round.\n\n     alice: 0     bob: 1\n\nbob, please enter a word expression:\n\u003e\n```\n\nThe game is far from perfect, and the automated judging can be aggravating at times (try with `--soft_score`), but it can lead to some fun times given the right crowd 💻🍻🎉. Increase the dictionary size and vector dimensions for best results:\n\n```bash\npython3 word_game.py --vector_dim 200 --num_words 100000 --soft_score\n```\n\n```\nusage: word_game.py [-h] [--vector_dim {50,100,200,300}]\n                    [--num_words NUM_WORDS] [--soft_score]\n                    [--glove_path GLOVE_PATH]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --vector_dim {50,100,200,300}, -d {50,100,200,300}\n                        What vector GloVe vector depth to use (default: 100).\n  --num_words NUM_WORDS, -n NUM_WORDS\n                        The number of lines to read from the GloVe vector file\n                        (default: 10000).\n  --soft_score, -s      points are scored relative to the distance a player's\n                        word is from the result of the input expression. This\n                        is in contrast to the default 1 point per-round\n                        scoring system. Soft scoring is recommended for a more\n                        fair-and-balanced game experience (default: false)\n  --glove_path GLOVE_PATH, -i GLOVE_PATH\n                        GloVe vector file path (default: data/glove)\n```\n\n## Word Clustering\n\n`word_clustering.py` uses unsupervised learning to clusters words into related groups using K-Means.\n\n```bash\npython3 word_clustering.py\nNo cached cluster found. Clustering using K-Means...\nSaved 1000 clusters to data/cache/100D_10000-words_1000-clusters.json. Cached for later use.\nCLUSTER 1: athens, stockholm, oslo, helsinki\nCLUSTER 2: long, short, longer, normal, usual, periods, lengthy, shorter, duration\nCLUSTER 3: current, term, future, key, position, primary, internal, existing, core, external\nCLUSTER 4: newton, luther, canon\nCLUSTER 5: ball, pitch, catch, throw, balls, swing, bat, kicked, opener, slip, spell, foul, knock, pitches, toss, kicking, bounced, kicks, scoreboard, bounce\nCLUSTER 6: popular, famous, prominent, notable, influential, renowned, well-known, famed, acclaimed, finest\nCLUSTER 7: affected, affect, affecting, affects\nCLUSTER 8: assassination, murdered, slain, assassinated\nCLUSTER 9: jordan, carter, jimmy\nCLUSTER 10: 1999, 1994, 1995, 1993, 1992, 1991, 1990, 1989, 1988, 1986, 1987, 1984, 1980, 1985, 1979, 1983, 1982, 1981\nCLUSTER 11: alongside, joining, touring, completing, toured, thereafter, whilst, filming, assignment, boarding, stint\nCLUSTER 12: 10, 20, 15, 30, 11, 12, 18, 25, 14, 13, 16, 17, 24, 19, 22, 21, 23, 26, 28, 27, 31, 29\nCLUSTER 13: support, provide, aid, access, provided, additional, offers, relief, provides, assistance, providing, funding\nCLUSTER 14: communist, regime, dictator, suharto, dictatorship, communism, monarchy\n...\n--- 28.54 seconds ---\n```\n\nClusters are printed to the screen and also saved as JSON arrays in `data/cache`. By default, the script clusters the 10,000 most-common words from GloVe.6B into 1,000 clusters using 100-D vector embeddings. This can be changed like so:\n\n```bash\n# note: this will take a *long* time to run...\npython3 word_clustering.py --num_words 100000 --num_clusters 10000 --vector_dim 300\n```\n\n```\nusage: word_clustering.py [-h] [--vector_dim {50,100,200,300}]\n                          [--num_words NUM_WORDS]\n                          [--num_clusters NUM_CLUSTERS] [--n_jobs N_JOBS]\n                          [--glove_path GLOVE_PATH]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --vector_dim {50,100,200,300}, -d {50,100,200,300}\n                        What vector GloVe vector dimension to use (default:\n                        100).\n  --num_words NUM_WORDS, -n NUM_WORDS\n                        The number of lines to read from the GloVe vector file\n                        (default: 10000).\n  --num_clusters NUM_CLUSTERS, -k NUM_CLUSTERS\n                        Number of resulting word clusters. The number of K in\n                        K-Means (default: 1000).\n  --n_jobs N_JOBS, -j N_JOBS\n                        Number of cores to use when fitting K-Means. -1 = all\n                        cores. More cores = less time, more memory (default:\n                        -1).\n  --glove_path GLOVE_PATH, -i GLOVE_PATH\n                        GloVe vector file path (default: data/glove)\n```\n\n## Running with Docker\n\nThese experiments, and the GloVe data they use, are available via a Docker image on Docker Hub. If you have Docker installed on your machine you can pull the images and run them inside of containers instead of installing them on your host machine.\n\n```bash\ndocker run --rm -it brannondorsey/glove-experiments python word_arithmetic.py\ndocker run --rm -it brannondorsey/glove-experiments python word_game.py\ndocker run --rm -it brannondorsey/glove-experiments python word_clustering.py\n```\n\nThese images have been built for 64-bit x86 CPU architectures.\n\n## License and Attribution\n\nAll code is released under an [MIT license](LICENSE). You are free to copy, edit, share, or sell it under those terms.\n\n### GloVe citation\n\nJeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. [GloVe: Global Vectors for Word Representation](https://nlp.stanford.edu/pubs/glove.pdf).\n\n```\n@inproceedings{pennington2014glove,\n  author = {Jeffrey Pennington and Richard Socher and Christopher D. Manning},\n  booktitle = {Empirical Methods in Natural Language Processing (EMNLP)},\n  title = {GloVe: Global Vectors for Word Representation},\n  year = {2014},\n  pages = {1532--1543},\n  url = {http://www.aclweb.org/anthology/D14-1162},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrannondorsey%2Fglove-experiments","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrannondorsey%2Fglove-experiments","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrannondorsey%2Fglove-experiments/lists"}