{"id":19608126,"url":"https://github.com/jfilter/hyperwords","last_synced_at":"2026-07-16T08:32:48.775Z","repository":{"id":68654675,"uuid":"189011901","full_name":"jfilter/hyperwords","owner":"jfilter","description":"Updated Version of Omer Levy's hyperwords","archived":false,"fork":false,"pushed_at":"2019-06-25T17:17:23.000Z","size":358,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-07-16T08:32:45.585Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jfilter.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2019-05-28T11:05:42.000Z","updated_at":"2025-04-26T15:57:05.000Z","dependencies_parsed_at":"2023-03-11T04:05:34.815Z","dependency_job_id":null,"html_url":"https://github.com/jfilter/hyperwords","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/jfilter/hyperwords","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jfilter%2Fhyperwords","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jfilter%2Fhyperwords/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jfilter%2Fhyperwords/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jfilter%2Fhyperwords/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jfilter","download_url":"https://codeload.github.com/jfilter/hyperwords/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jfilter%2Fhyperwords/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35537751,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-16T02:00:06.687Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T10:14:19.680Z","updated_at":"2026-07-16T08:32:48.770Z","avatar_url":"https://github.com/jfilter.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# hyperwords: Hyperparameter-Enabled Word Representations\n\nhyperwords is a collection of scripts and programs for creating word representations, designed to facilitate academic\nresearch and prototyping of word representations. It allows you to tune many hyperparameters that are pre-set or\nignored in other word representation packages.\n\nhyperwords is free and open software. If you use hyperwords in scientific publication, we would appreciate citations:  \n\"Improving Distributional Similarity with Lessons Learned from Word Embeddings\"\nOmer Levy, Yoav Goldberg, and Ido Dagan. TACL 2015.\n\n## Requirements\n\nRunning hyperwords may require a lot of computational resources:\n\n- disk space for independently pre-processing the corpus\n- internal memory for loading sparse matrices\n- significant running time; hyperwords is neither optimized nor multi-threaded\n\nhyperwords assumes a \\*nix shell, and requires Python 2.7 (or later, excluding 3+) with the following packages installed:\nnumpy, scipy, sparsesvd, docopt.\n\n```bash\nconda create --name hyperwords python=2.7\nconda activate hyperwords\nconda install -y -c conda-forge scipy\nconda install -y -c anaconda cython docopt\npip install sparsesvd\n```\n\n## Quick-Start\n\n1. Download the latest version from BitBucket, unzip, and make sure all scripts have running permissions (chmod 755 \\*.sh).\n2. Download a text corpus of your choice.\n3. To create word vectors...\n   - ...with SVD over PPMI, use: _corpus2svd.sh_\n   - ...with SGNS (skip-grams with negative sampling), use: _corpus2sgns.sh_\n4. The vectors should be available in textual format under \u003coutput_path\u003e/vectors.txt\n\nTo explore the list of hyperparameters, use the _-h_ or _--help_ option.\n\n## Pipeline\n\nThe following figure shows the hyperwords' pipeline:\n\n**DATA:** raw corpus =\u003e corpus =\u003e pairs =\u003e counts =\u003e vocab  \n**TRADITIONAL:** counts + vocab =\u003e pmi =\u003e svd  \n**EMBEDDINGS:** pairs + vocab =\u003e sgns\n\n**raw corpus =\u003e corpus**\n\n- _scripts/clean_corpus.sh_\n- Eliminates non-alphanumeric tokens from the original corpus.\n\n**corpus =\u003e pairs**\n\n- _corpus2pairs.py_\n- Extracts a collection of word-context pairs from the corpus.\n\n**pairs =\u003e counts**\n\n- _scripts/pairs2counts.sh_\n- Aggregates identical word-context pairs.\n\n**counts =\u003e vocab**\n\n- _counts2vocab.py_\n- Creates vocabularies with the words' and contexts' unigram distributions.\n\n**counts + vocab =\u003e pmi**\n\n- _counts2pmi.py_\n- Creates a PMI matrix (_scipy.sparse.csr_matrix_) from the counts.\n\n**pmi =\u003e svd**\n\n- _pmi2svd.py_\n- Factorizes the PMI matrix using SVD. Saves the result as three dense numpy matrices.\n\n**pairs + vocab =\u003e sgns**\n\n- _word2vecf/word2vecf_\n- An external program for creating embeddings with SGNS. For more information, see:  \n  **\"Dependency-Based Word Embeddings\". Omer Levy and Yoav Goldberg. ACL 2014.**\n\nAn example pipeline is demonstrated in: _example_test.sh_\n\n## Evaluation\n\nhyperwords also allows easy evaluation of word representations on two tasks: word similarity and analogies.\n\n### Word Similarity\n\n- _hyperwords/ws_eval.py_\n- Compares how a representation ranks pairs of related words by similarity versus human ranking.\n- 5 readily-available datasets\n\n### Analogies\n\n- _hyperwords/analogy_eval.py_\n- Solves analogy questions, such as: \"man is to woman as king is to...?\" (answer: queen).\n- 2 readily-available datasets\n- Shows results of two analogy recovery methods: 3CosAdd and 3CosMul. For more information, see:  \n  **\"Linguistic Regularities in Sparse and Explicit Word Representations\". Omer Levy and Yoav Goldberg. CoNLL 2014.**\n\nThese programs assume that the representation was created by hyperwords, and can be loaded by\n_hyperwords.representations.embedding.Embedding_. Dense vectors in textual format (such as the ones produced by word2vec\nand GloVe) can be converted to hyperwords' format using _hyperwords/text2numpy.py_.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjfilter%2Fhyperwords","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjfilter%2Fhyperwords","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjfilter%2Fhyperwords/lists"}