{"id":20669473,"url":"https://github.com/iitis/wikiwords","last_synced_at":"2025-08-28T23:49:20.043Z","repository":{"id":5657132,"uuid":"6866267","full_name":"iitis/wikiwords","owner":"iitis","description":"Word frequencies in the 2012 English Wikipedia","archived":false,"fork":false,"pushed_at":"2012-11-26T14:10:58.000Z","size":2252,"stargazers_count":8,"open_issues_count":1,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-08-21T12:36:40.836Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iitis.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-11-26T14:09:06.000Z","updated_at":"2021-12-16T04:24:46.000Z","dependencies_parsed_at":"2022-09-09T22:11:58.510Z","dependency_job_id":null,"html_url":"https://github.com/iitis/wikiwords","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/iitis/wikiwords","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iitis%2Fwikiwords","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iitis%2Fwikiwords/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iitis%2Fwikiwords/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iitis%2Fwikiwords/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iitis","download_url":"https://codeload.github.com/iitis/wikiwords/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iitis%2Fwikiwords/sbom","scorecard":{"id":483363,"data":{"date":"2025-08-11","repo":{"name":"github.com/iitis/wikiwords","commit":"b0369b9d493ba1edd144557d8b3d02422c8d44b8"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.6,"checks":[{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Code-Review","score":0,"reason":"Found 0/1 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"License","score":0,"reason":"license file not detected","details":["Warn: project does not have a license file"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-19T17:10:33.862Z","repository_id":5657132,"created_at":"2025-08-19T17:10:33.863Z","updated_at":"2025-08-19T17:10:33.863Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272582517,"owners_count":24959422,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-28T02:00:10.768Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T20:14:29.401Z","updated_at":"2025-08-28T23:49:20.024Z","avatar_url":"https://github.com/iitis.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Frequency of words in the English 2012 Wikipedia\n================================================\n\nThis Python module can be used to quickly retrieve absolute word frequency\nfor English language, as used in Wikipedia articles as of year 2012.\n\nAuthor: Paweł Foremski \u003cpjf@iitis.pl\u003e, IITiS PAN www.iitis.pl\n\nUsage\n-----\n\t\u003e\u003e\u003e import wikiwords\n\t\u003e\u003e\u003e wikiwords.freq(\"monty\")\n\t6.348454761413523e-06\n\t\u003e\u003e\u003e wikiwords.occ(\"python\")\n\t18972\n\t\u003e\u003e\u003e wikiwords.freq(\"no such word\", lambda x: 1./len(x))\n\t0.08333333333333333\n\nInstallation\n------------\n\t$ sudo pip install wikiwords\n\nDetails\n-------\n\n\nWikipedia files were downloaded on 23.11.2012\n\nThe corpus was filtered:\n\n1. all words containing non-latin letters were removed (A-Z)\n2. in single files (see below), words with frequency \u003c 5e-7 were removed\n3. in the final file (28 files merged), words with f \u003c 5e-8 were removed\n4. all words shorter than 2 characters were removed (see below)\n\nThe final corpus contains over 100,000 words with over 2 billion occurances.\n\nFor copyright information on data, see\n\thttp://en.wikipedia.org/wiki/Wikipedia:Copyrights\n\nThe Python source code is licensed to you under the GNU GPL v3\n\nSee \"the making of/\" subdirectory for details on how the data was created.\n\nThe single letter words\n-----------------------\nThe single-letter words were removed by the gensim parser. In order to address\nthis, relevant frequencies and number of occurances were artifically injected\nfrom the Google Web Trillion Word Corpus, described by Thorsten Brants and Alex\nFranz in 2006 [1]. For more information -- and for an example of possible\napplication of wikiwords.py -- see Peter Norvig ngrams site at [2].\n\n1. http://googleresearch.blogspot.com/2006/08/all-our-n-gram-are-belong-to-you.html\n2. http://norvig.com/ngrams/\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiitis%2Fwikiwords","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiitis%2Fwikiwords","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiitis%2Fwikiwords/lists"}