{"id":17963658,"url":"https://github.com/anthonycalandra/wikipedia-tfidf","last_synced_at":"2025-07-28T15:11:41.457Z","repository":{"id":77252146,"uuid":"171359539","full_name":"AnthonyCalandra/wikipedia-tfidf","owner":"AnthonyCalandra","description":"A Hadoop-powered search index of Wikipedia articles.","archived":false,"fork":false,"pushed_at":"2019-06-05T21:20:39.000Z","size":13,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-03T19:25:39.244Z","etag":null,"topics":["hadoop","mapreduce","tf-idf"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AnthonyCalandra.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-18T21:36:14.000Z","updated_at":"2020-03-11T15:01:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"3f7ddc26-6a2d-4d44-a0c3-5b04975d5899","html_url":"https://github.com/AnthonyCalandra/wikipedia-tfidf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AnthonyCalandra/wikipedia-tfidf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AnthonyCalandra%2Fwikipedia-tfidf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AnthonyCalandra%2Fwikipedia-tfidf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AnthonyCalandra%2Fwikipedia-tfidf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AnthonyCalandra%2Fwikipedia-tfidf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AnthonyCalandra","download_url":"https://codeload.github.com/AnthonyCalandra/wikipedia-tfidf/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AnthonyCalandra%2Fwikipedia-tfidf/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267535951,"owners_count":24103399,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-28T02:00:09.689Z","response_time":68,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hadoop","mapreduce","tf-idf"],"created_at":"2024-10-29T11:43:19.886Z","updated_at":"2025-07-28T15:11:41.429Z","avatar_url":"https://github.com/AnthonyCalandra.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Wikipedia tf-idf\n\nThis project creates a tf-idf of Wikipedia articles. The user provides a query of terms they would like to search for, and the application will return the specified articles ordered by descending computed tf-idf. The indexer finds terms in every article and maps them to `List(\u003cbyte offset, document frequency, term frequency, article ID\u003e)` where the `byte offset` is the location of the article the term is found in, in the original dataset.\n\nThe example dataset I used was retrieved from [here](https://blog.lateral.io/2015/06/the-unknown-perils-of-mining-wikipedia/). `clean.py` was ran on the dataset before feeding it into the indexer.\n\nFor example, to search for all articles containing the keywords \"big\" and \"data\":\n```\n$ java -cp target/wikipedia-indexer-1.0.jar com.anthony_calandra.wikipedia_indexer.ArticleRetriever -index wikipedia-index -collection data/wikipedia_utf8_filtered_20pageviews.csv -query \"big data AND\"\nQuery: big data AND\n\ntf-idf\tArticle ID\tArticle\n0.18663820291830818\t40008710\tData discovery  Data discovery is a Business intelligence architecture ...\n0.1821862684791389\t40131150\tData Defined Storage  New advancements in technology, such as growing i...\n0.18069245905296735\t35458904\tData science  Data science incorporates varying elements and builds on ...\n0.1480663058315223\t13000371\tData feed  Data feed is a mechanism for users to receive updated data f...\n0.141699277026621\t34229379\tOracle Big Data Appliance  The Oracle Big Data Appliance consists of ha...\n0.13064673814923958\t2654483\tEnterprise data management  Enterprise Data Management (EDM) is: EDM aro...\n0.11763211917592149\t168680\tTeradata  Teradata Corporation is an American computer company that sells...\n0.10724731887453147\t39799017\tSoftware-defined data center  Software-defined data center (SDDC) is an...\n0.10416278891178483\t27051151\tBig data  Big data is the term for a collection of data sets so large a...\n0.10013070852294043\t31882591\tRCFile  RCFile (Record Columnar File) is a data placement structure tha...\n\nquery completed in 671ms\n```\nThe first column is the article ID (can be queried using Wikipedia's APIs), the second is the article title, and the third is the article contents shortened.\n\nQueries are in reverse Polish notation and can be nested:\n```\n$ java -cp target/wikipedia-indexer-1.0.jar com.anthony_calandra.wikipedia_indexer.ArticleRetriever -index wikipedia-index -collection data/wikipedia_utf8_filtered_20pageviews.csv -query \"big small OR data AND\"\nQuery: big small OR data AND\n\ntf-idf\tArticle ID\tArticle\n0.2266929587483404\t303703\tData mart  A data mart is the access layer of the data warehouse environm...\n0.19748924922986472\t11658036\tData administration  Data administration or data resource management is...\n0.18663820291830818\t40008710\tData discovery  Data discovery is a Business intelligence architecture ...\n0.1821862684791389\t40131150\tData Defined Storage  New advancements in technology, such as growing i...\n0.18069245905296735\t35458904\tData science  Data science incorporates varying elements and builds on ...\n0.16986938808605723\t7990\tData warehouse  In computing, a data warehouse or enterprise data warehouse...\n0.16152687554931874\t988114\tData logger  A data logger (also datalogger or data recorder) is an elect...\n0.15628255942761757\t1381282\tData loss  Data loss is an error condition in information systems in whi...\n0.15370204316370414\t466099\tData processing system  A data processing system is a combination of mach...\n0.15191481383877772\t168753\tData haven  A data haven, like a corporate haven or tax haven, is a refug...\n\nquery completed in 1145ms\n```\nSearches for all articles with contains containing the keywords \"big\" and \"data\", or \"small\" and \"data\".\n\n## Instructions\n\nBuild project:\n```\nmvn clean package\n```\n\nBuild the index:\n```\nhadoop jar target/wikipedia-indexer-1.0.jar \\\n com.anthony_calandra.wikipedia_indexer.BuildWikipediaIndex \\\n -input data/wikipedia_utf8_filtered_20pageviews.csv \\\n -output wikipedia-index -reducers 4\n```\n\nRun queries:\n```\njava -cp target/wikipedia-indexer-1.0.jar \\\n com.anthony_calandra.wikipedia_indexer.ArticleRetriever \\\n -index wikipedia-index -collection data/wikipedia_utf8_filtered_20pageviews.csv \\\n -query \"big data AND\"\n```\n\n## License\n\nMIT\n\n## Author\n\nAnthony Calandra\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanthonycalandra%2Fwikipedia-tfidf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanthonycalandra%2Fwikipedia-tfidf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanthonycalandra%2Fwikipedia-tfidf/lists"}