{"id":23273102,"url":"https://github.com/zhaytam/pagerank","last_synced_at":"2025-06-29T22:35:42.104Z","repository":{"id":114360349,"uuid":"124706614","full_name":"zHaytam/PageRank","owner":"zHaytam","description":"An implementation of the PageRank algorithm in Hadoop MapReduce","archived":false,"fork":false,"pushed_at":"2018-03-11T23:21:10.000Z","size":144,"stargazers_count":1,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-12T15:53:13.531Z","etag":null,"topics":["hadoop","java","pagerank-algorithm"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zHaytam.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-10T23:17:17.000Z","updated_at":"2019-05-29T22:44:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"9ba59a3b-3f65-460b-a3a9-00243a61ce2f","html_url":"https://github.com/zHaytam/PageRank","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zHaytam%2FPageRank","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zHaytam%2FPageRank/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zHaytam%2FPageRank/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zHaytam%2FPageRank/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zHaytam","download_url":"https://codeload.github.com/zHaytam/PageRank/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247465488,"owners_count":20943185,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hadoop","java","pagerank-algorithm"],"created_at":"2024-12-19T19:32:33.053Z","updated_at":"2025-04-06T10:23:19.338Z","avatar_url":"https://github.com/zHaytam.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PageRank\n\nAn implementation of the Page Rank algorithm using Hadoop (Java).\nThis was tested using the inputs (available above):\n- pagerank_data.txt - A testing example\n- hollins.data - A dataset that can be found [here](https://web2.qatar.cmu.edu/~gdicaro/15381/hw/hw4-files/hollins.dat).\n\n## Input Format\n\nThe input used in this implementation (inputs) is as follows:\n*Note: Each line is 2 values seperated by a space.*\n - First line: NodesCount EdgesCount *(e.g. \"5 9\")*\n - NC next lines: NodeID NodeURL *(e.g. \"1 http://example.com\")*\n - EC next lines: NodeID OutlinkToNodeID *(e.g. \"1 2\")*\n\n## Step 1\n\n- **Mapper:** Reads the initial input file, ignores the first line and the NC next lines and outputs the EC edges `\u003cIntWritable, IntWritable\u003e` (e.g. `\u003c1, 2\u003e`).\n- **Reducer:** Receives the EC edges (Each node and an Iterable of its outlinks) and outputs for each node a concatenated list of its outlinks prefixed with an initial page rank (separated by a tab) `\u003cIntWritable, Text\u003e` (e.g. `\u003c1, \"1.0 3,2\"\u003e`). \n\n## Step 2\n\nThis is the most important step in the process. This is ran multiple times since the output of this step will be the same as in the first step.\n\n- **Mapper:** Reads the last output generated (wheither from the first step or from a previous iteration) and outputs for each `[node, rank, \"outlink1,outlink2,...\"]`:\n  - For each outlink it outputs the outlink and `rank / len(outlinks)`: `\u003cText, Text\u003e` (e.g. `\u003c\"3\", \"0.5\"\u003e`.\n  - The node and its outlinks prefixed with an open bracket: \u003cText, Text\u003e (e.g. `\u003c\"1\",  \"[3,2\"\u003e`).\n-  **Reducer:** Receives for each node a list of values (the values can be either page ranks or the list of the outlinks), calculates the pagerank by using the formula `(1 - d) + (d * sum(pageRank))` and outputs the same structure as the first step: \u003cText, Text\u003e `(e.g. \u003c\"1\", \"0.7875 3,2\"\u003e)`.\n\nThis can be hard to understand using sentences, so here is a pseudo-code:\n```\nMap(Offset, Text):\n\tnode, rank, outlinks = Parse(Text)\n\t\n\tif outlinks == None:\n\t\treturn\n\n\tfor (outlink in outlinks):\n\t\tWrite(outlink, rank / len(outlinks))\n\n\tWrite(node, '[' + outlinks)\n\nReduce(Node, Text[]):\n\toutlinks = []\n\ttotalRank = 0\n\n\tfor (text in Text):\n\t\tif text.startswith('['):\n\t\t\toutlinks = text[1:]\n\t\telse\n\t\t\ttotalRank += text\n\n\ttotalRank = (1 - 0.85) + (0.85 * totalRank)\n\tWrite(Node, totalRank + '\\t' + outlinks)\n```\n\nThis step is ran X times (The X must be given in the arguments) or until a minimum score is met.\nThe score is calculated as follows: `∑i(|lastRanks[i] - newRanks[i]|)`.\n\n## Step 3\n\nThis step basically only needs a mapper and the use of the shuffle\u0026sort phase to print the rankings but can use a Reducer to, for example, print the top N ranked pages.\n- **Setup:** Before mapping, we read the input data (the NC nodes lines) and fill a `HashMap` with each node and its corresponding url, this is simply to be able to print the url of the pages in the ranking instead of just the node ids.\n- **Mapper:** Receives the last ranks output (node, rank, outlinks) and outputs for each: `\u003cFloatWritable, Text\u003e` (e.g. `\u003c1.6375, \"http://www.hollins.edu/\"\u003e`). We write the rank as the key so that the shuffle\u0026sort phase (using a custom SortComparator) sorts our entries in a descending order (thus not needing a Reducer).\n\n## Testing\n\n 1. Download the sources.\n 2. Compile to a .jar file.\n 3. Create an `input` folder and put `pagerank_data.txt` in there.\n 4. Use the following command `hadoop jar File.jar PageRank /input /output /input/pagerank_data.txt 0.85 5 true true`: \n    - **/input:** The input folder\n    - **/output:** The output folder\n    - **/input/pagerank_data.txt:** The data file (used to get the urls in step 3)\n    - **0.85:** The damping factor.\n    - **5:** The maximum number of iterations.\n    - **0.01:** The criterion to break from the iterations (minimum difference).\n    - **true:** Delete the output folder before starting (if found).\n    - **true:** Show the results at the end (/ranking/part-r-00000 file).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhaytam%2Fpagerank","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzhaytam%2Fpagerank","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhaytam%2Fpagerank/lists"}