{"id":21025760,"url":"https://github.com/koderkow/wow_scraper","last_synced_at":"2026-05-01T05:39:44.793Z","repository":{"id":134309491,"uuid":"133195720","full_name":"KoderKow/wow_scraper","owner":"KoderKow","description":"Web Scraper for World of Warcraft Weapons","archived":false,"fork":false,"pushed_at":"2018-05-27T19:36:04.000Z","size":1352,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-20T14:28:13.085Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KoderKow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-13T01:14:59.000Z","updated_at":"2023-11-16T21:12:01.000Z","dependencies_parsed_at":"2023-08-09T13:03:26.150Z","dependency_job_id":null,"html_url":"https://github.com/KoderKow/wow_scraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KoderKow%2Fwow_scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KoderKow%2Fwow_scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KoderKow%2Fwow_scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KoderKow%2Fwow_scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KoderKow","download_url":"https://codeload.github.com/KoderKow/wow_scraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243461194,"owners_count":20294701,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T11:36:29.797Z","updated_at":"2025-12-30T05:38:25.121Z","avatar_url":"https://github.com/KoderKow.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Web Scraper - World of Warcraft\n\nThe main goal of this project was to learn how to use the R libraries rvest and RSelenium. The web scraping would be taking place on the website wowhead.com. There is about 10,000 rows of data to collect.\n\n## Learning Points\n-\tNavigate the DOM to capture desired element values\n-\tUsed R’s _RSelenium_ and _rvest_ library to capture data\n-\tConstruct a R script to automate the complete web scraping process\n-\tMade a data frame with captured data to save to a csv file for data analysis\n\n## Web Scraping Process\nThe web scraper goes to the wowhead.com weapon page. There is about 10,000 weapons worth of data. All of the data we want to collect from these weapons are not presented in the rows of the table. For each page, there are 50 weapons.\n\n#### Example of table view of weapons\n\u003cimg src=\"https://github.com/KoderKow/wow_scraper/blob/master/readme_images/table_view.png\" width=\"70%\" height=\"70%\"\u003e\n\nIn order to capture all of the data we want we will need to click on each row, collect the data, go back to the previous page, and then click on the next item.\n\n#### Example of weapon view for all attributes\n\u003cimg src=\"https://github.com/KoderKow/wow_scraper/blob/master/readme_images/item_view.png\" width=\"70%\" height=\"70%\"\u003e\n\nAt the start I was collecting the data directly from the elements shown on the top left box of the page. Issues started coming up when a weapon had a different amount of attributes. For example, if a weapon had two attributes such as strength and agility, that takes up 2 spaces on the DOM. Then if the next weapon had only one attribute, the script would return an error. After exploring the DOM I was able to find all of the data in a \\\u003cnoscript\\\u003e tag. All of the attributes had different code tags (ie;'!--stat3--\u0026gt;\\\\+' = agility). After sorting out all of the tags I was able to use string manipulation to return the information desired.\n  \nFirst time running the script all the way through with no errors showed the next hurdle to jump. The original default weapon data only showed about 1,000 of the 10,000 total weapons. The next step was to automate a filtering process to collect all the weapons.\n\n#### By default, the page displays around 1000 max\n\u003cimg src=\"https://github.com/KoderKow/wow_scraper/blob/master/readme_images/item_displayed_vs_total.png\" width=\"50%\" height=\"50%\"\u003e\n\nI decided to search for weapons based on their item level. For example, levels 1-25, 26-35, etc. This would return all the weapons in chunks. Once the first chunk of weapons was collected the script clears the filter and inputs the next number range.\n\nAfter thinking this project was complete I showed the data to my professor. We found out the weapon attributes were not correct. My string manipulation was not collecting stats correctly that had a comma in it (ex: 1,247). This was a simple fix, but then I noticed something completely different on the website. When you click a weapon and then go back, the order the weapons on the table were not static. The order changed! Eventually I found a solution, constantly sort the table by the weapon name everytime the script goes back to the main page.\n\nUnfortunately during this time, World of Warcraft introduced weapon scalability to their weapons. To put it simply and in terms I understand, a large amount of weapons in their game now 'upgrade' as you level up. For example, I am level one and have a level one sword that does 1-2 damage and has +1 strength. When I level up to level two the sword now does 2-3 damage and has +2 strength. World of Warcraft has 120 levels. This now means a lot of the weapons have different stats on 120 levels. This really put a thud on this web scraper.\n\nAfter talking to my professor, if I chose to continue with this project and wanted to do data analysis on the data, I could make an assumption that all weapons would be looked at in a maxed level view (player level = 120). Sadly, however, this will require going back and changing how a lot of the data is collected.\n\n## Reflection\nIt is unfortunate to get hit by that update, but the process of collecting the data and fixing all the issues that came up was an amazing learning experience. There is still possibility of data analysis with the old data I collected that had the comma issue.\n\n## Data\nThe data (.csv file) is on my [github](https://github.com/KoderKow/wow_scraper). I wanted to share what the data looked like and that I had success in gathering a lot of different attributes for every weapon.\n\n## Acknowledgments\nI want to thank Professor Lourens for sharing his idea for this project. Professor Lourens assisted me on learning web scraping through rvest/RSelenium and addressed errors I came across.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoderkow%2Fwow_scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkoderkow%2Fwow_scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoderkow%2Fwow_scraper/lists"}