{"id":13401184,"url":"https://github.com/MUSA-620-Spring-2018/MUSA-620-Week-6","last_synced_at":"2025-03-14T06:32:16.776Z","repository":{"id":217838672,"uuid":"121998200","full_name":"MUSA-620-Spring-2018/MUSA-620-Week-6","owner":"MUSA-620-Spring-2018","description":"Web scraping 1","archived":false,"fork":false,"pushed_at":"2018-02-22T02:09:02.000Z","size":2695,"stargazers_count":2,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-07-21T06:44:25.915Z","etag":null,"topics":["css","html","selectors","web-inspector","web-scraping"],"latest_commit_sha":null,"homepage":null,"language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MUSA-620-Spring-2018.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-18T22:52:05.000Z","updated_at":"2018-02-25T18:42:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"25fb5e3a-3a39-4bbd-a4fa-d00082bb44f1","html_url":"https://github.com/MUSA-620-Spring-2018/MUSA-620-Week-6","commit_stats":null,"previous_names":["musa-620-spring-2018/musa-620-week-6"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MUSA-620-Spring-2018%2FMUSA-620-Week-6","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MUSA-620-Spring-2018%2FMUSA-620-Week-6/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MUSA-620-Spring-2018%2FMUSA-620-Week-6/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MUSA-620-Spring-2018%2FMUSA-620-Week-6/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MUSA-620-Spring-2018","download_url":"https://codeload.github.com/MUSA-620-Spring-2018/MUSA-620-Week-6/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243538060,"owners_count":20307100,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["css","html","selectors","web-inspector","web-scraping"],"created_at":"2024-07-30T19:00:59.590Z","updated_at":"2025-03-14T06:32:16.769Z","avatar_url":"https://github.com/MUSA-620-Spring-2018.png","language":"R","funding_links":[],"categories":["R"],"sub_categories":[],"readme":"# MUSA-620-Week-6: Web Scraping Part 1\n\n\n\n### Links:\n* [HTML vs DOM, Example 1](https://blueshift.io/selectors2.html)\n* [HTML vs DOM, Example 2](https://blueshift.io/selectors2.html)\n* [rvest package](https://cran.r-project.org/web/packages/rvest/rvest.pdf) for parsing HTML\n\n## Getting to Know Your Browser's Web Inspector\n\n### Elements tab\n\nUse the elements tab to explore a webpage's DOM.\n\n##### Examples:\n* [Simple example](https://blueshift.io/selectors2.html)\n* Grab text from sites that have copying disabled: [NY Times](https://www.nytimes.com/2016/08/23/upshot/50-years-of-electoral-college-maps-how-the-us-turned-red-and-blue.html)\n* Find and download a hidden video file: [Imgur](https://i.imgur.com/9M4Rsf0.gifv)\n\n### Network tab\n\nUse the network tab to find the data used in these webpages.\n\n##### [FiveThirtyEight: Congressional Districts](https://projects.fivethirtyeight.com/redistricting-maps/)\n* [Congressional district map](https://projects.fivethirtyeight.com/redistricting-maps/US-current.topo.json)\n\n##### [WSJ: California Drought](http://graphics.wsj.com/californias-long-challenge-with-drought/)\n* [CA outline](http://graphics.wsj.com/californias-long-challenge-with-drought/data/shared/california.topo.json)\n* [Drought isocrones](http://graphics.wsj.com/californias-long-challenge-with-drought/data/drought/drought.ca.topo.json)\n* [Major cities](http://graphics.wsj.com/californias-long-challenge-with-drought/data/shared/major_cities.topo.json)\n* [Population affected](http://graphics.wsj.com/californias-long-challenge-with-drought/data/drought/population-affected.csv)\n\n![California drought map](https://github.com/MUSA-620-Spring-2018/MUSA-620-Week-6/blob/master/california-drought-map.png)\n\n*Data: [Wall Street Journal](http://graphics.wsj.com/californias-long-challenge-with-drought/) / code: [web-inspector-data.r](https://github.com/MUSA-620-Spring-2018/MUSA-620-Week-6/blob/master/web-inspector-data.R)*\n\n##### [Washington Post: US Sunlight Map](https://www.washingtonpost.com/news/wonk/wp/2015/07/13/map-where-americas-sunniest-and-least-sunny-places-are/)\n* See if you can you can find the data to recreate their sunlight map yourself. *The steps for doing this have now been added to [web-inspector-data.r](https://github.com/MUSA-620-Spring-2018/MUSA-620-Week-6/blob/master/web-inspector-data.R)*\n\n![Map of average daily sunlight by county](https://github.com/MUSA-620-Spring-2018/MUSA-620-Week-6/blob/master/daily-sunlight-by-county-2.png)\n\n*Data: [Washington Post](https://www.washingtonpost.com/news/wonk/wp/2015/07/13/map-where-americas-sunniest-and-least-sunny-places-are/) / code: [web-inspector-data.r](https://github.com/MUSA-620-Spring-2018/MUSA-620-Week-6/blob/master/web-inspector-data.R)*\n\n##### [The Economist: Property Values by US County](https://www.economist.com/blogs/graphicdetail/2015/04/daily-chart-2)\n* Sometimes data will be hidden in .js files: [county-level real estate values](https://infographics.economist.com/2015/ASBTest/Land/js/countyData.js?__sbCache=0.26521743179319657)\n* Used as the data source for the map below\n\n[Total residential property value of each US county](http://metrocosm.com/the-housing-value-of-every-county-in-the-u-s/)\n\n![map of property values USA](http://i0.wp.com/metrocosm.com/wp-content/uploads/2015/10/cartogram-property-values.gif)\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMUSA-620-Spring-2018%2FMUSA-620-Week-6","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMUSA-620-Spring-2018%2FMUSA-620-Week-6","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMUSA-620-Spring-2018%2FMUSA-620-Week-6/lists"}