{"id":19065082,"url":"https://github.com/nathan-lindstedt/coal_study","last_synced_at":"2026-06-25T07:31:11.477Z","repository":{"id":235960079,"uuid":"228533116","full_name":"nathan-lindstedt/coal_study","owner":"nathan-lindstedt","description":"ProQuest Index Scraping w/ Selenium and Beautiful Soup","archived":false,"fork":false,"pushed_at":"2020-06-22T01:09:21.000Z","size":8130,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-02T14:12:26.590Z","etag":null,"topics":["beatifulsoup","climate-change","coal","selenium-python","sql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nathan-lindstedt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2019-12-17T04:33:41.000Z","updated_at":"2024-11-17T18:21:12.000Z","dependencies_parsed_at":"2024-04-25T04:49:26.537Z","dependency_job_id":null,"html_url":"https://github.com/nathan-lindstedt/coal_study","commit_stats":null,"previous_names":["nathan-lindstedt/coal_study"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nathan-lindstedt%2Fcoal_study","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nathan-lindstedt%2Fcoal_study/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nathan-lindstedt%2Fcoal_study/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nathan-lindstedt%2Fcoal_study/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nathan-lindstedt","download_url":"https://codeload.github.com/nathan-lindstedt/coal_study/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240118432,"owners_count":19750491,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beatifulsoup","climate-change","coal","selenium-python","sql"],"created_at":"2024-11-09T00:48:32.886Z","updated_at":"2026-05-15T03:30:19.582Z","avatar_url":"https://github.com/nathan-lindstedt.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# coal_study\n\nThis project involves parsing and scraping text data on climate change and coal energy from ProQuest Congressional and ProQuest Newspaper indexes utilizing the Selenium and Beautiful Soup packages for Python. \n\nIncluded in this project are four main component files: **coal_query.sql**, **congress_parser.py**, **congress_scraper.py**, and **news_parser.py**\n\nThe script **coal_query.sql** contains the PostgreSQL queries used to create the tables and views needed for storage and analysis of text data.\n\nThe script **congress_parser.py** parses text data from ProQuest Congressional and inserts the transcript date of publication and url of Congressional hearings into a PostgreSQL local host. \n\nThe script **congress_scraper.py** uses the table created by congress_parser.py to retrieve the url; scrapes the associated webpage text data from ProQuest Congressional; and inserts the document title, committee, text, and url into a PostgreSQL local host. (Note: To be able to run, this script requires ChromeDriver to be installed in its local directory. For more information, see: https://chromedriver.chromium.org/).\n\nThe script **news_parser.py** parses text data from ProQuest Newspaper and inserts the newspaper article ID and text into a PostgreSQL local host.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnathan-lindstedt%2Fcoal_study","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnathan-lindstedt%2Fcoal_study","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnathan-lindstedt%2Fcoal_study/lists"}