{"id":19784063,"url":"https://github.com/openeventdata/scraper","last_synced_at":"2025-04-30T22:31:59.348Z","repository":{"id":13551987,"uuid":"16244037","full_name":"openeventdata/scraper","owner":"openeventdata","description":"Scrapes sites. Gets news. Eventually events.","archived":false,"fork":false,"pushed_at":"2016-03-06T18:58:25.000Z","size":178,"stargazers_count":85,"open_issues_count":2,"forks_count":52,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-04-06T05:34:31.674Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openeventdata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-01-26T01:18:53.000Z","updated_at":"2025-04-05T21:58:19.000Z","dependencies_parsed_at":"2022-09-04T07:42:34.369Z","dependency_job_id":null,"html_url":"https://github.com/openeventdata/scraper","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openeventdata%2Fscraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openeventdata%2Fscraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openeventdata%2Fscraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openeventdata%2Fscraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openeventdata","download_url":"https://codeload.github.com/openeventdata/scraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251791721,"owners_count":21644447,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T06:10:13.716Z","updated_at":"2025-04-30T22:31:59.112Z","avatar_url":"https://github.com/openeventdata.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"scraper\n=======\n\n[![Join the chat at https://gitter.im/openeventdata/scraper](https://badges.gitter.im/openeventdata/scraper.svg)](https://gitter.im/openeventdata/scraper?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\nScrapes sites. Gets news. Eventually events.\n\nMore information can be found in the\n[documentation](http://oeda-scraper.readthedocs.org/en/latest/).\n\n\n###Installation\n\nYou should probably create a [virtual environment](http://www.virtualenv.org/en/latest/), but\nin any event doing `pip install -r requirements.txt` should do the trick. You\nmight (probably will) have to specify something along the lines of \n`--allow-all-external pattern --allow-unverified pattern` for the pattern\nlibrary since it gets downloaded from its homepage. \n\nThe scraper requires  a running MongoDB instance to dump the scraped stories into. \nMake sure you have MongoDB [installed](http://docs.mongodb.org/manual/installation/) \nand type `mongod` at the terminal to begin the instance if your install method\ndidn't set up the MongoDB process to run automatically. MongoDB doesn't require you to prepare\nthe collection or database ahead of time, so when you run the program it should automatically\ncreate a database called `event_scrape` with a collection called `stories`. Once you've run  `python scraper.py`, \nyou can verify that the stories are in the Mongo database by opening a new terminal window and typing `mongo`. \n \nTo interface with Mongo, enter `mongo` at the command line. From inside Mongo, type `show dbs` to verify that there's a database called `event_scrape`. \nEnter the database with `use event_scrape` and type `show collections` to make sure there's a `stories` collection. \n `db.stories.find()` will show you the first 20 entries.\n\n###Running\n\nAfter everything is installed, it's as simple as `python scraper.py`. That is\nassuming, of course, that you wish to use the configuration seen in the\n`default_config.ini` file. If not, just modify that. For the source type\nsection of the config, the three types of sources are `wire`, `international`,\nand `local`. It is possible to specify any combination of those source types,\nwith the source types separated by commas in the config file. For more\ninformation on the source types, see the **Contributing** section below.\n\n###Contributing\n\nMore RSS feeds are always useful. If there's something specific you want to\nsee, just add it in and open a pull request with the source's raw XML RSS feed,\na unique source ID, a label indicating whether the source is\n\"international\" or \"local,\" and what language the site uses. We currently\nsupport English and Arabic in the scraper.\n\nWe face a tradeoff between seeking the broadest geographic coverage we can get\n(meaning including every local paper we can find) and accuracy and relevance\n(which would lead us to include only large, well-known, and high quality news\noutlets). We're trying to balance the two objectives by including a third\ncolumn indicating whether the source is one is a wire service, a dependable\nnews source with solid international coverage, or a local source that may\ncontribute extra noise to the data and may require specialized actor\ndictionaries. The distinction between the latter two is hazy and requires a\njudgement call. Eventually, these labels can be used to build event datasets\nthat are either optimized for accuracy and stability (at the cost of\nsparseness), or micro-level, geographically dispersed (but noisy) coverage.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopeneventdata%2Fscraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopeneventdata%2Fscraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopeneventdata%2Fscraper/lists"}