{"id":19784071,"url":"https://github.com/openeventdata/eldiablo","last_synced_at":"2025-04-30T22:32:01.421Z","repository":{"id":14803670,"uuid":"17525931","full_name":"openeventdata/eldiablo","owner":"openeventdata","description":"Event data in a box, basically.","archived":false,"fork":false,"pushed_at":"2014-11-04T01:44:26.000Z","size":389,"stargazers_count":15,"open_issues_count":0,"forks_count":11,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-06T05:34:33.648Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openeventdata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-03-07T20:11:21.000Z","updated_at":"2023-07-07T14:03:45.000Z","dependencies_parsed_at":"2022-09-16T04:30:30.047Z","dependency_job_id":null,"html_url":"https://github.com/openeventdata/eldiablo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openeventdata%2Feldiablo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openeventdata%2Feldiablo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openeventdata%2Feldiablo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openeventdata%2Feldiablo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openeventdata","download_url":"https://codeload.github.com/openeventdata/eldiablo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251791724,"owners_count":21644447,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T06:10:14.418Z","updated_at":"2025-04-30T22:32:01.117Z","avatar_url":"https://github.com/openeventdata.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Logo][logo]][logo]\n\nEvent data in a box, basically.\n\n##About\n\nThe main goal of EL:DIABLO is to overcome [dependency hell](https://en.wikipedia.org/wiki/Dependency_hell)\nand make it much easier for the end user to get up and running with event data\ncoding. Using software designed for development operations allows us to easily\nshare the setup we use to develop the tools and software for creating and\nworking with event data. In short, no matter what hardware or operating system\na user chooses, it is possible to replicate our exact event-data coding\nplatform on that specific configuration. This goal is important to us for two\nprimary reasons. First, we are striving to make the generation of event data\nmore open than it has historically been. Things such as copyright and licensing\nagreements make it difficult to share source texts for coded event data, but we\n**can** make the **process** as transparent as possible. This is especially\nimportant since there are a multitude of seemingly minor choices that go into\nevent data coding that can have a significant impact on the final product. The\nsecond reason we are pursuing EL:DIABLO as a project is to enable\ncollaboration. It will no longer be unclear what steps are taken to generate\nevent data, or what the various moving pieces within the system are. If someone\nwishes, for example, to develop a new event coder, they can simply drop that in\nto the existing pipeline. The same holds for the various dictionaries,\ngeocoders, web scrapers, etc. It's like Legos. But for event data.\n\n##Components\n\nOn the technical side of things, EL:DIABLO provides the information and scripts\nnecessary to set up a [virtual machine](https://en.wikipedia.org/wiki/Virtual_machine)\non a user's computer. For those not familiar, this can be thought of as a\ncomputer within a computer. EL:DIABLO relies on [Vagrant](https://www.vagrantup.com/),\nand by extension [VirtualBox](https://www.virtualbox.org/), to set up this\nvirtual environment. These two pieces of software allow for the easy setup and\nuse of a virtual machine. Thus, two of the files contained within EL:DIABLO are\na `Vagrantfile`, which gives instructions to Vagrant on how to setup the virtual\nmachine, and `bootstrap.sh`, which is a [shell script](https://en.wikipedia.org/wiki/Shell_script)\nthat installs the necessary software within the virtual machine.\n\nThe EL:DIABLO event coding platform is comprised of two primary applications:\na web scraper and a processing pipeline ([`scraper`](https://github.com/openeventdata/scraper) and\n[`phoenix_pipeline`](https://github.com/openeventdata/phoenix_pipeline) specifically). The scraper is a simple web scraper that\nmakes use of a whitelist of RSS feeds to pull stories from popular news\noutlets. The pipeline moves the news stories from storage in a database to the\nevent coder, such as TABARI or PETRARCH, and outputs event data. More\ninformation about the details of these projects can be found in their\nrespective documentation, linked to above. If you use the standard\n`bootstrap.sh` script provided with EL:DIABLO, the web scraper will run once an\nhour, and the pipeline will run once a day at 01:00.\n\n##Setting up\n\nAs mentioned above, EL:DIABLO relies on Vagrant and VirtualBox for most of the\nheavy lifting. This means that the only things that\na user needs to install on their local machine are these two pieces of\nsoftware. The creators of this software describe the install process better\nthan we can, so a user should look [here](https://www.vagrantup.com/downloads.html)\nfor Vagrant and [here](https://www.virtualbox.org/wiki/Downloads)\nfor VirtualBox. Once that software is installed, EL:DIABLO needs to be\ndownloaded from the [Github repository](https://github.com/openeventdata/eldiablo).\nFor those familiar with `git`, a `git clone` should work fine. For those\nunfamiliar with `git`, it is possible to download the repository as a zip file\nas shown in the picture below.\n\n*Note: We've tested this setup on Vagrant 1.6.5*\n\n[![Github][git]][git]\n\nOnce this file is downloaded and unzipped, you should use the command line to cd into the\ndirectory and do `vagrant up`. This will take awhile to download the operating system image\n(this will only be done once) and then install the relevant software within the virtual\nmachine. Seriously, this is going to take time; the process hasn't stalled out. Then `vagrant ssh`\nto get into the box. You're now in the virtual machine. Overall, this should look something like:\n\n[![Shell][first]][first]\n\n[![Shell][second]][second]\n\nAs a note, all of this will create a folder somewhere on your local machine\nthat contains the operating system images. On OSX it's in the home directory and\nnamed `VirtualBox VMs`. \n\nTo get out of the virtual machine, type `exit`, which will bring you back to your local machine.\nThere are three methods for ending the Vagrant box: `vagrant suspend`, `vagrant halt`, and\n`vagrant destroy`. The main difference between these three is the amount of\nsystem resources used while in the \"down\" state. If you are completely done\nwith the virtual machine, and do not wish to keep any of the data, make use of\n`vagrant destroy`. Again, this *will remove* all of the data within the virtual\nmachine and all software will have to be reinstalled. If you wish to just\ntemporarily bring down the virtual machine, the other two commands should be\nexplored in the [Vagrant documentation](https://docs.vagrantup.com/v2/getting-started/teardown.html).\n\n##Other Information\n\nDue to the way Vagrant sets up the virtual machine, it is necessary to prepend nearly\nevery command with `sudo`.\n\nThe filepaths in the config file for the `stanford_pipeline` need to be changed\nto use absolute paths. For example:\n\n```\ncd ~/stanford_pipeline\nsudo vim default_config.ini\n```\n\nOnce in the config, change the `~/` characters to `/home/vagrant/`. \n\nThe `bootstrap.sh` script is specifically configured for use with the Vagrant\nbox, but with slight modifications can be used on any Linux box (it's what we\nuse to bootstrap our machines). This means that the script can serve as the\nbasis for setting up a high-performance computer running EL:DIABLO, an\nindividual's laptop, etc. \n\nCurrently the virtual machine takes up 4GB of RAM. Less than this doesn't\nreally work since the shift-reduce parser needs a fair amount of memory to\noperate. \n\nFor the two Github repositories, `scraper` and `phoenix_pipeline`, each time\n`vagrant up` is run the most recent version of the code is pulled from Github.\nIf you have a long-running virtual machine and wish to obtain the latest code,\nyou can `cd` into the appropriate directory and run `sudo git pull`. \n\n[git]: http://i.imgur.com/YTT6Ppy.png \"Github example\"\n[first]: http://i.imgur.com/UJtjy3N.png \"Terminal example\"\n[second]: http://i.imgur.com/206UtDs.png \"Second terminal example\"\n[logo]: https://i.imgur.com/3R7gtyr.png \"EL:DIABLO logo\"\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopeneventdata%2Feldiablo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopeneventdata%2Feldiablo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopeneventdata%2Feldiablo/lists"}