https://github.com/9-volt/press-kit
The imperative evolution of attempts
https://github.com/9-volt/press-kit
Last synced: 6 months ago
JSON representation
The imperative evolution of attempts
- Host: GitHub
- URL: https://github.com/9-volt/press-kit
- Owner: 9-volt
- Created: 2014-04-19T12:51:08.000Z (over 12 years ago)
- Default Branch: master
- Last Pushed: 2016-10-27T00:56:44.000Z (almost 10 years ago)
- Last Synced: 2025-04-12T15:05:55.642Z (over 1 year ago)
- Language: Ruby
- Homepage:
- Size: 4.26 MB
- Stars: 0
- Watchers: 3
- Forks: 5
- Open Issues: 1
-
Metadata Files:
- Readme: Readme.md
Awesome Lists containing this project
README
## Press Kit
Yet another attempt at creating a full-stack parsing/analysis framework.
We do it for fun anyway.
**Datasets**
- [Fetched pages - 4.7 Gb (updated 29-06-2014)](https://mega.co.nz/#!asQHTJ6D!VYiKiKKZISQsYPlAa0XeodcO1ImxrTd-oTIzrwjCIx4)
- [Parsed pages - 100 Mb (mongo dump, updated 29-06-2014)](https://mega.co.nz/#!iwYTSJoK!nhlFpGNYxpafyUoh1KDNViirjXzK3GQ7oz0inXmmwfg)
## Running
There are several steps:
- Obtain the URLs
- Save the articles in a folder
- Parse the articles
- Squash the results into one big JSON
- Analyze the full data set
- Create a visualization
Run `make run`, wait around 6 hours and you'll have everything.
You'll also need around 7GB of disk space.
Use rake tasks to trigger fetchers and parsers.
Ex. `rake fetch:publika` or `rake parse:unimedia`
Currently in stock:
- ProTv
- Unimedia
- Publika
- Timpul