https://github.com/sunlightpolicy/fedwebanalysis
Web analysis tool for Sunlight Foundation Web Integrity Project
https://github.com/sunlightpolicy/fedwebanalysis
federal-websites scraping wip
Last synced: 3 months ago
JSON representation
Web analysis tool for Sunlight Foundation Web Integrity Project
- Host: GitHub
- URL: https://github.com/sunlightpolicy/fedwebanalysis
- Owner: sunlightpolicy
- Created: 2019-08-26T19:53:52.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2019-08-26T22:42:42.000Z (over 5 years ago)
- Last Synced: 2025-01-14T13:14:49.412Z (4 months ago)
- Topics: federal-websites, scraping, wip
- Language: Python
- Homepage: http://54.166.131.120.xip.io:2000/
- Size: 64.5 KB
- Stars: 1
- Watchers: 4
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# fedwebanalysis
A Flask-based website and Python scripts automating web analysis on federal government domains for the Sunlight Foundation's Web Integrity Project. The website is designed to be hosted on an AWS EC2 instance, which calls Python programs to run on another AWS EC2 instance, and stores output onto a Google Drive account. This site structure flows from what capacities AWS provides in its free student account, so `fedwedanalysis` will work on a AWS Free Tier.
It is recommended reaching the EC2 instances via Putty and FileZilla.
## Flask EC2
* Contains `templates` folder, with HTML files of the website.
* Contains `static` folder, with CSS file of the website.
* Contains an `uploads` folder.
* Contains your access `.pem` file for the computing instance inside your `.ssh` folder
* Contains `app.py`, `helpers.py`, `client_secrets.json` file (from your Google OAuth access), and optionally a `settings.yaml` file
## Computing EC2
* Contains domain specific scraping Python programs, which obtain all the URLs on a federal domains' sitemap
* Contains term finding Python program adapted highly from [Eric Nost's EDGI controlf.py](https://github.com/ericnost/EDGI), which counts the usage of a file of terms over a set of urls
* Contains term aggregating Python program conceptually adapted from [Eric Nost's EDGI termcount.R](https://github.com/ericnost/EDGI)
* Contains web_monitoring folder with two files `internetarchive.py` and `utils.py` taken from [a component of EDGI's Web Monitoring Project](https://github.com/edgi-govdata-archiving/web-monitoring-processing)
* Contains bash files for running each Python program
* Contains your access `.pem` file for the Flask instance inside your `.ssh` folder