https://github.com/nickstenning/epsrc-gow

A scrape of data from the UK Engineering and Physical Sciences Research Council "Grants on the Web" service.
https://github.com/nickstenning/epsrc-gow

Last synced: over 1 year ago
JSON representation

A scrape of data from the UK Engineering and Physical Sciences Research Council "Grants on the Web" service.

• Host: GitHub
• URL: https://github.com/nickstenning/epsrc-gow
• Owner: nickstenning
• Created: 2011-03-08T15:39:05.000Z (over 15 years ago)
• Default Branch: master
• Last Pushed: 2011-07-14T23:10:43.000Z (about 15 years ago)
• Last Synced: 2025-02-09T21:12:11.909Z (over 1 year ago)
• Language: Python
• Homepage:
• Size: 166 KB
• Stars: 2
• Watchers: 3
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.mkd

Awesome Lists containing this project

README

          
# [EPSRC Past Grant portfolio](http://gow.epsrc.ac.uk/SearchPastGrant.aspx)

As of 2011-03-08, the EPSRC "Grants on the Web" database contains grants dating back to 1985:

- 35,966 grants

- £6,055,173,655 of committed funding

This repository contains tools for scraping and manipulating this data into a useful format.

## Quick access

Instead of using the code in this repository, why not download the database generated by it? A listing of these can be found [at CKAN](http://storage.ckan.net/epsrc-gow).

- [Latest database](http://storage.ckan.net/epsrc-gow/grants-20110310.sql.bz2) (SQL format, updated 2011-03-10)

- [Extra grants detail](http://storage.ckan.net/epsrc-gow/grants-detail-20110311.yaml.bz2) (YAML format, updated 2011-03-11)

## Setup

To get up and running, you will need `python` and `pip`. Instructions for installing `python` are beyond the scope of this document, but you should be able to get `pip` if you don't have it by running:

    $ easy_install pip

If you use `virtualenv`, you might want to create an environment to play around in:

    $ virtualenv --distribute pyenv

    $ source pyenv/bin/activate

Now install the scraper and its dependencies:

    $ pip install -e .

Create the database (or migrate your existing database to the current schema version) by running:

    $ ./migrate.sh epsrc.db 3

## Usage

To get a database of basic grants information for the financial year beginning 1 Apr 1993, run:

    $ epsrc-scrape --basic --year 1993 epsrc.db

More usefully, to download all the information in the EPSRC GOW repository (this will take a long time to complete):

    $ epsrc-scrape epsrc.db

Each grant has a unique grant reference, e.g. `GR/J50118/01`. We can check that the example grant above made it into the database with the following command

    $ sqlite3 -line epsrc.db "select * from grants where id='GR/J50118/01'"

                           id = GR/J50118/01

                        title = USE OF CATALYTIC MEMBRANES FOR IN SITU REACTION AND SEPARATION

                        value = 114608

    principal_investigator_id = 40792

                department_id = 5273

                   created_at = 2011-06-16 12:36:35.164790

                  modified_at = 2011-06-16 12:36:35.164800

## Warnings

Be careful with the data this scraper generates. Care has been taken to sanitize the data, but as with any scraper, there may be inconsistencies or missing entries. Please [let me know](https://github.com/nickstenning/epsrc-gow/issues) if you find any problems.