https://github.com/pkqk/adaptive-panel-collection

https://github.com/pkqk/adaptive-panel-collection

Last synced: about 2 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/pkqk/adaptive-panel-collection
• Owner: pkqk
• Created: 2012-12-10T10:36:08.000Z (about 12 years ago)
• Default Branch: master
• Last Pushed: 2012-12-14T19:14:12.000Z (about 12 years ago)
• Last Synced: 2024-10-12T13:31:20.246Z (3 months ago)
• Language: Python
• Size: 152 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
Twitter panel data collection

=============================

A tool for fetching twitter profile, follower and friend info for SoMA

currently requires a redis database to write it's intermediate results too

TODO: turn this into a process that requires less manual intervention so that

panelist data can be continuously updated

Installation

------------

    virtualenv env

    . env/bin/activate

    pip install -r requirements.txt

    # install redis

    apt-get install redis-server

    # or

    brew install redis

Setup

-----

It needs a config.ini file (look at config.ini.example) to give it a twitter api account

and to tell it which redis server/database to connect to

    [twitter]

    consumer_key = client key value

    consumer_secret = client secret value

    access_token = token

    access_token_secret = token secret

    [redis]

    host = localhost

    port = 6379

    db = 0

Running

-------

collect.py will take a newline separated list of twitter names and collect profile,

followers and friend ids, it will take a while unless you have a less rate limited

twitter account

    python collect.py list-of-twitter-names.txt

Outputing results

-----------------

output.py will write out lines of json represening the panelist info for SoMA and

also write out a list of twitter names where it couldn't collect some of the data

it needs a mapping of twitter screen names to yougov panelist ids (a previous

panel_default.json will suffice or a csv of [screen_name, id] rows)

    python output.py panel.json mapping_for_yougov_id.[csv,json] missing-names.txt

This uses the panoptic proxy provided by beta.pulse.yougov.com, access to the us panoptic

is trickier at the moment.

open an ssh connection to the alab-pulse1 machine and set up a socks proxy to port 8050

    # connect to yougov vpn and then set up this proxy

    ssh -ND 8050 username@alab-pulse1

    # in another terminal

    python us_output.py us_panel.json mapping_for_yougov_id.csv missing-us-names.txt