Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/pablobarbera/twitter_ideology

Estimating Ideological Positions with Twitter Data
https://github.com/pablobarbera/twitter_ideology

Last synced: 3 days ago
JSON representation

Estimating Ideological Positions with Twitter Data

Awesome Lists containing this project

README 

        
Estimating Ideological Positions with Twitter Data

----------------



This GitHub repository contains code and materials related to the article "[Birds of a Feather Tweet Together. Bayesian Ideal Point Estimation Using Twitter Data](http://pan.oxfordjournals.org/content/23/1/76.full)," published in Political Analysis in 2015. 



The original replication code can be found in the `replication` folder. See also [Dataverse](https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/26589) for the full replication materials, including data and output.



For updated versions of the code, that implement a more computationally efficient approach introduced in [Barberá et al (2015, Psychological Science)](https://journals.sagepub.com/doi/abs/10.1177/0956797615594620), see the folders [2016-update/](https://github.com/pablobarbera/twitter_ideology/tree/master/2016-update), [2018-update/](https://github.com/pablobarbera/twitter_ideology/tree/master/2018-update), and [2020-update/](https://github.com/pablobarbera/twitter_ideology/tree/master/2020-update).



As an application of the method, in June 2015 I wrote a blog post on The Monkey Cage / Washington Post entitled ["Who is the most conservative Republican candidate for president?."](http://www.washingtonpost.com/blogs/monkey-cage/wp/2015/06/16/who-is-the-most-conservative-republican-candidate-for-president/) The replication code for the figure in the post is available in the `primary` folder.



Finally, this repository also contains an R package (`tweetscores`) with several functions to facilitate the application of this method in future research. The rest of this README file provides a tutorial with instructions showing how to use it.



**NOTE**: the package currently only support v1.1 of Twitter's API, which at some point in the near future will be deprecated. This package is not currently maintained, so use at your own risk.



### Authentication



In order to download data from Twitter’s API, you will first need to create a developer account in [developer.twitter.com](developer.twitter.com). After receiving your approval, it’s necessary to follow these steps:



1. Go to [Twitter's Developer Portal](https://developer.twitter.com/en/portal/dashboard) and sign in

2. Click on "Overview", then “Create New App”

3. Follow the instructions.

4. Copy the API Key and Secret, as well as the Access Token and Secret, and paste them below:

```

my_oauth <- list(consumer_key = "CONSUMER_KEY",

    consumer_secret = "CONSUMER_SECRET",

    access_token="ACCESS_TOKEN",

    access_token_secret = "ACCESS_TOKEN_SECRET")

```

5. Change current folder into a folder where you will save all your tokens

```

setwd("~/Dropbox/credentials/twitter&quot")

```

6. Now you can save oauth token for use in future sessions with R

```

save(my_oauth, file="my_oauth")

```



Installing the tweetscores package


The following code will install the tweetscores package, as well as all other R packages necessary for the functions to run.


toInstall <- c("ggplot2", "scales", "R2WinBUGS", "devtools", "yaml", "httr", "RJSONIO")

install.packages(toInstall, repos = "http://cran.r-project.org")

library(devtools)

install_github("pablobarbera/twitter_ideology/pkg/tweetscores")






Estimating the ideological positions of a US Twitter user


We can now go ahead and estimate ideology for any Twitter users in the US. In order to do so, the package includes pre-estimated ideology for political accounts and media outlets, so here we’re just replicating the second stage in the method – that is, estimating a user’s ideology based on the accounts they follow.


# load package

library(tweetscores)


# downloading friends of a user

user <- "p_barbera"

friends <- getFriends(screen_name=user, oauth="~/Dropbox/credentials/twitter")


## /Users/pablobarbera/Dropbox/credentials/twitter/oauth_token_32 

## 15  API calls left

## 1065 friends. Next cursor:  0 

## 14  API calls left


# estimate ideology with MCMC method

results <- estimateIdeology(user, friends)


## p_barbera follows 11 elites: nytimes maddow caitlindewey carr2n fivethirtyeight 

NickKristof nytgraphics nytimesbits NYTimeskrugman nytlabs thecaucus

## Chain 1

  |=================================================================| 100%

## Chain 2

  |=================================================================| 100%



Once we have this set of estimates, we can analyze them with a series of built-in functions.


# summarizing results

summary(results)


##        mean   sd  2.5%   25%   50%   75% 97.5% Rhat n.eff

## beta  -2.30 0.57 -3.37 -2.72 -2.25 -1.92 -1.26 1.02   200

## theta -1.78 0.30 -2.28 -1.99 -1.82 -1.59 -1.11 1.00   200


# assessing chain convergence using a trace plot

tracePlot(results, "theta")




# comparing with other ideology estimates

plot(results)








Faster ideology estimation


The previous function relies on a Metropolis-Hastings sampling algorithm to estimate ideology. However, we can also use Maximum Likelihood estimation to compute the distribution of the latent parameters. This method is much faster, since it’s not sampling from the posterior distribution of the parameters, but it will tend to give smaller standard errors. However, overall the results should be almost identical. (See here for the actual estimation functions for each of these two approaches.)


# faster estimation using maximum likelihood

results <- estimateIdeology(user, friends, method="MLE")


## p_barbera follows 11 elites: nytimes maddow caitlindewey carr2n fivethirtyeight 

NickKristof nytgraphics nytimesbits NYTimeskrugman nytlabs thecaucus


summary(results)


##        mean   sd  2.5%   25%   50%   75% 97.5% Rhat n.eff

## beta  -2.30 0.57 -3.37 -2.72 -2.25 -1.92 -1.26 1.02   200

## theta -1.78 0.30 -2.28 -1.99 -1.82 -1.59 -1.11 1.00   200






Estimation using correspondence analysis


One limitation of the previous method is that users need to follow at least one political account. To partially overcome this problem, in a recently published article in Psychological Science, we add a third stage to the model where we add additional accounts (not necessarily political) followed predominantely by liberal or by conservative users, under the assumption that if other users also follow this same set of accounts, they are also likely to be liberal or conservative. To reduce computational costs, we rely on correspondence analysis to project all users onto the latent ideological space (see Supplementary Materials), and then we normalize all the estimates so that they follow a normal distribution with mean zero and standard deviation one. This package also includes a function that reproduces the last stage in the estimation, after all the additional accounts have been added:



# estimation using correspondence analysis

results <- estimateIdeology2(user, friends)


## p_barbera follows 22 elites: andersoncooper, billclinton, BreakingNews, 

## cnnbrk, davidaxelrod, Gawker, HillaryClinton, maddow, MaddowBlog, mashable, mattyglesias,

## NateSilver538, NickKristof, nytimes, NYTimeskrugman, repjoecrowley, RonanFarrow, 

## SCOTUSblog, StephenAtHome, TheDailyShow, TheEconomist, UniteBlue


results


## [1] -1.06158







Additional functions


The package also contains additional functions that I use in my research, which I’m providing here in case they are useful:


    

  • 

    scrapeCongressData is a scraper of the list of Twitter accounts for Members of the US congress from the unitedstates Github account.
    • 

  • 

    getUsersBatch scrapes user information for more than 100 Twitter users from Twitter’s REST API.
    • 

  • 

    getFollowers scrapes followers lists from Twitter’ REST API.
    • 

  • 

    getTimeline downloads up to 3,200 most recent tweets for any given Twitter user.
    • 

  • 

    CA is a modified version of the ca function in the ca package (available on CRAN) that computes simple correspondence analysis with a much lower memory usage.
    • 

  • 

    supplementaryColumns and supplementaryRows takes additional columns of a follower matrix and projects them to the latent ideological space using the parameters of an already-fitted correspondence analysis model.
    •