https://github.com/clemsos/mitras

Investigate social dynamics of memes based on Sina Weibo
https://github.com/clemsos/mitras

Last synced: about 1 month ago
JSON representation

Investigate social dynamics of memes based on Sina Weibo

• Host: GitHub
• URL: https://github.com/clemsos/mitras
• Owner: clemsos
• License: mit
• Created: 2013-08-07T07:34:03.000Z (over 11 years ago)
• Default Branch: master
• Last Pushed: 2014-05-04T14:24:08.000Z (over 10 years ago)
• Last Synced: 2024-10-14T01:23:18.703Z (2 months ago)
• Language: Python
• Size: 13.8 MB
• Stars: 0
• Watchers: 3
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Mitras : Mining memes

Mitras is a set of scripts used to detect, analyse and visualize memes the Chinese microblog Sina Weibo. You will need the Weiboscope dataset prepared by HKU

It has been developed by Clément Renaud for his phD research.

## Mining workflow : 

The different workflows used in this research are documented in iPython notebook in the 

```/doc``` folder.

* ```es_*``` : plain-text search and mining with ElasticSearch and Kibana

* ```hashtags_*``` : build and analyze a corpus of all hashtags in the datasets

* ```pm_*``` : meme detection clustering algorithm using protomemes [(Ferrara, 2013)](http://www.emilio.ferrara.name/2013/08/01/clustering-memes-in-social-media/)

For visualisation, we use Matplotlib, d3js, Networkx and Gephi.

## Data : the Weiboscope  corpus 

To create this project we will use the data provided by the project Weiboscope from HKU University, JMSC - [link](http://147.8.142.179/datazip/). The dataset contains sample data from 52 weeks of 2012 from more than 350,000 Chinese microbloggers who have more than 1,000 followers (Fu, Chan, Chau, 2013 ; Fu, Chau, 2013).

Note : this data has been anonymized

**Data Set Statistics:**

    * Number of weibo messages: 226841122

    * Number of deleted messages: 10865955

    * Number of censored ('Permission Denied') messages: 86083

    * Number of unique weibo users: 14387628

    * 57 files, 18G

**Download & Prepare Data**

    # to download the data

    bash bin/get_raw_data.sh

    #  Downloaded: 57 files, 18G in 6h 42m 3s (803 KB/s)

    # move the files to the data folder

    mv 147.8.142.179/datazip data/datazip

    rm -R 147.8.142.179

    # remove zip files

    ls data/datazip/*zip | xargs -i rm {}