https://github.com/vspiewak/twitter-sentiment-analysis

Streaming tweets with spark, language detection & sentiment analysis, dashboard with Kibana
https://github.com/vspiewak/twitter-sentiment-analysis

dashboard kibana nlp scala sentiment-analysis spark tiwtter

Last synced: 2 days ago
JSON representation

Streaming tweets with spark, language detection & sentiment analysis, dashboard with Kibana

• Host: GitHub
• URL: https://github.com/vspiewak/twitter-sentiment-analysis
• Owner: vspiewak
• License: mit
• Created: 2015-02-28T13:46:35.000Z (almost 10 years ago)
• Default Branch: master
• Last Pushed: 2015-12-21T04:44:19.000Z (about 9 years ago)
• Last Synced: 2024-04-14T20:05:00.252Z (9 months ago)
• Topics: dashboard, kibana, nlp, scala, sentiment-analysis, spark, tiwtter
• Language: Scala
• Homepage:
• Size: 1.27 MB
• Stars: 104
• Watchers: 14
• Forks: 71
• Open Issues: 5
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.md

Awesome Lists containing this project

README

        
Twitter Sentiment Analysis

==========================

* retrieve tweets using Spark Streaming

* language detection

* sentiment analysis (StanfordNLP)   

* index tweets in Elasticsearch 

* live dashboard using Kibana

Docker setup

------------

Configure Twitter OAuth in [`docker-compose.yml`](./docker-compose.yml) file.

After installing docker & docker-compose, you can launch the stack with the following commands:

    docker-compose build

    docker-compose up

After a while, Elasticsearch, Kibana, and Spark should be up:

 * http://0.0.0.0:4040/streaming

 * http://0.0.0.0:9200/_plugin/head

 * http://0.0.0.0:5601 

Manual setup

------------

    curl -O http://d3kbcqa49mib13.cloudfront.net/spark-1.5.2.tgz

    tar xvzf spark-1.5.2.tgz

    cd spark-1.5.2

    ./sbt/sbt assembly

    

    cd ..

    curl -O https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.1.1/elasticsearch-2.1.1.tar.gz

    tar xvzf elasticsearch-2.1.1.tar.gz

    cd elasticsearch-2.1.1

    bin/plugin -install mobz/elasticsearch-head

    bin/elasticsearch -d

    cd ..

    chmod a+x insert.dashboard.sh

    ./insert.dashboard.sh

    curl -O https://download.elastic.co/kibana/kibana/kibana-4.3.1-linux-x64.tar.gz

    tar xvzf kibana-4.3.1-linux-x64.tar.gz 

    cd kibana-4.3.1-linux-x64

    bin/kibana

    cd ..

    curl -L -O https://dl.bintray.com/sbt/native-packages/sbt/0.13.9/sbt-0.13.9.tgz

    tar xvzf sbt-0.13.9.tgz

    JAVA_OPTS=-Xmx2G sbt/bin/sbt assembly

    ../spark-1.5.2/bin/spark-submit \

    --class com.github.vspiewak.TwitterSentimentAnalysis \

    --master local[2] \

    target/scala-2.10/twitter-sentiment-analysis-assembly-0.1-SNAPSHOT.jar \

     \

     \

     \

     \

    []