{"id":22521579,"url":"https://github.com/r13i/twitter-sentiment-analysis","last_synced_at":"2025-08-03T20:31:16.155Z","repository":{"id":46331849,"uuid":"226485859","full_name":"r13i/twitter-sentiment-analysis","owner":"r13i","description":"What if we could see the emotions and moods of people through the breadcrumbs they leave on Twitter ?","archived":false,"fork":false,"pushed_at":"2022-12-08T03:23:49.000Z","size":233,"stargazers_count":19,"open_issues_count":2,"forks_count":14,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-04-24T12:15:17.080Z","etag":null,"topics":["docker","emotions","influxdb","kafka","natural-language-processing","nlp","python","sentiment-analysis","tweets","twitter"],"latest_commit_sha":null,"homepage":"https://grafana.redouaneachouri.com/d/_TYUeQCZk/twitter-sentiment-analysis?orgId=1\u0026from=now-6h\u0026to=now","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/r13i.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-07T09:24:40.000Z","updated_at":"2023-11-02T00:33:48.000Z","dependencies_parsed_at":"2023-01-25T04:01:01.813Z","dependency_job_id":null,"html_url":"https://github.com/r13i/twitter-sentiment-analysis","commit_stats":null,"previous_names":["r13i/twitter-sentiment-analysis"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r13i%2Ftwitter-sentiment-analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r13i%2Ftwitter-sentiment-analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r13i%2Ftwitter-sentiment-analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r13i%2Ftwitter-sentiment-analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/r13i","download_url":"https://codeload.github.com/r13i/twitter-sentiment-analysis/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228562317,"owners_count":17937234,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","emotions","influxdb","kafka","natural-language-processing","nlp","python","sentiment-analysis","tweets","twitter"],"created_at":"2024-12-07T05:12:05.123Z","updated_at":"2024-12-07T05:12:06.553Z","avatar_url":"https://github.com/r13i.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tales of Twitter\nWhat if we could see the sentiments of people through the breadcrumbs they leave on Twitter ?\n\n## Quick Access Links\n\nThis project is described in details in [this article](https://medium.com/@redouane.achouri/twitter-sentiment-analysis-a-tale-of-stream-processing-8fd92e19a6e6).\n\nFinal dashboard can be found here: [grafana.redouaneachouri.com](https://grafana.redouaneachouri.com/d/twitter-sentiment-analysis/twitter-sentiment-analysis).\n\nThe Modelization part is described at [./model/twitter-sentiment-analysis.ipynb](./model/twitter-sentiment-analysis.ipynb).\n\n![Architecture diagram](docs/twitter-sentiment-analysis-diagram.png)\n\n## Updates\n- __Saturday, 23rd May 2020__: Changed the InfluxDB retention policy to 30 days to limit disk space consumption. Previous retention was defaulting to infinite. Older data has been discarded and new data starts to be recorded as of this day. See below for [How to Setup a Retention Policy](#how-to-setup-a-retention-policy).\n\n- __Sunday, 24th May 2020__: Found a way to change the retention policy while keeping the old (accumulated, undeleted) records. See [Keep old records](#keep-old-records) below for instructions.\n\n- __Monday, 12th October 2020__: Twitter discontinues some Labs v1 API endpoints (including the sample stream endpoint used by this project), after having launched a new version of its API. I've taken changes into consideration and the project is up-to-date.\n\n- __Saturday, 17th October 2020__: Added a worldmap panel to visualize where tweets were emitted.\n\n\n## Deployment\n\nLocal deployment is done through this [docker-compose.yaml](docker-compose.yaml) file.\n\nLive deployment instructions will be published in a separate tutorial/instructions which will consist mainly of how to setup SSL Encryption with a reverse NGINX proxy to enable Let's Encrypt. Following tutorials will be helpful if you want to setup your own encryption machinery: [Nginx Proxy](https://github.com/nginx-proxy/nginx-proxy) and [Let's Encrypt](https://github.com/nginx-proxy/docker-letsencrypt-nginx-proxy-companion).\n\nCompose offers a way to control the order of statup and shutdown of containers through the `depends_on` option, but, by design, [there is no way to delay statup until a service becomes available](https://docs.docker.com/compose/startup-order/) - unless we run a script that waits for a service to become available with a `command` option.\n\n\u003e The problem of waiting for a database (for example) to be ready is really just a subset of a much larger problem of distributed systems. In production, your database could become unavailable or move hosts at any time. Your application needs to be resilient to these types of failures.\n\n\u003e To handle this, design your application to attempt to re-establish a connection to the database after a failure. If the application retries the connection, it can eventually connect to the database.\n\n### Requirements\n\n- After obtaining your set of Twitter API key and secret (See article mentioned above), you have to set those in the [secret.ini](./produce-tweets/secret.ini) file.\n\n- Run the Jupter Notebook [twitter-sentiment-analysis.ipynb](./model/twitter-sentiment-analysis.ipynb) to build a model and save it in a *.pickle* file. See the (README.md)[./model/README.md] file for running instructions.\n\n### Starting the Services\n\nServices need to be started in a specific order with the following commands:\n```\n# Start Kafka and InfluxDB\ndocker-compose up -d kafka influxdb\n\n# Start Grafana\ndocker-compose up -d grafana\n\n# Start the producer and the consumer\ndocker-compose up -d producer consumer\n```\n\n#### How to Setup a Retention Policy\n\nFollowing instructions show how to setup a retention policy of 30 days for InfluxDB. The DB names are defined in the [config.ini](./config.ini) file.\n\n```\n# On your server, log into the InfluxDB container:\ndocker exec -it twittersentimentanalysis_influxdb bash\n\n# Start the influx prompt (useful execute InfluxQL queries)\ninflux # you can add '-precision rfc3339' to print timestamps to a human-readable format\n```\n\n##### Keep old records\n```\n# List existing retention policies\nSHOW RETENTION POLICIES ON sentiments\nSHOW RETENTION POLICIES ON languages\n\n# Alter default policy (generally named 'autogen')\nALTER RETENTION POLICY autogen ON sentiments DURATION 30d\nALTER RETENTION POLICY autogen ON languages DURATION 30d\n```\n\n##### Start recording from scratch\n\n```\n# Create new retention policy and set the duration to 30 days\nCREATE RETENTION POLICY one_month ON sentiments DURATION 30d REPLICATION 1 DEFAULT\nCREATE RETENTION POLICY one_month ON languages DURATION 30d REPLICATION 1 DEFAULT\n```\n\nOnce all finished:\n```\n# Verify that it has been set successfully\nSHOW RETENTION POLICIES ON sentiments\nSHOW RETENTION POLICIES ON languages\n\n# Exit the terminal with Ctrl+D\n```\n\n### Useful Links\n- [Twitter Developers portal](https://developer.twitter.com/en/docs)\n- [Sentiment Analysis with Python NLTK](https://www.digitalocean.com/community/tutorials/how-to-perform-sentiment-analysis-in-python-3-using-the-natural-language-toolkit-nltk)\n- [Kafka Quickstart](https://kafka.apache.org/quickstart)\n- [A Practical Introduction to Kafka Storage Internals](https://medium.com/@durgaswaroop/a-practical-introduction-to-kafka-storage-internals-d5b544f6925f)\n- [Deploy a Kafka broker in a Docker container](https://www.kaaproject.org/kafka-docker)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fr13i%2Ftwitter-sentiment-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fr13i%2Ftwitter-sentiment-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fr13i%2Ftwitter-sentiment-analysis/lists"}