{"id":27334871,"url":"https://github.com/dmuth/twitter-sentiment-analysis","last_synced_at":"2025-08-23T13:34:21.808Z","repository":{"id":50186180,"uuid":"141210074","full_name":"dmuth/twitter-sentiment-analysis","owner":"dmuth","description":"Perform sentiment analysis for Twitter keywords with AWS Comphrehend and Splunk","archived":false,"fork":false,"pushed_at":"2022-12-08T01:01:04.000Z","size":1554,"stargazers_count":4,"open_issues_count":2,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-12T14:51:24.730Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dmuth.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-07-17T00:35:52.000Z","updated_at":"2022-09-14T19:30:24.000Z","dependencies_parsed_at":"2023-01-24T07:01:28.299Z","dependency_job_id":null,"html_url":"https://github.com/dmuth/twitter-sentiment-analysis","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dmuth/twitter-sentiment-analysis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmuth%2Ftwitter-sentiment-analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmuth%2Ftwitter-sentiment-analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmuth%2Ftwitter-sentiment-analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmuth%2Ftwitter-sentiment-analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dmuth","download_url":"https://codeload.github.com/dmuth/twitter-sentiment-analysis/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmuth%2Ftwitter-sentiment-analysis/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271749051,"owners_count":24814115,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-12T14:46:32.259Z","updated_at":"2025-08-23T13:34:21.779Z","avatar_url":"https://github.com/dmuth.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Twitter Sentiment Analysis\n\nThis app makes use of Twitter's API and AWS Comprehend to get insights about your company,\ncity, event, or convention by way of analyzing tweet sentiment.  It further\nallows drilldown by user, topic, and search string.\n\nNOTE: I am sorry at this app is as complex as it's gotten.  I have a refactoring effort all planned out, I just need some 36-hour days in which to pull it off. :-)\n\n\n## Requirements\n\n- A Twitter app (created at \u003ca href=\"$C \u0026\u0026 docker-compose run $C bash\"\u003ehttps://developer.twitter.com/en/apps\u003c/a\u003e)\n- Twitter credentials (these can be created on the fly when running the first script)\n- AWS Credentials\n- Working knowledge of \u003ca href=\"http://splunk.com/\"\u003eSplunk\u003c/a\u003e.  There is some documentation \u003ca href=\"http://docs.splunk.com/Documentation/Splunk/7.1.2/SearchTutorial/WelcometotheSearchTutorial\"\u003ehere\u003c/a\u003e, but existing dashboards should be enough to get you started.\n- Docker\n\n\n## Screenshots\n\n\u003ca href=\"https://raw.githubusercontent.com/dmuth/twitter-sentiment-analysis/master/img/sentiment-breakdown-by-keyword.png\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/dmuth/twitter-sentiment-analysis/master/img/sentiment-breakdown-by-keyword.png\" width=\"250\" /\u003e\u003c/a\u003e \u003ca href=\"https://raw.githubusercontent.com/dmuth/twitter-sentiment-analysis/master/img/tag-cloud.png\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/dmuth/twitter-sentiment-analysis/master/img/tag-cloud.png\" width=\"250\" /\u003e\u003c/a\u003e \u003ca href=\"https://raw.githubusercontent.com/dmuth/twitter-sentiment-analysis/master/img/tweet-volume.png\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/dmuth/twitter-sentiment-analysis/master/img/tweet-volume.png\" width=\"250\" /\u003e\u003c/a\u003e  \u003ca href=\"https://raw.githubusercontent.com/dmuth/twitter-sentiment-analysis/master/img/sentiment-for-twitter-mentions.png\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/dmuth/twitter-sentiment-analysis/master/img/sentiment-for-twitter-mentions.png\" width=\"250\" /\u003e\u003c/a\u003e \u003ca href=\"https://raw.githubusercontent.com/dmuth/twitter-sentiment-analysis/master/img/search-tweets.png\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/dmuth/twitter-sentiment-analysis/master/img/search-tweets.png\" width=\"250\" /\u003e\u003c/a\u003e \u003ca href=\"https://raw.githubusercontent.com/dmuth/twitter-sentiment-analysis/master/img/service-status.png\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/dmuth/twitter-sentiment-analysis/master/img/service-status.png\" width=\"250\" /\u003e\u003c/a\u003e\n\n\n## Installation\n\n\n### Setting up an Agent\n\nBecause I've had a need to pull in different search queries and keep them separate, tweets\nare pulled by one or more Agents.  Agents are created with the `create-agent.sh` script\nas follows:\n\n`./create-agent.sh linux linux,#linux s3://YOUR_S3_BACKUP/tweets/linux/`\n\nThe first argument is the name of the agent, which will then be created in `agents/linux`,\nthe second argument is the search string (multiple things to search for can be separated\nby commas), and the third argument is the S3 location to backup to.  Backups are optional,\nso if there are no plans to back up, just use the default above and you'll be fine.\n\nNow, cd into your Agent directory (`agents/linux` in the previous example) and proceed as follows:\n\n- Get your Twitter credentials:\n   - `docker build -f ./Dockerfile-0-get-twitter-credentials -t 0-get-twitter-credentials . \u0026\u0026 docker run -v $(pwd):/mnt -it 0-get-twitter-credentials`\n   - Running this script will walk your through the steps, so it should be straightforward.\n- Make sure you have \u003ca href=\"https://aws.amazon.com/cli/\"\u003eAWS CLI\u003c/a\u003e installed and ran `aws configure` to enter your AWS credentials.\n   - Access to a single S3 bucket and AWS Comprehend will be required\n   - Note that if you create a policy for your IAM credentials (as you should!), the ARN in the Resource array must end in `/*`.  Exammple: `arn:aws:s3:::tweets/*`. There is a byg in the policy generator where this won't happen and your backups will fail.  Be careful\n   - You can now copy your AWS credentials to the agent directory with `cp ~/.aws/credentials aws-credentials.txt`\n\n\nRunning `docker-compose up -d` should start all your containers.  Look in `logs/` to see what's happening.\n\n\n### Deleting an Agent\n\nSimply run the script `delete-agent.sh` and the Agent you specify will be deleted.  This\nincludes its configuration, SQLite database, and logfiles.\n\nNote that when deleting an Agent (as well as creating one), `splunk-app/default/inputs.conf`\nis rewritten the the locations of logs from all agents.\n\n\n### Setting up Splunk\n\n- Return to the root directory of this repo.\n- Copy `docker-compose.yml.example` to `docker-compose.yml` and set the port as you see fit.\n- `docker-compose up -d`\n\nSplunk should now be running on https://localhost:8000/ and ingesting logs written by\nagents.\n\n\n### Running the app\n\nNow you can run the app via Docker:\n\n- `cp ~/.aws/credentials aws-credentials.txt`\n- Copy `docker-compose.yml.example` to `docker-compose.yml` and edit the latter with things like your search string and other values\n- `docker-compose up -d`\n\nThis will start up several Docker containers in the background running various Python\nscripts and a copy of Splunk.  To access Splunk, go to http://localhost:8000/ and\nlog in with default credentials of `admin/password` and *immediately* change your password.  **Do not expose this port to the Internet. Use nginx with HTTPS as a proxy if you do!**\n\n\n### Exporting the Password file\n\nIf you create users and want them to persist between runs, you can export the passwd file with this script:\n\n`./bin/export-password-file-from-splunk`\n\nRunning that will export the passwd file from Splunk and sstore it in `splunk-config/passwd`.  If the container\nis deleted at any point and then re-run, the passwd file will be copied into Splunk so the users will be able \nto log back in.\n\n\n### Restoring From Backups\n\nLet's say you were running this app on your desktop/laptop (as one does in Docker...) and you're ready\nto move the app to a server, how do you take all of your data with you?\n\nFrom an Agent, run `./bin/aws-download-latest-backup`. \nThat will connect to AWS and download the latest backup!\nThen simply rename the file to `tweets.db`, start up the containers, and you'll be up and running!\n\n\n## Architecture Overview\n\nThe following docker containers are used:\n\n- `1-fetch-tweets`\n   - Downloads tweets from Twitter with a search string in them.\n- `2-analyze-tweets`\n   - Sends tweets off to AWS to be analyzed\n- `3-export-tweets`\n   - Exports analyzed Tweets from the SQLite database to disk, where they can be analyzed\n- `4-backup`\n   - Does regular backups ofthe SQLIte database to AWS S3.\n- `4-splunk`\n   - Runs Splunk.\n\n\n## Adding reports to Splunk\n\nFeel free to edit/save new reports in Splunk, they will all show up in the `splunk-app/` directory.\n\n\n## Data persistence\n\nSplunk will save its index to `splunk-data/` between runs.\n\n\n## Development\n\nTo do developmenton a container, first go into `docker-compose.yml` and uncomment the `DEBUG` line in\n`environment:`. Then run one of these commands:\n\n- `export C=\"1-fetch-tweets\"; docker-compose kill $C \u0026\u0026 docker-compose rm -f $C \u0026\u0026 docker-compose build $C \u0026\u0026 docker-compose up $C`\n- `export C=\"2-analyze-tweets\"; docker-compose kill $C \u0026\u0026 docker-compose rm -f $C \u0026\u0026 docker-compose build $C \u0026\u0026 docker-compose up $C`\n- `export C=\"3-export-tweets\"; docker-compose kill $C \u0026\u0026 docker-compose rm -f $C \u0026\u0026 docker-compose build $C \u0026\u0026 docker-compose up $C`\n- `export C=\"4-splunk\"; docker-compose kill $C \u0026\u0026 docker-compose rm -f $C \u0026\u0026 docker-compose build $C \u0026\u0026 docker-compose up $C`\n- `export C=\"4-backup\"; docker-compose kill $C \u0026\u0026 docker-compose rm -f $C \u0026\u0026 docker-compose build $C \u0026\u0026 docker-compose up $C`\n\nWith `DEBUG` enabled in `docker-compose.yml`, standard output and standard error will be written to the screen instead of\nto `logs/`.  Note that for `3-export-tweets` specifically, this means that tweets will not be written \nto logfiles and therefore not make it into Splunk.\n\nTo attach to the running Splunk instance for troubleshooting:\n\n- `docker-compose exec 4-splunk bash`\n\n\n### Interactive Bash Shells\n\nDo you want an interactive `bash` shell so that you can instead stay in the container and run the script repeatedly?\nHere, try these commands:\n\n- `export C=\"1-fetch-tweets\"; docker-compose kill $C \u0026\u0026 docker-compose rm -f $C \u0026\u0026 docker-compose build $C \u0026\u0026 docker-compose run $C bash`\n- `export C=\"2-analyze-tweets\"; docker-compose kill $C \u0026\u0026 docker-compose rm -f $C \u0026\u0026 docker-compose build $C \u0026\u0026 docker-compose run $C bash`\n- `export C=\"3-export-tweets\"; docker-compose kill $C \u0026\u0026 docker-compose rm -f $C \u0026\u0026 docker-compose build $C \u0026\u0026 docker-compose run $C bash`\n- `export C=\"4-backup\"; docker-compose kill $C \u0026\u0026 docker-compose rm -f $C \u0026\u0026 docker-compose build $C \u0026\u0026 docker-compose run $C bash`\n- `export C=\"4-splunk\"; docker-compose kill $C \u0026\u0026 docker-compose rm -f $C \u0026\u0026 docker-compose build $C \u0026\u0026 docker-compose run $C bash`\n\n\n### Resetting The App\n\nRun `./bin/stop-and-reset-agent` to kill and remove all services and remove all \nlogs files in the agent. The data in the database will remain untouched.\n\nRun `./bin/stop-and-reset-splunk` to kill and remove Splunk and its data.\nThe underlying logfiles will remain untouched.\n\n\n## Known Issues\n\nSince the SQLite file `tweets.db` is shared between several containers/services, there may occasionally\nbe some contention, resulting in an error from SQLite that it can't lock `tweets.db`.  When this happens,\nit triggers an exception in the Python code, causing it to exit, which causes the Docker container to\nexit as well.  It will be restarted in accordance with Docker's restart policy, and the service will resume \nrunning.  This means that all Tweets will eventually make it into Splunk, but there may be a slight delay\nif a lot of tweets are being processed.\n\n\n## Splunk Apps Included\n\nThe following Splunk apps are included in this Docker image:\n\n- \u003ca href=\"https://splunkbase.splunk.com/app/3212/\"\u003eWordcloud Custom Visualization\u003c/a\u003e\n- \u003ca href=\"https://splunkbase.splunk.com/app/2878/\"\u003eSlack Notification Alert\u003c/a\u003e\n\nAll apps are covered under their own license.\n\nSplunk has its own license.  Please abide by it.\n\n\n\n## TODO\n\n- Write up some \"priming instructions\" for large tweet volumes\n\n\n## Contact\n\nIf there are any issues, feel\nfree to file an issue against this project, \u003ca href=\"http://twitter.com/dmuth\"\u003ehit me up on Twitter\u003c/a\u003e\n\u003ca href=\"http://facebook.com/dmuth\"\u003eor Facebook\u003c/a\u003e, or drop me a line: **dmuth AT dmuth DOT org**.\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmuth%2Ftwitter-sentiment-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdmuth%2Ftwitter-sentiment-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmuth%2Ftwitter-sentiment-analysis/lists"}