{"id":19901389,"url":"https://github.com/lissy93/twitter-sentiment-visualisation","last_synced_at":"2025-09-10T14:40:26.685Z","repository":{"id":35360591,"uuid":"39623325","full_name":"Lissy93/twitter-sentiment-visualisation","owner":"Lissy93","description":"🌍 Sentiment analysis over real-time social media data, rendering live charts to visualise trends ","archived":false,"fork":false,"pushed_at":"2023-11-04T11:58:40.000Z","size":15238,"stargazers_count":203,"open_issues_count":8,"forks_count":72,"subscribers_count":16,"default_branch":"dev","last_synced_at":"2024-04-14T08:51:53.536Z","etag":null,"topics":["data-visualisation","sentiment-analysis","sentiment-visualisation","stream-tweets","trends","twitter"],"latest_commit_sha":null,"homepage":"","language":"CoffeeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Lissy93.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-07-24T09:25:24.000Z","updated_at":"2024-05-04T05:32:31.498Z","dependencies_parsed_at":"2024-05-04T05:42:40.626Z","dependency_job_id":null,"html_url":"https://github.com/Lissy93/twitter-sentiment-visualisation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lissy93%2Ftwitter-sentiment-visualisation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lissy93%2Ftwitter-sentiment-visualisation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lissy93%2Ftwitter-sentiment-visualisation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lissy93%2Ftwitter-sentiment-visualisation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Lissy93","download_url":"https://codeload.github.com/Lissy93/twitter-sentiment-visualisation/tar.gz/refs/heads/dev","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248008630,"owners_count":21032556,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-visualisation","sentiment-analysis","sentiment-visualisation","stream-tweets","trends","twitter"],"created_at":"2024-11-12T20:14:51.399Z","updated_at":"2025-04-09T09:08:45.201Z","avatar_url":"https://github.com/Lissy93.png","language":"CoffeeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\u003ch1 align=\"center\"\u003e🌍 Twitter Sentiment Visualisations\u003c/h1\u003e\n\u003cp align=\"center\"\u003e\n \u003ci\u003eVisualising sentiment trends from real-time social media data\u003c/i\u003e\u003cbr\u003e\n \u003cb\u003e\u003ca href=\"https://web.archive.org/web/20220810055104/https://sentiment-sweep.com/\"\u003esentiment-sweep.com\u003c/a\u003e\u003c/b\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"/docs/screenshots#readme\"\u003e\n    \u003cimg width=\"800\" src=\"https://i.ibb.co/W0f10Vv/sentiment-sweep-grid.png\" /\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003e [!IMPORTANT]\n\u003e As of Feb 2023, [X no longer provides free API](https://twitter.com/XDevelopers/status/1621026986784337922). This makes it impractical to continue running this app afforably or at scale.\u003cbr\u003e\u003cbr\u003e\n\u003e Further to this, now that general purpose AI models are faster, more availible and less expensive than ever before, the static analysis approach (used here) no longer makes sense, given that AI sentiment analysis will yeild much more accurate and insightful results.\u003cbr\u003e\u003cbr\u003e\n\u003e For these reasons, the public instance of Sentiment Sweep is no longer availible, and this project will cease to be maintained going forwards.\u003cbr\u003e\u003cbr\u003e\n\u003e We had a good run! The Sentiment Sweep website ran for nearly a decade, saw upwards of 1 million sessions, and won 2 awards. It was a great learning experience, and a fun little project. Thank you everybody who used, supported and visited the website! 💖\n\n\n### Contents\n\n- [About](#about)\n- [Demo](#demo)\n- [Build Instructions](#building)\n   - [Developing](#developing)\n   - [Deploying](#deploying)\n   - [Testing](#testing)\n- [Modules](#modules)\n- [Project Info](#project-info)\n- [Documentation](#documentation)\n- [License](#license)\n\n\n## About\n\n#### What\nA project to make large quantities of social media data more understandable.\n\n#### How\nThe app that streams live social media data, and runs it through a custom sentiment analysis algorithm, to determine trends which are then visualised with a series of dynamic real-time charts.\n\n#### Why\nThe aim of the app is to allow trends to be found between sentiment and other factors such as geographical location, time of day, demographics, similar topics, etc.\n\nIt has a range of uses, like analysing the effectiveness of a marketing campaign, comparing competing products, viewing local trends, gauging public opinion by location, determining best time of day to advertise to certain audiences, etc.\n\n#### Where\nA live demo is available at: [http://sentiment-sweep.com](https://web.archive.org/web/20220810055104/https://sentiment-sweep.com/) (archived)\n\n#### When\nThis project was initially developed in 2015.\nSome of the technologies used are a little out-dated now, although the app still works great.\nA few of the external services that were used to provide additional context (like HP Idol on Demand, and IBM Watson, and certain GCP features) have been discontinued, meaning certain features may now be unavailible on the live instance.\n\n---\n\n## Building\n\n\n#### Developing\n\n\u003e See the [Dev Setup](/docs/installation-instructions.md) docs for local dev setup.\n\n1. **Prerequisites** - You will need [Node.js], [MongoDB] and [git]  installed on\nyour system.\n2. **Get the files** - `git clone https://github.com/Lissy93/twitter-sentiment-visualisation.git tsv` then `cd tsv`\n3. **Install dependencies** -  `npm i` / `yarn` will download requirements into node_modules, then automatically kick off a `bower install` for frontend libraries\n4. **Set Config** - `yarn run config`  will generate the `config\\src\\keys.coffee` file, which you will then need to populate with your API keys and save. 5. **Apply Settings** - Check that your happy with the general app config in `config/src/app-config.coffee`\n6. **Build Project** - `yarn build` will compile the project from the source, outputting files into dist ready to be published\n7. **Start MongoDB** - `mongod` will start a [MongoDB] instance (run in separate terminal instance, see instructions: [Starting a MongoDB instance])\n8. **Run the project** - `yarn dev` will build, start the dev server, with live-reload and auto-testing\n9. **Open Browser** - Navigate to the specified port, to view running app, e.g. [http://localhost:8080]\n\n#### Deploying\n\n\u003e See the [Prod Deployment(/docs/build-environment.md) docs for more info.\n\nFollow the instructions above, then\n1. **Execute Tests** - `yarn test` Ensure all tests pass and everything is working as expected \n2. **Build for Prod** - `yarn build` Compile all source files to the dist directory\n3. **Start Server** - `yarn start` Spin up HTTP server to start API and serve up compiled files\n\n#### Testing\n\n\u003e See the [Test Strategy] Docs for more info.\u003cbr\u003e\n\u003e TSV is fully unit tested, and follows a BHD pattern. Unit, integration, coverage and depencency tests can be run using `yarn test`.\n\n\u003cdetails\u003e\n \u003csummary\u003ePass/ Fail Criteria\u003c/summary\u003e\n \n| Test Type          | Pass Condition                                                                          |\n|--------------------|-----------------------------------------------------------------------------------------|\n| Functional Testing | All acceptance criteria must be met, checked and documented                             |\n| Unit Tests         | 100% of unit tests must pass. It will be immediately clear when a unit test is failing  |\n| Integration Tests  | 100% pass rate after every commit                                                       |\n| Coverage Tests     | 80% or greater                                                                          |\n| Code Reviews       | B grade/ Level 4 or higher. Ideally A grade/ Level 5 if possible.                       |\n| Dependency Checks  | Mostly up-to-date dependencies except in justified circumstances.                       |\n\n \n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n \u003csummary\u003eTesting Tool\u003c/summary\u003e\n \n- Framework - [Mocha](https://github.com/mochajs/mocha)\n  - Used in order to store, write and run the tests in a structured way\n- Assertion Library - [Chai](https://github.com/chaijs/chai)\n  - Provides a structure and syntax in order to actually write the test cases\n- Coverage Testing - [Istanbul](https://github.com/gotwarlost/istanbul)\n  - Measures the proportion of your source code that is covered by your unit tests\n- Stubs, Spies and Mocking - [Sinon.js](https://github.com/sinonjs/sinon)\n  - Mocking removes the need to call production APIs while running frontend unit tests\n- Continuous Integration Testing - [Travis CI](https://github.com/travis-ci/travis-ci)\n  - Ensures that all the standalone modules function correctly when put together\n- Dependency Checking - [David](https://github.com/alanshaw/david)\n  - Checks that each dependency is present, correct, secure and functional\n- Automated Code Review's - [Code Climate](https://github.com/codeclimate/codeclimate)\n  - Scans for best practices, and fails in any part of the code could be improved upon\n- Headless Browser Testing - [PhantomJS](https://github.com/ariya/phantomjs)\n  - Runs frontend tests without the need for a GUI browser\n- Testing HTTP services - [SuperTest](https://github.com/visionmedia/supertest)\n  - Tests API endpoints and ensures routing is working correctly\n\n\u003c/details\u003e \n\n#### Automated Workflows\n \n\u003e TSV uses the Gulp streaming build tool to automate the prod and dev workflows. For more info, see the [Build Environment] docs.\n\nThe following tasks are useful for getting started:\n- `gulp generate-config` - Generates correctly structured default configuration files for settings and API keys\n- `gulp build` - Builds the project fully, including optimization, compilation, minification and validation\n- `gulp nodemon` - Runs the application on the default port (probably 8080), with live refresh\n- `gulp test` - Executes all unit and coverage tests, and generates a report containing the results\n- `gulp` - Default dev task - check the project is configured correctly, build ALL the files, run the server, watch for changes, recompile relevant files and reload browsers on change, and keep all browsers in sync, when a test condition changes it will also re-run tests - a lot going on!\n\n---\n\n## Modules\n\nThe project was developed in a modular approach, made up of several distinct components.\nEach is published as a fully tested, documented and MIT-licensed NPM module for easy re-use.\n\n- [sentiment-analysis] - Useses AFINN-111 approach to calculate overall sentiment of a given sentence\n- [fetch-tweets] - Fetches tweets from Twitter based on topic, location, timeframe or combination\n- [stream-tweets] - Streams live Twitter data in real-time, based on location, given term, etc\n- [remove-words] - Removes all non-key words from a given string\n- [place-lookup] - Finds the latitude and longitude for any fuzzy place name using the Google Places API\n- [hp-haven-sentiment-analysis] - A Node.js client library for HP Haven OnDemand Sentiment Analysis module\n- [haven-entity-extraction] - Node.js client for HP Haven OnDemand Entity Extraction\n- [tweet-location] - Calculates the location from geo-tagged Tweets using the Twitter Geo API\n- [find-region-from-location] - Given a latitude and longitude calculates which region that point belongs in\n\n---\n\n## Project Info\n\n#### Project Planning\n\nA set of [User Stories with Acceptance Criteria] and [Complexity Estimates](docs/story-points.md) were drawn up outlining what features the finished solution should have. These were expaned upon further with wireframes in the [Methodology](https://github.com/Lissy93/twitter-sentiment-visualisation/blob/dev/docs/methodology-frontend.md) section.\n\n#### Technologies\n\nView full tech stack at: [stackshare.io/Lissy93/sentiment-sweep](http://stackshare.io/Lissy93/sentiment-sweep)\n\nThe backend is primarily written in Node.js, with web-sockets facilitating the real-time communication with the frontend, and a data cache stored in MongoDB. Pages are rendered isomorphically, with data visualizations written using D3.js. Social data is fetched from Twitter, compute happens locally, and a few external APIs were used to provide additional context in the form of AI. Views are written in Pug, styles in Less, scripts in CoffeeScript and everything is compiled via a Gulp script.\n\nThe project and app are still functional, however 5 years on, this would not be an ideal tech stack. There are now better technologies available that would enable greater performance, less code, easier project management and improved developer experience. If I was to re-write this project in 2022, a better tech stack would likely be Go for the backend, Svelte + Svelte Kit for the frontend and TypeScript for the code, with Pixi.js for the interactive content, styled-components for styling and Rollup for putting it all together.\n\n#### Status\n\n[![Build Status](https://travis-ci.org/Lissy93/twitter-sentiment-visualisation.svg?branch=dev)](https://travis-ci.org/Lissy93/twitter-sentiment-visualisation)\n[![View on Snyk](https://snyk.io/test/github/lissy93/twitter-sentiment-visualisation/badge.svg)](https://snyk.io/test/github/Lissy93/Twitter-Sentiment-Visualisation)\n[![Code Climate](https://codeclimate.com/github/Lissy93/twitter-sentiment-visualisation/badges/gpa.svg)](https://codeclimate.com/github/Lissy93/twitter-sentiment-visualisation)\n![Size](https://img.shields.io/bundlephobia/min/sentiment-analysis)\n[![Website](https://img.shields.io/website?down_color=red\u0026down_message=Down\u0026up_color=green\u0026up_message=Online\u0026url=https%3A%2F%2Fsentiment-sweep.com%2F)](https://sentiment-sweep.com/)\n\n#### Demo\n\nA live demo of the application has been deployed to: [http://sentiment-sweep.com](http://sentiment-sweep.com)\n\nView [Screenshots](/docs/screenshots#readme) of each screen in the docs.\n\n[![Screenshots](https://i.ibb.co/873YkwP/sentiemnt-sweep-screenshots.png)](/docs/screenshots#readme)\n\n#### Awards\n\n\u003ca href=\"https://notes.aliciasykes.com/p/0s5s3uOtKj\"\u003e\u003cimg align=\"left\" width=\"240\" src=\"https://i.ibb.co/RzcxQGK/profile1.jpg\" alt=\"Alicia Sykes - StartHack Winner\" /\u003e\u003c/a\u003e\n\u003ca href=\"https://oxon.bcs.org/2016/06/27/annual-student-prizes-2016/\"\u003e\u003cimg align=\"left\" width=\"315\" src=\"https://oxon.bcs.org/wp-content/uploads/2016/06/IMG_0681-768x576.jpg\" alt=\"Alicia Sykes - Oxford Winner\" /\u003e\u003c/a\u003e\n\nThe first stages of the project were developed at StartHack Switzerland 2014, where it won first-place. \n\nIt was then further expanded upon, and used as part of my undergraduate thesis, where it won the Oxford BCS best Dissertation Award.\n\n\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\n\nThe University Project recieved 96%, so feel free to use it as an example - here's the [Final Report](https://www.docdroid.net/x8srrAf/final-report-with-appendix-asykes-12011471-pdf) in PDF format (warning - it's 300 pages!). And the deck used for the technical presentation, us available at: [presentation.sentiment-sweep.com](https://web.archive.org/web/20231104114344/https://presentation.sentiment-sweep.com/#/)\n\n---\n\n## Documentation\n\n- **Development Documentation**\n  - [Installation Instructions](docs/installation-instructions.md)\n  - [Build Environment](docs/build-environment.md)\n  - [Testing](docs/test-strategy.md)\n- **Project Information**\n  - [Project Introduction](docs/project-introduction.md)\n  - [Addressing Potential Risks](docs/project-risks.md)\n  - [System Development Life Cycle](docs/sdlc.md)\n  - [Licence](docs/LICENSE.md)\n- **Project Planning**\n  - [User Stories](docs/user-stories.md)\n  - [Complexity Estimates](docs/story-points.md)\n  - [High Level UML](docs/high-level-data-flow.png)\n  - [Methodology - frontend](docs/methodology-frontend.md)\n  - [Test Strategy](docs/test-strategy.md)\n- **Research**\n  - [The current sentiment analysis scene](docs/research-1-sa-current-uses.md)\n  - [Comparison of various sentiment analysis algorithm approaches](docs/research-2-sa-comparison.md)\n  - [References](docs/references.md)\n\n---\n\n## License\n\n[twitter-sentiment-visualisation](https://github.com/Lissy93/twitter-sentiment-visualisation) was developed by [Alicia Sykes](https://aliciasykes.com), licensed under [MIT](https://git.io/Jew4i) © 2014 - 2022.\n\n\u003csup\u003eFor information, see [TLDR Legal \u003e MIT](https://tldrlegal.com/license/mit-license)\u003c/sup\u003e\n\n```\nThe MIT License (MIT)\nCopyright (c) Alicia Sykes \u003calicia@omg.com\u003e \n\nPermission is hereby granted, free of charge, to any person\nobtaining a copy of this software and associated documentation files\n(the \"Software\"), to deal in the Software without restriction,\nincluding without limitation the rights to use, copy, modify, merge,\npublish, distribute, sub-license, and/or sell copies of the Software,\nand to permit persons to whom the Software is furnished to do so,\nsubject to the following conditions:\n\nThe above copyright notice and this permission notice shall be\nincluded install copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND,\nEXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF\nMERCHANT ABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON INFRINGEMENT.\nIN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY\nCLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,\nTORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE\nSOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n```\n\n---\n\n\n\u003c!-- License + Copyright --\u003e\n\u003cp  align=\"center\"\u003e\n  \u003ci\u003e© \u003ca href=\"https://aliciasykes.com\"\u003eAlicia Sykes\u003c/a\u003e 2015 - 2020\u003c/i\u003e\u003cbr\u003e\n  \u003ci\u003eLicensed under \u003ca href=\"https://gist.github.com/Lissy93/143d2ee01ccc5c052a17\"\u003eMIT\u003c/a\u003e\u003c/i\u003e\u003cbr\u003e\n  \u003ca href=\"https://github.com/lissy93\"\u003e\u003cimg src=\"https://i.ibb.co/4KtpYxb/octocat-clean-mini.png\" /\u003e\u003c/a\u003e\u003cbr\u003e\n  \u003csup\u003eThanks for visiting :)\u003c/sup\u003e\n\u003c/p\u003e\n\n\u003c!-- Dinosaurs are Awesome --\u003e\n\u003c!-- \n                        . - ~ ~ ~ - .\n      ..     _      .-~               ~-.\n     //|     \\ `..~                      `.\n    || |      }  }              /       \\  \\\n(\\   \\\\ \\~^..'                 |         }  \\\n \\`.-~  o      /       }       |        /    \\\n (__          |       /        |       /      `.\n  `- - ~ ~ -._|      /_ - ~ ~ ^|      /- _      `.\n              |     /          |     /     ~-.     ~- _\n              |_____|          |_____|         ~ - . _ _~_-_\n--\u003e\n\n\n\n\n   [Trello Board]: \u003chttps://trello.com/b/jWBg1vd1/twitter-sentiment-visualisation\u003e\n   [Node.js]: \u003chttps://nodejs.org/en/\u003e\n   [MongoDB]: \u003chttps://www.mongodb.org/\u003e\n   [Yarn]: \u003chttps://yarnpkg.com\u003e\n   [git]: \u003chttps://git-scm.com/\u003e\n   [Starting a MongoDB instance]: \u003chttp://docs.mongodb.org/master/tutorial/getting-started-with-the-mongo-shell/\u003e\n   [http://localhost:8080]: \u003chttp://localhost:8080\u003e\n   [detailed installation instructions]: \u003cdocs/installation-instructions.md\u003e\n   [test strategy]: \u003cdocs/test-strategy.md\u003e\n   [build environment]: \u003cdocs/build-environment.md\u003e\n   [user stories with acceptance criteria]: \u003cdocs/user-stories.md\u003e\n   [fetch-tweets]: \u003chttps://www.npmjs.com/package/fetch-tweets\u003e\n   [stream-tweets]: \u003chttps://www.npmjs.com/package/stream-tweets\u003e\n   [place-lookup]: \u003chttps://github.com/Lissy93/place-lookup\u003e\n   [tweet-location]: \u003chttps://www.npmjs.com/package/tweet-location\u003e\n   [remove-words]: \u003chttps://www.npmjs.com/package/remove-words\u003e\n   [sentiment-analysis]: \u003chttps://www.npmjs.com/package/sentiment-analysis\u003e\n   [hp-haven-sentiment-analysis]: \u003chttps://github.com/Lissy93/haven-sentiment-analysis\u003e\n   [haven-entity-extraction]: \u003chttps://github.com/Lissy93/haven-entity-extraction\u003e\n   [find-region-from-location]: \u003chttps://github.com/Lissy93/find-region-from-location\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flissy93%2Ftwitter-sentiment-visualisation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flissy93%2Ftwitter-sentiment-visualisation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flissy93%2Ftwitter-sentiment-visualisation/lists"}