https://github.com/pauserratgutierrez/sentiment-media
AI-powered application designed to analyze sentiments from Twitter posts
https://github.com/pauserratgutierrez/sentiment-media
Last synced: 3 months ago
JSON representation
AI-powered application designed to analyze sentiments from Twitter posts
- Host: GitHub
- URL: https://github.com/pauserratgutierrez/sentiment-media
- Owner: pauserratgutierrez
- Created: 2024-07-10T22:43:04.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2024-07-26T08:01:10.000Z (11 months ago)
- Last Synced: 2025-02-07T12:18:16.244Z (4 months ago)
- Language: JavaScript
- Homepage: https://ai.pausg.dev/
- Size: 128 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
This project is part of a Hackaton. See more details here: https://github.com/midudev/hackaton-vercel-2024/issues/51
# Sentiment Media
Sentiment Media is an AI-powered application designed to analyze sentiments from Twitter posts. It leverages a cost-effective and fast AI model to categorize and score emotions present in social media content. The project is structured with a clear separation between the frontend and backend, with communication facilitated through an internal API.
Test it now!
https://ai.pausg.dev/
## Project Structure
- **Frontend**: Contains HTML, CSS, and JavaScript files for the user interface. It uses native technologies without any frontend frameworks.
- Located in the `frontend` directory.
- Uses HTML templates, styled with CSS, and interactive elements managed by JavaScript.
- Serves static files and main pages from `frontend/html/pages`.
- **Backend**: Built with Node.js and Express.js, handling API requests and interacting with the database.
- Located in the `backend` directory.
- Handles server-side logic, database interactions, and API endpoints.
- Main entry point is `app.js`.
## Technologies Used
- **Node.js**: Server-side JavaScript runtime.
- **Express.js**: Web framework for handling routing and middleware.
- **MySQL**: Database for storing Twitter posts and sentiment analysis results.
- **Puppeteer**: Headless browser for scraping Twitter content.
- **Cheerio**: Fast manipulation of HTML.
- **OpenAI GPT-4o-mini**: AI model used for sentiment analysis.
- **Vercel AI SDK**: Facilitates interaction with the OpenAI model.
## Features
### Sentiment Analysis
- Utilizes the `GPT-4o-mini` model from OpenAI for analyzing sentiments.
- Categorizes emotions into joy, love, hope, pride, nostalgia, fear, sadness, disgust, anger, shame, guilt, and surprise.
- Scores each emotion on a scale from 0 to 10.
- Provides a short general summary.
- Detects the general emotion of the text, categorizing it into neutral (0), positive (1), or negative (2).
- Returns the analysis in a valid JSON format.
### Puppeteer (Twitter Integration)
- Fetches and processes content from Twitter posts.
- Utilizes a headless Puppeteer instance (instead of the Twitter API).
- Spawns a pool of Chromium instances to allow multiple runs in parallel using `puppeteer-cluster`.
### Database
- MySQL database used for storing posts and sentiment analysis results.
- Connection managed using `mysql2` with connection pooling for efficiency.
- Correct implementation to prevent MySQL Injection.
## Caching System
- A custom-defined MySQL event runs every hour to determine if a record must be flagged as stale (no one checked a specific post in a day or more).
- Cached data is stored indefinitely, ensuring data availability without frequent re-fetches. A flag indicates if data is up-to-date or stale.
- Two-layer cache to optimize performance and reduce AI costs:
- **Twitter Post Layer**: Caches fetched Twitter posts.
- **AI Analysis Layer**: Caches the results of AI sentiment analysis.
- **How it works**:
- When analyzing a Twitter post for the first time, the post content is retrieved from Twitter and saved to the database. Then, the AI runs the sentiment analysis.
- When querying the same Twitter post, if the content in the database is flagged as up-to-date, the app uses it. Otherwise, it retrieves the post info from Twitter and compares the fresh data with the stale data saved in the database. If the data is equal, it uses the saved sentiment analysis. Otherwise, a new sentiment analysis is executed with the new Twitter post content.
### API Endpoints
- `/api/twitter/post`: Fetches a specific Twitter post's content and sentiment analysis. Example `/api/twitter/post?url=https://x.com/midudev/status/1787465376130859511`
- `/api/twitter/posts`: Retrieves cached posts with pagination support. Example `http://localhost:3000/api/twitter/posts?page=2&limit=1`
## No Authentication Required
- The application does not require user authentication for accessing the API.
- Fair usage is expected!
## Installation
1. Clone the repository:
2. Install dependencies:
```
npm install
```
3. Set up environment variables in a `.env` file:
```
PRODUCTION=
PORT=
DB_HOST=
DB_USER=
DB_PASSWORD=
DB_DATABASE=
DB_PORT=
OPENAI_API_KEY=
```
4. Run database migrations:
```
mysql -u -p < db-migration.sql
```
5. Start the application:
```
npm start
```