https://github.com/strcoder4007/voice-sentiment-analysis

Customer call analysis app with: Speech-to-Text + speaker diarization via ElevenLabs Structured conversation analysis via OpenAI React frontend for multi-file upload and rich results display. Takes into account not only the text but tone, rythym etc.
https://github.com/strcoder4007/voice-sentiment-analysis

call-analysis speaker-diarization speech-to-text

Last synced: 5 days ago
JSON representation

Host: GitHub
URL: https://github.com/strcoder4007/voice-sentiment-analysis
Owner: strcoder4007
Created: 2025-08-21T05:29:30.000Z (11 months ago)
Default Branch: main
Last Pushed: 2025-10-17T04:57:21.000Z (9 months ago)
Last Synced: 2025-10-26T03:29:00.991Z (9 months ago)
Topics: call-analysis, speaker-diarization, speech-to-text
Language: JavaScript
Homepage:
Size: 10.5 MB
Stars: 0
Watchers: 0
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

# Voice Sentiment Analysis

Customer call analysis app with:
- Speech-to-Text + speaker diarization via ElevenLabs
- Structured conversation analysis via OpenAI
- React frontend for multi-file upload and rich results display
- FastAPI backend orchestrating transcription, diarization, and analysis

This repository contains both the backend API (FastAPI) and the frontend web app (React).

- GitHub: https://github.com/strcoder4007/voice-sentiment-analysis

---

## Features

- Upload one or more audio files (wav, mp3, m4a, flac)
- Automatic transcription with timestamps and speaker diarization
- Grouped human-readable transcript by speaker turns with HH:MM:SS.mmm ranges
- Structured JSON analysis including:
- emotion_overall + confidence
- satisfaction + confidence
- summary, customer_intent, issues
- action_items with owner and due date
- agent_speaker_label + identification confidence
- agent_improvement_opportunities (category, evidence, impact, recommendation)
- post_call_recommendations
- follow_up_message_draft
- sentiment_analysis narrative
- Health check endpoint and robust error handling
- CORS enabled for local development
- Simple UI with results cards and JSON viewer

---

## Architecture and Flow

- Frontend (React)
- Multi-file audio selection and upload
- Displays result cards and raw JSON
- Default backend URL: http://localhost:8000/analyze/

- Backend (FastAPI)
- POST /analyze/: accepts multipart form-data (field key: files, repeated)
- Transcribes using ElevenLabs STT with diarization and word-level timestamps
- Groups words into speaker turns; renders readable transcript with timecodes
- Sends transcript summary to OpenAI for structured conversation analysis
- Returns combined metadata, transcript, and analysis JSON per file

- External Services
- ElevenLabs Speech-to-Text API
- OpenAI Responses API

---

## Repository Structure

```
voice-sentiment-analysis/
├─ backend/
│ ├─ main.py # FastAPI app and orchestration logic
│ └─ requirements.txt # Python dependencies
├─ frontend/
│ ├─ package.json # React app and scripts
│ ├─ public/ # CRA public assets
│ └─ src/
│ ├─ App.js # Upload UI and results grid
│ ├─ App.css # Styles
│ └─ components/
│ ├─ ResultCard.jsx # Result card UI
│ └─ JsonViewer.jsx # JSON viewer component
├─ TODO.md # Project roadmap
├─ voxtral.py # (not used by app runtime)
└─ README.md # This file
```

---

## Prerequisites

- macOS, Linux, or Windows
- Python 3.10+ recommended
- Node.js 18+ and npm
- API keys:
- OpenAI API key (with access to the selected model)
- ElevenLabs API key

Costs: Using OpenAI and ElevenLabs APIs incurs usage charges. Ensure your accounts are configured appropriately.

---

## Quick Start

1) Configure environment variables (create a .env file in the project root):
```
OPENAI_API_KEY=sk-...
ELEVENLABS_API_KEY=eleven-...
```

2) Start the backend (FastAPI):
```
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r backend/requirements.txt

# Option A: run via uvicorn (recommended during dev)
uvicorn backend.main:app --reload --host 0.0.0.0 --port 8000

# Option B: run the module directly
python backend/main.py
```

3) Start the frontend (React):
```
cd frontend
npm install
npm start
```

- Frontend dev server: http://localhost:3000
- Backend API: http://localhost:8000
- The default frontend expects the backend at http://localhost:8000/analyze/

---

## Configuration

- Environment variables (in project root .env; loaded by python-dotenv):
- `OPENAI_API_KEY` — required
- `ELEVENLABS_API_KEY` — required

- Backend ports
- Default is 8000; change via uvicorn flag if desired
- CORS is enabled for all origins by default (FastAPI middleware)

- OpenAI model
- Backend code uses `model="gpt-5"` in `backend/main.py`
- Ensure your account has access to this model; otherwise update the model string to an available one (e.g., `gpt-4o` or `gpt-4o-mini`) in `backend/main.py`

- Frontend API URL
- Currently hardcoded in `frontend/src/App.js`:
```
fetch("http://localhost:8000/analyze/", { ... })
```
- If deploying or changing ports, update this URL accordingly

---

## API

Base URL: `http://localhost:8000`

- GET `/`
- Returns: `{ "message": "Voice Sentiment Analysis API is running" }`

- GET `/health`
- Returns:
```
{
"status": "healthy",
"openai_configured": true|false,
"elevenlabs_configured": true|false
}
```

- POST `/analyze/`
- Content-Type: multipart/form-data
- Field name for files: `files` (repeat for multiple)
- Returns:
```
{
"results": [
{
"filename": "call1.mp3",
"date": "YYYY-MM-DD",
"time": "HH:MM:SS",
"audio_length": "HH:MM:SS.mmm",
"file_size": 12345,
"transcription": "...",
"analysis": {
"emotion_overall": "very_negative | negative | neutral | positive | very_positive",
"emotion_confidence": 0.0,
"satisfaction": "very_unsatisfied | unsatisfied | neutral | satisfied | very_satisfied",
"satisfaction_confidence": 0.0,
"summary": "2-4 sentences...",
"customer_intent": "one sentence...",
"issues": ["..."],
"action_items": [
{ "owner": "agent|customer|other", "item": "...", "due": "YYYY-MM-DD|null" }
],
"agent_speaker_label": "Speaker 1 | Speaker 2 | Speaker 3 | unknown",
"agent_identification_confidence": 0.0,
"agent_improvement_opportunities": [
{
"category": "empathy|discovery|clarity|solution_quality|ownership|pace|listening|policy_adherence|product_knowledge",
"observation": "...",
"evidence": "\"short quote\"",
"recommended_change": "...",
"impact": "low|medium|high"
}
],
"post_call_recommendations": ["..."],
"follow_up_message_draft": "short paragraph...",
"sentiment_analysis": "2-4 sentences of critical-thinking analysis..."
}
}
],
"total_processed": 1
}
```

Example curl (single file):
```
curl -X POST http://localhost:8000/analyze/ \
-F "files=@/path/to/audio.mp3;type=audio/mpeg"
```

Example curl (multiple files):
```
curl -X POST http://localhost:8000/analyze/ \
-F "files=@/path/to/call1.wav;type=audio/wav" \
-F "files=@/path/to/call2.m4a;type=audio/m4a"
```

---

## Frontend

- Tech: React (CRA), Result cards and JSON viewer
- File input accepts multiple audio files
- Action button posts to `/analyze/`
- Error states shown inline

To change the API URL:
- Edit `frontend/src/App.js` and update the fetch URL to your backend endpoint

Run:
```
cd frontend
npm install
npm start
```

Build for production:
```
npm run build
```

---

## Backend

- Tech: FastAPI, httpx, python-dotenv
- Entrypoints:
- `backend/main.py` (direct run)
- `uvicorn backend.main:app --reload --port 8000`
- Key pipeline (per file):
1) Validate and read bytes
2) ElevenLabs STT with diarization and `timestamps_granularity="word"`
3) Group words into speaker turns, render readable transcript with time ranges
4) Send enriched prompt to OpenAI Responses API for structured analysis JSON
5) Safe-parse JSON; add metadata and return

Python dependencies: see `backend/requirements.txt`

---

## ElevenLabs and OpenAI Notes

- ElevenLabs Speech-to-Text API:
- Endpoint: `POST https://api.elevenlabs.io/v1/speech-to-text`
- Requires `xi-api-key` header
- This app requests diarization and word-level timestamps
- Supported formats include common audio types (mp3, wav, m4a, flac)

- OpenAI Responses API:
- Model string configurable in `backend/main.py` (`gpt-5` by default)
- If your account lacks access to the default model, change it to one you can use (e.g., `gpt-4o`)

---

## Troubleshooting

- 500: OpenAI API key not configured
- Ensure `.env` contains `OPENAI_API_KEY` and the backend process can read it
- 500: ElevenLabs API key not configured
- Ensure `.env` contains `ELEVENLABS_API_KEY`
- 502 from ElevenLabs STT
- Check file format, account plan/limits, and API key validity
- Empty or invalid OpenAI response
- Ensure model access; if needed, switch to a supported model in `backend/main.py`
- CORS or network errors in the browser
- Confirm backend running at http://localhost:8000
- Verify fetch URL in `frontend/src/App.js`
- Large files/slow responses
- Backend uses httpx timeout of 120s; adjust if needed

---

## Security and Privacy

- API keys are loaded from environment variables; do not commit them to source control
- Uploaded audio is processed in-memory for analysis then returned in results
- Be mindful of sensitive content in audio/transcripts and downstream storage

---

## Roadmap

See [TODO.md](./TODO.md). Planned items include:
- Improved multiple upload UX
- Robust error states and retries
- Accuracy validation and diarization quality checks
- Deployment docs and environment management
- CI, unit/integration tests with sample audio

---

## Development Tips

- Run backend with `--reload` for hot reload during API edits:
```
uvicorn backend.main:app --reload --port 8000
```
- Adjust the analysis schema/prompt in `backend/main.py` under `schema_template`, `system_msg`, and `user_prompt`
- To change diarization behavior or language hints, modify `transcribe_with_elevenlabs` parameters in `backend/main.py`

---

## License

No license specified.

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/strcoder4007/voice-sentiment-analysis

Awesome Lists containing this project

README