{"id":23454850,"url":"https://github.com/bunyaminergen/callytics","last_synced_at":"2025-04-03T03:13:07.307Z","repository":{"id":269319666,"uuid":"907062017","full_name":"bunyaminergen/Callytics","owner":"bunyaminergen","description":"Callytics is an advanced call analytics solution that leverages speech recognition and large language models (LLMs) technologies to analyze phone conversations from customer service and call centers.","archived":false,"fork":false,"pushed_at":"2025-03-10T13:25:11.000Z","size":25043,"stargazers_count":65,"open_issues_count":2,"forks_count":10,"subscribers_count":5,"default_branch":"develop","last_synced_at":"2025-04-03T03:12:59.895Z","etag":null,"topics":["denoising","diarization","forced-alignment","llama3","llm","openai","opensource","sentiment-analysis","speech-emotion-recognition","speech-processing","speech-recognition","speech-to-text","summary","topic-modeling","transcription","voice-activity-detection","voice-recognition"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bunyaminergen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-22T17:47:34.000Z","updated_at":"2025-03-16T20:50:43.000Z","dependencies_parsed_at":"2024-12-22T18:22:02.884Z","dependency_job_id":"463d90c0-c62a-49f5-90d8-4a9efd7e93f3","html_url":"https://github.com/bunyaminergen/Callytics","commit_stats":null,"previous_names":["bunyaminergen/callytics"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bunyaminergen%2FCallytics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bunyaminergen%2FCallytics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bunyaminergen%2FCallytics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bunyaminergen%2FCallytics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bunyaminergen","download_url":"https://codeload.github.com/bunyaminergen/Callytics/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246927839,"owners_count":20856198,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["denoising","diarization","forced-alignment","llama3","llm","openai","opensource","sentiment-analysis","speech-emotion-recognition","speech-processing","speech-recognition","speech-to-text","summary","topic-modeling","transcription","voice-activity-detection","voice-recognition"],"created_at":"2024-12-24T03:14:49.545Z","updated_at":"2025-04-03T03:13:07.284Z","avatar_url":"https://github.com/bunyaminergen.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003cimg src=\".docs/img/CallyticsIcon.png\" alt=\"CallyticsLogo\" width=\"200\"\u003e\n\n![License](https://img.shields.io/github/license/bunyaminergen/Callytics)\n![GitHub release (latest by date)](https://img.shields.io/github/v/release/bunyaminergen/Callytics)\n![GitHub Discussions](https://img.shields.io/github/discussions/bunyaminergen/Callytics)\n![GitHub Issues](https://img.shields.io/github/issues/bunyaminergen/Callytics)\n\n[![LinkedIn](https://img.shields.io/badge/LinkedIn-Profile-blue?logo=linkedin)](https://linkedin.com/in/bunyaminergen)\n\n# Callytics\n\n`Callytics` is an advanced call analytics solution that leverages speech recognition and large language models (LLMs)\ntechnologies to analyze phone conversations from customer service and call centers. By processing both the\naudio and text of each call, it provides insights such as sentiment analysis, topic detection, conflict detection,\nprofanity word detection and summary. These cutting-edge techniques help businesses optimize customer interactions,\nidentify areas for improvement, and enhance overall service quality.\n\nWhen an audio file is placed in the `.data/input` directory, the entire pipeline automatically starts running, and the\nresulting data is inserted into the database.\n\n**Note**: _This is only a `v1.1.0` version; many new features will be added, models\nwill be fine-tuned or trained from scratch, and various optimization efforts will be applied. For more information,\nyou can check out the [Upcoming](#upcoming) section._\n\n**Note**: _If you would like to contribute to this repository,\nplease read the [CONTRIBUTING](.docs/documentation/CONTRIBUTING.md) first._\n\n\u003c/div\u003e\n\n---\n\n### Table of Contents\n\n- [Prerequisites](#prerequisites)\n- [Architecture](#architecture)\n- [Math And Algorithm](#math-and-algorithm)\n- [Features](#features)\n- [Demo](#demo)\n- [Installation](#installation)\n- [File Structure](#file-structure)\n- [Database Structure](#database-structure)\n- [Datasets](#datasets)\n- [Version Control System](#version-control-system)\n- [Upcoming](#upcoming)\n- [Documentations](#documentations)\n- [License](#licence)\n- [Links](#links)\n- [Team](#team)\n- [Contact](#contact)\n- [Citation](#citation)\n\n---\n\n### Prerequisites\n\n##### General\n\n- `Python 3.11` _(or above)_\n\n##### Llama\n\n- `GPU (min 24GB)` _(or above)_\n- `Hugging Face Credentials (Account, Token)`\n- `Llama-3.2-11B-Vision-Instruct` _(or above)_\n\n##### OpenAI\n\n- `GPU (min 12GB)` _(for other process such as `faster whisper` \u0026 `NeMo`)_\n- At least one of the following is required:\n    - `OpenAI Credentials (Account, API Key)`\n    - `Azure OpenAI Credentials (Account, API Key, API Base URL)`\n\n---\n\n### Architecture\n\n![Architecture](.docs/img/Callytics.gif)\n\n---\n\n### Math and Algorithm\n\nThis section describes the mathematical models and algorithms used in the project.\n\n_**Note**: The mathematical concepts and algorithms specific to this repository, rather than the models used, will be\nprovided in this section. Please refer to the `RESOURCES` under the [Documentations](#documentations) section for the\nrepositories and models utilized or referenced._\n\n##### Silence Duration Calculation\n\nThe silence durations are derived from the time intervals between speech segments:\n\n$$S = \\{s_1, s_2, \\ldots, s_n\\}$$\n\nrepresent _the set of silence durations (in seconds)_ between consecutive speech segments.\n\n- **A user-defined factor**:\n\n$$\\text{factor} \\in \\mathbb{R}^{+}$$\n\nTo determine a threshold that distinguishes _significant_ silence from trivial gaps, two statistical methods can be\napplied:\n\n**1. Standard Deviation-Based Threshold**\n\n- _Mean_:\n\n$$\\mu = \\frac{1}{n}\\sum_{i=1}^{n}s_i$$\n\n- _Standard Deviation_:\n\n$$\n\\sigma = \\sqrt{\\frac{1}{n}\\sum_{i=1}^{n}(s_i - \\mu)^2}\n$$\n\n- _Threshold_:\n\n$$\nT_{\\text{std}} = \\sigma \\cdot \\text{factor}\n$$\n\n**2. Median + Interquartile Range (IQR) Threshold**\n\n- _Median_:\n\n_Let:_\n\n$$ S = \\{s_{(1)} \\leq s_{(2)} \\leq \\cdots \\leq s_{(n)}\\} $$\n\nbe an ordered set.\n\n_Then:_\n\n$$\nM = \\text{median}(S) =\n\\begin{cases}\ns_{\\frac{n+1}{2}}, \u0026 \\text{if } n \\text{ is odd}, \\\\\\\\[6pt]\n\\frac{s_{\\frac{n}{2}} + s_{\\frac{n}{2}+1}}{2}, \u0026 \\text{if } n \\text{ is even}.\n\\end{cases}\n$$\n\n- _Quartiles:_\n\n$$\nQ_1 = s_{(\\lfloor 0.25n \\rfloor)}, \\quad Q_3 = s_{(\\lfloor 0.75n \\rfloor)}\n$$\n\n- _IQR_:\n\n$$\n\\text{IQR} = Q_3 - Q_1\n$$\n\n- **Threshold:**\n\n$$\nT_{\\text{median\\\\_iqr}} = M + (\\text{IQR} \\times \\text{factor})\n$$\n\n**Total Silence Above Threshold**\n\nOnce the threshold\n\n$$T$$\n\neither\n\n$$T_{\\text{std}}$$\n\nor\n\n$$T_{\\text{median\\\\_iqr}}$$\n\nis defined, we sum only those silence durations that meet or exceed this threshold:\n\n$$\n\\text{TotalSilence} = \\sum_{i=1}^{n} s_i \\cdot \\mathbf{1}(s_i \\geq T)\n$$\n\nwhere $$\\mathbf{1}(s_i \\geq T)$$ is an indicator function defined as:\n\n$$\n\\mathbf{1}(s_i \\geq T) =\n\\begin{cases}\n1 \u0026 \\text{if } s_i \\geq T \\\\\n0 \u0026 \\text{otherwise}\n\\end{cases}\n$$\n\n**Summary:**\n\n- **Identify the silence durations:**\n\n$$\nS = \\{s_1, s_2, \\ldots, s_n\\}\n$$\n\n- **Determine the threshold using either:**\n\n_Standard deviation-based:_\n\n$$\nT = \\sigma \\cdot \\text{factor}\n$$\n\n_Median+IQR-based:_\n\n$$\nT = M + (\\text{IQR} \\cdot \\text{factor})\n$$\n\n- **Compute the total silence above this threshold:**\n\n$$\n\\text{TotalSilence} = \\sum_{i=1}^{n} s_i \\cdot \\mathbf{1}(s_i \\geq T)\n$$\n\n---\n\n### Features\n\n- [x] Speech Enhancement\n- [x] Sentiment Analysis\n- [x] Profanity Word Detection\n- [x] Summary\n- [x] Conflict Detection\n- [x] Topic Detection\n\n---\n\n### Demo\n\n![callyticsDemo](.docs/img/callyticsDemo.gif)\n\n---\n\n### Installation\n\n##### Linux/Ubuntu\n\n```bash\nsudo apt update -y \u0026\u0026 sudo apt upgrade -y\n```\n\n```bash\nsudo apt install ffmpeg -y\n```\n\n```bash\nsudo apt install -y ffmpeg build-essential g++\n```\n\n```bash\ngit clone https://github.com/bunyaminergen/Callytics\n```\n\n```bash\ncd Callytics\n```\n\n```bash\nconda env create -f environment.yaml\n```\n\n```bash\nconda activate Callytics\n```\n\n##### Environment\n\n`.env` file sample:\n\n```Text\n# CREDENTIALS\n# OPENAI\nOPENAI_API_KEY=\n\n# HUGGINGFACE\nHUGGINGFACE_TOKEN=\n\n# AZURE OPENAI\nAZURE_OPENAI_API_KEY=\nAZURE_OPENAI_API_BASE=\nAZURE_OPENAI_API_VERSION=\n\n# DATABASE\nDB_NAME=\nDB_USER=\nDB_PASSWORD=\nDB_HOST=\nDB_PORT=\nDB_URL=\n```\n\n---\n\n##### Database\n\n_In this section, an `example database` and `tables` are provided. It is a `well-structured` and `simple design`. If you\ncreate the tables\nand columns in the same structure in your remote database, you will not encounter errors in the code. However, if you\nwant to change the database structure, you will also need to refactor the code._\n\n*Note*: __Refer to the [Database Structure](#database-structure) section for the database schema and tables.__\n\n```bash\nsqlite3 .db/Callytics.sqlite \u003c src/db/sql/Schema.sql\n```\n\n##### Grafana\n\n_In this section, it is explained how to install `Grafana` on your `local` environment. Since Grafana is a third-party\nopen-source monitoring application, you must handle its installation yourself and connect your database. Of course, you\ncan also use it with `Granafa Cloud` instead of `local` environment._\n\n```bash\nsudo apt update -y \u0026\u0026 sudo apt upgrade -y\n```\n\n```bash\nsudo apt install -y apt-transport-https software-properties-common wget\n```\n\n```bash\nwget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -\n```\n\n```bash\necho \"deb https://packages.grafana.com/oss/deb stable main\" | sudo tee /etc/apt/sources.list.d/grafana.list\n```\n\n```bash\nsudo apt install -y grafana\n```\n\n```bash\nsudo systemctl start grafana-server\nsudo systemctl enable grafana-server\nsudo systemctl daemon-reload\n```\n\n```bash\nhttp://localhost:3000\n```\n\n**SQLite Plugin**\n\n```bash\nsudo grafana-cli plugins install frser-sqlite-datasource\n```\n\n```bash\nsudo systemctl restart grafana-server\n```\n\n```bash\nsudo systemctl daemon-reload\n```\n\n### File Structure\n\n```Text\n.\n├── automation\n│         └── service\n│             └── callytics.service\n├── config\n│         ├── config.yaml\n│         ├── nemo\n│         │         └── diar_infer_telephonic.yaml\n│         └── prompt.yaml\n├── .data\n│         ├── example\n│         │         └── LogisticsCallCenterConversation.mp3\n│         └── input\n├── .db\n│         └── Callytics.sqlite\n├── .docs\n│         ├── documentation\n│         │         ├── CONTRIBUTING.md\n│         │         └── RESOURCES.md\n│         └── img\n│             ├── Callytics.drawio\n│             ├── Callytics.gif\n│             ├── CallyticsIcon.png\n│             ├── Callytics.png\n│             ├── Callytics.svg\n│            └── database.png\n├── .env\n├── environment.yaml\n├── .gitattributes\n├── .github\n│         └── CODEOWNERS\n├── .gitignore\n├── LICENSE\n├── main.py\n├── README.md\n├── requirements.txt\n└── src\n    ├── audio\n    │         ├── alignment.py\n    │         ├── analysis.py\n    │         ├── effect.py\n    │         ├── error.py\n    │         ├── io.py\n    │         ├── metrics.py\n    │         ├── preprocessing.py\n    │         ├── processing.py\n    │         └── utils.py\n    ├── db\n    │         ├── manager.py\n    │         └── sql\n    │             ├── AudioPropertiesInsert.sql\n    │             ├── Schema.sql\n    │             ├── TopicFetch.sql\n    │             ├── TopicInsert.sql\n    │             └── UtteranceInsert.sql\n    ├── text\n    │         ├── llm.py\n    │         ├── model.py\n    │         ├── prompt.py\n    │         └── utils.py\n    └── utils\n        └── utils.py\n\n19 directories, 43 files\n```\n\n---\n\n### Database Structure\n\n![Database Diagram](.docs/img/database.png)\n\n\n---\n\n### Datasets\n\n- [Callytics Speaker Verification Dataset *(CSVD)*](.data/groundtruth/speakerverification/DatasetCard.md)\n\n---\n\n### Version Control System\n\n##### Releases\n\n- [v1.0.0](https://github.com/bunyaminergen/Callytics/archive/refs/tags/v1.0.0.zip) _.zip_\n- [v1.0.0](https://github.com/bunyaminergen/Callytics/archive/refs/tags/v1.0.0.tar.gz) _.tar.gz_\n\n\n- [v1.1.0](https://github.com/bunyaminergen/Callytics/archive/refs/tags/v1.1.0.zip) _.zip_\n- [v1.1.0](https://github.com/bunyaminergen/Callytics/archive/refs/tags/v1.1.0.tar.gz) _.tar.gz_\n\n##### Branches\n\n- [main](https://github.com/bunyaminergen/Callytics/tree/main)\n- [develop](https://github.com/bunyaminergen/Callytics/tree/develop)\n\n---\n\n### Upcoming\n\n- [ ] **Speech Emotion Recognition:** Develop a model to automatically detect emotions from speech data.\n- [ ] **New Forced Alignment Model:** Train a forced alignment model from scratch.\n- [ ] **New Vocal Separation Model:** Train a vocal separation model from scratch.\n- [ ] **Unit Tests:** Add a comprehensive unit testing script to validate functionality.\n- [ ] **Logging Logic:** Implement a more comprehensive and structured logging mechanism.\n- [ ] **Warnings:** Add meaningful and detailed warning messages for better user guidance.\n- [ ] **Real-Time Analysis:** Enable real-time analysis capabilities within the system.\n- [ ] **Dockerization:** Containerize the repository to ensure seamless deployment and environment consistency.\n- [ ] **New Transcription Models:** Integrate and test new transcription models\n  suchas [AIOLA’s Multi-Head Speech Recognition Model](https://venturebeat.com/ai/aiola-drops-ultra-fast-multi-head-speech-recognition-model-beats-openai-whisper/).\n- [ ] **Noise Reduction Model:** Identify, test, and integrate a deep learning-based noise reduction model. Consider\n  existing models like **Facebook Research Denoiser**, **Noise2Noise**, **Audio Denoiser CNN**. Write test scripts for\n  evaluation, and if necessary, train a new model for optimal performance.\n\n##### Considerations\n\n- [ ] Detect CSR's identity via Voice Recognition/Identification instead of Diarization and LLM.\n- [ ] Transform the code structure into a pipeline for better modularity and scalability.\n- [ ] Publish the repository as a Python package on **PyPI** for wider distribution.\n- [ ] Convert the repository into a Linux package to support Linux-based systems.\n- [ ] Implement a two-step processing workflow: perform **diarization** (speaker segmentation) first, then apply *\n  *transcription** for each identified speaker separately. This approach can improve transcription accuracy by\n  leveraging speaker separation.\n- [ ] Enable **parallel processing** for tasks such as diarization, transcription, and model inference to improve\n  overall system performance and reduce processing time.\n- [ ] Explore using **Docker Compose** for multi-container orchestration if required.\n- [ ] Upload the models and relevant resources to **Hugging Face** for easier access, sharing, and community\n  collaboration.\n- [ ] Consider writing a **Command Line Interface (CLI)** to simplify user interaction and improve usability.\n- [ ] Test the ability to use **different language models (LLMs)** for specific tasks. For instance, using **BERT** for\n  profanity detection. Evaluate their performance and suitability for different use cases as a feature.\n\n---\n\n### Documentations\n\n- [RESOURCES](.docs/documentation/RESOURCES.md)\n- [CONTRIBUTING](.docs/documentation/CONTRIBUTING.md)\n- [PRESENTATION](.docs/presentation/CallyticsPresentationEN.pdf)\n\n---\n\n### Licence\n\n- [LICENSE](LICENSE)\n\n---\n\n### Links\n\n- [Github](https://github.com/bunyaminergen/Callytics)\n- [Website](https://bunyaminergen.com)\n- [Linkedin](https://www.linkedin.com/in/bunyaminergen)\n\n---\n\n### Team\n\n- [Bunyamin Ergen](https://www.linkedin.com/in/bunyaminergen)\n\n---\n\n### Contact\n\n- [Mail](mailto:info@bunyaminergen.com)\n\n---\n\n### Citation\n\n```bibtex\n@software{       Callytics,\n  author       = {Bunyamin Ergen},\n  title        = {{Callytics}},\n  year         = {2024},\n  month        = {12},\n  url          = {https://github.com/bunyaminergen/Callytics},\n  version      = {v1.1.0},\n}\n```\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbunyaminergen%2Fcallytics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbunyaminergen%2Fcallytics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbunyaminergen%2Fcallytics/lists"}