{"id":21484943,"url":"https://github.com/deepbiolab/customer-complaint-classification","last_synced_at":"2026-04-13T21:05:42.354Z","repository":{"id":264304113,"uuid":"892627356","full_name":"deepbiolab/customer-complaint-classification","owner":"deepbiolab","description":"An GenAI-powered pipeline leveraging Whisper, DALL-E, and GPT to transform customer complaints into actionable insights with automated transcription, visualization, and classification.","archived":false,"fork":false,"pushed_at":"2024-11-25T14:34:43.000Z","size":54239,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-02T13:12:04.060Z","etag":null,"topics":["azure","dalle","gpt","whisper"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deepbiolab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-22T13:24:25.000Z","updated_at":"2025-01-29T20:48:22.000Z","dependencies_parsed_at":"2025-03-17T09:55:07.553Z","dependency_job_id":null,"html_url":"https://github.com/deepbiolab/customer-complaint-classification","commit_stats":null,"previous_names":["deepbiolab/customer-complaint-classification"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/deepbiolab/customer-complaint-classification","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepbiolab%2Fcustomer-complaint-classification","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepbiolab%2Fcustomer-complaint-classification/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepbiolab%2Fcustomer-complaint-classification/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepbiolab%2Fcustomer-complaint-classification/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deepbiolab","download_url":"https://codeload.github.com/deepbiolab/customer-complaint-classification/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepbiolab%2Fcustomer-complaint-classification/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31770781,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-13T20:17:16.280Z","status":"ssl_error","status_checked_at":"2026-04-13T20:17:08.216Z","response_time":93,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","dalle","gpt","whisper"],"created_at":"2024-11-23T13:13:11.171Z","updated_at":"2026-04-13T21:05:42.325Z","avatar_url":"https://github.com/deepbiolab.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Customer Complaint Classification\n\n![](imgs/proj.png)\n\n## Overview\n\nThis project implement a generative AI-based solution for classifying customer complaints. The project involves various steps such as transcribing customer audio complaints, generating images, describing and annotating those images, and finally classifying the complaints into appropriate categories.\n\n1. **Transcribing Customer Audio Complaint**:\n\n   - The first step is to convert the customer's audio complaint into text using a speech-to-text model. This involves using the `whisper.py` module.\n\n2. **Create Prompt from Transcription**:\n\n   - Once the audio is transcribed, a prompt is created from the transcription that will be used to generate a visual representation of the complaint.\n\n3. **Generate Image Representing the Issue**:\n\n   - Using the prompt created from the transcription, an image is generated to visually represent the customer complaint. This is managed by the `dalle.py` module.\n\n4. **Describe the Generated Image**:\n\n   - The generated image is then analyzed to provide a description of its contents, using the `vision.py` module. This helps identify the key elements related to the issue.\n\n5. **Annotate the Reported Issue in the Image**:\n\n   - The key reported issue in the image is highlighted through annotation, which includes identifying specific objects or areas related to the complaint.\n\n6. **Classify Complaint into Category/Subcategory Pair**:\n   - Use the generated image description and the catalog metadata to classify the complaint into a category and subcategory. This is handled by the `gpt.py` module.\n\n\n## File Structure\n\n### Overall Structure\n```\ncustomer-complaint-classification/\n│\n├── main.py              # Main orchestrator for the project workflow\n├── whisper.py           # Module for audio transcription\n├── dalle.py             # Module for image generation\n├── vision.py            # Module for image description and annotation\n├── gpt.py               # Module for complaint classification\n├── categories.json      # File containing category and subcategory metadata\n├── audio/               # Directory for input audio files\n├── dev/                 # demo code for how to interact with different models and how to use azure openai service\n├── imgs/                # Images used in Readme file\n└── output/              # Directory for storing intermediate results and log file.\n```\n\n\n### Detailed explanation for following files:\n\n1. **`whisper.py`**:\n\n   - This file contains a function to transcribe audio complaints into text using the Whisper model.\n\n2. **`dalle.py`**:\n\n   - Contains the function `generate_image()` to create an image representing the issue.\n\n3. **`vision.py`**:\n\n   - This file contains a function to describe the generated image and annotate it with the key elements identified.\n\n4. **`gpt.py`**:\n\n   - Contains a function `classify_with_gpt()` that takes in an image description and classifies the complaint into an appropriate category/subcategory.\n\n5. **`main.py`**:\n\n   - Orchestrates the entire workflow, calling each of the modules in sequence.\n\n\n## Prerequisites\n\n- Azure OpenAI Service:\n   - Deployments for `Whisper`, `DALL-E`, and `GPT` models are necessary.\n   - Obtain API keys and set up the endpoint URL in the Azure portal.\n- Azure Resource Group: \n   - Create a resource group in your Azure account for managing related resources (e.g., `OpenAI services`).\n\n- Set up environment variables for secure access to Azure services:\n   1. Create a `.env` file in the root directory of the project.\n   2. Add the following environment variables to the `.env` file (for method how to get below information, see [How to deploy models on azure](#how-to-deploy-models-on-azure):\n      ```txt\n      AZURE_OPENAI_API_KEY=your_api_key_here \n      AZURE_OPENAI_ENDPOINT=your_endpoint_url_here \n      WHISPER_DEPLOYMENT=your_whisper_deployment_name \n      WHISPER_VERSION=your_whisper_api_version \n      GPT_DEPLOYMENT=your_gpt_deployment_name \n      GPT_VERSION=your_gpt_api_version\n      DALLE_DEPLOYMENT=your_dalle_deployment_name \n      DALLE_VERSION=your_dalle_api_version\n      ```\n      \u003e Replace the placeholders with your actual values from the Azure portal. \n      \u003e If you use different regions in your models, make sure differentiate the version of the API for each model by name.\n   3. Ensure that the `.env` file is added to your `.gitignore` to prevent sensitive information from being committed to version control.\n\n## Data Formats\n\n- Input:\n   - Audio files: Supported formats include `MP3` and `WAV` for transcription.\n- Output (see details for each sample in `output` folder):\n   - Text files for transcription (`transcription.txt`).\n   - PNG images for visual representations (`generated_image.png`, `annotated_image.png`).\n   - Text files for image descriptions and classifications (`image_description.txt`, `classification.txt`).\n   - `output.log`: Log file for tracking the execution of the project.\n- Other files:\n   - `categories.json`: Contains the category and subcategory metadata for classification.\n   - `prediction.json`: Contains the final classification results in JSON format.\n   - `labels.json`: Contains the labels used for classification, corresponding with `audio` folder.\n\n## Usage\n\n### Environment Setup\n- Clone the repository to your local machine:\n   ```bash\n   git clone https://github.com/deepbiolab/customer-complaint-classification.git\n   ```\n- Navigate into the project folder:\n   ```bash\n   cd customer-complaint-classification\n   ```\n\n- Ensure `Python 3.12` is installed on your system\n   - If not, using `conda` create a virtual environment(recommend)\n      ```bash\n      conda create -n complaint_clf python=3.12\n      conda activate complaint_clf\n      ```\n- Install dependencies:\n\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n### Quickstart\n```bash\npython main.py\n```\n\nThe project is a sequential execution of below tasks:\n   1. Run `main.py` to initiate the process.\n   2. Transcription via `whisper.py` processes audio file in `audio` folder and returns text.\n   3. Prompt generation in `main.py` feeds into `dalle.py` for image creation.\n   4. Image analysis and annotation are performed in `vision.py`.\n   5. Classification is finalized in `gpt.py` based on text and image outputs.\n\n## Resources\n\n- [OpenAI API Documentation](https://beta.openai.com/docs/)\n\n\n## Detailed Report\n- [Detailed Report](./dev/Report.md)\n\n\n## How to deploy models on Azure\n- [Deployment instruction](./dev/Deploy.md)\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepbiolab%2Fcustomer-complaint-classification","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeepbiolab%2Fcustomer-complaint-classification","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepbiolab%2Fcustomer-complaint-classification/lists"}