{"id":15164814,"url":"https://github.com/zzteam-rccup-2024/aurora-echo","last_synced_at":"2025-09-30T20:31:39.412Z","repository":{"id":248863720,"uuid":"829952949","full_name":"zzteam-rccup-2024/aurora-echo","owner":"zzteam-rccup-2024","description":"We propose a new feedback system, named Aurora Echo} which provides audio-visual feedback for the resource provider. Through numerous artificial intelligence, such as LLM, we can make the system more friendly to not only user but also the resource provider.","archived":true,"fork":false,"pushed_at":"2024-08-24T02:49:26.000Z","size":1849,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-09-30T03:21:41.549Z","etag":null,"topics":["ai","cv","llama","llm","mediapipe","nlp","opencv","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zzteam-rccup-2024.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-17T10:14:10.000Z","updated_at":"2024-08-24T13:14:27.000Z","dependencies_parsed_at":"2024-07-17T15:18:46.709Z","dependency_job_id":"32947bcb-2210-425a-9854-7519ed4f9402","html_url":"https://github.com/zzteam-rccup-2024/aurora-echo","commit_stats":null,"previous_names":["zzteam-rccup/aurora-echo","zzteam-rccup-2024/aurora-echo"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zzteam-rccup-2024%2Faurora-echo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zzteam-rccup-2024%2Faurora-echo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zzteam-rccup-2024%2Faurora-echo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zzteam-rccup-2024%2Faurora-echo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zzteam-rccup-2024","download_url":"https://codeload.github.com/zzteam-rccup-2024/aurora-echo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234774940,"owners_count":18884527,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","cv","llama","llm","mediapipe","nlp","opencv","pytorch"],"created_at":"2024-09-27T04:00:35.946Z","updated_at":"2025-09-30T20:31:34.077Z","avatar_url":"https://github.com/zzteam-rccup-2024.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Aurora Echo\n\nAurora Echo is a visual-audio feedback system with maximized security.\n\n## Get Started\n\n### Requirements\n\n\u003e Alternatively, you can run the setup script directly:\n\u003e \n\u003e - Windows:\n\u003e  ```pwsh\n\u003e   .\\setup.ps1\n\u003e  ```\n\u003e - macOS / Linux:\n\u003e  ```sh\n\u003e  . ./setup.sh\n\u003e  ```\n\n#### Port Audio\n\nBefore install the dependencies in Python, you should check the installation of `portaudio` for installing PyAudio.\n\n```shell\nsudo apt update\nsudo apt install portaudio19-dev\n```\n\nOr if you are using macOS, please run:\n\n```shell\nbrew install portaudio\n```\n\nFirst, you need to install requirements:\n\n```shell\npip install -r requirements.txt\n```\n\nThen you need to install the language library for `spaCy`.\n\n```shell\npython -m spacy download en_core_web_sm\n```\n\n### Devices\n\nThen, you need to prepare GPUs, including NVIDIA GPUs and CUDA Toolkit, or using MPS by Apple.\n\n#### NVIDIA GPUs\n\nYou need to use `nvidia-smi` to check the status of GPUs.\n\n#### CUDA Toolkit\n\nYou need to install CUDA Toolkit to use NVIDIA GPUs.\n\n#### MPS by Apple\n\nIf you are using Apple Silicon, feel free to use MPS device for acceleration except the LLMs.\n\n### Configuration\n\nYou need to configure the system by editing `config.yaml`.\n\n#### OpenAI Key\n\nIf you are using ChatGPT in Aurora Echo, you need to provide OpenAI key. Aurora Echo guarantees that your key will not be leaked.\n\nFor example, your key is `sk-proj-114514`, then you need to edit `config.yaml` as follows:\n\n```yaml\nopenai:\n  key: sk-proj-114514\n```\n\nIf you are using other servers, e.g. Microsoft Azure, please specify the server in `config.yaml`:\n\n```yaml\nopenai:\n  base: https://models.inference.ai.azure.com\n```\n\n#### Llama Installation Path\n\nWe use `Llama` or `Qwen` as the local LLM.\n\nYou should download the `Llama` model from Hugging Face, and move the LLM to `./data/models`, and then specify the model.\n\n```yaml\nmodels:\n  llama:\n    path: ./data/models\n    name: meta-llama/Meta-Llama-3.1-8B-Instruct\n```\n\nSams as `Qwen`:\n\n```yaml\nmodels:\n  qwen:\n    path: ./data/models\n    name: Qwen/Qwen2-1.5B-Instruct\n```\n\n#### Wav2Vec Installation Path\n\nWe use `Wav2Vec` as the local ASR.\n\nYou should download the `Wav2Vec` model from Hugging Face, and move the ASR to `./data/models`, and then specify the model.\n\n```yaml\nmodels:\n  wav2vec:\n    path: ./data/models/\n    name: facebook/wav2vec2-base-960h\n```\n\n### Training Datasets\n\nIf you want to train the model, you are required to specify train datasets.\n\n#### Facial Sentiment Analysis\n\nYou need to tell facial expressions apart with `angry`, `disgust`, `fear`, `happy`, `sad`, `surprise`, and `neutral`. If there's also unknown, you can use `others`.\n\nWe will use the `ImageFolder` to serialize the dataset.\n\nOr you can use the pretrained file and move it to `./data/sentiment/model.pth`.\n\n#### Sentiment Analysis\n\nYou can use `IMDB` dataset provided by Standford University, and using `PyTorch`, it will automatically download to `.data`.\n\nOr you can use the pretrained file and move it to `./data/models/facial.pth`.\n\n### Run the application\n\nBefore starting to run, you need to install `uvicorn` to run the server.\n\n```shell\npip install uvicorn\n```\n\nThen you can run the server by:\n\n```shell\nuvicorn main:app --reload\n```\n\nNow, please open [https://aurora-echo.zzdev.org](https://aurora-echo.zzdev.org) to use Aurora Echo. If you don't have the chance to access the International Internet, please manually build the frontend by running:\n\n```shell\ncd client\n# Suppose you have installed nodejs and pnpm\npnpm install\npnpm run build\n```\n\nThen you can use `http.server` module to serve the frontend.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzzteam-rccup-2024%2Faurora-echo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzzteam-rccup-2024%2Faurora-echo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzzteam-rccup-2024%2Faurora-echo/lists"}