Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/zzteam-rccup-2024/aurora-echo

We propose a new feedback system, named Aurora Echo} which provides audio-visual feedback for the resource provider. Through numerous artificial intelligence, such as LLM, we can make the system more friendly to not only user but also the resource provider.
https://github.com/zzteam-rccup-2024/aurora-echo

ai cv llama llm mediapipe nlp opencv pytorch

Last synced: about 2 months ago
JSON representation

We propose a new feedback system, named Aurora Echo} which provides audio-visual feedback for the resource provider. Through numerous artificial intelligence, such as LLM, we can make the system more friendly to not only user but also the resource provider.

Awesome Lists containing this project

README

        

# Aurora Echo

Aurora Echo is a visual-audio feedback system with maximized security.

## Get Started

### Requirements

> Alternatively, you can run the setup script directly:
>
> - Windows:
> ```pwsh
> .\setup.ps1
> ```
> - macOS / Linux:
> ```sh
> . ./setup.sh
> ```

#### Port Audio

Before install the dependencies in Python, you should check the installation of `portaudio` for installing PyAudio.

```shell
sudo apt update
sudo apt install portaudio19-dev
```

Or if you are using macOS, please run:

```shell
brew install portaudio
```

First, you need to install requirements:

```shell
pip install -r requirements.txt
```

Then you need to install the language library for `spaCy`.

```shell
python -m spacy download en_core_web_sm
```

### Devices

Then, you need to prepare GPUs, including NVIDIA GPUs and CUDA Toolkit, or using MPS by Apple.

#### NVIDIA GPUs

You need to use `nvidia-smi` to check the status of GPUs.

#### CUDA Toolkit

You need to install CUDA Toolkit to use NVIDIA GPUs.

#### MPS by Apple

If you are using Apple Silicon, feel free to use MPS device for acceleration except the LLMs.

### Configuration

You need to configure the system by editing `config.yaml`.

#### OpenAI Key

If you are using ChatGPT in Aurora Echo, you need to provide OpenAI key. Aurora Echo guarantees that your key will not be leaked.

For example, your key is `sk-proj-114514`, then you need to edit `config.yaml` as follows:

```yaml
openai:
key: sk-proj-114514
```

If you are using other servers, e.g. Microsoft Azure, please specify the server in `config.yaml`:

```yaml
openai:
base: https://models.inference.ai.azure.com
```

#### Llama Installation Path

We use `Llama` or `Qwen` as the local LLM.

You should download the `Llama` model from Hugging Face, and move the LLM to `./data/models`, and then specify the model.

```yaml
models:
llama:
path: ./data/models
name: meta-llama/Meta-Llama-3.1-8B-Instruct
```

Sams as `Qwen`:

```yaml
models:
qwen:
path: ./data/models
name: Qwen/Qwen2-1.5B-Instruct
```

#### Wav2Vec Installation Path

We use `Wav2Vec` as the local ASR.

You should download the `Wav2Vec` model from Hugging Face, and move the ASR to `./data/models`, and then specify the model.

```yaml
models:
wav2vec:
path: ./data/models/
name: facebook/wav2vec2-base-960h
```

### Training Datasets

If you want to train the model, you are required to specify train datasets.

#### Facial Sentiment Analysis

You need to tell facial expressions apart with `angry`, `disgust`, `fear`, `happy`, `sad`, `surprise`, and `neutral`. If there's also unknown, you can use `others`.

We will use the `ImageFolder` to serialize the dataset.

Or you can use the pretrained file and move it to `./data/sentiment/model.pth`.

#### Sentiment Analysis

You can use `IMDB` dataset provided by Standford University, and using `PyTorch`, it will automatically download to `.data`.

Or you can use the pretrained file and move it to `./data/models/facial.pth`.

### Run the application

Before starting to run, you need to install `uvicorn` to run the server.

```shell
pip install uvicorn
```

Then you can run the server by:

```shell
uvicorn main:app --reload
```

Now, please open [https://aurora-echo.zzdev.org](https://aurora-echo.zzdev.org) to use Aurora Echo. If you don't have the chance to access the International Internet, please manually build the frontend by running:

```shell
cd client
# Suppose you have installed nodejs and pnpm
pnpm install
pnpm run build
```

Then you can use `http.server` module to serve the frontend.