{"id":13574460,"url":"https://github.com/jim-schwoebel/nala_assistant","last_synced_at":"2025-08-09T23:05:05.288Z","repository":{"id":172265055,"uuid":"643245530","full_name":"jim-schwoebel/nala_assistant","owner":"jim-schwoebel","description":"🔊😊 A fastapi voice-assistant framework to quickly prototype LLM-powered voice assistants in \u003c5 minutes.","archived":false,"fork":false,"pushed_at":"2024-01-15T14:21:40.000Z","size":33121,"stargazers_count":27,"open_issues_count":2,"forks_count":2,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-11T20:21:27.626Z","etag":null,"topics":["chatbot","chatgpt","dolly2","fastapi","fastapi-boilerplate","fastapi-sqlalchemy","fastapi-template","large-language-models","llm","llms","speech-recognition","speech-to-text","speecht5","tts","voice","voice-assistant","voice-assistants","wakeword","whisper","whisper-model"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jim-schwoebel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-20T14:55:07.000Z","updated_at":"2025-01-19T18:21:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"61b783e4-c9b6-42e0-b8a9-58798f6b0fc8","html_url":"https://github.com/jim-schwoebel/nala_assistant","commit_stats":null,"previous_names":["jim-schwoebel/bark_assistant","jim-schwoebel/nala_assistant"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/jim-schwoebel/nala_assistant","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jim-schwoebel%2Fnala_assistant","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jim-schwoebel%2Fnala_assistant/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jim-schwoebel%2Fnala_assistant/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jim-schwoebel%2Fnala_assistant/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jim-schwoebel","download_url":"https://codeload.github.com/jim-schwoebel/nala_assistant/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jim-schwoebel%2Fnala_assistant/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269649848,"owners_count":24453541,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-09T02:00:10.424Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatbot","chatgpt","dolly2","fastapi","fastapi-boilerplate","fastapi-sqlalchemy","fastapi-template","large-language-models","llm","llms","speech-recognition","speech-to-text","speecht5","tts","voice","voice-assistant","voice-assistants","wakeword","whisper","whisper-model"],"created_at":"2024-08-01T15:00:51.774Z","updated_at":"2025-08-09T23:05:05.263Z","avatar_url":"https://github.com/jim-schwoebel.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"\n\u003cp align=\"center\"\u003e\n   \u003cimg src=\"https://camo.githubusercontent.com/b8be743a9ccf7b6f8f83b5037e933dd3ec7c8c23ca5f267efd4cd34a521ad16d/68747470733a2f2f6d656469612e67697068792e636f6d2f6d656469612f56447a5647386c764e527566752f67697068792e676966\"\u003e\n\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n[![Python](https://img.shields.io/badge/python-v3.9+-blue.svg)]()\n[![Dependencies](https://img.shields.io/badge/dependencies-up%20to%20date-brightgreen.svg)]()\n[![GitHub Issues](https://img.shields.io/github/issues/anfederico/Clairvoyant.svg)](https://github.com/jim-schwoebel/bark_assistant/issues)\n[![License](https://img.shields.io/badge/license-Apache%202-blue)](https://www.apache.org/licenses/LICENSE-2.0.html)\n[![Contributions welcome](https://img.shields.io/badge/contributions-welcome-orange.svg)](https://github.com/users/jim-schwoebel/projects/2)\n\n \u003c/p\u003e\n \n# Nala\n![](https://img.shields.io/github/stars/jim-schwoebel/bark_assistant?style=social) ![](https://img.shields.io/github/forks/jim-schwoebel/bark_assistant?label=Fork) \n\nNala is a voice-assistant framework to quickly build and prototype voice assistants in \u003c5 minutes within the greater context of the emerging large-language-model (LLM) landscape. With Nala you can easily intgrate with state-of-the-art (SOTA) transcription like [Whisper API](https://huggingface.co/docs/transformers/model_doc/whisper), text-to-speech synthesis engines like Microsoft's [SpeechT5 model](https://huggingface.co/microsoft/speecht5_tts), and LLMs like [Dolly-v2-3b](https://huggingface.co/databricks/dolly-v2-3b) within a nice front-end - across any arbitrary wake word powered with the [Web Speech API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API/Using_the_Web_Speech_API#javascript).\n\n![](https://github.com/jim-schwoebel/bark_assistant/blob/main/static/images/output.gif)\n\nHere are some of Nala's key features:\n\n- **Extensible Architecture:** Nala offers a flexible and modular, python-centric FastAPI architecture that allows developers to extend its functionality with ease. Integrate new response models or TTS voice skins into your projects effortlessly.\n- **Native LLM Integration:** Nala integrates directly with the the [Dolly-v2-3b](https://huggingface.co/databricks/dolly-v2-3b) LLM model - and makes it easy for you to integrate with others using an easy-to-follow strategy with [helper functions](https://github.com/jim-schwoebel/nala_assistant/blob/main/helpers.py#L269).\n- **Multi-Platform Support:** Nala is designed to work seamlessly across various platforms and operating systems (e.g. Mac/Linux and Chrome/Safari). Whether you're building web applications, mobile apps, or even IoT devices, Nala can be easily integrated into your technology stack.\n- **Audio-to-Audio API**: Nala's FastAPI design allows for you to submit an audio file and get back audio file responses through the query-response model; few projects out there exist to help guide you through how to do this, so this may help accelerate learning for your voice assistant projects.\n- **Simple UI**: Nala provides a simple user interface for users to quickly rate responses with thumbs up or thumbs-down to aid in building reinforcement learning models with [Reinforcement Learning with Human Feedback](https://medium.com/aiguys/reinforcement-learning-from-human-feedback-instructgpt-and-chatgpt-693d00cb9c58).\n- **Privacy and Security:** Nala allows for downloads to be administered by superusers as specified in the `settings.json` - as well as authenticates users and sessions with standard JSON web tokens. Other features like encryption at rest, deletion of audio files, and other defaults are being worked on right now to preserve user privacy.\n\nNote that this is a version 2.0, web-enabled version of [a prior voice assistant app here](https://github.com/jim-schwoebel/nala).\n\n## getting started\n\n### mac (locally)\nInstall basic dependencies:\n```\nsudo apt-get install ffmpeg\ngit clone git@github.com:jim-schwoebel/nala_assistant.git\ncd nala_assistant\nvirtualenv env \nsource env/bin/activate\npip3 install -r requirements.txt\npip3 install git+https://github.com/suno-ai/bark.git\n```\nGenerate a secret key for `SESSION_SECRET`, `JWT_SECRET_KEY`, `JWT_REFRESH_SECRET_KEY` and environment vars using the following line of code 3 times (save this in `.env`)\n```\npython -c 'import secrets; print(secrets.token_hex())'\n```\nAlso, you need a `WEB_URL` and `TERMS_URL` for your website and the terms of use, accordingly. These also are in the `.env` file. \n\nTo open and edit .env file:\n```\nnano .env\n```\nThen run the app:\n```\nuvicorn app:app --reload\n```\nNote if you having trouble with the `uvicorn app:app --reload` command, you can try:\n```\npython3 -m uvicorn app:app --reload\n```\n\nAnd sometimes this make it work.\n\nYou will now be able to visit localhost (`http://127.0.0.1:8000`) to use application.\n\n### linux with GPU (locally)\nInstall basic dependencies:\n```\nsudo apt-get install ffmpeg\ngit clone git@github.com:jim-schwoebel/nala_assistant.git\ncd nala_assistant\nvirtualenv env \nsource env/bin/activate\npip3 install -r gpu_requirements.txt\npip3 install git+https://github.com/suno-ai/bark.git\n```\nGenerate a secret key for `SESSION_SECRET`, `JWT_SECRET_KEY`, `JWT_REFRESH_SECRET_KEY` and environment vars using the following line of code 3 times (save this in `.env`)\n```\npython -c 'import secrets; print(secrets.token_hex())'\n```\nAlso, you need a `WEB_URL` and `TERMS_URL` for your website and the terms of use, accordingly. These also are in the `.env` file. \n\nTo open and edit .env file:\n```\nnano .env\n```\nThen run the app:\n```\nuvicorn app:app --reload\n```\n\nYou will now be able to visit localhost (`http://127.0.0.1:8000`) to use appication.\n\n### api docs (locally)\nOnce you have setup the app locally, you can get to the api docs @ `http://127.0.0.1:8000/docs` (for swagger docs) or `http://127.0.0.1:8000/redoc` (for redoc). The recommended set of docs to use is `http://127.0.0.1:8000/docs` (swagger) as there is greater support for authentication with JSON web tokens and audio-to-audio routes. A screenshot is shown below of the docs to give you an idea of what they look like. The auto-generated docs via FastAPI make it much easier to expand the routes to your particular need as a developer.\n\n![](https://github.com/jim-schwoebel/nala_assistant/blob/main/static/images/docs.png)\n\n### deploying to server (externally)\n\nFollow these instructions to deploy on a server.\n\n1. Buy a domain on namecheap.com.\n2. Get a vultr account / forward DNS to cloudflare from domain. Note that you will need at least [1 NVIDIA V100 GPU](https://www.vultr.com/products/cloud-gpu/nvidia-a100/) to have a seamless user experience with the Bark model and various LLMs [like Dolly](https://github.com/databrickslabs/dolly).\n3. Get a cert.pem and private.pem file on cloudflare for the server.\n4. Create a virtual machine on vultr or a similar platform, forward CNAME on cloudflare to IP address of host.\n5. Set up the server with at least 1 NVIDIA V100 GPU (e.g. `pip3 install -r gpu_requirements.txt`), as described in the [linux with GPU (locally)](https://github.com/jim-schwoebel/nala_assistant#linux-with-gpu-locally) section above.\n6. Run the command on the server with uvicorn below.\n\nEnable firewall rules for SSL (port 443)\n```\nsudo ufw allow 80\nsudo ufw allow 443\n```\n```\nnohup gunicorn --bind {ip_address}:443 main:app --certfile=cert.pem --keyfile=private.pem -w 10 --graceful-timeout 30 -t 30 --worker-class=uvicorn.workers.UvicornWorker --workers 10 \u003c/dev/null \u0026\u003e/dev/null \u0026\n```\n\n`\u003c/dev/null \u0026\u003e/dev/null \u0026` is a statement means that it is a background job, and you need to change [ip_address] with the right IP adddress.\n\n## settings\n\nHere are the current settings that you can edit in th `settings.json` file:\n```json\n{\"website_name\": \"Nala\",\n    \"wake_word\": \"hey\", \n    \"super_users\": [\"jim@schwoebel.me\"],\n    \"audio_delete\": {\"default\": false, \"options\": [true,false]},\n    \"sounds\": {\"default\": \"chime\", \"options\": [\"chime\", \"bell\"]}, \n    \"voice\": {\"default\": \"bark\", \"options\": [\"microsoft\", \"bark\"]}, \n    \"response_type\": {\"default\": \"dolly\", \"options\": [\"blender\",\"dolly\", \"echo\"]}, \n    \"language\": {\"default\": \"en-us\", \"options\": [\"en-us\"]}}\n```\n\nYou can edit the website name, wakeword, super_users (registered users who can download data), sounds (after query), voice (response skin), response_type (e.g.  LLM models), and language (e.g. en-us only supported for now) here in the file. Note that the options listed here are currently the only options provided in the repository, but they are easy-to-extend as a framework later in the `helpers.py` file.\n\n## browser compatibility\nCurrently, Nala works on **Chrome** and **Safari**-based browsers based on [Web Speech API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API/Using_the_Web_Speech_API#javascript) standards. If you load Nala on any other browser, it will give an error message like this.\n\n![](https://github.com/jim-schwoebel/bark_assistant/blob/main/static/images/error.png)\n\nNote that you can find a current list of browsers that support the Web Speech API [here](https://caniuse.com/?search=Web%20Speech%20API) or in the figure below.\n\n![](https://github.com/jim-schwoebel/bark_assistant/blob/main/static/images/wspeeech_api.png?raw=true)\n\n## maintainers\n![](https://static.wixstatic.com/media/55f531_a204ce54c8484c4294297f1252de3752~mv2.png/v1/fill/w_160,h_160,al_c,q_85,usm_0.66_1.00_0.01,enc_auto/Erd%C5%91sLogoNewSmall.png)\n\nThis project was incubated as a result of the [Erdos Fellowship program](https://www.erdosinstitute.org/become-a-member) - and since has resulted in a larger independent initiative.\n\nHere is a list of active maintainers to this project:\n\n- [Jim](https://github.com/jim-schwoebel) - chief maintainer, Erdos Institute mentor\n- [Jin](https://github.com/jxu81) - Erdos Institute fellow\n- [Nathan](https://github.com/Finer-Things) -  Erdos Institute fellow\n- [Collin](https://github.com/cjoverbay) - Data scientist @ Indeed.com (project advisor)\n\nIf you'd like to help maintain this project, reach out to Jim Schwoebel @ jim@schwoebel.me and he can invite you to our weekly call to ship PRs and delegate work in our sprint cycle.\n\n## references \nHere is a quick list of references for additional reading. \n\n### javascript front-end \n- [audio.js](http://kolber.github.io/audiojs/) - playback audio alternative (setting)\n- [bootstrap icons](https://icons.getbootstrap.com/) - use bootstrap and bootstrap icons for javascript front-end\n- [howlers.js](https://github.com/goldfire/howler.js) - playback audio for assistant\n- [recorder.js](https://github.com/mattdiamond/Recorderjs) - to record audio files with bootstrap icon buttons\n- [Web Speech API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API/Using_the_Web_Speech_API#javascript)\n- [wavesurfer.js](https://wavesurfer-js.org/) - for enumerating last audio file generated in the browser\n### feature extraction\n- [python_speech_features](https://github.com/jim-schwoebel/allie/blob/master/features/audio_features/pspeech_features.py) - audio feature extraction method useed\n### ML models (used)\n- [Dolly-v2-3b](https://huggingface.co/databricks/dolly-v2-3b) - LLM (Databricks) \n- [SpeechT5 model](https://huggingface.co/microsoft/speecht5_tts) - text-to-speech synthesis (Microsoft)\n- [Whisper API](https://huggingface.co/docs/transformers/model_doc/whisper) - speech-to-text (SOTA)\n- [RLHF](https://medium.com/aiguys/reinforcement-learning-from-human-feedback-instructgpt-and-chatgpt-693d00cb9c58) - human feedback.\n### future tools used\n- [auth0](https://github.com/auth0) - authentication / tokens\n- [minio](https://github.com/minio/minio) - minio is an object storage platform\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjim-schwoebel%2Fnala_assistant","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjim-schwoebel%2Fnala_assistant","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjim-schwoebel%2Fnala_assistant/lists"}