Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/kimtth/visual-genius

🪅Visual Learning aids for Autism spectrum disorder children. Built w/ Azure OpenAI, Azure Cognitive Search, Azure AI services, FastAPI, Next.js
https://github.com/kimtth/visual-genius

applied-behavior-analysis autism autism-spectrum-disorder azure-openai bing card-generator childrens education healthcare neurodiversity visual-aid visual-learning

Last synced: 1 day ago
JSON representation

🪅Visual Learning aids for Autism spectrum disorder children. Built w/ Azure OpenAI, Azure Cognitive Search, Azure AI services, FastAPI, Next.js

Awesome Lists containing this project

README 

        

# 🪅Visual Genius: Communication Assistant



Most children with autism spectrum disorders (ASD) are visual learners. They tend to comprehend visual information better than auditory input, making visual supports more effective for their learning process.



The project was initiated due to the laborious creation of a visual card and a market demand for a better, cost-effective product. The aim is also to democratize AI, making it accessible to all and overcoming previous product limitations with the help of new technologies.



- Visual aids product in the market: [Bing Search Results](https://www.bing.com/images/search?q=ASD+for+visual+aids) / Can be quite costly [Amazon Search Result](https://www.amazon.com/Special-Communication-Speech-Verbal-Children/dp/B08CFNDHYY)

- Applied behavior analysis (ABA) is a therapeutic approach for treating ASD. 

- Visual aids in applied behavior analysis (ABA) [URL](https://centerforautism.com/)



> **Important:** The code in this repository was developed during a hackathon and implemented within a limited timeline. It is intended for demonstration purposes only.



## 🤖 Overview



#### Features 🪂:



1. Switching between personas and modes of generation (List, Steps, Manual / Parents and Caregivers, Childs)

1. Visual Card generation and management (Set the order of images by Drag and Drop)

1. Vector Image search

1. Video generation from images (To teach work procedures)

1. Text-to-Image



https://github.com/kimtth/visual-genius/assets/13846660/7a39a3ba-32e7-4742-aea6-c288df2bc766



#### Azure services ☁️:



1. Vector-based image search: Azure Cognitive Search & Computer Vision API for Vector embedding

1. Text-to-image generation: Azure OpenAI & Image Generation by Azure OpenAI Dall-E

   > Due to the Generation speed issue, only the last image will be generated by Dall-E.

1. Bing Image Search

1. Fluent emoji dataset

1. Azure Cognitive Services Speech to Text (Read the text on the card)

1. [Optional] Microsoft Coco dataset (Everyday Life Images)

   > The test dataset for Vector Image Search. Vector search seeks images based on their features, not by the associated metadata tags or the image file name.



## 📦 Installation and Setup



#### Deploy to Azure 🪟:



1. Set parameters under `infra\parameter.json`

2. Execute `deploy.ps1` to upload the dataset, deploy Azure resources, initialize the database, and set up the search index.

3. Open the `backend` directory. Deploy the application code to Azure App Service: It is recommended to use the `Azure Extension` in VS Code to deploy the code to Azure App Service. You can follow the [Quickstart: Deploy a Python app](https://learn.microsoft.com/en-us/azure/app-service/quickstart-python)



> [!IMPORTANT]  

> 1. Create a DALL·E model on the Azure Portal and set the deployment model name in `Azure > WebApp > Environment variables > 'AZURE_OPENAI_IMG_MODEL_DEPLOYMENT_NAME'`. When attempting to deploy the model using Bicep, it was not possible to deploy at that time.

> 1. Azure Bing Search cannot be deployed using Bicep; it must be deployed through the Azure Portal. Once deployed, set the key in `Azure > WebApp > Configuration > Application settings` under the variable name `'BING_IMAGE_SEARCH_KEY'`.



- The deployment step using Azure CLI, without relying on VS Code, is commented out in `deploy.ps1`. It requires SCM Basic Auth credentials.



   ```powershell

   <# 

   az webapp deployment source config-zip `

      --resource-group $ResourceGroup `

      --name $APP_SERVICE_NAME `

      --src "app.zip"

   Write-Host "Application deployed to Azure App Service." 

   #>

   ```



- Please ensure you have installed nodejs, yarn, Azure CLI, Azure Functions Core Tools, psql, and  python3.



   ```powershell

   # Install Chocolatey

   Set-ExecutionPolicy Bypass -Scope Process -Force; `

   [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; `

   iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))



   # Install Node.js

   choco install nodejs -y



   # Install Yarn

   choco install yarn -y



   # Install Azure CLI

   choco install azure-cli -y



   # Install Azure Functions Core Tools

   choco install azure-functions-core-tools -y



   # Install PostgreSQL

   choco install postgresql -y



   # Install Python 3.11

   choco install python --version 3.11.0 -y

   ```



> [!NOTE]  

> 1. Azure Storage account should have `Allow storage account key access` enabled in the Configuration settings for access by the application and indexer.

> 1. The test user, called `sys`, will be registered during database initialization. However, you should update the value in the password column of the user table by the `hashed_pwd` you created (refer to `hashed_pwd.py` for password creation).

> 1. The `deploy.ps1` script generates a `_dev.env` file, which you can use for development after renaming it to `.env`.



#### To dev 🖥️:



1. Open project folder in Visual Studio Code

2. Rename `.env.template` to `.env`, then fill in the values in `.env`.

3. In the terminal, run `yarn install`.

4. Run `yarn run dev` to view the project in a browser.

5. Run `python app.py` to launch the backend.



!important: `react-beautiful-dnd` was not able to work well with `reactStrictMode: true` in NextJs.

Turn off the option at `next.config.js`.`



### TODO



- [ ] Option for PostgreSQL as a vector database.

- [X] Regularly removing unexpectedly failed or deleted data from the database, Azure AI Search, and Blob storage to maintain data consistency.

- [X] Download: If an image URL points to a web address that is not stored in the blob, it should be downloaded using HTTPS.

- [X] Key access to the Azure Storage Account for embedding API.

- [ ] Bug fix (wip): Script deployment issue fixed, and other bugs.