{"id":18359002,"url":"https://github.com/ev2900/opensearch_knn_vector_search","last_synced_at":"2025-04-06T13:31:42.902Z","repository":{"id":191860983,"uuid":"685557099","full_name":"ev2900/OpenSearch_kNN_Vector_Search","owner":"ev2900","description":"Tokenize and convert sample text data into vectors using BERT. Load the vector representation of the text to OpenSearch and use kNN for semantic search","archived":false,"fork":false,"pushed_at":"2025-03-30T01:10:35.000Z","size":793,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-30T02:22:21.271Z","etag":null,"topics":["aws","bert","huggingface-transformers","knn","opensearch","python","search","vector-search"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ev2900.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-31T13:52:07.000Z","updated_at":"2025-03-30T01:10:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"a193e8db-7f3e-4683-9d15-701c3663f338","html_url":"https://github.com/ev2900/OpenSearch_kNN_Vector_Search","commit_stats":null,"previous_names":["ev2900/opensearch_knn_vector_search"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ev2900%2FOpenSearch_kNN_Vector_Search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ev2900%2FOpenSearch_kNN_Vector_Search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ev2900%2FOpenSearch_kNN_Vector_Search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ev2900%2FOpenSearch_kNN_Vector_Search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ev2900","download_url":"https://codeload.github.com/ev2900/OpenSearch_kNN_Vector_Search/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247488551,"owners_count":20946958,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","bert","huggingface-transformers","knn","opensearch","python","search","vector-search"],"created_at":"2024-11-05T22:20:18.425Z","updated_at":"2025-04-06T13:31:42.892Z","avatar_url":"https://github.com/ev2900.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenSearch kNN Vector Search\n\n\u003cimg width=\"275\" alt=\"map-user\" src=\"https://img.shields.io/badge/cloudformation template deployments-16-blue\"\u003e \u003cimg width=\"85\" alt=\"map-user\" src=\"https://img.shields.io/badge/views-2533-green\"\u003e \u003cimg width=\"125\" alt=\"map-user\" src=\"https://img.shields.io/badge/unique visits-948-green\"\u003e\n\nThis example uses the publicly avaiable [Amazon Product Question Answer](https://registry.opendata.aws/amazon-pqa/) (PQA) data set. In this example, the questions in the PQA data set are tokenized and represented as vectors. BERT via. Hugging Face is used to generate the embeddings. The vector representation of the questions (embeddings) are loading to an OpenSearch index as a *knn_vector* data type\n\nSearches are executed against OpenSearch by transforming search text into embeddings and determining similarity using kNN. The most similar result answers are returned as search results\n\n# Deployment on AWS\n\nTo deploy this example on AWS you can click on the button below to launch a CloudFormation stack\n\n[![Launch CloudFormation Stack](https://sharkech-public.s3.amazonaws.com/misc-public/cloudformation-launch-stack.png)](https://console.aws.amazon.com/cloudformation/home#/stacks/new?stackName=open-search-kNN\u0026templateURL=https://sharkech-public.s3.amazonaws.com/misc-public/OpenSearch_kNN_Vector_Search.yaml)\n\nThe stack will deploy an Amazon OpenSearch domain and a Cloud9 environment with this GitHub repository downloaded. Before using the Cloud9 enviorment run the [resize_EBS.sh](https://github.com/ev2900/OpenSearch_kNN_Vector_Search/blob/main/resize_EBS.sh) from the Cloud9 termial.\n\nExecute the following in terminal from the *OpenSearch_kNN_Vector_Search* directory\n\n```bash resize_EBS.sh```\n\nThe bash script resizes the EBS volume attached to the Cloud9 instance from 10 GB to 100 GB.\n\nOnce the resize is complete, uou can update and run the [kNN.py](https://github.com/ev2900/OpenSearch_kNN_Vector_Search/blob/main/kNN.py) python script in the Cloud9 environment. The only parts of the Python script that need to updated before running it is the section below\n\n```\n# Configure re-usable variables for Opensearch domain URL, user name and password\nopensearch_url = 'https://\u003copensearch_domain_url' # DO NOT INCLUDE TRAILING SLASH\nopensearch_user_name = '\u003cuser_name\u003e'\nopensearch_password = '\u003cpassword\u003e'\n```\n\nThe user name and password are part of the CloudFormation outputs. The domain URL can be found on the [OpenSearch page](https://us-east-1.console.aws.amazon.com/aos/home) in the AWS console under the domains section. Once you update these values in the [kNN.py](https://github.com/ev2900/OpenSearch_kNN_Vector_Search/blob/main/kNN.py) file make sure to save the file.\n\nInstall the required Python libraries by running in the Cloud9 terminal\n\n```pip install -r requirements.txt```\n\nOnce the required libraries are install run the python script by executing\n\n```python kNN.py```\n\n# How does the Example Python Script Work\n\nThis section explains how the python script [kNN.py](https://github.com/ev2900/OpenSearch_kNN_Vector_Search/blob/main/kNN.py) works. The script has 6 sections each section in the full [kNN.py](https://github.com/ev2900/OpenSearch_kNN_Vector_Search/blob/main/kNN.py) script is clearly defined by comments. Each is explained below\n\n## 1. Prepare the headset production question answer (PQA) data\n\nEach JSON document in the raw PQA data set has a question with many potential answers in additon to other information about the product in question. The code below creates a pandas data frame (df) where each row is a single question and answer pair. The other product information is also removed.\n\nFor example a JSON document from the raw PQA data set is below\n\n```\n{\n\t\"question_id\": \"Tx39GCUOS5AYAFK\",\n\t\"question_text\": \"does this work with cisco ip phone 7942\",\n\t\"asin\": \"B000LSZ2D6\",\n\t\"bullet_point1\": \"Noise-Canceling microphone filters out background sound\",\n\t\"bullet_point2\": \"HW251N P/N 75100-06\",\n\t\"bullet_point3\": \"Uses Plantronics QD Quick Disconnect Connector. Must be used with Plantronics Amp or with proper phone or USB adapter cable\",\n\t\"bullet_point4\": \"Connectivity Technology: Wired, Earpiece Design: Over-the-head, Earpiece Type: Monaural, Host Interface: Proprietary, Microphone Design: Boom, Microphone Technology: Noise Canceling, Product Model: HW251N, Product Series: SupraPlus, Standard Warranty: 2 Year\",\n\t\"bullet_point5\": \"Easy Lightweight Wear -Leaving One Ear Uncovered For Person-to-Person Conversations\", \"product_description\": \"\", \"brand_name\": \"Plantronics\", \"item_name\": \"Plantronics HW251N SupraPlus Wideband Headset (64338-31)\",\n\t\"question_type\": \"yes-no\",\n\t\"answer_aggregated\": \"neutral\",\n\t\"answers\": [\n\t\t{\"answer_text\": \"Use the Plantronics compatibility guide to see what is compatible with your phone. http://www.plantronics.com/us/compatibility-guide/\"},\n\t\t{\"answer_text\": \"I think that you will need a extra cord, but, To avoid offering you any false information,   We highly recommend contacting the manufacturer of this product for more specific information.   if you are not sure about it, you can go first to :  http://www.plantronics.com/us/support/  or call Plantronics TOLL FREE SUPPORT: 1-855-765-7878 24-HOUR SUPPORT SUNDAY 2PM-FRIDAY 5PM (PT)  they will answer all the questions you need to know about it.\"},\n\t\t{\"answer_text\": \"I'm really not positive. It works with our phones that include model numbers 7941, 7945 and 7961.\"}\n\t]\n}\n```\nAfter processing the document the df data frame will have a question and answer column\n\n\tQuestion: \tdoes this work with cisco ip phone 7942\n\tAnswer: \tUse the Plantronics compatibility guide to see what is compatible with your phone. http://www.plantronics.com/us/compatibility-guide/\n\n## 2. Convert the question text in the PQA data set into vector(s)\n\nAfter preparing the PQA dataset, we need to tokenize the question text and convert it into a vector representation. We do this using BERT via. Hugging Face. This process has 3 steps. Each is explained below\n\n### Tokenize the question text\n\nInput:  ```df[\"question\"].tolist()``` \u003cbr\u003e\nOutput: ```inputs_tokens```\n\ntokenizer()\n\tpadding - Ensure that all sequences in a batch have the same length. If the padding argument is set to True, the function will pad sequences up to the length of the longest sequence in the batch\n\treturn_tensors - Return output as a PyTorch torch.Tensor object\n\n### Convert tokenized questions into vectors using BERT\n\nInput:  ```inputs_tokens``` \u003cbr\u003e\nOutput: ```outputs```\n\n```outputs``` is 3 dimensional tensor object. Working with 1000 rows of data the dimension of outputs could be [1000, 64, 768]\n\n### Use mean pooling to condense the\n\nInput: ```outputs``` \u003cbr\u003e\nOuput: ```question_text_embeddings```\n\n```question_text_embeddings``` is a 2 dimensional tensor object. Working with 1000 rows of data the dimension of output could be [1000, 768]\n\n## 3. Create an OpenSearch index\n\nMake an API call to the OpenSearch domain to create an OpenSearch index named ```nlp_pqa``` with 3 fields. These fields include\n\n1. question_vector\n2. question\n3. answer\n\nThe data type of the ```question_vector``` field is ```knn_vector```\n\n## 4. Load data into the index\n\nMake API calls to the OpenSearch domain to load the data (plain text and vector representation) into the OpenSearch index that was just created\n\n## 5. Convert user input/search into a vector\n\nTokenize and convert the user input / search of *does this work with xbox?* into a vector. The vector representation of this search will be used in the next step\n\nInput: ```query_raw_sentences = ['does this work with xbox?']```\nOuput: ```search_vector```\n\n## 6. Search OpenSearch using the vector representation of the user input/search\n\nMake an API call to the OpenSearch domain to run the run the search *does this work with xbox* by passing the vector-ized version of the search to OpenSearch. Print the top results to the console\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fev2900%2Fopensearch_knn_vector_search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fev2900%2Fopensearch_knn_vector_search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fev2900%2Fopensearch_knn_vector_search/lists"}