{"id":20703849,"url":"https://github.com/euiyounghwang/python-fastapi-vector-search","last_synced_at":"2025-10-14T02:19:05.018Z","repository":{"id":190037903,"uuid":"681821189","full_name":"euiyounghwang/python-fastapi-vector-search","owner":"euiyounghwang","description":"python-fastapi-vector-search","archived":false,"fork":false,"pushed_at":"2024-02-27T03:17:26.000Z","size":1325,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-14T02:19:04.426Z","etag":null,"topics":["circleci","docker","docker-compose","faiss","fastapi","model","orm","pip","poetry","prometheus","pytest","shell-script","sqlalchemy","vector-search"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/euiyounghwang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-08-22T20:40:11.000Z","updated_at":"2024-02-10T05:14:37.000Z","dependencies_parsed_at":"2023-10-15T05:08:38.896Z","dependency_job_id":"c3440ad5-8a3c-4aee-ab12-3b1b0164edb3","html_url":"https://github.com/euiyounghwang/python-fastapi-vector-search","commit_stats":null,"previous_names":["euiyounghwang/python-fastapi-vector-search"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/euiyounghwang/python-fastapi-vector-search","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/euiyounghwang%2Fpython-fastapi-vector-search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/euiyounghwang%2Fpython-fastapi-vector-search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/euiyounghwang%2Fpython-fastapi-vector-search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/euiyounghwang%2Fpython-fastapi-vector-search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/euiyounghwang","download_url":"https://codeload.github.com/euiyounghwang/python-fastapi-vector-search/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/euiyounghwang%2Fpython-fastapi-vector-search/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279017675,"owners_count":26086125,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["circleci","docker","docker-compose","faiss","fastapi","model","orm","pip","poetry","prometheus","pytest","shell-script","sqlalchemy","vector-search"],"created_at":"2024-11-17T01:09:47.580Z","updated_at":"2025-10-14T02:19:04.983Z","avatar_url":"https://github.com/euiyounghwang.png","language":"HTML","readme":"\n### Python-Vector-Search Project based on Python-Fastapi\n- We use Facebook AI Similarity Search (FAISS) to efficiently search for similar text (https://medium.com/loopio-tech/how-to-use-faiss-to-build-your-first-similarity-search-bf0f708aa772). Finding items that are similar is commonplace in many applications. Perhaps you want to find products in your store that match the description input by a customer. Or perhaps you want to find related keyword\n- A vector or an embedding is a numerical representation of text data. For example, using an embedding framework, text like ‘name’ can be transformed into a numerical representation\n- Semantic search consists of retrieving texts whose meaning matches a search query. For example, if your search query is “car,” the retrieved texts could include words such as “car,” “automobile,” “vehicle,” and so on. In contrast, keyword search only returns text passages with words of the search query.\n- The results of a semantic search are the texts whose embeddings are most similar to the query's embedding (https://blog.dataiku.com/semantic-search-an-overlooked-nlp-superpower?ref=dylancastillo.co)\n- Normalization is the process of transforming numerical data so that it uses a common scale.\n- Reference : https://pytest-with-eric.com/pytest-advanced/pytest-fastapi-testing/ \n\n\n### Install Tools\n- Postgres\n```\ndocker run --name postgres-local --network bridge -e POSTGRES_PASSWORD=1234 -p 15432:5432 -d postgres\n```\n- Elasticsearch/Kibana v8.\n```\ndocker run --name kibaba-run --network bridge -e \"ELASTICSEARCH_URL=http://host.docker.internal:9209\" -e \"ES_JAVA_OPTS=-Xms1g -Xmx1g\" -e \"ELASTICSEARCH_HOSTS=http://host.docker.internal:9209\" -p 5801:5601 docker.elastic.co/kibana/kibana:8.8.0\ndocker run --name es8-run --network bridge -p 9209:9200 -p 9114:9114 -p 9309:9300 -e \"http.cors.enabled=true\" -e \"http.cors.allow-origin=\\\"*\\\"\" -e \"http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization\" -e \"http.cors.allow-credentials=true\" -e \"xpack.security.enabled=false\" -e \"discovery.type=single-node\" -e \"ES_JAVA_OPTS=-Xms2g -Xmx2g\" docker.elastic.co/elasticsearch/elasticsearch:8.8.0\n```\n- RabbitMQ\n```\ndocker run --name rabbitmq -e RABBITMQ_DEFAULT_USER=euiyoung.hwang -e RABBITMQ_DEFAULT_PASS=1234 -p 5672:5672 -p 15672:15672 -p 25672:25672 rabbitmq:3.12-management\n- Ubutu Docker Install\nsudo wget -qO- http://get.docker.com/ | sh  (Install All-in-One) Othewise, \nsudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg\nsudo echo \\\n  \"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \\\n  $(lsb_release -cs) stable\" | sudo tee /etc/apt/sources.list.d/docker.list \u003e /dev/null\nsudo apt-get update\nsudo apt-get install -y docker-ce docker-ce-cli containerd.io\nsudo docker version\nsudo systemctl enable docker\nsudo systemctl start docker\nsudo systemctl enable containerd\nsudo systemctl start containerd\n```\n\n### REST API Realtime Performance using Prometheus\n![Alt text](screenshot/image-6.png)\n\n### Flow\n\n![Alt text](image.png)\n- Controllers - Contains application logic and passing user input data to service\n- Services - The middleware between controller and repository. Gather data from controller, performs validation and business logic, and calling repositories for data manipulation.\n- Repositories - layer for interaction with models and performing DB operations\n- Models - common laravel model files with relationships defined\n\n### Environment\n- \u003ci\u003eNo module named 'sentence_transformers' based on Poetry on Python/.Venv Environment. So try to make an enviroment on Conda and builder Docker \u0026 Docker-compose.yml\n\n### Swagger for FAISS Model\n- __\u003ci\u003eSupport Similarity Search using FAISS Model from trained sample datasets\u003c/i\u003e__\n- \u003ci\u003eI'll try to make it to REST API Endpoint with building /train, /reloading the model and search\u003c/i\u003e\n\n![Alt text](screenshot/image-3.png)\n\n### Swagger for Elasticsearch\n- __\u003ci\u003eSupport Enterprise Search using Elasticsearch Docker Instance\u003c/i\u003e__\n![Alt text](screenshot/image-4.png)\n\n## Docker build\n```\ndocker build \\\n  -f \"$(dirname \"$0\")/Dockerfile\" \\\n  -t fn-vector-search-api:test \\\n  --target fta_test \\\n  \"$(dirname \"$0\")/.\"\n\ndocker build \\\n  -f \"$(dirname \"$0\")/Dockerfile\" \\\n  -t fn-vector-search-api:es \\\n  --target fta_runtime \\\n  \"$(dirname \"$0\")/.\"\n```\n![Alt text](screenshot/image.png)\n\n## Docker run\n```\ndocker run --rm --platform linux/amd64 -it -d \\\n  --name fn-vector-search-api --publish 7001:7000 --expose 7000 \\\n  --network bridge \\\n  -e DATABASE_URL=postgresql://postgres:1234@host.docker.internal:15432/postgres \\\n  -e ES_HOST=http://host.docker.internal:9209 \\\n  -v \"$SCRIPTDIR:/app/FN-FTA-Services/\" \\\n  fn-vector-search-api:es\n```\n\n## services_start.sh for local env\n```\n#!/bin/bash\nset -e\n\nSCRIPTDIR=\"$( cd -- \"$(dirname \"$0\")\" \u003e/dev/null 2\u003e\u00261 ; pwd -P )\"\n\ncd $SCRIPTDIR\nsource .venv/bin/activate\nuvicorn main:app --reload --port=7000 --workers 4\n```\n![Alt text](screenshot/image-1.png)\n\n\n## FastAPI with Swagger UI\n```\nWilling to build FAISS (Facebook AI Similarity Search) with train model to search for similar text\nBuild Model/Schema with Postgres\nBuild Search with Elasticsearch\n```\n\n![Alt text](screenshot/image-5.png)","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feuiyounghwang%2Fpython-fastapi-vector-search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feuiyounghwang%2Fpython-fastapi-vector-search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feuiyounghwang%2Fpython-fastapi-vector-search/lists"}