{"id":25238093,"url":"https://github.com/danjamk/vector-database-loader","last_synced_at":"2026-02-26T12:04:58.735Z","repository":{"id":275474996,"uuid":"924224493","full_name":"danjamk/vector-database-loader","owner":"danjamk","description":"Utility library for curating and loading vector databases","archived":false,"fork":false,"pushed_at":"2025-04-18T13:12:57.000Z","size":140,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-12T22:08:06.113Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/danjamk.png","metadata":{"files":{"readme":"README-PYPI.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-29T16:27:13.000Z","updated_at":"2025-04-18T13:12:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"e4712361-a468-4a53-9397-2cb3264ece99","html_url":"https://github.com/danjamk/vector-database-loader","commit_stats":null,"previous_names":["danjamk/vector-database-loader"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/danjamk/vector-database-loader","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danjamk%2Fvector-database-loader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danjamk%2Fvector-database-loader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danjamk%2Fvector-database-loader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danjamk%2Fvector-database-loader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/danjamk","download_url":"https://codeload.github.com/danjamk/vector-database-loader/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danjamk%2Fvector-database-loader/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29858461,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-26T08:51:08.701Z","status":"ssl_error","status_checked_at":"2026-02-26T08:50:19.607Z","response_time":89,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-11T16:45:39.050Z","updated_at":"2026-02-26T12:04:58.719Z","avatar_url":"https://github.com/danjamk.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# vector-database-loader\nLoading content info a vector database is relatively easy to do, especially with frameworks like [LangChain](https://www.langchain.com/).\nHowever, the process of curating the content and loading it into the database can be a bit more complex.  If you are building \na RAG application or similar, the quality and relevance of the content is critical.  This project is meant to help with that process.\n\nA use case for this type of project is discussed in more depth in the blog [A Cost-Effective AI Chatbot Architecture with AWS Bedrock, Lambda, and Pinecone](https://medium.com/@dan.jam.kuhn/a-cost-effective-ai-chatbot-architecture-with-aws-bedrock-lambda-and-pinecone-40935b9ec361)\n\n## Features\n- **Vector Database Support** - The framework is built to support multiple vector databases but is currently implementing support for [Pinecone](https://www.pinecone.io/).\nMore vector databases will be added, but if needed you can fork the project and handle your own needs by extending the base class.  \n- **Embedding Support** - You can use any embedding provided by [Langchain](https://python.langchain.com/docs/integrations/text_embedding/), which includes OpenAI, AWS Bedrock, HuggingFace, Cohere and much, much more.\n- **Content Curation** - The framework is built configure some common content types and sources, but again is meant to be extended a needed.\n  - Sources include websites, local folders and google drive\n  - Types include PDF, Word, and Web content and google docs\n- **Text Splitter** - This framework uses the [RecursiveCharacterTextSplitter](https://python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/recursive_text_splitter/) from LangChain.  \nThis is a powerful tool that can split text into chunks of a specified size, while maintaining the context of the text.  This is especially useful for long documents like web pages or PDFs.\n\n## Example\n**Install the package with pip:**\n\nNote: The dependencies are kind of beefy!\n\n```pip install vector-database-loader```\n\n**Configuration details:**\n\nAdd your Pinecone API and OpenAI keys to your .env file, see sample.env for an examples.\n\n\n**Code**\n\n```python\nimport time\n\nfrom dotenv import load_dotenv, find_dotenv\nfrom langchain_openai import OpenAIEmbeddings\n\nfrom vector_database_loader.pinecone_vector_db import PineconeVectorLoader, PineconeVectorQuery\n\n# Define your content sources and add them to the array\nweb_page_content_source = {\"name\": \"SpaceX\", \"type\": \"Website\", \"items\": [\n    \"https://en.wikipedia.org/wiki/SpaceX\"\n\n], \"chunk_size\": 512}\ncontent_sources = [web_page_content_source]\n\n# Load into your vector database.  Be sure to add your Pinecone and OpenAI API keys to your .env file\nload_dotenv(find_dotenv())\nembedding_client = OpenAIEmbeddings()\nindex_name = \"my-vectordb-index\"\nvector_db_loader = PineconeVectorLoader(index_name=index_name,\n                                        embedding_client=embedding_client)\nvector_db_loader.load_sources(content_sources, delete_index=True)\n\n# Query your vector database\nprint(\"Waiting 30 seconds before running the query, to make sure the data is available\")\ntime.sleep(30)  # This is needed because there is a latency in the data being available\nvector_db_query = PineconeVectorQuery(index_name=index_name,\n                                      embedding_client=embedding_client)\nquery = \"What is SpaceX's most recent rocket model being tested?\"\ndocuments = vector_db_query.query(query)\nprint(f\"Query: {query} returned {len(documents)} results\")\nfor doc in documents:\n    print(f\"   {doc.metadata['title']}\")\n    print(f\"   {doc.page_content}\")\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanjamk%2Fvector-database-loader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdanjamk%2Fvector-database-loader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanjamk%2Fvector-database-loader/lists"}