https://github.com/dcarpintero/github-semantic-search
Semantic Search on Langchain Github Issues with Weaviate
https://github.com/dcarpintero/github-semantic-search
bm25 embedding-vectors hybrid-search langchain large-language-models python semantic-search streamlit weaviate
Last synced: 6 months ago
JSON representation
Semantic Search on Langchain Github Issues with Weaviate
- Host: GitHub
- URL: https://github.com/dcarpintero/github-semantic-search
- Owner: dcarpintero
- License: apache-2.0
- Created: 2023-09-16T16:34:10.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2023-09-25T10:16:03.000Z (about 2 years ago)
- Last Synced: 2025-04-13T15:04:16.063Z (6 months ago)
- Topics: bm25, embedding-vectors, hybrid-search, langchain, large-language-models, python, semantic-search, streamlit, weaviate
- Language: Python
- Homepage: https://gh-semantic-search.streamlit.app/
- Size: 2.36 MB
- Stars: 4
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://gh-semantic-search.streamlit.app/)
[](https://www.python.org/)
[](https://www.codefactor.io/repository/github/dcarpintero/github-semantic-search)
[](https://github.com/dcarpintero/st-newsapi-connector/blob/main/LICENSE)
# 🦜 Semantic Search on Langchain Github Issues with Weaviate 🔍
## 🔍 What's Semantic Search?
> *Semantic search refers to search algorithms that consider the intent and contextual meaning of search phrases when generating results, rather than solely focusing on keyword matching. The goal is to provide more accurate and relevant results by understanding the semantics, or meaning, behind the query.*
## 📋 How does it work?
- **Ingesting Github Issues**: We use the [Langchain Github Loader](https://js.langchain.com/docs/modules/data_connection/document_loaders/integrations/web_loaders/github) to connect to the [Langchain Repository](http://github.com/langchain-ai/langchain) and fetch the GitHub issues (nearly 2.000), which are then converted to a pandas dataframe and stored in a pickle file. See [./data-pipeline/ingest.py](./data-pipeline/ingest.py).
- **Generate and Index Vector Embeddings with Weaviate**: Weaviate generates vector embeddings at the object level (rather than for individual properties), it includes by default properties that use the text data type, in our case we skip the 'url' field (which will be also not filterable and not searchable) and set up the 'text2vec-openai' vectorizer. Given that our use case values fast queries over loading time, we have opted for the [HNSW](https://arxiv.org/abs/1603.09320) vector index type, which incrementally builds a multi-layer structure consisting from hierarchical set of proximity graphs (layers).
```python
class_obj = {
"class": "GitHubIssue",
"description": "This class contains GitHub Issues from the langchain repository.",
"vectorIndexType": "hnsw",
"vectorizer": "text2vec-openai",
"moduleConfig": {
"text2vec-openai": {
"model": "ada",
"modelVersion": "002",
"type": "text"
}
},
"properties": [
{
"name": "title",
"dataType": ["text"]
},
{
"name": "url",
"dataType": ["text"],
"indexFilterable": False,
"indexSearchable": False,
"vectorizePropertyName": False
},
{
"name": "description",
"dataType": ["text"]
},
{
"name": "creator",
"dataType": ["text"],
},
{
"name": "created_at",
"dataType": ["date"]
},
{
"name": "state",
"dataType": ["text"],
},
]
}
```
The ingestion follows in batches of 100 records:
```python
with client.batch as batch:
batch.batch_size = 100
for item in df.itertuples():
properties = {
"title": item.title,
"url": item.url,
"labels": item.labels,
"description": item.description,
"creator": item.creator,
"created_at": item.created_at,
"state": item.state,
}
batch.add_data_object(
data_object=properties,
class_name="GitHubIssue")
```
- **Searching with Weaviate**: Our App supports:
[Near-Text-Vector-Search](https://weaviate.io/developers/weaviate/search/similarity):
```python
@st.cache_data
def query_with_near_text(_w_client: weaviate.Client, query, max_results=10) -> pd.DataFrame:
"""
Search GitHub Issues in Weaviate with Near Text.
Weaviate converts the input query into a vector through the inference API (OpenAI) and uses that vector as the basis for a vector search.
"""
response = (
_w_client.query
.get("GitHubIssue", ["title", "url", "labels", "description", "created_at", "state"])
.with_near_text({"concepts": [query]})
.with_limit(max_results)
.do()
)
data = response["data"]["Get"]["GitHubIssue"]
return pd.DataFrame.from_dict(data, orient='columns')
```
[BM25-Search](https://weaviate.io/developers/weaviate/search/bm25):
```python
@st.cache_data
def query_with_bm25(_w_client: weaviate.Client, query, max_results=10) -> pd.DataFrame:
"""
Search GitHub Issues in Weaviate with BM25.
Keyword (also called a sparse vector search) search that looks for objects that contain the search terms in their properties according to
the selected tokenization. The results are scored according to the BM25F function. It is .
"""
response = (
_w_client.query
.get("GitHubIssue", ["title", "url", "labels", "description", "created_at", "state"])
.with_bm25(query=query)
.with_limit(max_results)
.with_additional("score")
.do()
)
data = response["data"]["Get"]["GitHubIssue"]
return pd.DataFrame.from_dict(data, orient='columns')
```
[Hybrid-Search](https://weaviate.io/developers/weaviate/search/hybrid):
```python
@st.cache_data
def query_with_hybrid(_w_client: weaviate.Client, query, max_results=10) -> pd.DataFrame:
"""
Search GitHub Issues in Weaviate with BM25.
Keyword (also called a sparse vector search) search that looks for objects that contain the search terms in their properties according to
the selected tokenization. The results are scored according to the BM25F function. It is .
"""
response = (
_w_client.query
.get("GitHubIssue", ["title", "url", "labels", "description", "created_at", "state"])
.with_hybrid(query=query)
.with_limit(max_results)
.with_additional(["score"])
.do()
)
```
## 🚀 Quickstart
1. Clone the repository:
```
git@github.com:dcarpintero/github-semantic-search.git
```
2. Create and Activate a Virtual Environment:
```
Windows:
py -m venv .venv
.venv\scripts\activate
macOS/Linux
python3 -m venv .venv
source .venv/bin/activate
```
3. Install dependencies:
```
pip install -r requirements.txt
```
4. Ingest Data
```
python ./data-pipeline/ingest.py
```
5. Index Data
```
python ./data-pipeline/index.py
```
6. Launch Web Application
```
streamlit run ./app.py
```
## 👩💻 Streamlit Web App
Demo Web App deployed to [Streamlit Cloud](https://streamlit.io/cloud) and available at https://gh-semantic-search.streamlit.app/
## 📚 References
- [Langchain Document Loaders - Github](https://js.langchain.com/docs/modules/data_connection/document_loaders/integrations/web_loaders/github)
- [Weaviate Vector Search](https://weaviate.io/developers/weaviate/search/similarity)
- [Weaviate BM25 Search](https://weaviate.io/developers/weaviate/search/bm25)
- [Weaviate Hybrid Search](https://weaviate.io/developers/weaviate/search/hybrid)
- [Weaviate Schema Configuration](https://weaviate.io/developers/weaviate/configuration/schema-configuration)
- [Weaviate - How to efficiently add data objects and cross-references to Weaviate](https://weaviate.io/developers/weaviate/manage-data/import)
- [Get Started with Streamlit Cloud](https://docs.streamlit.io/streamlit-community-cloud/get-started)