https://github.com/capjamesg/nanosearch
Build a search engine from a website sitemap.
https://github.com/capjamesg/nanosearch
search-engine web-search
Last synced: over 1 year ago
JSON representation
Build a search engine from a website sitemap.
- Host: GitHub
- URL: https://github.com/capjamesg/nanosearch
- Owner: capjamesg
- License: mit
- Created: 2024-05-28T23:01:39.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-05-29T12:53:46.000Z (about 2 years ago)
- Last Synced: 2025-04-24T23:42:50.621Z (over 1 year ago)
- Topics: search-engine, web-search
- Language: Python
- Homepage: https://jamesg.blog/2024/05/29/nanosearch/
- Size: 9.77 KB
- Stars: 12
- Watchers: 2
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# nanosearch
Nanosearch is an in-memory search engine designed for small (< 10,000 URL) websites.
With Nanosearch, you can build a search engine in a few lines of code.
Nanosearch supports the BM25 and TF/IDF algorithms.
Nanosearch also computes a link graph and uses the number of inlinks to a page as a ranking factor. This is useful for ranking results for queries where there are multiple relevant pages by keyword.
## Installation
```bash
pip install nanosearch
```
## Quickstart
### Build a Search Engine from a Sitemap
```python
from nanosearch import NanoSearchBM25
engine = NanoSearchBM25().from_sitemap(
"https://jamesg.blog/sitemap.xml",
title_transforms=[lambda x: x.split("|")[0]]
)
results = engine.search("coffee")
print(results)
```
### Build a Search Engine from a List of URLs
```python
from nanosearch import NanoSearchBM25
urls = [
"https://jamesg.blog/",
"https://jamesg.blog/coffee",
]
engine = NanoSearchBM25().from_urls(urls)
results = engine.search("coffee")
print(results)
```
### Save an Index to Disk
You can save an index to disk and load it later with:
```python
engine.to_nanosearch_json("index.json")
engine = NanoSearchBM25().from_nanosearch_json("index.json")
```
## Supported Algorithms
Nanosearch supports the following search algorithms:
- TF/IDF
- BM25
## License
This project is licensed under an [MIT license](LICENSE).