Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/mubelotix/admarus
Peer-to-Peer Search Engine for IPFS
https://github.com/mubelotix/admarus
decentralized distributed full-text-search ipfs libp2p p2p peer-to-peer rust search search-engine site-search
Last synced: 5 days ago
JSON representation
Peer-to-Peer Search Engine for IPFS
- Host: GitHub
- URL: https://github.com/mubelotix/admarus
- Owner: Mubelotix
- License: agpl-3.0
- Created: 2023-06-10T15:42:34.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2024-12-04T09:33:59.000Z (about 2 months ago)
- Last Synced: 2025-01-09T09:17:01.813Z (12 days ago)
- Topics: decentralized, distributed, full-text-search, ipfs, libp2p, p2p, peer-to-peer, rust, search, search-engine, site-search
- Language: Rust
- Homepage: https://admarus.net
- Size: 5.97 MB
- Stars: 72
- Watchers: 6
- Forks: 1
- Open Issues: 101
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Admarus
Peer-to-Peer Search Engine for IPFS
Admarus unlocks the full potential of IPFS by making it searchable. It is an open, decentralized network of peers indexing their IPFS documents. Admarus relies on no central authority, and is censorship-resistant by design.
🔥 [**Try the gateway-based demo!**](https://admarus.net/) 🔥
## ✨ Features
- **No storage use**: Admarus indexes data that's already in your Kubo node
- **Search operators**: `AND`, `OR`, and `NOT`
- **Trustless**: Results are verified, not trusted
- **Language detection**: Language is detected from text
- [**Scalable**](#scalability): Gets faster as more peers join
- **Censorship-resistant**: Censors would need full control of more than 95% of the network
- **Open**: Nodes don't discriminate on obscure criteria (hi emails)
- **Decentralized**: No central authority. Multiple peer discovery mechanisms available
- **Blockchain-free**: No blockchain, no token, just peer-to-peer magic
- **Developer-friendly**: Practical API for building apps and bots
## 🌍 Scalability
There is no use in having a search engine if it breaks under load.
While previous attempts at building a peer-to-peer, powerful search engine have all failed, **Admarus was designed with scalability in mind**.
Actually, Admarus gets faster and more reliable as the network grows to thousands of peers.
This is all thanks to the [Kamilata protocol](https://github.com/mubelotix/kamilata) and its routing algorithm for queries.
It allows Admarus to download results at constant speed, regardless of the size of the network.
Of course, the client cannot afford to download and rank millions of results, but we actually don't need a [recall](https://en.wikipedia.org/wiki/Precision_and_recall) of 100%.
That's because some documents are more popular than others, and these are the ones we want to find.
(By default, 50% of the score of a document is based on its popularity.)
The more popular a document is, the faster we will find it, hence we don't need 100% recall to find the top `n` results.
I have only been able to run simulations with up to `30000` peers, so the behavior of the network with millions of peers is still unknown.
However, we can confidently say that Admarus can scale to at least a `200k` peers and billions of documents.
Even if problems were to arise, these would be limited to queries made of a single common word, as for each additional term, you divide the pool of peers to query by a huge factor.
The current slowness of the network is due to the fact that there are not enough peers on it.
They are doing their best at generating results for queries. Each result is generated from the document.
The thing is, each document must be read from the Kubo store on the disk, which only yields 11 documents per second on my machines.
As we parallelize the process by querying multiple peers concurrently, search will get significantly faster.
## 📚 Guides and Documentation
We have a [publicly-editable wiki](https://github.com/Mubelotix/admarus/wiki) for guides and documentation.
* [Getting Started](https://github.com/Mubelotix/admarus/wiki)
* [Building](https://github.com/Mubelotix/admarus/wiki/building-from-source)
* [Installing](https://github.com/Mubelotix/admarus/wiki/installation)
* [Searching](https://github.com/Mubelotix/admarus/wiki/searching)
- [Basics](https://github.com/Mubelotix/admarus/wiki/searching#basics)
- [Operators](https://github.com/Mubelotix/admarus/wiki/searching#operators)
- [Filters](https://github.com/Mubelotix/admarus/wiki/searching#filters)
* [Indexing](https://github.com/Mubelotix/admarus/wiki/indexing)
- [Pages](https://github.com/Mubelotix/admarus/wiki/indexing#pages)
- [Websites](https://github.com/Mubelotix/admarus/wiki/indexing#complete-websites)
## 👨💻 About
Admarus is developped by [Mubelotix](https://github.com/Mubelotix), a french cypherpunk.
Feel free to [get in touch](mailto:[email protected]) if you have any question or suggestion.
I'm available to help people run nodes to make the network grow.
Admarus means Google in Gaulish (literally "very big").
## 📜 License
Admarus is licensed under the GNU AGPL v3.
However, the underlying [Kamilata protocol](https://github.com/Mubelotix/kamilata) uses the less restrictive MIT license.