Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/SJTU-IPADS/DeSearch
a decentralized search engine with a decentralized verifiable dataflow
https://github.com/SJTU-IPADS/DeSearch
decentralized-applications search-engine sgx-enclave steem-blockchain web3
Last synced: about 1 month ago
JSON representation
a decentralized search engine with a decentralized verifiable dataflow
- Host: GitHub
- URL: https://github.com/SJTU-IPADS/DeSearch
- Owner: SJTU-IPADS
- Created: 2021-05-24T13:12:21.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-04-16T01:31:58.000Z (over 1 year ago)
- Last Synced: 2024-08-03T01:24:25.443Z (4 months ago)
- Topics: decentralized-applications, search-engine, sgx-enclave, steem-blockchain, web3
- Language: C
- Homepage:
- Size: 8.01 MB
- Stars: 54
- Watchers: 5
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- Awesome-SGX-Open-Source - https://github.com/SJTU-IPADS/DeSearch
README
# Desearch
Desearch is an experimental system towards a decentralized search engine.
To achieve scalability and fault tolerance, desearch decouples computation and storage using a *stateless* trusted network and a *stateful* cloud store. The network is traced by witnesses and the store is regulated by blockchain.
The current implementation of desearch uses Intel SGX as the trusted hardware and Redis as the store.
**Warning: This repo hosts an academic proof-of-concept prototype and has not received a careful code review.**
## Publication
Please refer to [our OSDI'21 paper](https://www.usenix.org/conference/osdi21/presentation/li) for more details.
~~~
@inproceedings{li2021desearch,
author = {Mingyu Li and Jinhao Zhu and Tianxu Zhang and Cheng Tan and Yubin Xia and Sebastian Angel and Haibo Chen},
title = {Bringing Decentralized Search to Decentralized Services},
booktitle = {15th {USENIX} Symposium on Operating Systems Design and Implementation, {OSDI} 2021},
pages = {331--347},
publisher = {{USENIX} Association},
year = {2021},
}
~~~
## Getting Started
Hardware Requirement: SGX-capable desktops or SGX-capable cloud machines. To check whether your machine supports SGX, please refer to [Intel SGX Hardware](https://github.com/ayeks/SGX-hardware).
Note that if you wish to run several SGX nodes on your local machine without support of scalable SGX, it might take longer to bootstrap the whole system because of the scarce encrypted memory (usually 128MB/256MB on SGXv1).
This repo provides a non-SGX version in the `executor` folder, and an SGX version in the `sgx-executor` folder. The non-SGX version helps you debug more easily if you want to build extensions to desearch; it uses the same folder structure as the SGX version. Note that only the SGX version contains ORAM-based queriers.
### Prerequisite
Under Ubuntu 20.04, building desearch executors requires the following depecencies installed:
- [SGX SDK + SGX PSW](https://01.org/intel-software-guard-extensions/downloads)
- [Boost C++ Libraries](https://www.boost.org/)
- [Hiredis](https://github.com/redis/hiredis)
- [Redis++](https://github.com/sewenew/redis-plus-plus)
You can refer to the `deps` folder for the correct versions of Redis, Hiredis, and Redis++.
```shell
apt install -y libboost-all-dev
cd deps
tar zxf redis-6.2.6.tar.gz
tar zxf redis-plus-plus-1.2.3.tar.gz
pushd redis-6.2.6/deps/hiredis/
make -j$(nproc) && make install
popd
pushd redis-plus-plus-1.2.3
mkdir build && cd build && cmake ..
make -j$(nproc) && make install
popd
```
### Kanban
Kanban is an unmodified [Redis](https://redis.io/) that leverages relatively cheap cloud storage.
Simply start the Redis server:
```shell
cd Kanban
cd deps && make hdr_histogram linenoise jemalloc
cd - && make all
./src/redis-server ./redis.conf
```
To clear all states from Kanban, you can issue `./src/redis-cli FLUSHALL`.
### Manager
Manager is a special executor that makes Kanban resistant to tampering.
```shell
cd manager
make all
./manager
```
### Executor
Executor consist of the whole search pipeline. This release consolidates all search roles within one executable. You are free to modify `executor/start.sh` to launch more executors as you like.
```shell
cd executor
make all
bash ./start.sh
```
### Client
Client is a Web server that serves as the entry for desearch.
```sh
cd client
make all
./client
```
Then double click the `client/index.html`, or use a fancy Web entry `client/WebUI/index.html`.
### How to build a distributed setup
To extend to a WAN setup, you need to modify a few network configurations:
- `config.hpp`: change `KANBAN_ADDR` to a global public IP
- `executor/start.sh`: change querier IP address to a public one that clients can reach
### Limitations and FAQs
See [executor/README.md](executor/README.md)
## Contributors
- Mingyu Li: maxullee [at] sjtu [dot] edu [dot] cn
- Jinhao Zhu: zhujinhao [at] sjtu [dot] edu [dot] cn
- Tianxu Zhang: zhangtianxu [at] sjtu [dot] edu [dot] cn
- Sajin Sasy: [ZeroTrace](https://github.com/sshsshy/ZeroTrace)
## License
MulanPSL-2.0 (see [here](https://opensource.org/licenses/MulanPSL-2.0))