Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/ConsenSys/Mahuta
IPFS Storage service with search capability
https://github.com/ConsenSys/Mahuta
caching elasticsearch ipfs search-engine
Last synced: 2 months ago
JSON representation
IPFS Storage service with search capability
- Host: GitHub
- URL: https://github.com/ConsenSys/Mahuta
- Owner: Consensys
- License: apache-2.0
- Created: 2018-02-15T17:51:49.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2024-06-24T13:23:57.000Z (7 months ago)
- Last Synced: 2024-11-06T06:20:49.721Z (3 months ago)
- Topics: caching, elasticsearch, ipfs, search-engine
- Language: Java
- Size: 1020 KB
- Stars: 239
- Watchers: 40
- Forks: 49
- Open Issues: 17
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- DeFi-Developer-Road-Map - Mahuta - IPFS Storage service with added search capability, formerly IPFS-Store (Roadmap)
README
Mahuta
======
**Mahuta** (formerly known as IPFS-Store) is a library to aggregate and consolidate files or documents stored by your application on the IPFS network. It provides a solution to collect, store, index, cache and search IPFS data handled by your system in a convenient way.
## Project status
| Service | Master | Development |
| -------- | -------- | -------- |
| CI Status |  |  |
| Test Coverage | [](https://coveralls.io/github/ConsenSys/Mahuta?branch=master) | [](https://coveralls.io/github/ConsenSys/Mahuta?branch=development) |
| Bintray | [  ](https://bintray.com/consensys/kauri/mahuta/_latestVersion) |
| Docker | [](https://cloud.docker.com/repository/docker/gjeanmart/mahuta)|
| Sonar | [](https://sonarcloud.io/dashboard?id=gjeanmart_IPFS-Store) |
## Features
- **Indexation**: Mahuta stores documents or files on IPFS and index the hash with optional metadata.
- **Discovery**: Documents and files indexed can be searched using complex logical queries or fuzzy/full text search)
- **Scalable**: Optimised for large scale applications using asynchronous writing mechanism and caching
- **Replication**: Replica set can be configured to replicate (pin) content across multiple nodes (standard IPFS node or IPFS-cluster node)
- **Multi-platform**: Mahuta can be used as a simple embedded Java library for your JVM-based application or run as a simple, scalable and configurable Rest API.
-------------------------------------------------------------------------
## Getting Started
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.
#### Prerequisites
Mahuta depends of two components:
- an IPFS node ([go](https://github.com/ipfs/go-ipfs) or [js](https://github.com/ipfs/js-ipfs) implementation)
- a search engine (currently only ElasticSearch is supported)
See how to run those two components first [run IPFS and ElasticSearch](https://github.com/ConsenSys/Mahuta/blob/master/mahuta-docs/run_ipfs_and_elasticsearch.md)
## Java library
1. Import the Maven dependencies (core module + indexer)
```xml
consensys-kauri
consensys-kauri
https://consensys.bintray.com/kauri/
```
````xml
net.consensys.mahuta
mahuta-core
${MAHUTA_VERSION}
net.consensys.mahuta
mahuta-indexing-elasticsearch
${MAHUTA_VERSION}
````
2. Configure Mahuta to connect to an IPFS node and an indexer
```
Mahuta mahuta = new MahutaFactory()
.configureStorage(IPFSService.connect("localhost", 5001))
.configureIndexer(ElasticSearchService.connect("localhost", 9300, "cluster-name"))
.defaultImplementation();
```
3. Execute high-level operations
```
IndexingResponse response = mahuta.prepareStringIndexing("article", "## This is my first article")
.contentType("text/markdown")
.indexDocId("article-1")
.indexFields(ImmutableMap.of("title", "First Article", "author", "greg"))
.execute();
GetResponse response = mahuta.prepareGet()
.indexName("article")
.indexDocId("article-1")
.loadFile(true)
.execute();
SearchResponse response = mahuta.prepareSearch()
.indexName("article")
.query(Query.newQuery().equals("author", "greg"))
.pageRequest(PageRequest.of(0, 20))
.execute();
```
For more info, [Mahuta Java API](https://github.com/ConsenSys/Mahuta/blob/master/mahuta-docs/mahuta_java_api.md)
## Spring-Data
1. Import the Maven dependencies
````
net.consensys.mahuta
mahuta-springdata
${MAHUTA_VERSION}
````
2. Configure your spring-data repository
```
@IPFSDocument(index = "article", indexConfiguration = "article_mapping.json", indexContent = true)
public class Article {
@Id
private String id;
@Hash
private String hash;
@Fulltext
private String title;
@Fulltext
private String content;
@Indexfield
private Date createdAt;
@Indexfield
private String createdBy;
}
public class ArticleRepository extends MahutaRepositoryImpl {
public ArticleRepository(Mahuta mahuta) {
super(mahuta);
}
}
```
For more info, [Mahuta Spring Data](https://github.com/ConsenSys/Mahuta/blob/master/mahuta-docs/mahuta_spring_data.md)
## HTTP API with Docker
### Prerequisites
- Docker
- [run IPFS and ElasticSearch with Docker](https://github.com/ConsenSys/Mahuta/blob/master/mahuta-docs/run_ipfs_and_elasticsearch.md#Docker)
### Docker
```
$ docker run -it --name mahuta \
-p 8040:8040 \
-e MAHUTA_IPFS_HOST=ipfs \
-e MAHUTA_ELASTICSEARCH_HOST=elasticsearch \
gjeanmart/mahuta
```
### Docker Compose
Check out the [documentation](https://github.com/ConsenSys/Mahuta/blob/master/mahuta-docs/mahuta_docker-compose.md) to configure Mahuta HTTP-API with Docker.
### Examples
To access the API documentation, go to [Mahuta HTTP API](https://github.com/ConsenSys/Mahuta/blob/master/mahuta-docs/mahuta_http-api.md)
#### Create the index `article`
- *Sample Request:*
```
curl -X POST \
http://localhost:8040/mahuta/config/index/article \
-H 'Content-Type: application/json'
```
- *Success Response:*
- Code: 200
Content:
```
{
"status": "SUCCESS"
}
```
#### Store and index an article and its metadata
- *Sample Request:*
```
curl -X POST \
'http://localhost:8040/mahuta/index' \
-H 'content-type: application/json' \
-d '{"content":"# Hello world,\n this is my first file stored on **IPFS**","indexName":"article","indexDocId":"hello_world","contentType":"text/markdown","index_fields":{"title":"Hello world","author":"Gregoire Jeanmart","votes":10,"date_created":1518700549,"tags":["general"]}}'
```
- *Success Response:*
- Code: 200
Content:
```
{
"indexName": "article",
"indexDocId": "hello_world",
"contentId": "QmWHR4e1JHMs2h7XtbDsS9r2oQkyuzVr5bHdkEMYiqfeNm",
"contentType": "text/markdown",
"content": null,
"pinned": true,
"indexFields": {
"title": "Hello world",
"author": "Gregoire Jeanmart",
"votes": 10,
"createAt": 1518700549,
"tags": [
"general"
]
},
"status": "SUCCESS"
}
```
#### Search by query
- *Sample Request:*
```
curl -X POST \
'http://localhost:8040/mahuta/query/search?index=article' \
-H 'content-type: application/json' \
-d '{"query":[{"name":"title","operation":"CONTAINS","value":"Hello"},{"name":"author.keyword","operation":"EQUALS","value":"Gregoire Jeanmart"},{"name":"votes","operation":"GT","value":"5"}]}'
```
- *Success Response:*
- Code: 200
Content:
```
{
"status": "SUCCESS",
"page": {
"pageRequest": {
"page": 0,
"size": 20,
"sort": null,
"direction": "ASC"
},
"elements": [
{
"metadata": {
"indexName": "article",
"indexDocId": "hello_world",
"contentId": "Qmd6VkHiLbLPncVQiewQe3SBP8rrG96HTkYkLbMzMe6tP2",
"contentType": "text/markdown",
"content": null,
"pinned": true,
"indexFields": {
"author": "Gregoire Jeanmart",
"votes": 10,
"title": "Hello world",
"createAt": 1518700549,
"tags": [
"general"
]
}
},
"payload": null
}
],
"totalElements": 1,
"totalPages": 1
}
}
```