Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/cool-japan/ipfrs

IPFRS - Inter-Planet File RUST System
https://github.com/cool-japan/ipfrs

blockchain ipfs ipfs-api ipfs-blockchain rust rust-lang

Last synced: 3 months ago
JSON representation

IPFRS - Inter-Planet File RUST System

Awesome Lists containing this project

README 

          
# IPFRS - Inter-Planet File RUST System



![IPFRS](ipfrs.jpg)



**Version:** 0.1.0 "Foundation Release"



**Status:** Ready for Production (Local-First Focus)



A next-generation distributed file system built in Rust, combining content-addressed storage with semantic search and logic programming capabilities.



---



## 🚀 Quick Start



```bash

# Install (requires Rust 1.70+)

cargo install --path crates/ipfrs-cli



# Initialize repository

ipfrs init



# Add a file

ipfrs add myfile.txt

# Output: CID: bafybeig...



# Retrieve content

ipfrs cat bafybeig...



# Get statistics

ipfrs stats

```



---



## 🎯 What is IPFRS?



IPFRS revolutionizes distributed storage by adding **intelligence** to content-addressed systems. While traditional IPFS is a "static file warehouse," IPFRS transforms it into a "thinking highway."



**Key Innovations:**

- 🧠 **Semantic Search**: Find content by meaning, not just hash

- 🎓 **Logic Programming**: Content-addressed reasoning and inference

- ⚡ **Zero-Copy I/O**: Apache Arrow integration for performance

- 🦀 **Pure Rust**: Memory safety and ARM optimization



---



## 📦 Installation



### From Source (Recommended for 0.1.0)



```bash

git clone https://github.com/yourusername/ipfrs.git

cd ipfrs

cargo build --release

cargo install --path crates/ipfrs-cli

```



### Requirements

- Rust 1.70 or later

- ~100MB disk space

- Linux, macOS, or Windows



---



## 📖 Usage



### Command-Line Interface



#### Basic File Operations



```bash

# Initialize a repository

ipfrs init

# Creates .ipfrs/ directory



# Add files

ipfrs add document.pdf

ipfrs add image.png



# Output: CID: bafybeig...



# Retrieve by CID

ipfrs get bafybeig... --output recovered.pdf



# View content

ipfrs cat bafybeig... | less



# List all blocks

ipfrs list



# Show statistics

ipfrs stats

# Output:

# Number of blocks: 42

# Total size: 52.43 MB

# Average block size: 1.24 MB

```



#### HTTP Gateway



```bash

# Start HTTP gateway

ipfrs gateway --listen 127.0.0.1:8080



# Access via HTTP

curl http://localhost:8080/ipfs/bafybeig...



# Use REST API

curl -X POST http://localhost:8080/api/v0/add \

  -F file=@myfile.txt

```



### Rust API



```rust

use ipfrs::{Node, NodeConfig};



#[tokio::main]

async fn main() -> ipfrs::Result<()> {

    // Create and start node

    let mut node = Node::new(NodeConfig::default())?;

    node.start().await?;



    // Add content

    let content = b"Hello, IPFRS!";

    let cid = node.add_bytes(content).await?;

    println!("Added: {}", cid);



    // Retrieve content

    if let Some(block) = node.get(&cid).await? {

        println!("Retrieved: {:?}", block.data());

    }



    // Semantic search

    if node.is_semantic_enabled() {

        let embedding = vec![0.1, 0.2, 0.3]; // Your embedding

        node.index_content(&cid, &embedding).await?;



        let results = node.search_similar(&embedding, 10).await?;

        for result in results {

            println!("Found: {} (score: {})", result.cid, result.score);

        }

    }



    Ok(())

}

```



---



## 🌐 HTTP API



IPFRS provides a comprehensive REST API compatible with Kubo (go-ipfs) and extended with semantic/logic features.



### Block Operations



```bash

# Add file

curl -X POST -F file=@document.pdf \

  http://localhost:8080/api/v0/add



# Get block

curl http://localhost:8080/ipfs/bafybeig...



# Block statistics

curl -X POST -d '{"arg":"bafybeig..."}' \

  http://localhost:8080/api/v0/block/stat

```



### DAG Operations



```bash

# Store DAG node

curl -X POST --data-binary @dag.cbor \

  http://localhost:8080/api/v0/dag/put



# Resolve IPLD path

curl -X POST -d '{"arg":"/ipfs/Qm.../path/to/data"}' \

  http://localhost:8080/api/v0/dag/resolve

```



### Semantic Search (NEW!)



```bash

# Index content with embedding

curl -X POST -H "Content-Type: application/json" \

  -d '{

    "cid": "bafybeig...",

    "embedding": [0.1, 0.2, ..., 0.768]

  }' \

  http://localhost:8080/api/v0/semantic/index



# Search similar content

curl -X POST -H "Content-Type: application/json" \

  -d '{

    "query": [0.15, 0.25, ...],

    "k": 10,

    "filter": {"min_score": 0.8}

  }' \

  http://localhost:8080/api/v0/semantic/search



# Get statistics

curl http://localhost:8080/api/v0/semantic/stats

```



### Logic Programming (NEW!)



```bash

# Store logical term

curl -X POST -H "Content-Type: application/json" \

  -d '{"term": {"Variable": "X"}}' \

  http://localhost:8080/api/v0/logic/term



# Store inference rule

curl -X POST -H "Content-Type: application/json" \

  -d '{

    "rule": {

      "head": {"name": "ancestor", "args": [...]},

      "body": [...]

    }

  }' \

  http://localhost:8080/api/v0/logic/rule



# Retrieve term

curl http://localhost:8080/api/v0/logic/term/bafybeig...

```



**Complete API Reference:** [See HTTP API docs](#http-api-reference)



---



## 🏗️ Architecture



IPFRS follows a bi-layer architecture combining intelligence with infrastructure:



### Logical Layer (The Brain)

- **Semantic Router**: HNSW vector search with LRU query caching

- **TensorLogic Store**: Content-addressed logic programming



### Physical Layer (The Body)

- **Block Storage**: Sled embedded database with content addressing

- **Zero-Copy I/O**: Apache Arrow integration (planned)

- **Network Stack**: libp2p with QUIC transport (planned for 0.2.0)



```

┌─────────────────────────────────────┐

│      Application Layer              │

│   (Your Code / HTTP Clients)        │

└──────────────┬──────────────────────┘

               │

┌──────────────┴──────────────────────┐

│         Node API (Rust)             │

│  ┌──────────┐  ┌────────────────┐   │

│  │ Semantic │  │  TensorLogic   │   │

│  │  Router  │  │     Store      │   │

│  │  (HNSW)  │  │   (Logic IR)   │   │

│  └──────────┘  └────────────────┘   │

└──────────────┬──────────────────────┘

               │

┌──────────────┴──────────────────────┐

│       Block Storage (Sled)          │

│   Content-Addressed Blocks (CID)   │

└─────────────────────────────────────┘

```



---



## 📚 Project Structure



```

ipfrs/

├── Cargo.toml                 # Workspace manifest

├── crates/

│   ├── ipfrs-core/            # Core types (Block, CID, Error, IPLD)

│   ├── ipfrs-storage/         # Block storage (Sled), caching

│   ├── ipfrs-semantic/        # Semantic router, HNSW

│   ├── ipfrs-tensorlogic/     # TensorLogic store, logic IR

│   ├── ipfrs-interface/       # HTTP gateway, zero-copy interface

│   ├── ipfrs-network/         # libp2p networking (0.2.0)

│   ├── ipfrs-transport/       # TensorSwap, Bitswap (0.2.0)

│   ├── ipfrs/                 # Main library (unified API)

│   └── ipfrs-cli/             # Command-line interface

└── README.md

```



---



## ✨ Key Features



### 1. Content-Addressed Storage ✅

- Immutable blocks identified by CID (Content Identifier)

- Sled embedded database for persistence

- DAG operations with IPLD support

- Directory tree handling



### 2. Semantic Search ✅

- HNSW (Hierarchical Navigable Small World) index

- k-NN similarity search with configurable distance metrics

- Query result caching (LRU)

- Hybrid filtered search (by score, prefix, etc.)



### 3. Logic Programming ✅

- Content-addressed terms, predicates, and rules

- JSON serialization for portability

- Foundation for distributed reasoning (0.2.0)

- Compatible with TensorLogic IR



### 4. Comprehensive Observability ✅

- Storage statistics (block count, total size)

- Semantic index stats (vectors, dimension, cache)

- TensorLogic statistics

- HTTP API monitoring endpoints



### 5. HTTP Gateway ✅

- 20 REST API endpoints

- Kubo (go-ipfs) compatibility

- HTTP 206 range request support

- JSON responses throughout



---



## 🎓 Examples



### Example 1: Basic File Storage



```rust

use ipfrs::{Node, NodeConfig};



#[tokio::main]

async fn main() -> ipfrs::Result<()> {

    let mut node = Node::new(NodeConfig::default())?;

    node.start().await?;



    // Add a file

    let cid = node.add_file("./document.pdf").await?;

    println!("Stored as: {}", cid);



    // Retrieve it

    node.get_to_file(&cid, "./recovered.pdf").await?;

    println!("Retrieved successfully!");



    Ok(())

}

```



### Example 2: Semantic Document Search



```rust

use ipfrs::{Node, NodeConfig};

use ipfrs_semantic::RouterConfig;



#[tokio::main]

async fn main() -> ipfrs::Result<()> {

    let mut config = NodeConfig::default();

    config.semantic_config = Some(RouterConfig {

        dimension: 768,

        max_elements: 100_000,

        ..Default::default()

    });



    let mut node = Node::new(config)?;

    node.start().await?;



    // Add documents with embeddings

    let doc1_cid = node.add_bytes(b"AI research paper").await?;

    let doc1_embedding = get_embedding("AI research paper"); // Your embedding function

    node.index_content(&doc1_cid, &doc1_embedding).await?;



    // Search for similar documents

    let query_embedding = get_embedding("machine learning");

    let results = node.search_similar(&query_embedding, 5).await?;



    for result in results {

        println!("Found: {} (similarity: {:.2})", result.cid, result.score);

    }



    Ok(())

}



fn get_embedding(text: &str) -> Vec {

    // Use your favorite embedding model (BERT, Sentence Transformers, etc.)

    vec![0.1; 768] // Placeholder

}

```



### Example 3: Logic Programming



```rust

use ipfrs::{Node, NodeConfig};

use ipfrs_tensorlogic::{Term, Predicate, Rule};



#[tokio::main]

async fn main() -> ipfrs::Result<()> {

    let mut node = Node::new(NodeConfig::default())?;

    node.start().await?;



    // Store a term

    let term = Term::Variable("X".to_string());

    let term_cid = node.put_term(&term).await?;

    println!("Term stored: {}", term_cid);



    // Store a predicate: parent(alice, bob)

    let predicate = Predicate {

        name: "parent".to_string(),

        args: vec![

            Term::Constant("alice".to_string()),

            Term::Constant("bob".to_string()),

        ],

    };

    let pred_cid = node.store_predicate(&predicate).await?;

    println!("Predicate stored: {}", pred_cid);



    // Retrieve it

    if let Some(retrieved) = node.get_predicate(&pred_cid).await? {

        println!("Retrieved: {:?}", retrieved);

    }



    Ok(())

}

```



### Example 4: DAG Operations



```rust

use ipfrs::{Node, NodeConfig};

use ipfrs_core::Ipld;

use std::collections::BTreeMap;



#[tokio::main]

async fn main() -> ipfrs::Result<()> {

    let mut node = Node::new(NodeConfig::default())?;

    node.start().await?;



    // Create a DAG structure

    let mut metadata = BTreeMap::new();

    metadata.insert("title".to_string(), Ipld::String("My Document".to_string()));

    metadata.insert("author".to_string(), Ipld::String("Alice".to_string()));



    let dag_node = Ipld::Map(metadata);

    let cid = node.dag_put(dag_node).await?;

    println!("DAG node stored: {}", cid);



    // Retrieve DAG node

    if let Some(node_data) = node.dag_get(&cid).await? {

        println!("Retrieved: {:?}", node_data);

    }



    // Resolve path

    if let Some(resolved_cid) = node.dag_resolve(&cid, "/title").await? {

        println!("Resolved path to: {}", resolved_cid);

    }



    Ok(())

}

```



---



## 🧪 Testing



```bash

# Run all tests

cargo test



# Run with logging

RUST_LOG=debug cargo test



# Test specific crate

cargo test -p ipfrs-core



# Integration tests

cargo test --test integration

```



---



## 📊 Performance



### Benchmarks (0.1.0)



| Operation | Time | Throughput |

|-----------|------|------------|

| Block put | ~50µs | 20,000 ops/sec |

| Block get | ~30µs | 33,000 ops/sec |

| DAG put | ~80µs | 12,500 ops/sec |

| Semantic search (k=10) | ~1ms | 1,000 queries/sec |

| HNSW insertion | ~100µs | 10,000 inserts/sec |



*Tested on: AMD Ryzen 9 5900X, NVMe SSD*



### Scalability



- **Storage**: Limited only by disk space

- **HNSW Index**: Scales to millions of vectors

- **Concurrent Operations**: Async I/O with Tokio

- **Memory**: ~50MB base + index data



---



## 🗺️ Roadmap



### ✅ Version 0.1.0 "Foundation" (Current)

- Content-addressed storage with DAG support

- Semantic search (HNSW)

- Logic programming (TensorLogic)

- HTTP API (20 endpoints)

- CLI (13 commands)

- Comprehensive observability



### 🚧 Version 0.2.0 "Network" (Next - ETA: +1 month)

- libp2p networking integration

- DHT bootstrap and peer discovery

- Distributed inference engine

- Network CLI commands

- Circuit relay support



### 📅 Version 0.3.0 "Performance" (+2 months)

- Persistent HNSW index

- Performance optimizations

- Advanced query features

- Production hardening



### 📅 Version 0.4.0 "Ecosystem" (+3 months)

- Language bindings (Python, JavaScript)

- GraphQL API

- Enhanced tooling

- Monitoring & metrics



### 📅 Version 1.0.0 "Stable" (+6 months)

- API stability guarantees

- Comprehensive documentation

- Production deployments

- Security audit complete



---



## 🤝 Contributing



IPFRS is part of the COOLJAPAN ecosystem. Contributions are welcome!



### Development Setup



```bash

git clone https://github.com/yourusername/ipfrs.git

cd ipfrs

cargo build

cargo test

```



### Guidelines



- Follow Rust style guidelines (rustfmt)

- Maintain zero warnings policy

- Add tests for new features

- Update documentation



---



## 📄 License



MIT OR Apache-2.0



---



## 🙏 Acknowledgments



- **IPFS** - Content-addressed foundation

- **libp2p** - Networking stack

- **Sled** - Embedded database

- **HNSW** - Vector search algorithm

- **TensorLogic** - Reasoning framework



---



## 📞 Support



- **Issues**: [GitHub Issues](https://github.com/yourusername/ipfrs/issues)

- **Discussions**: [GitHub Discussions](https://github.com/yourusername/ipfrs/discussions)

- **Documentation**: [docs.rs/ipfrs](https://docs.rs/ipfrs)



---



## 🔖 HTTP API Reference



### Block Operations

- `POST /api/v0/add` - Upload file

- `POST /api/v0/block/get` - Get raw block

- `POST /api/v0/block/put` - Store raw block

- `POST /api/v0/block/stat` - Block statistics

- `POST /api/v0/cat` - Output content

- `GET /ipfs/{cid}` - Retrieve content (HTTP 206 support)



### DAG Operations

- `POST /api/v0/dag/put` - Store DAG node

- `POST /api/v0/dag/get` - Retrieve DAG node

- `POST /api/v0/dag/resolve` - Resolve IPLD path



### Semantic Search

- `POST /api/v0/semantic/index` - Index content

- `POST /api/v0/semantic/search` - Search similar

- `GET /api/v0/semantic/stats` - Index statistics



### Logic Programming

- `POST /api/v0/logic/term` - Store term

- `GET /api/v0/logic/term/{cid}` - Retrieve term

- `POST /api/v0/logic/predicate` - Store predicate

- `POST /api/v0/logic/rule` - Store rule

- `GET /api/v0/logic/stats` - Logic statistics



### Utility

- `GET /health` - Health check

- `POST /api/v0/version` - Version information



---



## 💭 Philosophy



> "IPFRS is not just a storage reinvention. It is an attempt to unify human knowledge (data) and machine intelligence (reasoning) under the same physical law (Protocol)."



By fusing Rust's robust implementation (The Body) with TensorLogic's flexible reasoning (The Brain), IPFRS becomes the core of an autonomous distributed knowledge mesh.



---



**Status**: v0.1.0 - Production Ready (Local-First)



🎉 **First stable release! Ready for local development and testing.**