https://github.com/fuellabs/data-systems
Fuel data systems related projects
https://github.com/fuellabs/data-systems
blockchain data-systems fuel indexer
Last synced: 3 months ago
JSON representation
Fuel data systems related projects
- Host: GitHub
- URL: https://github.com/fuellabs/data-systems
- Owner: FuelLabs
- License: apache-2.0
- Created: 2024-05-28T13:11:17.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-09-10T15:09:08.000Z (4 months ago)
- Last Synced: 2025-10-10T22:25:51.359Z (3 months ago)
- Topics: blockchain, data-systems, fuel, indexer
- Language: Rust
- Homepage:
- Size: 4.1 MB
- Stars: 18
- Watchers: 10
- Forks: 17
- Open Issues: 40
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
Fuel Data Systems
Official data streaming libraries and tools for the Fuel Network.
📦 Crates
🐛 Report Bug
✨ Request Feature
## 📝 About The Project
Fuel Data Systems is a comprehensive suite of libraries and tools designed to enable real-time and historical data streaming and processing from the Fuel Network. This repository houses the official data streaming ecosystem, offering developers a powerful and flexible API to interact with Fuel Network data in real-time.
With Fuel Data Systems, developers can build sophisticated applications that leverage the full potential of the Fuel Network's data, from simple block explorers to complex analytics engines and trading systems.
### Getting Started
To get started with local development and syncing blocks locally, see the [Syncing Blocks Locally](services/publisher/README.md#syncing-blocks-locally) section in the Publisher README.
The [CONTRIBUTING.md](CONTRIBUTING.md) file contains detailed information about setting up your development environment and contributing to this project.
## 🚀 Features
- Real-time streaming of Fuel blockchain data
- Historical streaming of Fuel blockchain data
- Support for various Fuel-specific data types
- Customizable filters for targeted data retrieval
- Flexible deliver policies for historical and real-time data
- Seamless integration with other Fuel ecosystem tools
## 📚 Documentation
To check this stream service up and running, visit:
- [https://stream.fuel.network](https://stream.fuel.network)
For the REST API documentation (Mainnet), visit:
- [https://api-rest-mainnet.fuel.network/swagger-ui](https://api-rest-mainnet.fuel.network/swagger-ui)
For codebase documentation, see the README files in the relevant directories:
- [Crates Documentation](crates/)
- [Services Documentation](services/)
- [Cluster Documentation](cluster/)
## 💻 Fuel Stream Rust SDK
The [fuel-streams](crates/fuel-streams/README.md) library provides a simple Rust SDK for connecting to and consuming data from the Fuel blockchain.
### Installing
```sh
cargo add fuel-streams futures tokio
```
### Basic Usage Example
```rust
use fuel_streams::prelude::*;
use futures::StreamExt;
#[tokio::main]
async fn main() -> Result<(), Box> {
// Create a client and establish connection
let mut client = Client::new(FuelNetwork::Local).with_api_key("your_key");
let mut connection = client.connect().await?;
println!("Listening for blocks...");
// Choose which subjects you want to filter
let subjects = vec![BlocksSubject::new().into()];
// Subscribe to blocks with new deliver policy
let mut stream = connection
.subscribe(subjects, DeliverPolicy::New)
.await?;
while let Some(block) = stream.next().await {
println!("Received block: {:?}", block);
}
Ok(())
}
```
### Filtering Transactions
```rust
use fuel_streams::prelude::*;
use futures::StreamExt;
#[tokio::main]
async fn main() -> Result<(), Box> {
let mut client = Client::new(FuelNetwork::Local).with_api_key("your_key");
let mut connection = client.connect().await?;
// Create a subject for script transactions
let subjects = vec![
TransactionsSubject::new()
.with_tx_type(Some(TransactionType::Script))
.into()
];
// Subscribe to the filtered transaction stream
let mut stream = connection
.subscribe(subjects, DeliverPolicy::New)
.await?;
while let Some(transaction) = stream.next().await {
println!("Received transaction: {:?}", transaction);
}
Ok(())
}
```
For more examples and detailed documentation, see the [fuel-streams crate documentation](https://docs.rs/fuel-streams/).
## 📑 Architecture Components
### Services
| Service | Description |
| ------------------------------------------------- | ---------------------------------------------------------------- |
| [API Service](services/api/README.md) | REST API for retrieving blockchain data from an indexed database |
| [Consumer Service](services/consumer/README.md) | Processes and stores blockchain data in a database |
| [Publisher Service](services/publisher/README.md) | Subscribes to new blocks and publishes them to a message broker |
| [WebSocket Service](services/webserver/README.md) | Real-time data streaming via WebSocket connections |
| [Dune Service](services/dune/README.md) | Processes blockchain data for analytics with Dune |
### Libraries
| Library | Description |
| ------------------------------------------------- | ------------------------------------------------------------- |
| [fuel-streams](crates/fuel-streams/README.md) | Main library for streaming Fuel blockchain data |
| [web-utils](crates/web-utils/README.md) | Web utilities for building web services in the Fuel ecosystem |
| [domains](crates/domains/README.md) | Domain models and database infrastructure |
| [subject](crates/subject/README.md) | Subject derive macro for type-safe subject definitions |
| [types](crates/types/README.md) | Core type definitions and utilities |
| [core](crates/core/README.md) | Core functionalities and shared components |
| [data-parser](crates/data-parser/README.md) | Parser for Fuel blockchain data |
| [message-broker](crates/message-broker/README.md) | Message broker implementation for event publishing |
### Deployment and Infrastructure
| Component | Description |
| ---------------------------------- | ---------------------------------------------- |
| [Cluster](cluster/README.md) | Deployment infrastructure and configuration |
| [Docker](cluster/docker/README.md) | Docker configuration for local development |
| [Charts](cluster/charts/README.md) | Helm charts for Kubernetes deployment |
| [Scripts](scripts/README.md) | Utility scripts for development and deployment |
## 🛠️ Development
For local development:
1. **Setup Environment**:
```bash
make create-env
make setup
```
2. **Start Required Services**:
```bash
make start-docker
make setup-db
```
3. **Run Services**:
- API Service: `make run-api`
- Publisher Service: `make run-publisher`
- Consumer Service: `make run-consumer`
- WebSocket Service: `make run-webserver`
- Dune Service: `make run-dune`
See the [CONTRIBUTING.md](CONTRIBUTING.md) for more detailed development instructions.
## 💪 Contributing
We welcome contributions to Fuel Streams! Please check our [contributing guidelines](CONTRIBUTING.md) for more information on how to get started.
## 📜 License
This project is licensed under the `Apache-2.0` license. See [`LICENSE`](./LICENSE) for more information.