https://github.com/loggdme/kyro

Collection of utilities and examples for creating efficient data pipelines in go with parallel queues and, rate limitiers and much more.
https://github.com/loggdme/kyro

data package

Last synced: 5 months ago
JSON representation

Collection of utilities and examples for creating efficient data pipelines in go with parallel queues and, rate limitiers and much more.

• Host: GitHub
• URL: https://github.com/loggdme/kyro
• Owner: loggdme
• License: mit
• Created: 2025-05-05T13:57:40.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2025-12-13T21:07:19.000Z (7 months ago)
• Last Synced: 2025-12-15T01:58:27.109Z (6 months ago)
• Topics: data, package
• Language: Go
• Homepage:
• Size: 258 KB
• Stars: 0
• Watchers: 0
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# 🚀 Kyro

Welcome to the **Kyro** project! This repository contains a collection of utilities and examples for working with pipelines, parallel queues, and rate limiters in Go. It is designed to help you build efficient and scalable data processing workflows.

## 📦 Core Packages

You can find various examples demonstrating the usage of the core packages in the `examples` directory. Each example is self-contained and can be run independently.

### 1. **Pipeline** (`pkg/pipeline`)

The `pipeline` package provides utilities for creating and executing sequential and parallel workflows.

```go

result, err := pipeline.Execute(

    pipeline.InSequence(

        generateItems,

        stringLength,

        pipeline.InParallel(double, triple),

        add,

    ),

)

```

### 2. **Parallel Queue** (`pkg/pqueue`)

The `pqueue` package enables parallel processing of items with configurable workers, progress notifications, and error handling.

```go

limiter := kyro.NewRateLimiter(40, 40)

erroredItems, err := pqueue.NewParallelQueue[int](40).

    Enqueue(&ids).

    WithProgressNotifier(100, func(curr int, duration time.Duration, itemsPerSecond float64) {

        log.Printf("Processed %d items in %s (%.2f items/sec)", curr, duration, itemsPerSecond)

    }).

    OnProcessQueue(func(item int) error {

        limiter.Wait()

        

        if item%420 == 0 {

            return fmt.Errorf("simulated error for item %d", item)

        }

        

        return nil

    }).

    Done()

```

## 📄 License

This project is licensed under the MIT License. See the `LICENSE` file for details.