https://github.com/hupe1980/dagster-pipes-go

Go library for integrating with Dagster Pipes, enabling seamless communication between external processes and Dagster workflows. It provides a set of utilities for managing context, logging, reporting asset materializations, and more.
https://github.com/hupe1980/dagster-pipes-go

dagster data-engineering golang orchestration pipes

Last synced: 6 days ago
JSON representation

Go library for integrating with Dagster Pipes, enabling seamless communication between external processes and Dagster workflows. It provides a set of utilities for managing context, logging, reporting asset materializations, and more.

• Host: GitHub
• URL: https://github.com/hupe1980/dagster-pipes-go
• Owner: hupe1980
• License: mit
• Created: 2024-12-10T20:13:15.000Z (10 months ago)
• Default Branch: main
• Last Pushed: 2024-12-21T12:00:56.000Z (10 months ago)
• Last Synced: 2025-10-11T13:46:43.031Z (6 days ago)
• Topics: dagster, data-engineering, golang, orchestration, pipes
• Language: Go
• Homepage:
• Size: 103 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# dagster-pipes-go

![Build Status](https://github.com/hupe1980/dagster-pipes-go/workflows/Build/badge.svg) 

[![Go Reference](https://pkg.go.dev/badge/github.com/hupe1980/dagster-pipes-go.svg)](https://pkg.go.dev/github.com/hupe1980/dagster-pipes-go)

[![goreportcard](https://goreportcard.com/badge/github.com/hupe1980/dagster-pipes-go)](https://goreportcard.com/report/github.com/hupe1980/dagster-pipes-go)

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

`dagster-pipes-go` is a Go library for integrating with [Dagster Pipes](https://docs.dagster.io/concepts/pipes), enabling seamless communication between external processes and Dagster workflows. It provides a set of utilities for managing context, logging, reporting asset materializations, and more.

## Features

- **Context Management**: Open and manage Pipes contexts.

- **Message Handling**: Send and receive messages in a structured format.

- **Asset Reporting**: Report asset materializations and checks.

- **Custom Messaging**: Send custom messages for advanced use cases.

- **Error Handling**: Report exceptions gracefully.

## Installation

To install the library, use `go get`:

```sh

go get github.com/hupe1980/dagster-pipes-go

```

## How to use

```golang

package main

import (

	"log"

	dagsterpipes "github.com/hupe1980/dagster-pipes-go"

)

func main() {

	session, err := dagsterpipes.New[map[string]any]()

	if err != nil {

		log.Fatalf("Error creating dagster pipes session: %v", err)

	}

	defer session.Close()

	if err := session.Run(func(context *dagsterpipes.Context[map[string]any]) error {

		if err := context.ReportAssetMaterialization(&dagsterpipes.AssetMaterialization{

			AssetKey:    "asset",

			DataVersion: "1.0",

			Metadata: map[string]any{

				"foo": "bar",

			},

		}); err != nil {

			return err

		}

		return nil

	}); err != nil {

		log.Fatalf("Error running dagster pipes session: %v", err)

	}

}

```

## Contributing

Contributions are welcome! If you find bugs or want to suggest features, please open an issue or submit a pull request.

## License

This project is licensed under the MIT License. See the [LICENSE](./LICENSE) file for details.