https://github.com/usedatabrew/open_ai_benthos_processor
Open AI processor for Benthos
https://github.com/usedatabrew/open_ai_benthos_processor
benthos benthos-plugin data-engineering data-ops etl etl-pipeline event-sourcing golang-library openai stream-processor streaming-data
Last synced: 19 days ago
JSON representation
Open AI processor for Benthos
- Host: GitHub
- URL: https://github.com/usedatabrew/open_ai_benthos_processor
- Owner: usedatabrew
- Created: 2023-11-01T14:37:58.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2023-11-05T12:19:02.000Z (about 2 years ago)
- Last Synced: 2024-04-19T21:00:44.323Z (over 1 year ago)
- Topics: benthos, benthos-plugin, data-engineering, data-ops, etl, etl-pipeline, event-sourcing, golang-library, openai, stream-processor, streaming-data
- Language: Go
- Homepage: https://databrew.tech
- Size: 35.2 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Open AI Processor for Benthos
Benthos processors are functions applied to messages passing through a pipeline. The function signature allows a processor to mutate or drop messages depending on the content of the message
Welcome to the Open AI Processor for Benthos! This processor allows you to enrich the context of your messages with requests to Open AI.
## Getting Started
To get started you have to run benthos with custom processor. Since this processor is not adopted by benthos itself
you have to create a new benthos build with plugin registered
```go
package main
import (
"context"
"github.com/benthosdev/benthos/v4/public/service"
_ "github.com/benthosdev/benthos/v4/public/components/all"
// import open_ai processor
_ "github.com/usedatabrew/open_ai_benthos_processor"
)
func main() {
// here we initialize benthos
service.RunCLI(context.Background())
}
```
### Create benthos configuration with processor
```yaml
pipeline:
label: open_ai_processor
processors:
- open_ai:
source_field: "comment"
target_field: "ai_processed_comment"
prompt: "Check the field for hateful speech"
api_key: ""
api_endpoint: "" ## Optional for azure driver
driver: "azure" ## azure or openai
model: "gpt-3.5-turbo-instruct"
```