{"id":29482618,"url":"https://github.com/ConduitIO/conduit","last_synced_at":"2025-07-15T02:02:09.498Z","repository":{"id":36952115,"uuid":"446826327","full_name":"ConduitIO/conduit","owner":"ConduitIO","description":"Conduit streams data between data stores. Kafka Connect replacement. No JVM required.","archived":false,"fork":false,"pushed_at":"2025-07-09T10:04:42.000Z","size":13700,"stargazers_count":535,"open_issues_count":120,"forks_count":54,"subscribers_count":17,"default_branch":"main","last_synced_at":"2025-07-09T22:07:15.417Z","etag":null,"topics":["conduit","data-engineering","data-integration","data-pipeline","data-stream","etl","go","kafka","kafkaconnect"],"latest_commit_sha":null,"homepage":"https://conduit.io","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ConduitIO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-01-11T13:12:14.000Z","updated_at":"2025-07-09T13:37:30.000Z","dependencies_parsed_at":"2023-10-15T00:46:48.694Z","dependency_job_id":"ccfceb7a-d507-4b81-9027-f80b6e2846aa","html_url":"https://github.com/ConduitIO/conduit","commit_stats":null,"previous_names":[],"tags_count":403,"template":false,"template_full_name":null,"purl":"pkg:github/ConduitIO/conduit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ConduitIO","download_url":"https://codeload.github.com/ConduitIO/conduit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265386079,"owners_count":23756747,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["conduit","data-engineering","data-integration","data-pipeline","data-stream","etl","go","kafka","kafkaconnect"],"created_at":"2025-07-15T02:01:17.646Z","updated_at":"2025-07-15T02:02:09.451Z","avatar_url":"https://github.com/ConduitIO.png","language":"Go","readme":"# Conduit\n\n![Logo](https://raw.githubusercontent.com/ConduitIO/.github/main/images/conduit-logo-outlined.svg)\n\n_Data Integration for Production Data Stores. :dizzy:_\n\n![scarf pixel](https://static.scarf.sh/a.png?x-pxid=ae2e6c1b-5c87-4c4f-80bd-dd598b2e3c3b)\n[![License](https://img.shields.io/badge/license-Apache%202-blue)](https://github.com/ConduitIO/conduit/blob/main/LICENSE.md)\n[![Test](https://github.com/ConduitIO/conduit/actions/workflows/test.yml/badge.svg)](https://github.com/ConduitIO/conduit/actions/workflows/test.yml)\n[![Go Report Card](https://goreportcard.com/badge/github.com/conduitio/conduit)](https://goreportcard.com/report/github.com/conduitio/conduit)\n[![Discord](https://img.shields.io/discord/828680256877363200?label=discord\u0026logo=discord)](https://discord.meroxa.com)\n[![Twitter](https://img.shields.io/static/v1?label=X/Twitter\u0026message=Follow\u0026color=1DA1F2\u0026logo=twitter\u0026logoColor=white)](https://x.com/ConduitIO)\n[![Go Reference](https://pkg.go.dev/badge/github.com/conduitio/conduit.svg)](https://pkg.go.dev/github.com/conduitio/conduit)\n[![Conduit docs](https://img.shields.io/badge/conduit-docs-blue)](https://conduit.io/docs/getting-started)\n[![API docs](https://img.shields.io/badge/HTTP_API-docs-blue)](https://docs.conduit.io/api)\n\n## Overview\n\nConduit is a data streaming tool written in Go. It aims to provide the best user\nexperience for building and running real-time data pipelines. Conduit comes with\ncommon connectors, processors and observability data out of the box.\n\nConduit pipelines are built out of simple building blocks which run in their own\ngoroutines and are connected using Go channels. This makes Conduit pipelines\nincredibly performant on multi-core machines. Conduit guarantees the order of\nreceived records won't change, it also takes care of consistency by propagating\nacknowledgements to the start of the pipeline only when a record is successfully\nprocessed on all destinations.\n\nConduit connectors are plugins that communicate with Conduit via a gRPC\ninterface. This means that plugins can be written in any language as long as\nthey conform to the required interface.\n\nConduit was created and open-sourced by [Meroxa](https://meroxa.io).\n\n- [Quick start](#quick-start)\n- [Installation guide](#installation-guide)\n- [Configuring Conduit](#configuring-conduit)\n- [Storage](#storage)\n- [Connectors](#connectors)\n- [Processors](#processors)\n- [API](#api)\n- [Documentation](#documentation)\n- [Contributing](#contributing)\n\n## Quick start\n\n\u003chttps://conduit.io/docs/getting-started\u003e\n\n## Installation guide\n\n### Download binary and run\n\nDownload a pre-built binary from\nthe [latest release](https://github.com/conduitio/conduit/releases/latest) and\nsimply run it!\n\n```sh\n./conduit run\n```\n\nOnce you see that the service is running, the configured pipeline should start\nprocessing records automatically. You can also interact with\nthe [Conduit API](#api) directly, we recommend navigating\nto `http://localhost:8080/openapi` and exploring the HTTP API through Swagger\nUI.\n\nConduit can be configured through command line parameters. To view the full list\nof available options, run `./conduit run --help` or see\n[configuring Conduit](#configuring-conduit).\n\n### Homebrew\n\nMake sure you have [homebrew](https://brew.sh/) installed on your machine, then run:\n\n```sh\nbrew update\nbrew install conduit\n```\n\n### Debian\n\nDownload the right `.deb` file for your machine architecture from the\n[latest release](https://github.com/conduitio/conduit/releases/latest), then run:\n\n```sh\ndpkg -i conduit_0.14.0_Linux_x86_64.deb\n```\n\n### RPM\n\nDownload the right `.rpm` file for your machine architecture from the\n[latest release](https://github.com/conduitio/conduit/releases/latest), then run:\n\n```sh\nrpm -i conduit_0.14.0_Linux_x86_64.rpm\n```\n\n### Build from source\n\nRequirements:\n\n- [Go](https://golang.org/)\n- [Make](https://www.gnu.org/software/make/)\n\n```shell\ngit clone git@github.com:ConduitIO/conduit.git\ncd conduit\nmake\n./conduit run\n```\n\n### Docker\n\nOur Docker images are hosted on GitHub's Container Registry. To run the latest\nConduit version, you should run the following command:\n\n```sh\ndocker run -p 8080:8080 conduit.docker.scarf.sh/conduitio/conduit:latest\n```\n\n## Configuring Conduit\n\nConduit accepts CLI flags, environment variables and a configuration file to\nconfigure its behavior. Each CLI flag has a corresponding environment variable\nand a corresponding field in the configuration file. Conduit uses the value for\neach configuration option based on the following priorities:\n\n- **CLI flags (highest priority)** - if a CLI flag is provided it will always be\n  respected, regardless of the environment variable or configuration file. To\n  see a full list of available flags run `conduit run --help`.\n- **Environment variables (lower priority)** - an environment variable is only\n  used if no CLI flag is provided for the same option. Environment variables\n  have the prefix `CONDUIT` and contain underscores instead of dots and\n  hyphens (e.g. the flag `-db.postgres.connection-string` corresponds to\n  `CONDUIT_DB_POSTGRES_CONNECTION_STRING`).\n- **Configuration file (lowest priority)** - By default, Conduit loads a\n  configuration file named `conduit.yaml` located in the same directory as the\n  Conduit binary. You can customize the directory path to this file using the\n  CLI flag `--config.path`. The configuration file is optional, as any value\n  specified within it can be overridden by an environment variable or a CLI\n  flag.\n\n  The file must be a YAML document, and keys can be hierarchically structured\n  using `.`. For example:\n\n  ```yaml\n  db:\n    type: postgres # corresponds to flag -db.type and env variable CONDUIT_DB_TYPE\n    postgres:\n      connection-string: postgres://localhost:5432/conduitdb # -db.postgres.connection-string or CONDUIT_DB_POSTGRES_CONNECTION_STRING\n  ```\n\n  To generate a configuration file with default values, use:\n  `conduit init --path \u003cdirectory where conduit.yaml will be created\u003e`.\n\nThis parsing configuration is provided thanks to our own CLI\nlibrary [ecdysis](https://github.com/conduitio/ecdysis), which builds on top\nof [Cobra](https://github.com/spf13/cobra) and\nuses [Viper](https://github.com/spf13/viper) under the hood.\n\n## Storage\n\nConduit's own data (information about pipelines, connectors, etc.) can be stored\nin the following databases:\n\n- BadgerDB (default)\n- PostgreSQL\n- SQLite\n\nIt's also possible to store all the data in memory, which is sometimes useful\nfor development purposes.\n\nThe database type used can be configured with the `db.type` parameter (through\nany of the [configuration](#configuring-conduit) options in Conduit).\nFor example, the CLI flag to use a PostgreSQL database with Conduit is as\nfollows: `-db.type=postgres`.\n\nChanging database parameters (e.g. the PostgreSQL connection string) is done\nthrough parameters of the following form: `db.\u003cdb type\u003e.\u003cparameter name\u003e`. For\nexample, the CLI flag to use a PostgreSQL instance listening on `localhost:5432`\nwould be: `-db.postgres.connection-string=postgres://localhost:5432/conduitdb`.\n\nThe full example in our case would be:\n\n```shell\n./conduit run -db.type=postgres -db.postgres.connection-string=\"postgresql://localhost:5432/conduitdb\"\n```\n\n## Connectors\n\nFor the full list of available connectors, see\nthe [Connector List](https://conduit.io/docs/using/connectors/list). If\nthere's a connector that you're looking for that isn't available in Conduit,\nplease file an [issue](https://github.com/ConduitIO/conduit/issues/new?assignees=\u0026labels=triage\u0026template=3-connector-request.yml\u0026title=Connector%3A+%3Cresource%3E+%5BSource%2FDestination%5D)\n.\n\nConduit loads standalone connectors at startup. The connector binaries need to\nbe placed in the `connectors` directory relative to the Conduit binary so\nConduit can find them. Alternatively, the path to the standalone connectors can\nbe adjusted using the CLI flag `-connectors.path`.\n\nConduit ships with a number of built-in connectors:\n\n- [File connector](https://github.com/ConduitIO/conduit-connector-file) provides\n  a source/destination to read/write a local file (useful for quickly trying out\n  Conduit without additional setup).\n- [Kafka connector](https://github.com/ConduitIO/conduit-connector-kafka)\n  provides a source/destination for Apache Kafka.\n- [Postgres connector](https://github.com/ConduitIO/conduit-connector-postgres)\n  provides a source/destination for PostgreSQL.\n- [S3 connector](https://github.com/ConduitIO/conduit-connector-s3) provides a\n  source/destination for AWS S3.\n- [Generator connector](https://github.com/ConduitIO/conduit-connector-generator)\n  provides a source which generates random data (useful for testing).\n- [Log connector](https://github.com/ConduitIO/conduit-connector-log)\n  provides a destination which logs all records (useful for testing).\n  \nAdditionally, we have prepared\na [Kafka Connect wrapper](https://github.com/conduitio/conduit-kafka-connect-wrapper)\nthat allows you to run any Apache Kafka Connect connector as part of a Conduit\npipeline.\n\nIf you are interested in writing a connector yourself, have a look at\nour [Go Connector SDK](https://github.com/ConduitIO/conduit-connector-sdk).\nSince standalone connectors communicate with Conduit through gRPC they can be\nwritten in virtually any programming language, as long as the connector follows\nthe [Conduit Connector Protocol](https://github.com/ConduitIO/conduit-connector-protocol).\n\n## Processors\n\nA processor is a component that operates on a single record that flows through a\npipeline. It can either change the record (i.e. **transform** it) or **filter**\nit out based on some criteria.\n\nConduit provides a number of built-in processors, which can be used to\nmanipulate fields, send requests to HTTP endpoints, and more,\ncheck [built-in processors](https://conduit.io/docs/using/processors/builtin/)\nfor the list of built-in processors and documentations.\n\nConduit also provides the ability to write your\nown [standalone processor](https://conduit.io/docs/developing/processors/building),\nor you can use the built-in processor [`custom.javascript`](https://conduit.io/docs/using/processors/builtin/custom.javascript)\nto write custom processors in JavaScript.\n\nMore detailed information as well as examples can be found in\nthe [Processors documentation](https://conduit.io/docs/using/processors/getting-started).\n\n## API\n\nConduit exposes a gRPC API and an HTTP API.\n\nThe gRPC API is by default running on port 8084. You can define a custom address\nusing the CLI flag `-api.grpc.address`. To learn more about the gRPC API please have\na look at\nthe [protobuf file](https://github.com/ConduitIO/conduit/blob/main/proto/api/v1/api.proto).\n\nThe HTTP API is by default running on port 8080. You can define a custom address\nusing the CLI flag `-api.http.address`. It is generated\nusing [gRPC gateway](https://github.com/grpc-ecosystem/grpc-gateway) and is thus\nproviding the same functionality as the gRPC API. To learn more about the HTTP\nAPI please have a look at the [API documentation](https://www.conduit.io/api),\n[OpenAPI definition](https://github.com/ConduitIO/conduit/blob/main/pkg/web/openapi/swagger-ui/api/v1/api.swagger.json)\nor run Conduit and navigate to `http://localhost:8080/openapi` to open\na [Swagger UI](https://github.com/swagger-api/swagger-ui) which makes it easy to\ntry it out.\n\n## Documentation\n\nTo learn more about how to use Conduit\nvisit [Conduit.io/docs](https://conduit.io/docs).\n\nIf you are interested in internals of Conduit we have prepared some technical\ndocumentation:\n\n- [Pipeline Semantics](https://conduit.io/docs/core-concepts/pipeline-semantics) explains the internals of how\n  a Conduit pipeline works.\n- [Pipeline Configuration Files](https://conduit.io/docs/using/pipelines/configuration-file)\n  explains how you can define pipelines using YAML files.\n- [Processors](https://conduit.io/docs/using/processors/getting-started) contains examples and more information about\n  Conduit processors.\n- [Conduit Architecture](https://conduit.io/docs/core-concepts/architecture)\n  will give you a high-level overview of Conduit.\n- [Conduit Metrics](docs/metrics.md)\n  provides more information about how Conduit exposes metrics.\n- [Conduit Package structure](docs/package_structure.md)\n  provides more information about the different packages in Conduit.\n\n## Contributing\n\nFor a complete guide to contributing to Conduit, see\nthe [contribution guide](https://github.com/ConduitIO/conduit/blob/master/CONTRIBUTING.md).\n\nWe welcome you to join the community and contribute to Conduit to make it\nbetter! When something does not work as intended please check if there is\nalready an [issue](https://github.com/ConduitIO/conduit/issues) that describes\nyour problem, otherwise\nplease [open an issue](https://github.com/ConduitIO/conduit/issues/new/choose)\nand let us know. When you are not sure how to do something\nplease [open a discussion](https://github.com/ConduitIO/conduit/discussions) or\nhit us up on [Discord](https://discord.meroxa.com).\n\nWe also value contributions in the form of pull requests. When opening a PR please\nensure:\n\n- You have followed\n  the [Code Guidelines](https://github.com/ConduitIO/conduit/blob/main/docs/code_guidelines.md).\n- There is no other [pull request](https://github.com/ConduitIO/conduit/pulls)\n  for the same update/change.\n- You have written unit tests.\n- You have made sure that the PR is of reasonable size and can be easily\n  reviewed.\n","funding_links":[],"categories":["Go"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FConduitIO%2Fconduit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FConduitIO%2Fconduit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FConduitIO%2Fconduit/lists"}