{"id":16630664,"url":"https://github.com/mrhamburg/airbyte.cdk.dotnet","last_synced_at":"2025-10-30T03:32:10.507Z","repository":{"id":45060942,"uuid":"435998611","full_name":"mrhamburg/airbyte.cdk.dotnet","owner":"mrhamburg","description":"Airbyte dotnet/c# Connector Development Kit","archived":false,"fork":false,"pushed_at":"2022-01-21T20:52:32.000Z","size":107,"stargazers_count":15,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-13T04:48:46.257Z","etag":null,"topics":["airbyte","airbyte-connectors"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mrhamburg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-12-07T19:13:30.000Z","updated_at":"2024-04-03T07:45:35.000Z","dependencies_parsed_at":"2022-09-05T05:01:00.157Z","dependency_job_id":null,"html_url":"https://github.com/mrhamburg/airbyte.cdk.dotnet","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrhamburg%2Fairbyte.cdk.dotnet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrhamburg%2Fairbyte.cdk.dotnet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrhamburg%2Fairbyte.cdk.dotnet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrhamburg%2Fairbyte.cdk.dotnet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mrhamburg","download_url":"https://codeload.github.com/mrhamburg/airbyte.cdk.dotnet/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238930130,"owners_count":19554122,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airbyte","airbyte-connectors"],"created_at":"2024-10-12T04:48:29.164Z","updated_at":"2025-10-30T03:32:05.226Z","avatar_url":"https://github.com/mrhamburg.png","language":"C#","funding_links":[],"categories":["Community Connector Developer Kits"],"sub_categories":[],"readme":"# Airbyte .NET CDK  \n\n [![CI](https://github.com/mrhamburg/airbyte.cdk.dotnet/actions/workflows/build.yml/badge.svg?query=branch%3Amain)](https://github.com/mrhamburg/airbyte.cdk.dotnet/actions/workflows/build.yml?query=branch%3Amain) [![CI](https://github.com/mrhamburg/airbyte.cdk.dotnet/actions/workflows/release.yml/badge.svg)](https://github.com/mrhamburg/airbyte.cdk.dotnet/actions/workflows/release.yml)  \n\n- [Quick Start](#quick-start)\n- [Components](#components)\n- [Concepts \u0026 Documentation](#concepts--documentation)\n- [Contributing - Connector](#contributing---connector)\n- [Contributing - CDK](#contributing---cdk)\n\nThe Airbyte Dotnet CDK is a framework for rapidly developing production-grade Airbyte connectors. The CDK currently offers helpers specific for creating Airbyte source connectors for:\n\n* HTTP APIs \\(REST APIs, GraphQL, etc..\\)\n* Generic .NET sources \\(anything not covered by the above\\)\n\nThe CDK provides an improved developer experience by providing basic implementation structure and abstracting away low-level glue boilerplate.\n\n## Quick Start\n\nGenerate a boilerplate connector using the code generator. Create a new directory and run:\n\n```bash\ngit clone https://github.com/mrhamburg/airbyte.cdk.dotnet.git\ncd airbyte.cdk.dotnet\ndotnet run init --project Airbyte.Cdk\n```\n\n## Components\n\nComponent | Type |Code | Installation | Version\n----------|------|-----|------|--------\nAirbyte CDK | CDK | [Airbyte.Cdk](Airbyte.Cdk) | `dotnet add package Airbyte.Cdk` |[![NetMQ NuGet version](https://img.shields.io/nuget/v/Airbyte.Cdk.svg)](https://www.nuget.org/packages/Airbyte.Cdk/)\nExchange Rates Free | Source | [source-exchange-rates-free](airbyte-integrations/connectors/source-exchange-rates-free) | `docker pull airbytedotnet/source-exchange-rates-free` | ![Docker Image Version (latest semver)](https://img.shields.io/docker/v/airbytedotnet/source-exchange-rates-free?sort=semver\u0026style=flat)\n\n## Concepts \u0026 Documentation\nThis readme is a general introduction to the CDK. Readers should have basic familiarity with the [Airbyte Specification](https://docs.airbyte.io/architecture/airbyte-specification) before proceeding.\n\nIf you have any issues with troubleshooting or want to learn more about the CDK from the Airbyte team, head to the \\#connector-development channel in [Slack](https://airbytehq.slack.com/ssb/redirect) to inquire further!\n\n### Basic Concepts\nIf you want to learn more about the classes required to implement an Airbyte Source, head to our [basic concepts doc](https://docs.airbyte.io/connector-development/cdk-python/basic-concepts) as described in the python-cdk.\n### Full Refresh Streams\nA `Stream` is the atomic unit for reading data from a Source. A stream can read data from anywhere: a relational database, an API, or even scrape a web page! \\(although that might be stretching the limits of what a connector should do\\).\nTo implement a stream, there are two minimum requirements: 1. Define the stream's schema 2. Implement the logic for reading records from the underlying data source\nSchema's should be stored in the schemas folder part of your project.\nUsing the fluentbuilder we can define a source. The short and simple example below, does the following steps:\n1. Create a string `url` as a baseurl for all subsequent requests\n2. Create a base implementation by converting the string to a HttpStream and set a default response parser being the whole object (this function expects a [JsonPath](https://github.com/json-path/JsonPath) expression to extract the data)\n3. Create a new http stream using the create statement and name it symbols (the name `symbols` will also be appended to the url when executing the request, thus the example below will result in an executed endpoint being: `https://api.exchangerate.host/symbols`)\n```csharp\npublic override Stream[] Streams(JsonElement config)\n{\n    string url = \"https://api.exchangerate.host\";\n    var baseimpl = url.HttpStream().ParseResponseObject(\"$\");\n    return new Stream[] { baseimpl.Create(\"symbols\") };\n}\n```\n\n### Incremental Streams\nAn incremental Stream is a stream which reads data incrementally. That is, it only reads data that was generated or updated since the last time it ran, and is thus far more efficient than a stream which reads all the source data every time it runs. If possible, developers are encouraged to implement incremental streams to reduce sync times and resource usage.\n\nSeveral new pieces are essential to understand how incrementality works with the CDK:\n\n* Cursor fields\n* `AirbyteStateMessage`\n* `Stream.GetUpdatedState`\n\nThe `AirbyteStateMessage` is sent based on the `StateCheckpointInterval` setting of a stream object. Every N number of requests will result in sending out an `AirbyteStateMessage`. In the example below, this is every 25th request for the `symbols` stream.\n\nThe `CursorField` refers to the field in the stream's output records used to determine the \"recency\" or ordering of records. An example is a `date` field in an API, as shown in the example below.\n\nCursor fields can be input by the user \\(e.g: a user can choose to use an auto-incrementing `id` column in a DB table\\) or they can be defined by the source e.g: where an API defines that `date` is what determines the ordering of records.\n\nIn the context of the CDK, setting the `Stream.CursorField` property to any truthy value informs the framework that this stream is incremental.\n\n`Stream.GetUpdatedState`, this function helps the stream keep track of the latest state by inspecting every record output by the stream \\(as returned by the `Stream.ReadRecords` method\\) and comparing it against the most recent state object. This allows sync to resume from where the previous sync last stopped, regardless of success or failure. This function typically compares the state object's and the latest record's cursor field, picking the latest one.\n\n\n```csharp\npublic override Stream[] Streams(JsonElement config)\n{\n...\n    Dictionary\u003cstring, DateTime\u003e _currentstate = new Dictionary\u003cstring, DateTime\u003e();\n    string basesymbol = config.GetProperty(\"symbol\").GetString();\n    var incremental = baseimpl\n        .CursorField(new[] {\"date\"})\n        .BackoffTime(((i, _) =\u003e TimeSpan.FromMinutes(i * 10)))\n        .GetUpdatedState((_, _) =\u003e _currentstate.AsJsonElement())\n        .RequestParams((_, _, _) =\u003e new Dictionary\u003cstring, object\u003e {{ \"date\", _currentstate[basesymbol] } })\n        .BackoffTime((i, response) =\u003e\n            response.StatusCode == 429 ? i * TimeSpan.FromSeconds(15) : TimeSpan.FromMinutes(1))\n        .HttpMethod(HttpMethod.Get)\n        .PageSize(150)\n        .StateCheckpointInterval(25)\n        .MaxRetries(15)\n        .ShouldRetry(exc =\u003e exc.StatusCode \u003e 300)\n        .WithAuth(new BasicAuth(new[] {config.GetProperty(\"api-token\").GetString()}))\n        .Create(\"symbols\");\n...\n}\n```\n\n### Practical Tips\n\nAirbyte recommends using the CDK template generator to develop with the CDK. The template generates created all the required scaffolding, with convenient TODOs, allowing developers to truly focus on implementing the API.\n\n## Contributing - Connector\n\n### First time setup\n\nMake sure the latest version of dotnet is installed, this can be found using the following link: [Dotnet SDK](https://dotnet.microsoft.com/download). Also, make sure you have docker installed, information to do so can be found here: [Get Docker](https://docs.docker.com/get-docker/)\n\nRun the following commands to get a boilerplate for creating a new connector:\n```bash\ngit clone https://github.com/mrhamburg/airbyte.cdk.dotnet.git\ncd airbyte.cdk.dotnet\ndotnet run init --project Airbyte.Cdk\n```\n\nOnce ready, check the README.md file in your connector directory for instructions. Include your connector in this readme as well, see the components section at the top.\n\n#### Iteration\n\n* Iterate on the code locally\n* Run tests via `dotnet test`\n* Try to build the connector using docker `docker build .`\n\n#### Testing\n\nAll tests are located in the `*.Tests` directory where your connector is created. Run `dotnet test` to run them.\n\n#### Publishing a new version of a connector\n\n1. Open a PR\n2. Once ready, update the CHANGELOG.md file, which should include a new version number \n3. Connectors are automatically published to Docker Hub after updates to the main branch. If the connector is updated without incrementing the version, GitHub will NOT overwrite the existing image in Docker Hub.\n\n## Contributing - CDK\n\n### First time setup\n\nMake sure the latest version of dotnet is installed, this can be found using the following link: [Dotnet SDK](https://dotnet.microsoft.com/download). Also, make sure you have docker installed, information to do so can be found here: [Get Docker](https://docs.docker.com/get-docker/)\n\n#### Iteration\n\n* Iterate on the code locally\n* Run tests via `dotnet test`\n* Try to build the nuget package using docker `docker build --target build .`\n\n#### Testing\n\nAll tests are located in the `Airbyte.Cdk.Test` directory. Run `dotnet test` to run them.\n\n#### Publishing a new version to NuGet\n\n1. Open a PR\n2. Testing and releasing is part of the CI/CD process\n\n## Coming Soon\n\n* Don't see a feature you need? [Create an issue and let us know how we can help!](https://github.com/mrhamburg/airbyte.cdk.dotnet/issues/new)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrhamburg%2Fairbyte.cdk.dotnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmrhamburg%2Fairbyte.cdk.dotnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrhamburg%2Fairbyte.cdk.dotnet/lists"}