{"id":24024785,"url":"https://github.com/cognitedata/dotnet-extractor-utils","last_synced_at":"2026-04-27T08:03:10.000Z","repository":{"id":36965446,"uuid":"250268765","full_name":"cognitedata/dotnet-extractor-utils","owner":"cognitedata","description":"Common utilities for developing extractors in .NET","archived":false,"fork":false,"pushed_at":"2025-06-24T11:57:29.000Z","size":7575,"stargazers_count":5,"open_issues_count":8,"forks_count":1,"subscribers_count":68,"default_branch":"master","last_synced_at":"2025-06-24T12:51:43.082Z","etag":null,"topics":["c-sharp","dotnet","library"],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cognitedata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-03-26T13:40:05.000Z","updated_at":"2025-06-24T11:57:34.000Z","dependencies_parsed_at":"2023-10-17T04:37:16.298Z","dependency_job_id":"7cfb3592-b06f-488f-813f-a93368e63adf","html_url":"https://github.com/cognitedata/dotnet-extractor-utils","commit_stats":null,"previous_names":[],"tags_count":129,"template":false,"template_full_name":null,"purl":"pkg:github/cognitedata/dotnet-extractor-utils","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cognitedata%2Fdotnet-extractor-utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cognitedata%2Fdotnet-extractor-utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cognitedata%2Fdotnet-extractor-utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cognitedata%2Fdotnet-extractor-utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cognitedata","download_url":"https://codeload.github.com/cognitedata/dotnet-extractor-utils/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cognitedata%2Fdotnet-extractor-utils/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263279275,"owners_count":23441682,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["c-sharp","dotnet","library"],"created_at":"2025-01-08T15:34:53.095Z","updated_at":"2026-04-23T14:02:19.601Z","avatar_url":"https://github.com/cognitedata.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ca href=\"https://cognite.com/\"\u003e\n    \u003cimg src=\"https://images.squarespace-cdn.com/content/5bd167cf65a707203855d3c0/1540463676940-6USHZRRF36KCAZLUPM2P/Logo-H.jpg?format=300w\u0026content-type=image%2Fjpeg\" alt=\"Cognite logo\" title=\"Cognite\" align=\"right\" height=\"40\" /\u003e\n\u003c/a\u003e\n\n.Net Utilities for Building Cognite Extractors\n=======================\n![Build and Test](https://github.com/cognitedata/dotnet-extractor-utils/workflows/Build%20and%20Test/badge.svg?branch=master\u0026event=push)\n![Release](https://github.com/cognitedata/dotnet-extractor-utils/workflows/Create%20Release/badge.svg)\n[![codecov](https://codecov.io/gh/cognitedata/dotnet-extractor-utils/branch/master/graph/badge.svg?token=2IX9UN9ING)](https://codecov.io/gh/cognitedata/dotnet-extractor-utils)\n[![Nuget](https://img.shields.io/nuget/vpre/Cognite.ExtractorUtils)](https://www.nuget.org/packages/Cognite.ExtractorUtils/)\n\nA library containing utilities for building extractors in .Net.\n\n[Documentation](https://cognitedata.github.io/dotnet-extractor-utils/index.html)\n\n## Installation\n\nThe Cognite Extractor Utils can be downloaded from [NuGet](https://www.nuget.org/packages/Cognite.ExtractorUtils). \n\nTo create a console application and add the library:\n\nUsing .NET CLI:\n```sh\nmkdir NewExtractor\ncd NewExtractor\ndotnet new console\ndotnet add package Cognite.ExtractorUtils\n```\n\n## Quickstart\n\nCreate a ```config.yml``` file containing the extractor configuration\n\n```yaml\nversion: 1\n\nlogger:\n    console:\n        level: \"debug\"\n\nmetrics:\n    push-gateways:\n      - host: \"http://localhost:9091\"\n        job: \"extractor-metrics\"\n\ncognite:\n    project: ${COGNITE_PROJECT}\n    # This is for microsoft as IdP, to use a different provider,\n    # set implementation: Basic, and use token-url instead of tenant.\n    # See the example config for the full list of options.\n    idp-authentication:\n        # Directory tenant\n        tenant: ${COGNITE_TENANT_ID}\n        # Application Id\n        client-id: ${COGNITE_CLIENT_ID}\n        # Client secret\n        secret: ${COGNITE_CLIENT_SECRET}\n        # List of resource scopes, ex:\n        # scopes:\n        #   - scopeA\n        #   - scopeB\n        scopes:\n          - ${COGNITE_SCOPE}\n```\n\nSee the [example configuration](ExtractorUtils/config/config.example.yml) for a full example with all available options.\n\nSet the ```COGNITE_PROJECT```, ```COGNITE_TENANT_ID```, ```COGNITE_CLIENT_ID```, ```COGNITE_CLIENT_SECRET```, and ```COGNITE_SCOPE``` environment variables. Set the ```metrics``` tag, only if collecting metrics is required by the extractor. If using a [Prometheus pushgateway](https://prometheus.io/docs/practices/pushing/), set ```host```to a valid endpoint.\n\nThe easiest way to use the library utilities is by using the ```BaseExtractor``` class. The following is a working implementation of an extractor writing a sine wave to CDF.\n\n```c#\nusing Microsoft.Extensions.DependencyInjection;\nusing Cognite.Extractor.Utils;\nusing Cognite.Extensions;\nusing CogniteSdk;\n\nclass MyExtractor : BaseExtractor\u003cBaseConfig\u003e\n{\n    public MyExtractor(BaseConfig config, CogniteDestination destination)\n        : base(config, destination)\n    {\n    }\n    \n    protected override async Task Start() \n    {\n        await Destination.EnsureTimeSeriesExistsAsync(new[]\n        {\n            new TimeSeriesCreate {\n                ExternalId = \"sine-wave\",\n                Name = \"Sine Wave\"\n            }\n        }, RetryMode.OnError, SanitationMode.Clean, Source.Token);\n        CreateTimeseriesQueue(1000, TimeSpan.FromSeconds(1), null);\n        ScheduleDatapointsRun(\"datapoints\", TimeSpan.FromMilliseconds(100), token =\u003e\n        {\n            var dp = (\n                Identity.Create(\"sine-wave\"),\n                new Datapoint(DateTime.UtcNow, Math.Sin(DateTime.UtcNow.Ticks))\n            );\n            return Task.FromResult\u003cIEnumerable\u003c(Identity, Datapoint)\u003e\u003e(new [] { dp });\n        });\n    }\n}\n\n// Then, in the Main() method:\nclass Program\n{\n    static async Task Main()\n    {\n        await ExtractorRunner.Run\u003cBaseConfig, MyExtractor\u003e(\n            \"config.yml\",\n            new[] { 1 },\n            \"my-extractor\",\n            \"myextractor/1.0.0\",\n            addStateStore: false,\n            addLogger: true,\n            addMetrics: true,\n            restart: true,\n            CancellationToken.None);\n    }\n}\n\n```\n\n## Inserting data points:\n```c#\n// Create a dictonary of time series identities and datapoint objects\ndatapoints = new Dictionary\u003cIdentity, IEnumerable\u003cDatapoint\u003e\u003e() {\n    { new Identity(\"externalId1\"), new Datapoint[] { new Datapoint(DateTime.UtcNow, \"A\")}},\n    { new Identity(\"externalId2\"), new Datapoint[] { new Datapoint(DateTime.UtcNow, 1), \n                                                     new Datapoint(DateTime.UtcNow, 2)}},\n    { new Identity(12345789), new Datapoint[] { new Datapoint(DateTime.UtcNow, 1)}}}\n};\n\n// Insert the data points, ignoring and returning any errors.\nvar errors = await destination.InsertDataPointsIgnoreErrorsAsync(\n    datapoints,\n    cancellationToken);\nif (errors.IdsNotFound.Any() || errors.IdsWithMismatchedData.Any())\n{\n    logger.LogError(\"Ids not found: {NfIds}. Time series with mismatched type: {MmIds}\",\n        errors.IdsNotFound, errors.IdsWithMismatchedData);\n}\n```\n\n## Using Raw upload queues:\n```c#\n// Data type object representing raw columns\nprivate class ColumnsDto\n{\n    public string Name { get; set; }\n    public int Number { get; set; }\n}\n\n// Creates an queue that uploads rows to Raw every 5 seconds (or when the queue size reaches 1.000)\nusing (var queue = destination.CreateRawUploadQueue\u003cColumnsDto\u003e(\"myDb\", \"myTable\", TimeSpan.FromSeconds(5), 1_000,\n    result =\u003e { // handle result of upload here }))\n{\n    // Task to generate rows at regular intervals\n    var enqueueTask = Task.Run(async () =\u003e {\n        while (index \u003c 2_000)\n        {\n            queue.EnqueueRow($\"r{index}\", new ColumnsDto {Name = \"Test\", Number = index});\n            await Task.Delay(50, cancellationToken);\n            index++;\n        }\n    });\n    \n    // Task to start the upload queue\n    var uploadTask = queue.Start(cancellationToken);\n\n    // wait for either the enqueue task to finish or the upload task to fail\n    var t = Task.WhenAny(uploadTask, enqueueTask);\n    await t;\n    logger.LogInformation(\"Enqueueing task completed. Disposing of the upload queue\");\n} // disposing the queue will upload any rows left and stop the upload loop\n\n```\n\n## Using the State Store\n```c#\nservices.AddStateStore();\n\nusing (var provider = services.BuildServiceProvider()) {\n    var stateStore = provider.GetRequiredService\u003cIExtractionStateStore\u003e();\n    var destination = provider.GetRequiredService\u003cCogniteDestination\u003e();\n    \n    // Create a state for a node\n    var myState = new BaseExtractionState(\"myState\");\n    var states = new [] { myState };\n    var stateDict = states.ToDictionary(state =\u003e state.Id);\n    \n    await stateStore.RestoreExtarctionState(stateDict, \"someTableorcollection\", cancellationToken)\n   \n    // After uploading points cdf, update the ranges in your state\n    var now = DateTime.UtcNow;\n    var datapoints = new Dictionary\u003cIdentity, IEnumerable\u003cDatapoint\u003e\u003e() {\n        { new Identity(\"myState\"), new Datapoint[] { new Datapoint(now - TimeSpan.FromHours(2), \"B\"), new Datapoint(now, \"A\")}}}\n\n    await destination.InsertDataPointsIgnoreErrorsAsync(datapoints, cancellationToken);\n    \n    myState.UpdateDestinationRanges(now - TimeSpan.FromHours(2), now);\n    \n    await stateStore.StoreExtractionState(states, \"someTableorcollection\", cancellationToken);\n    // If the extractor stops here, the state is saved and can be restored after restart.\n}\n```\n\n# Using the installer template\n\nThe installer template is mostly configured from the command live and the config .json file, but more advanced changes may require modifying the project files. Feel free to copy the installer project and modifying it to your needs. The following instructions produce a single file executable for a .NET core project. It assumes that you have a config file named `config.example.yml` somewhere.\n\n - Update the `setup-config.json` file to suit your needs. The default fields are described below. All config fields except for `setup_project` is injected as a build property with msbuild into the `.wixproj` file.\n - Modify the `InstBanner.bmp` file in `Resources`, adding your extractor name to the blue field.\n - Modify or replace the `License.rtf` with a license adapted to your project.\n - (Optional) Add any new additional config files at the bottom of the `Product.wxs` file, by adding new `Component` blocks after the one for `config.example.yml`\n - (Optional) Add a cognite icon to your extractor by adding `Resources/black32x32.ico` to the folder for your actual extractor, and adding `\u003cApplicationIcon\u003eblack32x32.ico\u003c/ApplicationIcon\u003e` to its .csproj file.\n - Compile the installer in a windows environment with Wix utils and msbuild installed, using something like the following command:\n \n`build.ps1 -b Path\\To\\MSBuild.exe -v 1.0.0 -d \"some description\" -c Path\\To\\setup-config.json`\n\n - `-v` or `-version` is embedded as InformationalVersion when compiling, and can be retrieved at runtime. It is also used for the installer version, so it is required. A good way to retrieve this in a build environment is using a git tag, this way a github release can also be created based on tags.\n - `-b` or `-build` is the path to your MSBuild.exe.\n - `-d` or `-description` is embedded as Description when compiling. It must be set, either to something static, or something like the current git commit + git commit time.\n - `-c` or `-config` is the path to the json configuration file.\n\nThe following configuration parameters are used by default:\n\n - `target_project` is the csproj file for your extractor. Note that all paths are relative to the .wixproj file.\n - `target_configuration` is generally Release.\n - `target_runtime` is most likely either win-x64 or win-x86\n - `product_name` full display name of your product. This is used both in the installer and in the installed programs registry on the target computer.\n - `product_short_name` a short version of the product name without spaces. Should not contain \"Cognite\". It is used for registry keys and folder names.\n - `exe_name` the name of the final executable.\n - `config_dir` the path to the config.example.yml folder. This is also relative to the .wixproj folder.\n - `service` can be left out. If `true`, the installer will add and configure a windows service for the extractor.\n - `service-args` can be left out. Arguments to specify to the extractor when running it as a service. This is useful if the standalone and service versions use the same executable, but with different command line parameters.\n - `upgrade-guid` is a unique upper case GUID which you must generate yourself. It identifies the project when upgrading.\n - `setup-project` is the path to the .wixproj file used to build.\n - `output-name` is the name of the output msi, like `MyExtractorSetup`.\n \nSee the ExampleExtractorSetup project in this repository for a full example.\n\nIn general the setup template assumes that this is a cognite product, but changing this is no more difficult than replacing instances of `Cognite` in `Product.wxs` with whatever suits your purposes.\n \n## Modifying the installer template\n\nAdding to the installer template is relatively easy. New builds can be added in the `\u003cTarget Name=\"BeforeBuild\"\u003e` block in `SetupTemplate.wixproj`, these should output to new folders. New files going in the `bin/` folder can be added to `\u003cComponentGroup Id=\"ExecutableComponentGroup\"\u003e`. Note that the executable is duplicated here due to conditionals on `service`. New components can be added after `\u003c?endif?\u003e`.\n\nNew folders can be added by adding new `Directory` tags in the first `Fragment`, a new `ComponentGroupRef` at the bottom of `Product`, and a new `ComponentGroup` somewhere in the last `Fragment`.\n\n# Code of Conduct\n\nThis project follows https://www.contributor-covenant.org\n\n## License\n\nApache v2, see [LICENSE](https://github.com/cognitedata/dotnet-extractor-utils/blob/master/LICENSE).\n\n## Contributing\n\nDue to restrictions on integration tests, PRs from external forks should be merged into the `integration` branch.\n\nThe project requires test coverage, if your change adds code, make sure to create a test for it.\n\nExternal commits are _merged_. PRs should have clean commit history with descriptive commit messages.\n\n### Style\n\n - Newline before braces.\n - Private member names start with underscore.\n - Use PascalCase for methods and public members, and camelCase for locals.\n - Public members must be `const` or properties.\n - Methods must have inline XML documentation.\n - Make sure to fix any warnings generated by code analysis during build.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcognitedata%2Fdotnet-extractor-utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcognitedata%2Fdotnet-extractor-utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcognitedata%2Fdotnet-extractor-utils/lists"}