https://github.com/andre-maree/sequentialblobintegrator
Integrate sequentially by saving payloads to blobs
https://github.com/andre-maree/sequentialblobintegrator
blob-storage csharp d365 durable-functions integration powerapps sequential sql-server
Last synced: 7 months ago
JSON representation
Integrate sequentially by saving payloads to blobs
- Host: GitHub
- URL: https://github.com/andre-maree/sequentialblobintegrator
- Owner: andre-maree
- Created: 2025-05-14T10:03:58.000Z (9 months ago)
- Default Branch: master
- Last Pushed: 2025-05-29T16:06:09.000Z (8 months ago)
- Last Synced: 2025-06-26T14:05:14.897Z (7 months ago)
- Topics: blob-storage, csharp, d365, durable-functions, integration, powerapps, sequential, sql-server
- Language: C#
- Homepage:
- Size: 87.9 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Sequential Blob Integration Function App
This C# project can be used for integration. Sequential operations (creates, updates, and deletes) are enforced by uploading json payloads to blobs within transactions. No ServiceBus queues are needed (for first-in-first-out), as long as the blob is saved with a ticks timestamp for its name within an originating transaction. For example, when outbound integration is needed for PowerApps and D365 CRM, a plugin can run with a synchronous post-operation that saves a blob within the transaction for the row create, update, or delete. This means that the blobs will be ordered correctly sequentially by name. All that the the D365 plugin must do is to upload the blob with the payload with the correct name: {key}/{ticks}. An Azure Function blob trigger will start a Durable Function that will process the blobs sequentially per key.
This (SequentialBlobIntegrator) is an alternative to using ServiceBus. ServiceBus pricing is tiered. SequentialBlobIntegrator pricing is pay-as-you-go, cheap serverless FAAS pricing. ServiceBus is a good option, but this project shows that first-in-first-out processing is possible by using Durable Functions with no ServiceBus.
This will also work for creates, updates, and deletes in MS SQL by saving the blobs within transactions. For a MS SQL solution, use a code based transaction and call a stored procedure or in-line sql. If a sequential processing order is not required, then no transaction is needed.
Functionality include:
- Sequential outbound updates per key. The key should be the row id.
- Only 1 outbound call will execute concurrently per key, in the correct sequential order.
- Outbound calls will execute concurrently across row keys.
- Outbound throttling: Outbound calls executing concurrently across different keys can be limited by how many execute concurrently. This is to make sure the outbound calls do not overwhelm the destination service or system.
To get started locally, run the function app and call http://localhost:7161/testFunction1_HttpStart. This will save some blobs and execute them sequentially and call the test external endpoint https://httpbin.org/post. This endpoint is good for testing, because it will sometimes return with a failed http status code, and this is good to test that the retries are working.
local.settings.json Config:
```json
"Container": "integration-test", // set the blob container to use
"MaxConcurrentOutboundCalls": "5", // max concurrent outbound calls
"BlobBatchSize": "5000", // page size of the call to get blob names
"HttpBinBaseUrl": "https://httpbin.org", // test outbound base url for dev purposes
// retries
"RetryMaxIntervalMinutes": "5",
"RetryFirstIntervalSeconds": "5",
"RetryMaxIntervals": "1000",
"RetryBackOffCofecient": "1.125"
```
Use Azure Storage Explorer and Azurerite for local development. Azure Storage Explorer can be downloaded, and Azurite should be included within Visual Sudio.
Simply call [POST] http://localhost:7161/CreateIntegrationInstance with the following object model (included in the Models class). Look at the TestFunction class how to save the blobs with the needed json payload:
```csharp
public class IntegrationPayload()
{
public string Key { get; set; }
public long TicksStamp { get; set; }
public IntegrationHttpRequest IntegrationHttpRequest { get; set; }
}
public class IntegrationHttpRequest
{
public string HttpRoute { get; set; }
public Dictionary Headers { get; set; }
public string HttpMethod { get; set; }
public string Content { get; set; }
}
```