{"id":26541601,"url":"https://github.com/peter-juhasz/blob-snapshots","last_synced_at":"2026-05-06T11:33:32.956Z","repository":{"id":89618049,"uuid":"381489759","full_name":"Peter-Juhasz/blob-snapshots","owner":"Peter-Juhasz","description":"Use Azure Blob Storage as cache for remote HTTP responses.","archived":false,"fork":false,"pushed_at":"2021-06-29T20:59:58.000Z","size":11,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-17T06:45:50.701Z","etag":null,"topics":["azure","blob","cache","dotnet","http"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Peter-Juhasz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-06-29T20:32:31.000Z","updated_at":"2022-06-11T21:16:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"56980851-e1bd-43bb-8939-289be3ae9ea5","html_url":"https://github.com/Peter-Juhasz/blob-snapshots","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Peter-Juhasz/blob-snapshots","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Peter-Juhasz%2Fblob-snapshots","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Peter-Juhasz%2Fblob-snapshots/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Peter-Juhasz%2Fblob-snapshots/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Peter-Juhasz%2Fblob-snapshots/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Peter-Juhasz","download_url":"https://codeload.github.com/Peter-Juhasz/blob-snapshots/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Peter-Juhasz%2Fblob-snapshots/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32691857,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-06T08:33:17.875Z","status":"ssl_error","status_checked_at":"2026-05-06T08:33:17.221Z","response_time":117,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","blob","cache","dotnet","http"],"created_at":"2025-03-22T01:39:17.632Z","updated_at":"2026-05-06T11:33:32.951Z","avatar_url":"https://github.com/Peter-Juhasz.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HTTP Client Blob Snapshot\n\nThis concept saves responses of remote HTTP service calls to a still remote, but closer and cheaper Blob Storage for caching purposes.\n\n## Introduction\nA few example use cases:\n - bring remote data closer to a local storage to reduce latency\n - recover from remote service failures and fallback to a cached snapshot of data\n - reduce usage of a remote service to avoid rate limiting\n - reduce billing cost of paid service (e.g.: resolving municipalities from postal codes)\n\n## Solution\n\n### Intercepting HTTP calls\nYou can intercept all HTTP calls using a [DelegatingHandler](https://docs.microsoft.com/en-us/dotnet/api/system.net.http.delegatinghandler):\n\n```cs\nprotected override async Task\u003cHttpResponseMessage\u003e SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\n{\n\treturn await base.SendAsync(request, cancellationToken);\n}\n```\n\nIn this method, you are free to do whatever you want to generate a [HttpResponseMessage](https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpresponsemessage) based on a [HttpRequestMessage](https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httprequestmessage). You can also use this concept for testing, to mock remote services and have control over the responses returned in different test cases.\n\nTo use this `DelegatingHandler`, you have to register it to a specific `HttpClient`:\n\n```cs\nservices.AddHttpClient(nameof(MyClient))\n\t.AddHttpMessageHandler\u003cMyDelegatingHandler\u003e()\n```\n\nYou can chain as many handlers after each other as you like.\n\n### Saving responses to Blob Storage\nFirst, we need to determine which requests and responses to save. Let's use a simple approach for that:\n\n```cs\nstatic bool IsApplicable(HttpRequestMessage request) =\u003e\n    request.Method == HttpMethod.Get \u0026\u0026\n    request.Headers.Range == null;\n\nstatic bool IsApplicable(HttpResponseMessage response) =\u003e\n    response.StatusCode is HttpStatusCode.OK \u0026\u0026\n    response.Headers.CacheControl switch\n    {\n        CacheControlHeaderValue cacheControlHeader =\u003e !cacheControlHeader.NoCache,\n        null =\u003e true,\n    };\n```\n\nThen implement saving:\n\n```cs\n// check whether this request is applicable\nif (!IsApplicableTo(request))\n{\n    return await base.SendAsync(request, cancellationToken);\n}\n\n// execute request\nvar response = await base.SendAsync(request, cancellationToken);\n\n// save\nif (IsApplicable(response))\n{\n    // buffer content\n    await response.Content.LoadIntoBufferAsync();\n\n    // read\n    var stream = await response.Content.ReadAsStreamAsync(cancellationToken);\n\n    // upload\n    var key = uri.Host + uri.PathAndQuery;\n    var blob = ContainerClient.GetBlockBlobClient(key);\n    var options = new BlobUploadOptions\n    {\n        HttpHeaders = new BlobHttpHeaders\n        {\n            ContentType = response.Content.Headers.ContentType?.ToString(),\n        },\n        AccessTier = AccessTier.Hot,\n    };\n    try\n    {\n        await blob.UploadAsync(stream, options, cancellationToken);\n    }\n    // container may not exists on very first call\n    catch (RequestFailedException ex) when (ex.ErrorCode == BlobErrorCode.ContainerNotFound)\n    {\n        // create blob container\n        await ContainerClient.CreateIfNotExistsAsync(PublicAccessType.None, cancellationToken: cancellationToken);\n\n        // retry\n        stream.Seek(0L, SeekOrigin.Begin);\n        await blob.UploadAsync(stream, options, cancellationToken);\n    }\n    catch (Exception ex)\n    {\n        Logger.LogWarning(ex, \"Couldn't save HTTP snapshot to blob storage.\");\n    }\n\n    // rewind stream\n    stream.Seek(0L, SeekOrigin.Begin);\n}\n\nreturn response;\n```\n\n### Load content\nNow that we have saved responses on Blob Storage, we can try to serve responses from there:\n\n```cs\n// check whether this request is applicable\nif (!IsApplicableTo(request))\n{\n    return await base.SendAsync(request, cancellationToken);\n}\n\n// find blob\nvar key = uri.Host + uri.PathAndQuery;\nvar blob = ContainerClient.GetBlockBlobClient(key);\ntry\n{\n    // download\n    var blobResponse = (await blob.DownloadAsync(cancellationToken)).Value;\n\n    // create response\n    var content = new StreamContent(blobResponse.Content);\n    content.Headers.ContentType = MediaTypeHeaderValue.Parse(blobResponse.ContentType);\n    content.Headers.ContentLength = blobResponse.ContentLength;\n    var responseMessage = new HttpResponseMessage(HttpStatusCode.OK)\n    {\n        Content = content\n    };\n    return responseMessage;\n}\ncatch (RequestFailedException ex) when (\n    ex.ErrorCode == BlobErrorCode.ContainerNotFound ||\n    ex.ErrorCode == BlobErrorCode.BlobNotFound ||\n    ex.ErrorCode == BlobErrorCode.OperationTimedOut\n)\n{\n    // no snapshot found is expected\n}\ncatch (Exception ex)\n{\n    Logger.LogWarning(ex, \"Couldn't access HTTP blob snapshot storage.\");\n}\n```\n\n\n## Appendix\n\n### Ignore specific calls\nWe used a simple static rule earlier to decided what requests are applicable to snapshot:\n```cs\nstatic bool IsApplicable(HttpRequestMessage request) =\u003e\n    request.Method == HttpMethod.Get \u0026\u0026\n    request.Headers.Range == null;\n```\n\nWe could add a user-defined filter to our configuration options:\n```cs\npublic Func\u003cHttpRequestMessage, bool\u003e RequestFilter { get; set; }\n```\n\nSo we could take this filter into account as well when determining whether a request is applicable to snapshot or not:\n```cs\nbool IsApplicable(HttpRequestMessage request) =\u003e\n    request.Method == HttpMethod.Get \u0026\u0026\n    request.Headers.Range == null \u0026\u0026\n    Options.RequestFilter(request);\n```\n\nAnd then we are able to define our specific filtering logic:\n```cs\nnew HttpSnapshotOptions \n{\n    RequestFilter = request =\u003e !request.RequestUri.AbsolutePath.StartsWith(\"/weather\")\n}\n```\n\n### Ignore specific parts of the URI\nWe used the following formula to generate blob names:\n```cs\nvar key = uri.Host + uri.PathAndQuery;\n```\n\nBut in some cases the path or query part of the URI may contain either values we would like to ignore:\n - a tenant ID `/tenants/123/api`\n - a pre-shared subscription key `/api?subscription-key=123`\n - an access token `/api?jwt=123`\n - current date or time `/api?time=2020-06-04`\n - an operation or correlation ID `/api?correlationId=123`\n - ...\n\nSo we could add a user-definable key selection / uniqueness function to the configuration:\n```cs\npublic Func\u003cHttpRequestMessage, string\u003e KeySelector { get; set; }\n```\n\nAnd use this to determine the blob name for a request:\n```cs\nvar key = Options.KeySelector(request);\n```\n\n### Time-based expiration\nIn some cases the remote service responses may be static/immutable and never change, while in most cases they may expire after some time.\n\nBut now that we have better control over cache keys, we could append time as well:\n```cs\nvar expiration = TimeSpan.FromHours(1);\nvar currenTimeSlot = DateTimeOffset.Now.Floor(expiration);\nvar key = String.Join('/', request.RequestUri.Host, request.RequestUri.PathAndQuery, currentTimeSlot);\n```\n\nEven though we include time in cache keys, we should be able to delete expired contents. To do that, we can configure a [Lifecycle management rule](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-lifecycle-management-concepts) in Azure Blob Storage, to automatically delete blobs older than X days. But if we have this rule in place already, we can even forget about appending time slots to cache keys, because Azure is going to delete old blobs either way. So when a blob is expired, it is not going to be there anymore, so our logic is going to get a fresh version from the remote service and store it as a new blob with the same name.\n\n### Cache control\nThe remote service may send specific caching instructions for each response, where we can't have preset lifecycle management rules which are applied to all blobs in general.\n\nAs a first step, we should save them:\n```cs\nvar options = new BlobUploadOptions\n{\n    HttpHeaders = new BlobHttpHeaders\n    {\n        ContentType = response.Content.Headers.ContentType?.ToString(),\n        CacheControl = response.Content.Headers.CacheControl?.ToString(),\n    },\n    AccessTier = AccessTier.Hot,\n};\n```\n\nIn a next step, we could expire specific snapshots upon reading:\n```cs\nvar blobResponse = await blob.DownloadAsync(cancellationToken);\nif (CacheControlHeaderValue.TryParse(blobResponse.Details.CacheControl, out var cacheControl))\n{\n    if (DateTimeOffset.Now - blobResponse.Details.LastModified \u003e cacheControl.MaxAge)\n    {\n        // invalidate cache\n        await blob.DeleteIfExistsAsync(conditions: new BlobRequestConditions()\n        { \n            IfMatch = blobResponse.Details.ETag // for race condition\n        });\n\n        // TODO: replay request\n    }\n}\n```\n\n### Tiering and performance\nIn general, the `Hot` tier is recommended for frequently access data, this is why we used that tier in the example.\n\nBut, if your use case is routed in performance, you can have a few extra options to speed up and get even lower latencies:\n - as this concept uses only Blobs, you can use a [Premium tier Blob Storage](https://azure.microsoft.com/en-us/blog/azure-premium-block-blob-storage-is-now-generally-available/) with SSDs, instead of a General Purpose account\n - if your service is deployed to multiple regions, you can have either a CDN in front of the Blob Storage to add another layer of cache, or use [Object Replication](https://docs.microsoft.com/en-us/azure/storage/blobs/object-replication-overview) to replicate blobs to multiple regions\n\n### Use another storage mechanism\nThis implementation is specific to Azure Blob Storage, but the storage logic could be easily extracted from the main logic into something else:\n\n```cs\npublic interface IHttpSnapshotStorage\n{\n    Task StoreAsync(string key, HttpResponseMessage response, CancellationToken cancellationToken);\n\n    Task\u003cHttpResponseMessage?\u003e LoadAsync(string key, CancellationToken cancellationToken);\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpeter-juhasz%2Fblob-snapshots","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpeter-juhasz%2Fblob-snapshots","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpeter-juhasz%2Fblob-snapshots/lists"}