{"id":15348878,"url":"https://github.com/arvindshmicrosoft/blobcat","last_synced_at":"2025-04-05T22:17:50.960Z","repository":{"id":38005476,"uuid":"135664268","full_name":"arvindshmicrosoft/BlobCat","owner":"arvindshmicrosoft","description":"Concatenate Azure Storage Blobs (from other blobs and / or files on disk)","archived":false,"fork":false,"pushed_at":"2022-12-07T17:56:41.000Z","size":92,"stargazers_count":0,"open_issues_count":2,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-12T03:23:13.311Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arvindshmicrosoft.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-06-01T03:42:33.000Z","updated_at":"2019-03-08T02:41:40.000Z","dependencies_parsed_at":"2023-01-24T19:34:05.843Z","dependency_job_id":null,"html_url":"https://github.com/arvindshmicrosoft/BlobCat","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arvindshmicrosoft%2FBlobCat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arvindshmicrosoft%2FBlobCat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arvindshmicrosoft%2FBlobCat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arvindshmicrosoft%2FBlobCat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arvindshmicrosoft","download_url":"https://codeload.github.com/arvindshmicrosoft/BlobCat/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247406102,"owners_count":20933806,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-01T11:53:16.573Z","updated_at":"2025-04-05T22:17:50.910Z","avatar_url":"https://github.com/arvindshmicrosoft.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"### NOTE: the binary release provided under the \"Releases\" tab is out-of-date and there are no plans to refresh it. To use the latest features you should build the binaries from the source in this repo. That way you can target either .NET Core or .NET Framework.\n\n# BlobCat \nConcatenates blobs in Azure Storage. Source can be other block blobs or files on disk. Big Data platforms and SQL Polybase export files in a distributed fashion, which means a file called part.txt will actually be multiple files like part_1.txt, part_2.txt etc. The user then typically has to download and then concatenate them somehow. This utility is aimed at making that simple by offering a blob-to-blob or even local-disk-to-blob concatenation (why the latter? In case you have on-prem Hadoop or SQL Server with Polybase and then want to concat files directly into Azure storage).\n \n# Prerequisites\nThe utility is built on .NET Core, which means it runs on Windows, Mac and Linux (I have tested with Windows 10, Windows 2016 and Ubuntu 16.04). You will need .NET Core 2.1 runtime to run this utility. .NET Core runtime is a small download, around 30 MB max. [The .NET download page](https://www.microsoft.com/net/download) has all the links, again remember you just need the RUNTIME not the whole SDK to execute this utility. Scroll down on that page to get to the section where the runtimes are listed in the second column of that table.\n\nThere is also a .NET Framework version of the utility, which you can build from latest sources.\n\n# Usage \nUse command line execution. The output is self-explanatory and all the current command line options are shown on the screen.\n\n\tdotnet blobcat.dll \u003cverb\u003e \u003coptions\u003e\n\n\\\u003cverb\\\u003e is currently either of the following:\n\n* *concatblob*: Concatenates a set of blobs into a single blob.\n* *filestoblob*: Concatenates a set of on-disk files into a single blob.\n\n\\\u003coptions\\\u003e vary for each verb, see sections below for details.\n\n## Concatenate blob-to-blob syntax\nA sample command line (for the blob-to-blob scenario) is shown below. it concatenates any files beginning with the specified prefix in the specified Azure storage account and container, into the destination Azure storage account and container.\n\t\n\tdotnet blobcat.dll concatblob --SourceAccount \u003csource Azure storage account name\u003e --SourceContainer \u003csource Azure storage container name\u003e --SourceKey \u003csource Azure storage account key\u003e | --SourceSAS \u003csource SAS\u003e --SourceFilePrefix \u003cprefix\u003e --SortFilenames true --DestAccount \u003cAzure storage account name\u003e --DestContainer \u003csomecontainer\u003e --DestFilename \u003csomefilename can include foldername\u003e --DestKey \u003cStorage Key\u003e | --DestSAS \u003cdest SAS\u003e --ColHeader \u003cstring representing the column headers\u003e\n\nFor a short explanation of each of these switches, you can run:\n\t\n\tdotnet blobcat.dll concatblob --help\n\n## Concatenate files-to-blob syntax\nA sample command line (for the files-to-blob scenario) is shown below. it concatenates any files (on the local disk) beginning with the specified prefix into the destination Azure storage account and container.\n\n\tdotnet blobcat.dll filestoblob --SourceFolder \u003cfolder path\u003e --SourceFilePrefix \u003cprefix\u003e --SortFilenames true --DestAccount \u003cAzure storage account name\u003e --DestContainer \u003csomecontainer\u003e --DestFilename \u003csomefilename can include foldername\u003e --DestKey \u003cStorage Key\u003e | --DestSAS \u003cdest SAS\u003e --ColHeader \u003cstring representing the column headers\u003e\n\nFor a short explanation of each of these switches, you can run:\n\n\tdotnet blobcat.dll filestoblob --help\n\n# Acknowledgements\nA shout out to the following projects which are used in this utility:\n[CommandLineParser](https://github.com/commandlineparser/commandline) - command line options\n[Polly](https://github.com/App-vNext/Polly) - resiliency / auto-retry capabilities\n[Serilog](https://github.com/serilog/serilog) - logging\n[Data.HashFunction](https://github.com/brandondahler/Data.HashFunction/) - used mainly for CityHash implementation\n[ShellProgressBar](https://github.com/Mpdreamz/shellprogressbar) - for progress reporting\n[Azure Storage SDK](https://github.com/Azure/azure-storage-net/) - used via Nuget\n[Json.NET](https://github.com/JamesNK/Newtonsoft.Json) - Newtonsoft Json - used implicitly because of the Azure SDKs\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farvindshmicrosoft%2Fblobcat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farvindshmicrosoft%2Fblobcat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farvindshmicrosoft%2Fblobcat/lists"}