{"id":15699040,"url":"https://github.com/robertoprevato/azureblobasyncupload","last_synced_at":"2025-05-09T02:15:19.420Z","repository":{"id":50198808,"uuid":"164026996","full_name":"RobertoPrevato/AzureBlobAsyncUpload","owner":"RobertoPrevato","description":"Example of upload and download made with asyncio and aiohttp for Azure Blob Service","archived":false,"fork":false,"pushed_at":"2022-12-08T01:30:57.000Z","size":19,"stargazers_count":8,"open_issues_count":5,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-09T02:15:17.644Z","etag":null,"topics":["aiohttp","async","asyncio","azure","blob"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RobertoPrevato.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-01-03T21:56:34.000Z","updated_at":"2024-04-11T13:07:32.000Z","dependencies_parsed_at":"2023-01-24T03:15:30.334Z","dependency_job_id":null,"html_url":"https://github.com/RobertoPrevato/AzureBlobAsyncUpload","commit_stats":null,"previous_names":[],"tags_count":0,"template":true,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RobertoPrevato%2FAzureBlobAsyncUpload","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RobertoPrevato%2FAzureBlobAsyncUpload/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RobertoPrevato%2FAzureBlobAsyncUpload/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RobertoPrevato%2FAzureBlobAsyncUpload/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RobertoPrevato","download_url":"https://codeload.github.com/RobertoPrevato/AzureBlobAsyncUpload/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253176444,"owners_count":21866143,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aiohttp","async","asyncio","azure","blob"],"created_at":"2024-10-03T19:37:43.828Z","updated_at":"2025-05-09T02:15:19.396Z","avatar_url":"https://github.com/RobertoPrevato.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Azure Storage Blob Service, upload and download with asyncio\nThis repository contains code to download and upload files of any size to Azure Storage Blob Service, using its REST api with `asyncio` and `aiohttp`.\nThis code was shared, in relation to [this thread in GitHub](https://github.com/Azure/azure-storage-python/issues/534#issuecomment-451260323).\nIn this context, the official Python SDK is used only to generate shared access signatures.\n\n## Example of concurrent files upload\n\n```python\n\nasync with SSLClientSession() as http_client:\n    client = BlobsClient(http_client, BlockBlobService(ACCOUNT_NAME, ACCOUNT_KEY))\n\n    # NB: this code does not create containers automatically!\n\n    destination_container_name = 'test'\n\n    files = [\n        r'one.ext',\n        r'two.ext',\n        r'three.ext',\n        r'four.ext'\n    ]\n\n    await asyncio.gather(*[client.upload_file(file_path, destination_container_name) for file_path in files])\n```\n\n## Note\nWhen uploading big files to blob service, it is necessary to do several web requests for each file: one for every chunk and a\nlast one to commit the file. My code intentionally doesn't start parallel web requests to upload chunks of the same file,\nbecause it was designed having in mind a scenario where many files are read from file system and uploaded concurrently (concurrent upload of different files,\nnot concurrent uploads of _chunks_ of every single file!).\n\nMoreover, concurrent chunk uploads, or concurrent file uploads without limits, could cause to handle too many bytes in memory at the same time - potentially defeating the purpose of chunking big files\non the client side. For this reason, it is recommended to use a [semaphore](https://docs.python.org/3/library/asyncio-sync.html#semaphore), to limit the concurrency of upload operations. There is no perfect 'one-size-fits-all' solution.\n\nIf you need a scenario where you handle a few files at a given time, then you could benefit, from changing the code to support parallel uploads of chunks of every single file.\n\n## Example to download files in chunks\n\nTo download a file saving it to file system:\n```python\nasync with SSLClientSession() as http_client:\n    client = BlobsClient(http_client, BlockBlobService(ACCOUNT_NAME, ACCOUNT_KEY))\n    \n    source_container_name = 'test'\n    blob_name = 'some_blob_in_container.txt'\n    await client.download_file(source_container_name, blob_name) \n\n```\n\nTo download a file handling chunks in memory:\n```python\nasync with SSLClientSession() as http_client:\n    client = BlobsClient(http_client, BlockBlobService(ACCOUNT_NAME, ACCOUNT_KEY))\n    \n    source_container_name = 'test'\n    blob_name = 'some_blob_in_container.txt'\n\n    async for chunk in client.read_blob(source_container_name, blob_name):\n        # handle chunk in memory\n        pass  \n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobertoprevato%2Fazureblobasyncupload","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frobertoprevato%2Fazureblobasyncupload","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobertoprevato%2Fazureblobasyncupload/lists"}