{"id":20927267,"url":"https://github.com/davidsteiner/blobby","last_synced_at":"2025-05-13T17:34:02.909Z","repository":{"id":243634854,"uuid":"811863598","full_name":"davidsteiner/blobby","owner":"davidsteiner","description":"Cloud provider agnostic library for object storage.","archived":false,"fork":false,"pushed_at":"2024-06-12T16:56:17.000Z","size":77,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-24T05:26:27.484Z","etag":null,"topics":["aws","aws-s3","blob","blob-storage","boto3","cloud","gcp","google-cloud-storage","object-storage","s3"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/blobby/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davidsteiner.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-07T13:06:57.000Z","updated_at":"2025-01-14T12:58:51.000Z","dependencies_parsed_at":"2024-06-10T10:48:44.250Z","dependency_job_id":"a5a539e6-3649-4843-a981-78067cdcc0b7","html_url":"https://github.com/davidsteiner/blobby","commit_stats":null,"previous_names":["davidsteiner/blobby"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidsteiner%2Fblobby","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidsteiner%2Fblobby/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidsteiner%2Fblobby/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidsteiner%2Fblobby/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davidsteiner","download_url":"https://codeload.github.com/davidsteiner/blobby/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253993972,"owners_count":21996397,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-s3","blob","blob-storage","boto3","cloud","gcp","google-cloud-storage","object-storage","s3"],"created_at":"2024-11-18T20:47:12.320Z","updated_at":"2025-05-13T17:34:02.610Z","avatar_url":"https://github.com/davidsteiner.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# Blobby\n\n**A cloud agnostic object storage library.**\n\n[![pypi](https://img.shields.io/pypi/v/blobby.svg)](https://pypi.org/project/blobby/)\n[![versions](https://img.shields.io/pypi/pyversions/blobby.svg)](https://github.com/davidsteiner/blobby/)\n[![Codacy Badge](https://app.codacy.com/project/badge/Grade/5c800180fb3b466fb8964d798aecdcc2)](https://app.codacy.com/gh/davidsteiner/blobby/dashboard?utm_source=gh\u0026utm_medium=referral\u0026utm_content=\u0026utm_campaign=Badge_grade)\n[![Codacy Badge](https://app.codacy.com/project/badge/Coverage/5c800180fb3b466fb8964d798aecdcc2)](https://app.codacy.com/gh/davidsteiner/blobby/dashboard?utm_source=gh\u0026utm_medium=referral\u0026utm_content=\u0026utm_campaign=Badge_coverage)\n[![license](https://img.shields.io/github/license/davidsteiner/blobby.svg)](https://github.com/davidsteiner/blobby/blob/main/LICENSE)\n\n\u003c/div\u003e\n\n---\n\nBlobby provides uniform interface for object storage solutions of common cloud providers.\nIt also provides a local filesystem-based implementation and an in-memory implementation\nfor local development and testing.\n\nIn addition to the core APIs for manipulating and retrieving\nbinary data, blobby also provides convenient wrappers to \nwrite and read\n[pydantic](https://docs.pydantic.dev/latest/) objects\nserialised as JSON documents.\n\n## Provider support\n\n- [x] AWS S3\n- [x] Azure Blob Storage\n- [x] Filesystem\n- [x] Google Cloud Storage\n- [x] In-memory\n\n## Creating a storage\n\nAll storage implementations inherit from `blobby.Storage` and\noffer a uniform API.\n\n### AWS S3 storage\n\n\u003e :warning: **Install blobby with the `aws` extra, i.e.**\n\u003e `pip install blobby[aws]`\n\nThe S3 implementation uses a `boto3` client, which needs to be\npassed in when the storage is initialised. An S3 storage object\nrepresents a bucket, whose name also needs to be supplied.\n\n```python\nimport boto3\nfrom blobby.aws import S3Storage\n\nclient = boto3.client(\"s3\")\nstorage = S3Storage(client=client, bucket_name=\"my-bucket\")\n```\n\n### Azure Blob Storage\n\n\u003e :warning: **Install blobby with the `azure` extra, i.e.**\n\u003e `pip install blobby[azure]`\n\nThe Azure implementation leverages the Azure SDK for Python.\nThe storage expects the storage client to be provided.\n\n```python\nfrom azure.storage.blob import BlobServiceClient\nfrom blobby.azure import AzureBlobStorage\n\nurl = \"\u003cconnection_string\u003e\"\nservice_client = BlobServiceClient.from_connection_string(url)\ncontainer_client = service_client.create_container(\"my-container\")\n\nstorage = AzureBlobStorage(container_client)\n```\n\n### Google Cloud Storage\n\n\u003e :warning: **Install blobby with the `gcp` extra, i.e.**\n\u003e `pip install blobby[gcp]`\n\nThe Google Cloud Storage leverages the official SDK for \nCloud Storage. The bucket object needs to be supplied to the\nstorage when it's initialised.\n\n```python\nfrom google.cloud.storage import Client\nfrom blobby.gcp import GoogleCloudStorage\n\nclient = Client()\nbucket = client.bucket(\"my-bucket\")\nstorage = GoogleCloudStorage(bucket)\n```\n\n### Filesystem storage\n\nWhen creating a filesystem-based storage, the root directory\nneeds to be provided. All files will be relative to this \ndirectory.\n\n```python\nfrom blobby.filesystem import FileSystemStorage\n\nstorage = FileSystemStorage(root_dir=\"/my/storage/\", create_missing_dirs=True)\n```\n\nThe `create_missing_dirs` flag controls whether the root directory\nwill be automatically created if it doesn't already exist.\n\n### In-memory storage\n\nThe in-memory implementation is backed with a simple dictionary stored\nin memory.\n\n```python\nfrom blobby.memory import MemoryStorage\n\nstorage = MemoryStorage()\n```\n\n## Common operations\n\n### Putting objects\n\nThe `put` operation works with `bytes` and `str` inputs.\nIn either case, the object is stored as a binary blob.\n\n```python\nkey = \"my-object\"\ndata = b\"hello world\"\nstorage.put(key, data)\n```\n\nIn the case of filesystem storage, the key needs to be a \nvalid path.\n\n### Getting objects\n\n```python\nkey = \"my-object\"\nstorage.get(key)\n```\n\n### Deleting objects\n\n```python\nkey = \"my-object\"\nstorage.delete(key)\n```\n\n### Listing objects\n\nCurrently, only listing by object prefix is supported.\nThis isn't very natural for filesystems, but the primary focus\nof this library is object storage solutions, which often\ndon't have the concept of a folder or directory.\n\n```python\nprefix = \"my/prefix\"\nstorage.list(prefix)\n```\n\n## Pydantic objects\n\nPydantic objects can be written and read using \ndedicated APIs for convenience.\n\n```python\nclass MyData(pydantic.BaseModel):\n    foo: str\n    bar: int\n\nkey = \"my/data\"\ndata = MyData(foo=\"hello\", bar=1)\n\nstorage.put_model_object(key, data)\n```\n\n## Error handling\n\nStorage implementations map their internal errors\nto shared error types, which are contained in `blobby.error`.\n\n```python\nfrom blobby.error import NoSuchKeyError\n\ntry:\n    storage.get(\"test\")\nexcept NoSuchKeyError as err:\n    # do something with err\n    pass\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidsteiner%2Fblobby","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidsteiner%2Fblobby","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidsteiner%2Fblobby/lists"}