{"id":19227982,"url":"https://github.com/snakemake/snakemake-interface-storage-plugins","last_synced_at":"2026-02-20T14:02:08.699Z","repository":{"id":196575197,"uuid":"694761252","full_name":"snakemake/snakemake-interface-storage-plugins","owner":"snakemake","description":"This package provides a stable interface for interactions between Snakemake and its storage plugins.","archived":false,"fork":false,"pushed_at":"2024-03-18T05:10:39.000Z","size":110,"stargazers_count":1,"open_issues_count":8,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-04-07T11:00:29.250Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/snakemake.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-09-21T16:30:37.000Z","updated_at":"2024-04-15T09:49:22.214Z","dependencies_parsed_at":"2023-12-17T20:46:22.970Z","dependency_job_id":"edf00d8c-8c0c-4962-ae68-fd59f79b5114","html_url":"https://github.com/snakemake/snakemake-interface-storage-plugins","commit_stats":null,"previous_names":["snakemake/snakemake-interface-storage-plugins"],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snakemake%2Fsnakemake-interface-storage-plugins","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snakemake%2Fsnakemake-interface-storage-plugins/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snakemake%2Fsnakemake-interface-storage-plugins/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snakemake%2Fsnakemake-interface-storage-plugins/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/snakemake","download_url":"https://codeload.github.com/snakemake/snakemake-interface-storage-plugins/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249982555,"owners_count":21355718,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T15:26:10.874Z","updated_at":"2026-02-20T14:02:08.694Z","avatar_url":"https://github.com/snakemake.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# snakemake-interface-storage-plugins\n\nThis package provides a stable interface for interactions between Snakemake and its storage plugins.\n\nPlugins should implement the following skeleton to comply with this interface.\nIt is recommended to use [Snakedeploy to set up the skeleton](https://snakedeploy.readthedocs.io/en/stable/snakemake_developers/scaffold_snakemake_plugins.html) (and automated testing) within a python package.\n\n```python\nfrom dataclasses import dataclass, field\nfrom typing import Any, Iterable, Optional, List\nfrom snakemake_interface_storage_plugins.settings import StorageProviderSettingsBase\nfrom snakemake_interface_storage_plugins.storage_provider import (\n    StorageProviderBase,\n    StorageQueryValidationResult,\n    ExampleQuery,\n    Operation,\n)\nfrom snakemake_interface_storage_plugins.storage_object import (\n    StorageObjectRead,\n    StorageObjectWrite,\n    StorageObjectGlob,\n    StorageObjectTouch,\n    retry_decorator,\n)\nfrom snakemake_interface_storage_plugins.io import IOCacheStorageInterface\n\n\n# Optional:\n# Define settings for your storage plugin (e.g. host url, credentials).\n# They will occur in the Snakemake CLI as --storage-\u003cstorage-plugin-name\u003e-\u003cparam-name\u003e\n# Make sure that all defined fields are 'Optional' and specify a default value\n# of None or anything else that makes sense in your case.\n# Note that we allow storage plugin settings to be tagged by the user. That means,\n# that each of them can be specified multiple times (an implicit nargs=+), and\n# the user can add a tag in front of each value (e.g. tagname1:value1 tagname2:value2).\n# This way, a storage plugin can be used multiple times within a workflow with different\n# settings.\n@dataclass\nclass StorageProviderSettings(StorageProviderSettingsBase):\n    myparam: Optional[int] = field(\n        default=None,\n        metadata={\n            \"help\": \"Some help text\",\n            # Optionally request that setting is also available for specification\n            # via an environment variable. The variable will be named automatically as\n            # SNAKEMAKE_\u003cstorage-plugin-name\u003e_\u003cparam-name\u003e, all upper case.\n            # This mechanism should only be used for passwords, usernames, and other\n            # credentials.\n            # For other items, we rather recommend to let people use a profile\n            # for setting defaults\n            # (https://snakemake.readthedocs.io/en/stable/executing/cli.html#profiles).\n            \"env_var\": False,\n            # Optionally specify a function that parses the value given by the user.\n            # This is useful to create complex types from the user input.\n            \"parse_func\": ...,\n            # If a parse_func is specified, you also have to specify an unparse_func\n            # that converts the parsed value back to a string.\n            \"unparse_func\": ...,\n            # Optionally specify that setting is required when the executor is in use.\n            \"required\": True,\n            # Optionally specify multiple args with \"nargs\": True\n        },\n    )\n\n\n# Required:\n# Implementation of your storage provider\n# This class can be empty as the one below.\n# You can however use it to store global information or maintain e.g. a connection\n# pool.\n# Inside of the provider, you can use self.logger (a normal Python logger of type \n# logging.Logger) to log any additional informations or\n# warnings.\nclass StorageProvider(StorageProviderBase):\n    # For compatibility with future changes, you should not overwrite the __init__\n    # method. Instead, use __post_init__ to set additional attributes and initialize\n    # futher stuff.\n\n    def __post_init__(self):\n        # This is optional and can be removed if not needed.\n        # Alternatively, you can e.g. prepare a connection to your storage backend here.\n        # and set additional attributes.\n        pass\n\n    @classmethod\n    def example_queries(cls) -\u003e List[ExampleQuery]:\n        \"\"\"Return valid example queries (at least one) with description.\"\"\"\n        ...\n\n    def rate_limiter_key(self, query: str, operation: Operation) -\u003e Any:\n        \"\"\"Return a key for identifying a rate limiter given a query and an operation.\n\n        This is used to identify a rate limiter for the query.\n        E.g. for a storage provider like http that would be the host name.\n        For s3 it might be just the endpoint URL.\n        \"\"\"\n        ...\n\n    def default_max_requests_per_second(self) -\u003e float:\n        \"\"\"Return the default maximum number of requests per second for this storage\n        provider.\"\"\"\n        ...\n\n    def use_rate_limiter(self) -\u003e bool:\n        \"\"\"Return False if no rate limiting is needed for this provider.\"\"\"\n        ...\n\n    @classmethod\n    def is_valid_query(cls, query: str) -\u003e StorageQueryValidationResult:\n        \"\"\"Return whether the given query is valid for this storage provider.\"\"\"\n        # Ensure that also queries containing wildcards (e.g. {sample}) are accepted\n        # and considered valid. The wildcards will be resolved before the storage\n        # object is actually used.\n        ...\n\n    # If required, overwrite the method postprocess_query from StorageProviderBase\n    # in order to e.g. normalize the query or add information from the settings to it.\n    # Otherwise, remove this method as it will be inherited from the base class.\n    def postprocess_query(self, query: str) -\u003e str:\n        return query\n\n    # This can be used to change how the rendered query is displayed in the logs to\n    # prevent accidentally printing sensitive information e.g. tokens in a URL.\n    def safe_print(self, query: str) -\u003e str:\n        \"\"\"Process the query to remove potentially sensitive information when printing.\n        \"\"\"\n        return query\n\n\n# Required:\n# Implementation of storage object. If certain methods cannot be supported by your\n# storage (e.g. because it is read-only see\n# snakemake-storage-http for comparison), remove the corresponding base classes\n# from the list of inherited items.\n# Inside of the object, you can use self.provider to access the provider (e.g. for )\n# self.provider.logger, see above, or self.provider.settings).\nclass StorageObject(\n    StorageObjectRead,\n    StorageObjectWrite,\n    StorageObjectGlob,\n    StorageObjectTouch\n):\n    # For compatibility with future changes, you should not overwrite the __init__\n    # method. Instead, use __post_init__ to set additional attributes and initialize\n    # futher stuff.\n\n    def __post_init__(self):\n        # This is optional and can be removed if not needed.\n        # Alternatively, you can e.g. prepare a connection to your storage backend here.\n        # and set additional attributes.\n        pass\n\n    async def inventory(self, cache: IOCacheStorageInterface):\n        \"\"\"From this file, try to find as much existence and modification date\n        information as possible. Only retrieve that information that comes for free\n        given the current object.\n        \"\"\"\n        # This is optional and can be left as is\n\n        # If this is implemented in a storage object, results have to be stored in\n        # the given IOCache object, using self.cache_key() as key.\n        # Optionally, this can take a custom local suffix, needed e.g. when you want \n        # to cache more items than the current query: self.cache_key(local_suffix=...)\n        pass\n\n    def get_inventory_parent(self) -\u003e Optional[str]:\n        \"\"\"Return the parent directory of this object.\"\"\"\n        # this is optional and can be left as is\n        return None\n\n    def local_suffix(self) -\u003e str:\n        \"\"\"Return a unique suffix for the local path, determined from self.query.\"\"\"\n        ...\n\n    def cleanup(self):\n        \"\"\"Perform local cleanup of any remainders of the storage object.\"\"\"\n        # self.local_path() should not be removed, as this is taken care of by\n        # Snakemake.\n        ...\n\n    # Fallible methods should implement some retry logic.\n    # The easiest way to do this (but not the only one) is to use the retry_decorator\n    # provided by snakemake-interface-storage-plugins.\n    @retry_decorator\n    def exists(self) -\u003e bool:\n        # return True if the object exists\n        ...\n\n    @retry_decorator\n    def mtime(self) -\u003e float:\n        # return the modification time\n        ...\n\n    @retry_decorator\n    def size(self) -\u003e int:\n        # return the size in bytes\n        ...\n\n    @retry_decorator\n    def local_footprint(self) -\u003e int:\n        # Local footprint is the size of the object on the local disk.\n        # For directories, this should return the recursive sum of the\n        # directory file sizes.\n        # If the storage provider supports ondemand eligibility (see retrieve_object()\n        # below), this should return 0 if the object is not downloaded but e.g.\n        # mounted upon retrieval.\n        # If this method is not overwritten here, it defaults to self.size().\n        ...\n\n    @retry_decorator\n    def retrieve_object(self):\n        # Ensure that the object is accessible locally under self.local_path()\n        # Optionally, this can make use of the attribute self.is_ondemand_eligible,\n        # which indicates that the object could be retrieved on demand,\n        # e.g. by only symlinking or mounting it from whatever network storage this\n        # plugin provides. For example, objects with self.is_ondemand_eligible == True\n        # could mount the object via fuse instead of downloading it.\n        # The job can then transparently access only the parts that matter to it\n        # without having to wait for the full download.\n        # On demand eligibility is calculated via Snakemake's access pattern annotation.\n        # If no access pattern is annotated by the workflow developers,\n        # self.is_ondemand_eligible is by default set to False.\n        ...\n\n    # The following two methods are only required if the class inherits from\n    # StorageObjectReadWrite.\n\n    @retry_decorator\n    def store_object(self):\n        # Ensure that the object is stored at the location specified by\n        # self.local_path().\n        ...\n\n    @retry_decorator\n    def remove(self):\n        # Remove the object from the storage.\n        ...\n\n    # The following method is only required if the class inherits from\n    # StorageObjectGlob.\n\n    @retry_decorator\n    def list_candidate_matches(self) -\u003e Iterable[str]:\n        \"\"\"Return a list of candidate matches in the storage for the query.\"\"\"\n        # This is used by glob_wildcards() to find matches for wildcards in the query.\n        # The method has to return concretized queries without any remaining wildcards.\n        # Use snakemake_executor_plugins.io.get_constant_prefix(self.query) to get the\n        # prefix of the query before the first wildcard.\n        ...\n\n    # The following method is only required if the class inherits from\n    # StorageObjectTouch\n    @retry_decorator\n    def touch(self):\n        \"\"\"Touch the object, updating its modification date.\"\"\"\n        ...\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnakemake%2Fsnakemake-interface-storage-plugins","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnakemake%2Fsnakemake-interface-storage-plugins","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnakemake%2Fsnakemake-interface-storage-plugins/lists"}