{"id":28395237,"url":"https://github.com/databiosphere/ssds","last_synced_at":"2025-06-27T01:31:14.665Z","repository":{"id":37557300,"uuid":"267850835","full_name":"DataBiosphere/ssds","owner":"DataBiosphere","description":"Simple data storage system for AWS and GCP","archived":false,"fork":false,"pushed_at":"2022-06-30T17:21:08.000Z","size":311,"stargazers_count":2,"open_issues_count":8,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-06-01T06:52:36.137Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataBiosphere.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-29T12:20:44.000Z","updated_at":"2021-12-15T19:34:07.000Z","dependencies_parsed_at":"2022-08-31T10:22:47.291Z","dependency_job_id":null,"html_url":"https://github.com/DataBiosphere/ssds","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/DataBiosphere/ssds","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fssds","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fssds/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fssds/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fssds/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataBiosphere","download_url":"https://codeload.github.com/DataBiosphere/ssds/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fssds/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262172362,"owners_count":23269991,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-31T19:39:36.236Z","updated_at":"2025-06-27T01:31:14.633Z","avatar_url":"https://github.com/DataBiosphere.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ssds\nThe s\u003csup\u003e1\u003c/sup\u003e simple data store.\n\nUpload directory trees to S3 or GS cloud buckets as \"submissions\". Each submission takes a user assigned\nidentifier and a human readable name. The cloud location of the submission has the key structure\n`submissions/{uuid}--{name}/{tree}`.\n\nAll uploads are checksummed and verified. Multpart uploads use defined chunk sizes to consistently track compose S3\nETags.\n\n# Installation\n\n```\npip install git+https://github.com/DataBiosphere/ssds\n```\n\n# Usage\nMake a new submission\n\n```\nssds staging upload --submission-id my_submission_id --name my_cool_submission_name /local/path/to/my/submission\n```\n\nUpdate an existing submission\n\n```\nssds staging upload --submission-id my_existing_submission_id /local/path/to/my/submission\n```\n\nList all staging submissions\n\n```\nssds staging list\n```\n\nList contents of a staging submission\n\n```\nssds staging list-submission --submission-id my_submission_id\n```\n\nThe above commands can target staging deployments other than the default with the `--deployment` argument.\n\nAvailable deployments can be listed with\n\n```\nssds deployment list-staging\n```\n\nand\n\n```\nssds deployment list-release\n```\n\nSubmissions can be synced between staging deployments with\n```\nssds staging sync --submission-id my_existing_submission_id --dst-deployment my_dst_deployment\n```\n\n## Configuring Billing Projects for Requester Pays Google Buckets\n\nFor working with requester pays Google Storage buckets, the billing project is specified by setting the\nenvironment variable `GOOGLE_PROJECT`, e.g.\n```\nexport GOOGLE_PROJECT=\"my-gcp-billing-project\"\n```\n\n# Developing\n\nRun tests with \n```\nmake test\n```\nIf `mypy` linting fails, you may need to run\n```\nmypy --install-types\n```\n\nMany tests require access to test buckets listed in `ssds/deployment.py`.\n\nThese buckets are in the `pangenomics` AWS account.\nBe sure to configure your S3 and GS credentials have access.\n\n## Links\nProject home page [GitHub](https://github.com/DataBiosphere/ssds)  \n\n### Bugs\nPlease report bugs, issues, feature requests, etc. on [GitHub](https://github.com/DataBiosphere/ssds).\n\n![](https://travis-ci.org/DataBiosphere/ssds.svg?branch=master)\n\n\u003csup\u003e1\u003c/sup\u003esuper, splendidly, serendipitous, sometimes, sporadically, etc.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fssds","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabiosphere%2Fssds","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fssds/lists"}