{"id":18565640,"url":"https://github.com/quiltdata/quiltcore","last_synced_at":"2025-09-07T05:35:39.157Z","repository":{"id":202602788,"uuid":"651602370","full_name":"quiltdata/quiltcore","owner":"quiltdata","description":null,"archived":false,"fork":false,"pushed_at":"2024-11-07T13:44:57.000Z","size":568,"stargazers_count":1,"open_issues_count":19,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-12-19T19:01:39.480Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/quiltdata.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-09T15:55:02.000Z","updated_at":"2024-07-10T08:18:12.000Z","dependencies_parsed_at":"2023-11-19T19:28:58.780Z","dependency_job_id":"a7c724c4-929c-4dbb-b99c-f1129a556e56","html_url":"https://github.com/quiltdata/quiltcore","commit_stats":{"total_commits":72,"total_committers":4,"mean_commits":18.0,"dds":0.4722222222222222,"last_synced_commit":"641c78215b1187f9013542a7b60d359be8e5abb2"},"previous_names":["quiltdata/quiltcore"],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quiltdata%2Fquiltcore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quiltdata%2Fquiltcore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quiltdata%2Fquiltcore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quiltdata%2Fquiltcore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/quiltdata","download_url":"https://codeload.github.com/quiltdata/quiltcore/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239293947,"owners_count":19615043,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T22:19:36.909Z","updated_at":"2025-02-17T13:15:51.924Z","avatar_url":"https://github.com/quiltdata.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# QuiltCore\n\nQuiltCore is a library for building and running [Quilt](https://quiltdata.com) data packages.\nIt is designed to leverage standard open source technology and YAML configuration files\nso that it can easily be ported to other languages and platforms.\n\nThis initial implementation is in Python.\n\n## Key Technologies\n\n- Apache [Arrow](https://arrow.apache.org/) for reading, writing, and representing manifests\n  - [PyArrow](https://arrow.apache.org/docs/python/) for Python bindings to Arrow\n- fsspec [filesystems](https://filesystem-spec.readthedocs.io/en/latest/)\n  for reading and writing files from various sources\n- [PyYAML](https://pyyaml.org/) for reading and writing YAML configuration files\n\n## Example\n\n```bash\npoetry install\n```\n\n```python\n#!/usr/bin/env python\nimport os\nfrom quiltcore import Domain, UDI\nfrom tempfile import TemporaryDirectory\nfrom upath import UPath\n\nTEST_BKT = \"s3://quilt-example\"\nTEST_PKG = \"akarve/amazon-reviews\"\nTEST_TAG = \"1570503102\"\nTEST_HASH = \"ffe323137d0a84a9d1d6f200cecd616f434e121b3f53a8891a5c8d70f82244c2\"\nTEST_KEY = \"camera-reviews\"\nWRITE_BKT = os.environ.get(\"WRITE_BUCKET\")\nSOURCE_URI = f\"quilt+{TEST_BKT}#package={TEST_PKG}:{TEST_TAG}\"\nDEST_URI = f\"quilt+{TEST_BKT}#package={TEST_PKG}:{TEST_TAG}\"\n```\n\n### Get Manifest\n\n\u003c!--pytest-codeblocks:cont--\u003e\n```python\nremote = UDI.FromUri(SOURCE_URI)\nprint(f\"remote: {remote}\")\nwith TemporaryDirectory() as tmpdir:\n    local = UPath(tmpdir)\n    domain = Domain.FromLocalPath(local)\n    print(f\"domain: {domain}\")\n    folder = domain.pull(remote)\n    print(f\"folder: {folder}\")\n    if WRITE_BKT:\n        tag = domain.push(folder, remote=UDI.FromUri(DEST_URI))\n        print(f\"tag: {tag}\")\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquiltdata%2Fquiltcore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fquiltdata%2Fquiltcore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquiltdata%2Fquiltcore/lists"}