{"id":21643921,"url":"https://github.com/application-research/radiant-data-to-delta","last_synced_at":"2025-03-19T08:47:39.590Z","repository":{"id":82191772,"uuid":"606274102","full_name":"application-research/radiant-data-to-delta","owner":"application-research","description":"Radiant MLHub Data to Delta","archived":false,"fork":false,"pushed_at":"2023-03-01T00:50:17.000Z","size":80501,"stargazers_count":1,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-25T04:11:10.140Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/application-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-25T02:14:06.000Z","updated_at":"2023-03-13T04:39:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"e139399d-04a7-4945-8300-7597f2a920c2","html_url":"https://github.com/application-research/radiant-data-to-delta","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/application-research%2Fradiant-data-to-delta","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/application-research%2Fradiant-data-to-delta/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/application-research%2Fradiant-data-to-delta/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/application-research%2Fradiant-data-to-delta/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/application-research","download_url":"https://codeload.github.com/application-research/radiant-data-to-delta/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244394463,"owners_count":20445634,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-25T05:36:53.424Z","updated_at":"2025-03-19T08:47:39.577Z","avatar_url":"https://github.com/application-research.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Radiant MLHub Data to Delta\n\nA simple script to download datasets from Radiant MLHub and uploads all of them to Delta.\n\nNote: This will run multiple threads on the host server. \n\n## Flow\n- Pull all datasets from Radiant\n- Download and save it on the host PC / server\n- Upload all extracted and decompressed content to Delta\n\nEach upload will return a delta `content_id` which can be monitored using the following delta endpoint.\n```\ncurl --location --request GET 'http://shuttle-4-bs2.estuary.tech:1414/api/v1/stats/content/\u003ccontent_id\u003e' \\\n--header 'Authorization: Bearer [ESTUARY_API_KEY]'\n```\n\nReturns the following result from delta\n\n```\n{\n    \"piece_commitment\": [\n        {\n            \"ID\": 20884,\n            \"cid\": \"bafybeihiecc3nltivel4mckhsuzzmf42rcuzcrb357enha7bvbimohxewy\",\n            \"piece\": \"baga6ea4seaqbstd72jvvpdtylm5zlf4lv7eflrlbrou46nledefeacp2k2hjogy\",\n            \"size\": 3824,\n            \"padded_piece_size\": 4096,\n            \"unnpadded_piece_size\": 4064,\n            \"status\": \"open\",\n            \"last_message\": \"\",\n            \"created_at\": \"2023-02-25T02:20:55.195719Z\",\n            \"updated_at\": \"2023-02-25T02:20:55.195719Z\"\n        }\n    ],\n    \"content\": {\n        \"ID\": 21395,\n        \"name\": \"nasa_marine_debris_labels_20181124_155715_1049_16765-29692-16.json\",\n        \"size\": 3716,\n        \"cid\": \"bafybeihiecc3nltivel4mckhsuzzmf42rcuzcrb357enha7bvbimohxewy\",\n        \"piece_commitment_id\": 20884,\n        \"status\": \"deal-proposal-failed\",\n        \"connection_mode\": \"e2e\",\n        \"last_message\": \"connecting to f0123456: lotus error: failed to load miner actor state: actor code is not miner: account\",\n        \"created_at\": \"2023-02-25T02:20:54.221897Z\",\n        \"updated_at\": \"2023-02-25T02:20:57.354899Z\"\n    },\n    \"deal_proposals\": [\n        {\n            \"ID\": 19266,\n            \"content\": 21395,\n            \"label\": \"bafybeihiecc3nltivel4mckhsuzzmf42rcuzcrb357enha7bvbimohxewy\",\n            \"duration\": 1494720,\n            \"remove_unsealed_copy\": true,\n            \"skip_ipni_announce\": true,\n            \"created_at\": \"2023-02-25T02:20:54.777549Z\",\n            \"updated_at\": \"2023-02-25T02:20:54.777549Z\"\n        }\n    ],\n    \"deals\": null\n}\n```\n\n## Installation\n```bash\npip install radiant_mlhub\n```\n\n## Usage\nInitialize your Radiant API key\n```\nmlhub configure\nAPI Key: Enter your API key here...\n```\n\n## Download the files first\n```bash\npython main.py \u003cminer\u003e \u003cestuary_api_key\u003e all true false false\n```\n\n## Group the files\n```bash\npython main.py \u003cminer\u003e \u003cestuary_api_key\u003e all false true false\n```\n\n## Push the batch files to delta\n```bash\npython main.py \u003cminer\u003e \u003cestuary_api_key\u003e all false false true\n```\n\n# Author\nOutercore Engineering Team.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapplication-research%2Fradiant-data-to-delta","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapplication-research%2Fradiant-data-to-delta","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapplication-research%2Fradiant-data-to-delta/lists"}