{"id":19448669,"url":"https://github.com/neurodata/boss-export","last_synced_at":"2026-06-13T03:31:30.326Z","repository":{"id":140561005,"uuid":"187909311","full_name":"neurodata/boss-export","owner":"neurodata","description":"Export data from BOSS in bulk from w/in BOSS AWS environment","archived":false,"fork":false,"pushed_at":"2019-12-11T19:20:01.000Z","size":474,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-07T23:27:03.862Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/neurodata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-21T20:32:08.000Z","updated_at":"2022-04-15T14:11:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"d15179c5-d5f1-4df6-95c4-591e826d5667","html_url":"https://github.com/neurodata/boss-export","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neurodata%2Fboss-export","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neurodata%2Fboss-export/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neurodata%2Fboss-export/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neurodata%2Fboss-export/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/neurodata","download_url":"https://codeload.github.com/neurodata/boss-export/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240636663,"owners_count":19832922,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T16:28:02.809Z","updated_at":"2026-06-13T03:31:25.302Z","avatar_url":"https://github.com/neurodata.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# boss-export\n\nExport data from BOSS s3 bucket in bulk direct to another bucket\n\nThis repo contains necessary tools to convert a BOSS dataset to Neuroglancer precomputed format without going through the BOSS endpoint, but by accessing the cuboids directly from S3.\n\nIt works by publishing messages (cuboid metadata) to an SQS queue and having a lambda function process those messages, converting and compressing them, into Neuroglancer precomputed format.\n\nYou must have read access to the S3 bucket in the BOSS.  Additionally, to be able to compute the s3 keys, you need access to the database IDs for collections/experiments/channels in the BOSS that you wish to convert, as those are used in determining they s3 key names.\n\n## Deployment notes\n\n### Lambda\n\nIn addition to the basic lambda execution environment permissions, the lambda role also needs\n\n- s3 getobject from BOSS bucket\n- s3 putobject and putobject on destination bucket\n- ReceiveMessage/DeleteMessage on SQS\n\n### SQS\n\nQueue visibility needs to be [6 times](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html) the lambda timeout\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneurodata%2Fboss-export","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fneurodata%2Fboss-export","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneurodata%2Fboss-export/lists"}