{"id":19215867,"url":"https://github.com/vanvalenlab/caliban-toolbox","last_synced_at":"2025-05-12T23:28:00.771Z","repository":{"id":55894408,"uuid":"140756012","full_name":"vanvalenlab/caliban-toolbox","owner":"vanvalenlab","description":"Data engineering toolbox for DeepCell","archived":false,"fork":false,"pushed_at":"2021-07-16T19:06:10.000Z","size":159846,"stargazers_count":7,"open_issues_count":22,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-20T19:38:05.542Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vanvalenlab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-07-12T19:26:15.000Z","updated_at":"2023-07-22T05:24:49.000Z","dependencies_parsed_at":"2022-08-15T08:50:41.060Z","dependency_job_id":null,"html_url":"https://github.com/vanvalenlab/caliban-toolbox","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vanvalenlab%2Fcaliban-toolbox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vanvalenlab%2Fcaliban-toolbox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vanvalenlab%2Fcaliban-toolbox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vanvalenlab%2Fcaliban-toolbox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vanvalenlab","download_url":"https://codeload.github.com/vanvalenlab/caliban-toolbox/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253839431,"owners_count":21972321,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T14:15:06.477Z","updated_at":"2025-05-12T23:28:00.746Z","avatar_url":"https://github.com/vanvalenlab.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Caliban Toolbox: Data Engineering Tools for Single Cell Analysis\n\n[![Build Status](https://github.com/vanvalenlab/caliban-toolbox/workflows/build/badge.svg)](https://github.com/vanvalenlab/caliban-toolbox/actions)\n[![Coverage Status](https://coveralls.io/repos/github/vanvalenlab/caliban-toolbox/badge.svg?branch=master)](https://coveralls.io/github/vanvalenlab/caliban-toolbox?branch=master)\n\nCaliban Toolbox is a collection of data engineering tools to process and curate crowdsourced image annotations using [Caliban](https://github.com/vanvalenlab/caliban), our data annotation tool. The Toolbox and Caliban work together to generate  annotations for training [DeepCell](https://github.com/vanvalenlab/deepcell-tf).\n\nThe process is as follows:\n\n1. Raw data is imported using the data loader, which allows the user to select data based on imaging platform, cell type, and marker of interest. \n\n2. The raw data can then be run through deepcell-tf to produce predicted labels.\n\n3. After making predictions with deepcell, the raw data is processed to make it easier for annotators to view. This includes applying filters, adjusting the contrast, etc. Multiple channels can be combined together, including the user-generated modified channels, to create summed channels. Following these modifications, the user selects which of these channels will be included for the annotators to see.\n\n4. The size of the images is then modified to make annotation easier. In order to get high quality annotations, it is important that the images are not so large that the annotators miss errors. Therefore, the images can be cropped into overlapping 2D regions to break up large FOVs. Stacks of images can be further sliced into smaller, more manageable pieces. \n\n5. Once the image dimensions have been set, each unique crop or slice is saved as an NPZ file. During this process, a JSON file is created which stores the necessary data to reconstruct the original image after annotation.\n\n6. The NPZ files are then uploaded to a cloud bucket, where they can be accesssed by the crowdsource platform. During the upload process, the user specifies an existing job to use a template, which populates the instructions for the annotators and the job settings. A log file is also created with the necessary information to download the annotations once the job is completed.\n\n7. Once the job is completed, the corrected annotations are downloaded from the AWS bucket, where they are stored as the job progresses.\n\n8. These annotations are then stitched back together, and saved as full-size NPZ files to be manually inspected for errors.\n\n9. Following correction, the individual caliban NPZ files are combined together into a single training data NPZ, and saved in the appropriate location in the training data ontology. \n\n## Getting Started\n\n### Build a local docker container\n\n```bash\ngit clone https://github.com/vanvalenlab/caliban-toolbox.git\ncd caliban-toolbox\ndocker build -t $USER/caliban_toolbox .\n\n```\n\n### Run the new docker image\n\n```bash\nNV_GPU='0' nvidia-docker run -it \\\n  -p 8888:8888 \\\n  $USER/caliban_toolbox:latest\n```\n\nIt can also be helpful to mount the local copy of the repository and the scripts to speed up local development.\n\n```bash\nNV_GPU='0' nvidia-docker run -it \\\n  -p 8888:8888 \\\n  -v $PWD/caliban_toolbox:/usr/local/lib/python3.7/site-packages/caliban_toolbox/ \\\n  -v $PWD/notebooks:/notebooks \\\n  -v /data:/data \\\n  $USER/caliban_toolbox:latest\n```\n\n## Copyright\n\nCopyright © 2016-2020 [The Van Valen Lab](http://www.vanvalen.caltech.edu/) at the California Institute of Technology (Caltech), with support from the Paul Allen Family Foundation, Google, \u0026 National Institutes of Health (NIH) under Grant U24CA224309-01.  \nAll rights reserved.\n\n## License\n\nThis software is licensed under a modified [APACHE2](LICENSE).\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\nSee [LICENSE](LICENSE) for full details.\n\n## Trademarks\n\nAll other trademarks referenced herein are the property of their respective owners.\n\n## Credits\n\n[![Van Valen Lab, Caltech](https://upload.wikimedia.org/wikipedia/commons/7/75/Caltech_Logo.svg)](http://www.vanvalen.caltech.edu/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvanvalenlab%2Fcaliban-toolbox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvanvalenlab%2Fcaliban-toolbox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvanvalenlab%2Fcaliban-toolbox/lists"}