{"id":18964134,"url":"https://github.com/janeliascicomp/hortacloud-importer","last_synced_at":"2025-04-19T12:50:17.578Z","repository":{"id":39699292,"uuid":"450619273","full_name":"JaneliaSciComp/hortacloud-importer","owner":"JaneliaSciComp","description":"a tiff-to-octree converter for dask","archived":false,"fork":false,"pushed_at":"2024-11-15T20:18:12.000Z","size":97,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-29T08:02:55.123Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JaneliaSciComp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-01-21T19:41:32.000Z","updated_at":"2024-11-15T20:18:16.000Z","dependencies_parsed_at":"2024-04-24T21:26:46.522Z","dependency_job_id":"583a9c7d-3b35-41a1-ab28-98ec232c9583","html_url":"https://github.com/JaneliaSciComp/hortacloud-importer","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JaneliaSciComp%2Fhortacloud-importer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JaneliaSciComp%2Fhortacloud-importer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JaneliaSciComp%2Fhortacloud-importer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JaneliaSciComp%2Fhortacloud-importer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JaneliaSciComp","download_url":"https://codeload.github.com/JaneliaSciComp/hortacloud-importer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249201117,"owners_count":21229004,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T14:22:58.923Z","updated_at":"2025-04-16T05:31:48.279Z","avatar_url":"https://github.com/JaneliaSciComp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HortaCloud Data Importer\n\n[![DOI](https://zenodo.org/badge/450619273.svg)](https://zenodo.org/badge/latestdoi/450619273)\n\nImport microscopy data in various formats into the native KTX format for [HortaCloud](https://github.com/JaneliaSciComp/hortacloud).\n\nThis converter can convert TIFF slices, TIFF stacks, N5/Zarr and JPEG2 slices to the multi-resolution octree format (TIFF and/or KTX) for the Horta 3D viewer. The Horta 3D viewer can render the octree dataset and dynamically shift its resolution level depending on the zoom ratio. At low zoom, lower resolution images are displayed. When zoomed in, higher resolution images are loaded. This converter automatically determines the optimal number of levels for your data.\n\n## Initial Setup\n1. Install miniconda (https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html)\n2. Clone pyktx (https://github.com/JaneliaSciComp/pyktx)\n3. Run ```conda config --set channel_priority strict```\n3. Run ```conda env create -f environment.yml```\n4. Run ```conda activate octree```\n5. Run ```pip install /path/to/pyktx```\n6. Run ```conda config --set auto_activate_base false``` (for LSF cluster)\n\n\n## Convert Tiff Slices or N5/Zarr datasets on a Local PC\n1. Create an input folder for putting your tiff slices.\n2. Copy your tiff slices to the input folder.\n3. Create an output folder.\n4. Run the following command.\n```\nconda activate octree\npython tiff2octree.py -i /path/to/input_dir -o /path/to/output_dir -d 2ndmax -t 16 --ktx\n```\nThis command generates both tiff and ktx octrees.  \n```\n-i: set path to your input folder.  \n-o: set path to your output folder.  \n-d: downsampling method. you can use 2ndmax, area, aa (anti-aliasing), spline. (2ndmax is being used for the mousdlight project.)  \n-t: number of threads.  \n--ktx: generate a ktx compressed octree. You need to generate a KTX octree for browsing your data on Horta3D viewer. By default, this converter generates only a tiff octree.  \n```\nThis converter aoutomatically determines the optimal number of levels for your data if you do not set the number of levels by using -l option.\n\nIf you browse your data only on Horta3D, please use --ktxonly option. The converter will generate only a ktx octree without a tiff octree.\n```\npython tiff2octree.py -i /path/to/input_dir -o /path/to/output_dir -d 2ndmax -t 16 --ktxonly\n```\n\nYou can convert a multi-channel image by the following command. \n```\npython tiff2octree.py -i /path/to/input_dir/ch1,/path/to/input_dir/ch2 -o /output/octree/ -d 2ndmax -t 8 --ktx\n```\nYou need to create multiple folders for input data. (e.g. /input_slices/ch1, /input_slices/ch2)\n\nYou can load your local octree data to Janelia Workstation by File \u003e New \u003e Tiled Microscope Sample.\n\n## Convert a Tiff Stack on a Local PC\nThe datasize of your tiff stack must be smaller than system memory.\n1. Create an output folder.\n2. Run the following command.\n```\nconda activate octree\npython tiff2octree.py -f /input_slices/tiff -o /output/octree -d 2ndmax -t 16 --ktx\n```\n```\n-f: set path to your input tif stack.\n-o: set path to your output folder.\n-d: downsampling method. you can use 2ndmax, area, aa (anti-aliasing), spline. (2ndmax is being used for the mousdlight project.)\n-t: number of threads. \n--ktx: generate a ktx compressed octree. You need to generate a KTX octree for browsing your data on Horta3D viewer. By default, this converter generates only a tiff octree.\n```\nYou must use -f option for setting your tif stack as input.\n\n\n## Convert Tiff Slices or N5/Zarr datasets on the Janelia LSF cluster\n\n1. Create an input folder for putting your tiff slices.\n2. Copy your tiff slices to the input folder.\n3. Create an output folder.\n4. Run the following command.\n```\nconda activate octree\nbsub -n 1 -W 24:00 -o log_output.txt -P scicompsoft \"python tiff2octree.py -i /path/to/input_dir -o /path/to/output_dir -d 2ndmax -t 10 --ktx --lsf --project scicompsoft --memory 16GB --walltime 8:00\"\n```\n```\n-i: set path to your input folder.\n-o: set path to your output folder.\n-d: downsampling method. you can use 2ndmax, area, aa (anti-aliasing), spline. (2ndmax is being used for the mousdlight project.)\n-t: number of threads. \n--ktx: generate a ktx compressed octree. You need to generate a KTX octree for browsing your data on Horta3D viewer. By default, this converter generates only a tiff octree.\n--lsf: this option is necessary to use the lsf cluster.\n--project: set a project name to be charged the cost for the janelia lsf cluster.\n--memory: amount of memory per thread.\n--walltime: runtime limit of each job. The default runtime limit is 1:00. If you are trying to convert large data, you may need to set a longer time limit.\n```\n\n## Resume a Stopped Process\nIf a process is terminated in the middle of execution, you can resume it by using ```--resume``` option.\n\nif the following process is stopped in the middle of execution:\n```\nconda activate octree\nbsub -n 1 -W 24:00 -o log_output.txt -P scicompsoft \"python tiff2octree.py -i /path/to/input_dir -o /path/to/output_dir -d 2ndmax -t 10 --ktx --lsf --project scicompsoft --memory 16GB --walltime 8:00\"\n```\n\nYou can resume the process by the following command:\n```\nconda activate octree\nbsub -n 1 -W 24:00 -o log_output.txt -P scicompsoft \"python tiff2octree.py -i /path/to/input_dir -o /path/to/output_dir -d 2ndmax -t 10 --ktx --lsf --project scicompsoft --memory 16GB --walltime 8:00 --resume\"\n```\n\n## Usage\n```\ncommandline arguments:\n  -h, --help                              show this help message and exit\n  -t NUMBER, --thread NUMBER              number of threads (default: 16)\n  -i INPUT, --inputdir INPUT              input directory\n  -f FILE, --inputfile FILE               input image stack\n  -o OUTPUT, --output OUTPUT              output directory\n  -l LEVEL, --level LEVEL                 number of levels\n  -c CHANNEL, --channel CHANNEL           channel id\n  -d DOWNSAMPLE, --downsample DOWNSAMPLE  downsampling method: 2ndmax, area, aa (anti-aliasing), spline\n  -m, --monitor                           activate monitoring. \n                                          you can see the dask dashboard at \n                                          http://(NodeName).int.janelia.org:8989/status\n                                          (e.g. http://h07u01.int.janelia.org:8989/status)\n  \n  --origin ORIGIN       position of the corner of the top-level image in nanometers\n  --voxsize VOXSIZE     voxel size of the top-level image\n  --memory MEMORY       memory amount per thread (for LSF cluster)\n  --project PROJECT     project name (for LSF cluster)\n  --maxjobs MAXJOBS     maximum jobs (for LSF cluster)\n  --lsf                 use LSF cluster\n  --ktx                 generate ktx files\n  --ktxout KTXOUT       output directory for a ktx octree\n  --cluster CLUSTER     address of a dask scheduler server\n  --verbose             enable vorbose logging\n  --resume              resume processing\n\n\nexamples: \n\n1. use a local cluster. (process image slices)\nconda activate octree\npython tiff2octree.py -i /path/to/input_dir -o /path/to/output_dir -d 2ndmax -t 16\n\n2. use a local cluster. (process image stack)\nconda activate octree\npython tiff2octree.py -f /path/to/tiff_stack.tif -o /output/octree -d 2ndmax -t 16\n\n3. use a LSF cluster.\nconda activate octree\npython tiff2octree.py -i /path/to/input_dir -o /path/to/output_dir -d 2ndmax --lsf --project scicompsoft --memory 12GB --maxjobs 10 -t 10\n\n4. output a ktx octree without a tiff octree.\nconda activate octree\npython tiff2octree.py -i /path/to/input_dir/ch1,/path/to/input_dir/ch2 -o /path/to/output_dir -ktxonly -d 2ndmax -t 8\n\n5. specify a cluster by its address.\nconda activate octree\npython tiff2octree.py -i /path/to/input_dir -o /path/to/output_dir --cluster tcp://10.60.0.223:8786 -d spline -t 16\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaneliascicomp%2Fhortacloud-importer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjaneliascicomp%2Fhortacloud-importer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaneliascicomp%2Fhortacloud-importer/lists"}