{"id":26689308,"url":"https://github.com/opendatacube/odc-tools","last_synced_at":"2025-05-16T15:07:39.385Z","repository":{"id":37257942,"uuid":"152664678","full_name":"opendatacube/odc-tools","owner":"opendatacube","description":"Useful tools and libraries for working with Open Datacube","archived":false,"fork":false,"pushed_at":"2025-05-05T16:57:19.000Z","size":4627,"stargazers_count":61,"open_issues_count":41,"forks_count":33,"subscribers_count":20,"default_branch":"develop","last_synced_at":"2025-05-05T18:03:32.241Z","etag":null,"topics":["hacktoberfest","opendatacube","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opendatacube.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-10-11T22:46:28.000Z","updated_at":"2025-04-24T05:08:50.000Z","dependencies_parsed_at":"2024-02-12T18:27:21.349Z","dependency_job_id":"168fa2f6-d1c0-4f5c-b4f1-65b8735164e5","html_url":"https://github.com/opendatacube/odc-tools","commit_stats":{"total_commits":1600,"total_committers":49,"mean_commits":32.6530612244898,"dds":"0.40874999999999995","last_synced_commit":"e1f76a4d60c11a6ac83a85598d7e8af888e6c886"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opendatacube%2Fodc-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opendatacube%2Fodc-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opendatacube%2Fodc-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opendatacube%2Fodc-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opendatacube","download_url":"https://codeload.github.com/opendatacube/odc-tools/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254553958,"owners_count":22090417,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest","opendatacube","python3"],"created_at":"2025-03-26T14:23:37.008Z","updated_at":"2025-05-16T15:07:39.324Z","avatar_url":"https://github.com/opendatacube.png","language":"Python","funding_links":[],"categories":["Open Data Cube"],"sub_categories":["Testing your code"],"readme":"[![Test Status](https://github.com/opendatacube/odc-tools/actions/workflows/main.yml/badge.svg)](https://github.com/opendatacube/odc-tools/actions/workflows/main.yml)\n[![codecov](https://codecov.io/gh/opendatacube/odc-tools/branch/develop/graph/badge.svg?token=PovpVLRFwn)](https://codecov.io/gh/opendatacube/odc-tools)\n\nDEA Prototype Code\n==================\n\nThis repository provides developmental [libraries](https://github.com/opendatacube/odc-tools/tree/develop/libs)\nand [CLI tools](https://github.com/opendatacube/odc-tools/tree/develop/apps) for Open Datacube.\n\n- AWS S3 tools\n- CLIs for using ODC data from AWS S3 and SQS \n- Utilities for data visualizations in notebooks\n- Experiments on optimising Rasterio usage on AWS S3 \n\nFull list of libraries, and install instructions:\n\n- `odc.ui` tools for data visualization in notebook/lab\n- `odc.io` common IO utilities, used by apps mainly\n- `odc-cloud[ASYNC,AZURE,THREDDS]` cloud crawling support package\n  - `odc.aws` AWS/S3 utilities, used by apps mainly\n  - `odc.aio` faster concurrent fetching from S3 with async, used by apps `odc-cloud[ASYNC]`\n  - `odc.{thredds,azure}` internal libs for cloud IO `odc-cloud[THREDDS,AZURE]`\n\n## Promoted to their own repositories \n- `odc.stats` large scale processing framework (Moved to [odc-stats](http://github.com/opendatacube/odc-stats))\n- `odc.stac` STAC to ODC conversion tools (Moved to [odc-stac](https://github.com/opendatacube/odc-stac))\n- `odc.dscache` experimental key-value store where `key=UUID`, `value=Dataset` (moved to [odc-dscache](https://github.com/opendatacube/odc-dscache))\n\nInstallation\n============\n\nLibraries and applications in this repository are published to PyPI, and can be installed \\\nwith `pip` like so:\n\n```\npip install \\\n  odc-ui \\\n  odc-stac \\\n  odc-stats \\\n  odc-io \\\n  odc-cloud[ASYNC] \\\n  odc-dscache\n```\n\nFor Conda Users\n---------------\n\nSome **odc-tools** are available via `conda` from the `conda-forge` channel.\n\n\n```\nconda install -c conda-forge odc-apps-dc-tools odc-io odc-cloud \n\n```\n\n\nCloud Tools\n===========\n\nInstallation\n------------\n\nCloud tools depend on the `aiobotocore` package, which depends on specific\nversions of `botocore`. Another package we use, `boto3`, also depends on\nspecific versions of `botocore`. As a result, having both `aiobotocore` and\n`boto3` in one environment can be a bit tricky. The way to solve this\nis to install `aiobotocore[awscli,boto3]` before anything else, which will install\ncompatible versions of `boto3` and `awscli` into the environment.\n\n```\npip install -U \"aiobotocore[awscli,boto3]==1.3.3\"\n# OR for conda setups\nconda install \"aiobotocore==1.3.3\" boto3 awscli\n```\n\n\n1. For cloud (AWS only)\n   ```\n   pip install odc-apps-cloud\n   ```\n2. For cloud (GCP, THREDDS and AWS)\n   ```\n   pip install odc-apps-cloud[GCP,THREDDS]\n   ```\n2. For `dc-index-from-tar` (indexing to datacube from tar archive)\n   ```\n   pip install odc-apps-dc-tools\n   ```\n\nApps\n----\n\n1. `s3-find` list S3 bucket with wildcard\n2. `s3-to-tar` fetch documents from S3 and dump them to a tar archive\n3. `gs-to-tar` search GS for documents and dump them to a tar archive\n4. `dc-index-from-tar` read yaml documents from a tar archive and add them to datacube\n\n\nExample:\n\n```bash\n#!/bin/bash\n\ns3_src='s3://dea-public-data/L2/sentinel-2-nrt/**/*.yaml'\n\ns3-find \"${s3_src}\" | \\\n  s3-to-tar | \\\n    dc-index-from-tar --env s2 --ignore-lineage\n```\n\nFastest way to list regularly placed files is to use fixed depth listing:\n\n```bash\n#!/bin/bash\n\n# only works when your metadata is same depth and has fixed file name\ns3_src='s3://dea-public-data/L2/sentinel-2-nrt/S2MSIARD/*/*/ARD-METADATA.yaml'\n\ns3-find --skip-check \"${s3_src}\" | \\\n  s3-to-tar | \\\n    dc-index-from-tar --env s2 --ignore-lineage\n```\n\nWhen using Google Storage:\n\n```bash\n#!/bin/bash\n\n# Google Storage support\ngs-to-tar --bucket data.deadev.com --prefix mangrove_cover\ndc-index-from-tar --protocol gs --env mangroves --ignore-lineage metadata.tar.gz\n```\n\n\nLocal Development\n=================\n\nThe following steps are used in the GitHub Actions workflow `main.yml`\n\n```bash\n\n# build environment from file\nmamba env create -f tests/test-env.yml\n\n# this environment name is defined in tests/test-env.yml file\nconda activate odc-tools-tests\n\n# install additional packages\n./scripts/dev-install.sh --no-deps\n\n# setup database for testing\n./scripts/setup-test-db.sh\n\n# run test\necho \"Running Tests\"\npytest --cov=. \\\n--cov-report=html \\\n--cov-report=xml:coverage.xml \\\n--timeout=30 \\\nlibs apps\n\n# Optional, to delete the environment\nconda env remove -n odc-tools-tests\n```\n\nUse `conda env update -f \u003cfile\u003e` to install all needed dependencies for\n`odc-tools` libraries and apps.\n\n\u003cdetails\u003e\u003csummary\u003eConda `environment.yaml` (click to expand)\u003c/summary\u003e\u003cdiv markdown=\"1\"\u003e\n\n```yaml\nchannels:\n  - conda-forge\ndependencies:\n  # Datacube\n  - datacube\u003e=1.8.5\n\n  # odc.dscache\n  - python-lmdb\n  - zstandard\n\n  # odc.ui\n  - ipywidgets\n  - ipyleaflet\n  - tqdm\n\n  # odc-apps-dc-tools\n  - pystac\u003e=1\n  - pystac-client\u003e=0.2.0\n  - azure-storage-blob\n  - fsspec\n  - lxml  # needed for thredds-crawler\n\n  # odc.{aio,aws}: aiobotocore/boto3\n  #  pin aiobotocore for easier resolution of dependencies\n  - aiobotocore==1.3.3\n  - boto3\n\n  # eodatasets3 (used by odc-stats)\n  - boltons\n  - ciso8601\n  - python-rapidjson\n  - requests-cache\n  - ruamel.yaml\n  - structlog\n  - url-normalize\n\n  # for dev\n  - pylint\n  - autopep8\n  - flake8\n  - isort\n  - black\n  - mypy\n\n  # For tests\n  - pytest\n  - pytest-httpserver\n  - pytest-cov\n  - pytest-timeout\n  - moto\n  - deepdiff\n\n  - pip\u003e=20\n  - pip:\n      # odc.apps.dc-tools\n      - thredds-crawler\n\n      # odc.stats\n      - eodatasets3\n\n      # tests\n      - pytest-depends\n\n      # odc.ui\n      - jupyter-ui-poll\n\n      # odc-tools libs\n      - odc-stac\n      - odc-ui\n      - odc-dscache\n      - odc-stats\n\n      # odc-tools CLI apps\n      - odc-apps-cloud\n      - odc-apps-dc-tools\n```\n\u003c/div\u003e\u003c/details\u003e\n\nRelease Process\n===============\n\n1. Manually edit `{lib,app}/{pkg}/odc/{pkg}/_version.py` file to increase version number\n2. Merge changes to the `develop` branch via a Pull Request\n3. Fast-forward the `pypi/publish` branch to match `develop`\n4. Push to GitHub\n\nSteps 3 and 4 can be done by an authorized user with\n`./scripts/sync-publish-branch.sh` script.\n\n\nPublishing to [PyPi](https://pypi.org/) happens automatically when changes are\npushed to the protected `pypi/publish` branch. Only members of [Open Datacube\nAdmins](https://github.com/orgs/opendatacube/teams/admins) group have the\npermission to push to this branch.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendatacube%2Fodc-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopendatacube%2Fodc-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendatacube%2Fodc-tools/lists"}