{"id":24786966,"url":"https://github.com/mbari-org/aidata","last_synced_at":"2026-02-05T03:01:04.393Z","repository":{"id":274685168,"uuid":"773209695","full_name":"mbari-org/aidata","owner":"mbari-org","description":"Extract, transform, and load/download and augment images and annotations for detection, clustering or classification workflows","archived":false,"fork":false,"pushed_at":"2026-02-03T00:23:49.000Z","size":24133,"stargazers_count":5,"open_issues_count":1,"forks_count":2,"subscribers_count":7,"default_branch":"main","last_synced_at":"2026-02-03T12:26:17.386Z","etag":null,"topics":["ai","database","etl","machine-learning"],"latest_commit_sha":null,"homepage":"https://docs.mbari.org/internal/ai/data","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mbari-org.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-03-17T03:02:16.000Z","updated_at":"2026-02-03T00:23:53.000Z","dependencies_parsed_at":"2025-01-28T19:38:43.129Z","dependency_job_id":"41bf0882-611f-4e0d-9d9a-2d58b85e4242","html_url":"https://github.com/mbari-org/aidata","commit_stats":null,"previous_names":["mbari-org/aidata"],"tags_count":150,"template":false,"template_full_name":null,"purl":"pkg:github/mbari-org/aidata","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbari-org%2Faidata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbari-org%2Faidata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbari-org%2Faidata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbari-org%2Faidata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mbari-org","download_url":"https://codeload.github.com/mbari-org/aidata/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbari-org%2Faidata/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29108231,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-05T02:48:39.389Z","status":"ssl_error","status_checked_at":"2026-02-05T02:48:27.400Z","response_time":65,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","database","etl","machine-learning"],"created_at":"2025-01-29T15:17:57.346Z","updated_at":"2026-02-05T03:01:04.387Z","avatar_url":"https://github.com/mbari-org.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![MBARI](https://www.mbari.org/wp-content/uploads/2014/11/logo-mbari-3b.png)](http://www.mbari.org)\n[![semantic-release](https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg)](https://github.com/semantic-release/semantic-release)\n[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n[![downloads](https://img.shields.io/pypi/dm/mbari-aidata)](https://pypistats.org/packages/mbari-aidata)\n\n__mbari-aidata__\n---\nA command line tool to do extract, transform, load (ETL) and download operations\non AI data for a number of projects at MBARI that require detection, clustering or classification\nworkflows.  This tool is designed to work with [Tator](https://www.tator.io/), a web based\nplatform for video and image annotation and data management and [Redis](https://redis.io/) \nqueues for ingesting data from real-time workflows.\n\nMore documentation and examples are available at [https://docs.mbari.org/internal/ai/data](https://docs.mbari.org/internal/ai/data/).\n \n## 🚀 Features\n* 🧠 Object Detection/Clustering Integration: Loads detection/classification/clustering output from SDCAT formatted results.\n* Flexible Data Export: Downloads from Tator into machine learning formats like COCO, CIFAR, or PASCAL VOC.\n* Crop localization into optimized datasets for training classification models.\n* Real-Time Uploads: Pushes localizations to [Tator](https://www.tator.io/) via [Redis](https://redis.io/glossary/redis-queue/) queues for real-time workflows.\n* Metadata Extraction: Parses images metadata such as GPS/time/date through a plugin-based system (extractors).\n* Duplicate Detection \u0026 flexible media references: Supports duplicate media load checks with the --check-duplicates flag. \n* Images or video can be loaded through a web server without needing to upload or move them from your internal NFS project mounts (e.g. Thalassa)\n* Or, video can be uploaded without needing to figure out how to do the video transcoding required for web viewing.\n* Video tracks can be uploaded into Tator for training and evaluation, including time-decay weighted tracks.\n* Multiple data versions can be downloaded into a single dataset for training or evaluation using the --version flag with comma separated values, otherwise localization data is combined through Non-Maximum Suppression (NMS) to remove duplicate boxes.\n* Augmentation Support: Augment VOC datasets with [Albumentations](https://albumentations.ai/) to boost your object detection model performance.\n\n## Requirements\n- Python 3.10 or higher\n- A Tator API token and (optional) Redis password for the .env file. Contact the MBARI AI team for access.\n- 🐳Docker for development and testing only, but it can also be used instead of a local Python installation.\n- For video loads, you will need to install the required Python packages listed in the `requirements.txt` file, [ffmpeg](https://ffmpeg.org/), and the mp4dump tool from [https://www.bento4.com/](https://www.bento4.com/downloads/)\n\n## 📦 Installation \nInstall as a Python package:\n\n```shell\npip install mbari-aidata\n```\n \nCreate the .env file with the following contents in the root directory of the project:\n\n```text\nTATOR_TOKEN=your_api_token\nREDIS_PASSWORD=your_redis_password\nENVIRONMENT=testing or production\n```\n\nCreate a configuration file in the root directory of the project:\n```bash\ntouch config_cfe.yaml\n```\nOr, use the project specific configuration from our docs server at\nhttps://docs.mbari.org/internal/ai/projects/\n\n\nThis file will be used to configure the project data, such as mounts, plugins, and database connections.\n```bash\naidata download --version Baseline --labels \"Diatoms, Copepods\" --config https://docs.mbari.org/internal/ai/projects/uav-901902/config_uav.yml\n```\n\n⚙️Example configuration file:\n```yaml\n# config_cfe.yml\n# Config file for CFE project production\nmounts:\n  - name: \"image\"\n    path: \"/mnt/CFElab\"\n    host: \"https://mantis.shore.mbari.org\"\n    nginx_root: \"/CFElab\"\n\n  - name: \"video\"\n    path: \"/mnt/CFElab\"\n    host: \"https://mantis.shore.mbari.org\"\n    nginx_root: \"/CFElab\"\n\n\nplugins:\n  - name: \"extractor\"\n    module: \"mbari_aidata.plugins.extractors.tap_cfe_media\"\n    function: \"extract_media\"\n\nredis:\n  host: \"doris.shore.mbari.org\"\n  port: 6382\n\nvss:\n  project: \"902111-CFE\"\n  model: \"google/vit-base-patch16-224\"\n\ntator:\n  project: \"902111-CFE\"\n  host: \"https://mantis.shore.mbari.org\"\n  image:\n    attributes:\n      iso_datetime: #\u003c-------Required for images\n        type: datetime\n      depth:\n        type: float\n  video:\n    attributes:\n      iso_start_datetime:  #\u003c-------Required for videos\n        type: datetime\n  box:\n    attributes:\n      Label:\n        type: string\n      score:\n        type: float\n      cluster:\n        type: string\n      saliency:\n        type: float\n      area:\n        type: int\n      exemplar:\n        type: bool\n  tdwa_box: #\u003c-------Optional for videos track loads\n    attributes:\n      Label:\n        type: string\n      score:\n        type: float\n      verified:\n        type: bool\n      similarity_score:\n        type: float\n  track_state:  #\u003c-------Optional for videos track loads\n    attributes:\n      Label:\n        type: string\n      max_score:\n        type: float\n      num_frames:\n        type: int\n      verified:\n        type: bool\n    \n```\n\n## Track Format\n\nTracks are video frames with a label and score for each detected object in the frame, along with a tracker_id to link\ndetections across frames into tracks.\n\nTrack data is stored in a compressed .tar.gz file with the -tracks.tar.gz, e.g.\n\n```shell\naidata load tracks --input video-tracks/tracks.tar.gz --dry-run --config config_cfe.yml\n```\n\nvideo-tracks/tracks.tar.gz. This compressed file contains a structure like:\n\nThe detections.csv file contains the detections for each frame, e.g.\n\n| frame | tracker_id | label    | score           | x                   | y                   | xx                  | xy                  |\n|-------|------------|----------|-----------------------|----------------------|----------------------|----------------------|----------------------|\n| 3     | 2          | Copepod  | 0.6826763153076172    | 0.7003568708896637   | 0.4995344939055266   | 0.7221783697605133   | 0.5368460761176215   |\n| 3     | 1          | Copepod  | 0.7094097137451172    | 0.2693319320678711   | 0.6148265485410337   | 0.29686012864112854  | 0.6434915330674913   |\n| 3     | 3          | Detritus | 0.2776843011379242    | 0.2693319320678711   | 0.6148265485410337   | 0.29686012864112854  | 0.6434915330674913   |\n| 4     | 1          | Copepod  | 0.49819645285606384   | 0.2683655321598053   | 0.6125818323206018   | 0.2965434789657593   | 0.6455737643771702   |\n\nMetadata about video is in the metadata.json file, e.g.\n```json\n{ \"video_name\": \"video.mp4\", \n  \"video_path\": \"/data/input/video.mp4\", \n  \"processed_at\": \"2025-11-15T13:37:35.997007Z\", \n  \"total_frames\": 12000, \n  \"video_width\": 1920, \n  \"video_height\": 1080, \n  \"video_fps\": 10, \n  \"total_detections\": 3000, \n  \"unique_tracks\": 148, \n  \"detection_threshold\": 0.15, \n  \"min_track_frames\": 5, \n  \"slice_size\": 800, \n  \"rfdetr_model\": \"/mnt/models/best/checkpoint_best_total.pth\" }\n```\n\nThe tracks.csv file contains the tracks for each frame, e.g.\n\n| tracker_id | label         | first_frame | last_frame | frame_count | avg_score |\n|------------|---------------|-------------|------------|-------------|----------------------|\n| 2          | Detritus | 3           | 37         | 35          | 0.3780171153800829   |\n| 1          | Copepod  | 3           | 36         | 31          | 0.5898609180604258   |\n| 3          | Copepod  | 3           | 37         | 34          | 0.5619616565458914   |\n\n## 🐳 Docker usage\nA docker version is also available at `mbari/aidata:latest` or `mbari/aidata:latest:cuda-124`.\nFor example, to download data from version Baseline using the docker image:\n\n```shell\ndocker run -it --rm -v $(pwd):/mnt mbari/aidata:latest aidata download --version Baseline --labels \"Diatoms, Copepods\" --config config_cfe.yml\n```\n\nto download multiple versions\n```shell\ndocker run -it --rm -v $(pwd):/mnt mbari/aidata:latest aidata download --version Baseline,ver0 --labels \"Diatoms, Copepods\" --config config_cfe.yml`\n```\n \n## Commands\n\n* `aidata download --help` -  Download data, such as images, boxes, into various formats for machine learning e.g. COCO, CIFAR, or PASCAL VOC format. Augmentation supported for VOC exported data using Albumentations.\n* `aidata load --help` -  Load data, such as images, boxes, or clusters into either a Postgres or REDIS database\n* `aidata db --help` -  Commands related to database management\n* `aidata transform --help` - Commands related to transforming downloaded data\n* `aidata  -h` - Print help message and exit.\n \nSource code is available at [github.com/mbari-org/aidata](https://github.com/mbari-org/aidata/). \n\n## Development\nSee the [Development Guide](https://github.com/mbari-org/aidata/blob/main/DEVELOPMENT.md) for more information on how to set up the development environment or the [justfile](justfile)  \n \n🗓️ Last updated: 2026-01-03\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmbari-org%2Faidata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmbari-org%2Faidata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmbari-org%2Faidata/lists"}