{"id":20560083,"url":"https://github.com/cannylab/vdtk","last_synced_at":"2025-04-14T14:05:55.100Z","repository":{"id":39589027,"uuid":"480905200","full_name":"CannyLab/vdtk","owner":"CannyLab","description":"Visual Description Dataset Analysis Toolkit","archived":false,"fork":false,"pushed_at":"2023-07-06T16:38:54.000Z","size":42630,"stargazers_count":8,"open_issues_count":1,"forks_count":2,"subscribers_count":4,"default_branch":"main","last_synced_at":"2023-07-06T17:39:50.547Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CannyLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-04-12T17:16:17.000Z","updated_at":"2023-06-22T12:52:59.000Z","dependencies_parsed_at":"2023-01-21T23:03:50.669Z","dependency_job_id":null,"html_url":"https://github.com/CannyLab/vdtk","commit_stats":null,"previous_names":[],"tags_count":1,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CannyLab%2Fvdtk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CannyLab%2Fvdtk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CannyLab%2Fvdtk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CannyLab%2Fvdtk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CannyLab","download_url":"https://codeload.github.com/CannyLab/vdtk/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224873086,"owners_count":17384078,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T03:53:11.361Z","updated_at":"2024-11-16T03:53:11.898Z","avatar_url":"https://github.com/CannyLab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# vdtk: Visual Description Evaluation Tools\n\nThis tool is designed to allow for a deep investigation of diversity in visual description datasets, and to help users\nunderstand their data at a token, n-gram, description, and dataset level.\n\n## Installation\n\nTo use this tool, you can easily pip install with `pip install vdtk`.\n\n## Data format\n\nIn order to prepare datasets to work with this tool, datasets must be formatted as JSON files with the following schema:\n\n```python\n# List of samples in the dataset\n[\n    # JSON object for each sample\n    {\n        \"_id\": \"string\", # A string ID for each sample. This can help keep track of samples during use.\n        \"split\": \"string\", # A string corresponding to the split of the data. Default splits are \"train\", \"validate\" and \"test\"\n        \"references\": [\n            # List of string references\n            \"reference 1...\",\n            \"reference 2...\",\n        ],\n        \"candidates\": [\n            # List of string candidates (Optional)\n            \"candidate 1...\",\n            \"candidate 2...\",\n        ],\n        \"media_path\": \"string\", # (Optional) Path to the image/video (for image/video based metrics, recall experiemnts, etc.)\n        \"metadata\": {} # Any JSON object. This field is not used by the toolkit at this time.\n    }\n]\n```\n\n## Usage\n\nAfter installation, the basic menu of commands can be accessed with `vdtk --help`. We make several experiments/tools\navailable for use:\n\n| Command | Details |\n| ----------- | ----------- |\n| vocab-stats | Run with `vdtk vocab-stats DATASET_JSON_PATH`. Compute basic token-level vocab statistics |\n| ngram-stats | Run with `vdtk ngram-stats DATASET_JSON_PATH`. Compute n-gram statistics, EVS@N and ED@N  |\n| caption-stats | Run with `vdtk caption-stats DATASET_JSON_PATH`. Compute caption-level dataset statistics  |\n| semantic-variance | Run with `vdtk semantic-variance DATASET_JSON_PATH`. Compute within-sample BERT embedding semantic variance |\n| coreset | Run with `vdtk coreset DATASET_JSON_PATH`. Compute the caption coreset from the training split needed to solve the validation split |\n| concept-overlap | Run with `vdtk concept-overlap DATASET_JSON_PATH`. Compute the concept overlap between popular feature extractors, and the dataset |\n| concept-leave-one-out | Run with `vdtk concept-leave-one-out DATASET_JSON_PATH`. Compute the performance with a coreset of concept captions |\n| leave-one-out | Run with `vdtk leave-one-out DATASET_JSON_PATH`. Compute leave-one-out ground truth performance on a dataset with multiple ground truths |\n\nAdditionally, several commands take multiple dataset JSONs, which can be used to compare different runs, or different datasets. Appending (:baseline) to any\nof the JSON file paths will treat this run as a baseline, and compute relative values and coloring accordingly (example: `vdtk score cider-d ./baseline.json:baseline ./model.json`).\n\n| Command | Details |\n| ----------- | ----------- |\n| score | Run with `vdtk score [metric] DATASET_JSON_PATH_1, DATASET_JSON_PATH_2...`. Compute BLEU/METEOR/CIDEr-D/ROUGE/BERTScore/MAUVE/etc. Guaranteed to be consistent with the COCO captioning tools (for use externally). |\n| clip-recall | Run with `vdtk clip-recall DATASET_JSON_PATH_1, DATASET_JSON_PATH_2...`. Compute the MRR, and Recall@K values for candidate/reference captions based on the CLIP model. |\n| content-recall | Run with `vdtk content-recall DATASET_JSON_PATH_1, DATASET_JSON_PATH_2...`. Compute Noun/Verb recall for the candidates against the references. |\n\nFor more details and options, see the `--help` command for any of the commands above. Note that some tools are relatively\ncompute intensive. This toolkit will make use of a GPU if available and necessary, as well as a large number of CPU cores\nand RAM depending on the task.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcannylab%2Fvdtk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcannylab%2Fvdtk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcannylab%2Fvdtk/lists"}