https://github.com/om-ai-lab/ovdeval

A Comprehensive Evaluation Benchmark for Open-Vocabulary Detection (AAAI 2024)
https://github.com/om-ai-lab/ovdeval

object-detection open-vocabulary-detection vision-and-language

Last synced: 6 months ago
JSON representation

A Comprehensive Evaluation Benchmark for Open-Vocabulary Detection (AAAI 2024)

• Host: GitHub
• URL: https://github.com/om-ai-lab/ovdeval
• Owner: om-ai-lab
• License: apache-2.0
• Created: 2023-08-16T01:17:44.000Z (about 2 years ago)
• Default Branch: main
• Last Pushed: 2024-05-07T10:54:11.000Z (over 1 year ago)
• Last Synced: 2024-11-06T03:46:50.399Z (11 months ago)
• Topics: object-detection, open-vocabulary-detection, vision-and-language
• Language: Python
• Homepage:
• Size: 5.79 MB
• Stars: 35
• Watchers: 5
• Forks: 2
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# OVDEval

 A Comprehensive Evaluation Benchmark for Open-Vocabulary Detection



  [Paper 📄]   [Dataset 🗂️] 



***

OVDEval is a new benchmark for OVD model, which includes 9 sub-tasks and introduces evaluations on commonsense 

knowledge, attribute understanding, position understanding, object relation comprehension, and more. The dataset is 

meticulously created to provide hard negatives that challenge models' true understanding of visual and linguistic input.

Additionally, we identify a problem with the popular Average Precision (AP) metric when benchmarking models on these 

fine-grained label datasets and propose a new metric called Non-Maximum Suppression Average Precision (NMS-AP) to address this issue.

Check out Our AAAI24 paper [How to Evaluate the Generalization of Detection? A Benchmark for

Comprehensive Open-Vocabulary Detection] for more details about the Inflated AP Problem and NMS-AP.





***

## Dataset Statistics



***

## Benchmark









***

## How To Download

See Our [hugging face page](https://huggingface.co/datasets/omlab/OVDEval) for downloading OVDEval.

***

## Evaluate With NMS-AP

OVDEval should be evaluated using **NMS-AP** to avoid the inflated AP problem.

Please follow the [evaluation instructions](https://github.com/om-ai-lab/OVDEval/blob/main/Evaluation.md).

The "output" folder provides the final output JSON files obtained by applying NMS to the inference results of the GLIP model on the material test dataset.

***

## Citations

Please consider citing our papers if you use the dataset:

```

@article{yao2023evaluate,

  title={How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection},

  author={Yao, Yiyang and Liu, Peng and Zhao, Tiancheng and Zhang, Qianqian and Liao, Jiajia and Fang, Chunxin and Lee, Kyusong and Wang, Qing},

  journal={arXiv preprint arXiv:2308.13177},

  year={2023}

}

```