https://github.com/s-gupta/v-coco

Hooks for VCOCO
https://github.com/s-gupta/v-coco

Last synced: 2 months ago
JSON representation

Hooks for VCOCO

• Host: GitHub
• URL: https://github.com/s-gupta/v-coco
• Owner: s-gupta
• License: mit
• Created: 2016-01-13T00:24:11.000Z (about 9 years ago)
• Default Branch: master
• Last Pushed: 2017-06-16T18:10:12.000Z (over 7 years ago)
• Last Synced: 2024-08-04T01:17:12.530Z (5 months ago)
• Language: Jupyter Notebook
• Size: 5.45 MB
• Stars: 148
• Watchers: 7
• Forks: 36
• Open Issues: 7
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-human-object-interaction - V-COCO: Visual Semantic Role Labeling

README

        
# Verbs in COCO (V-COCO) Dataset

This repository hosts the Verbs in COCO (V-COCO) dataset and associated code to evaluate models for the Visual Semantic Role Labeling (VSRL) task as ddescribed in this technical report. 

### Citing

If you find this dataset or code base useful in your research, please consider citing the following papers:

    @article{gupta2015visual,

      title={Visual Semantic Role Labeling},

      author={Gupta, Saurabh and Malik, Jitendra},

      journal={arXiv preprint arXiv:1505.04474},

      year={2015}

    }

    

    @incollection{lin2014microsoft,

      title={Microsoft COCO: Common objects in context},

      author={Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C Lawrence},

      booktitle={Computer Vision--ECCV 2014},

      pages={740--755},

      year={2014},

      publisher={Springer}

    }

    

### Installation

1. Clone repository (recursively, so as to include COCO API).

    ```Shell

    git clone --recursive https://github.com/s-gupta/v-coco.git

    ```

2. This dataset builds off MS COCO, please download MS-COCO images and annotations. 

3. Current V-COCO release only uses a subset of MS-COCO images (Image IDs listed in ```data/splits/vcoco_all.ids```). Use the following script to pick out annotations from the COCO annotations to allow faster loading in V-COCO.  

    ```Shell

    # Assume you cloned the repository to `VCOCO_DIR'

    cd $VCOCO_DIR

    # If you downloaded coco annotations to coco-data/annotations

    python script_pick_annotations.py coco-data/annotations

    ```

    

4. Build ```coco/PythonAPI/pycocotools/_mask.so```, ```cython_bbox.so```. 

    ```Shell

    # Assume you cloned the repository to `VCOCO_DIR'

    cd $VCOCO_DIR/coco/PythonAPI/ && make

    cd $VCOCO_DIR && make

    ```

### Using the dataset

1. An IPython notebook, illustrating how to use the annotations in the dataset is available in ```V-COCO.ipynb```

2. The current release of the dataset includes annotations as indicated in Table 1 in the paper. We are collecting role annotations for the 6 categories (that are missing) and will make them public shortly.

### Evaluation

We provide evaluation code that computes ```agent AP``` and ```role AP```, as explained in the paper. 

In order to use the evaluation code, store your predictions as a ```pickle file (.pkl)``` in the following format:

  ```Shell

  [ {'image_id':        # the coco image id,

     'person_box':      #[x1, y1, x2, y2] the box prediction for the person,

     '[action]_agent':  # the score for action corresponding to the person prediction,

     '[action]_[role]': # [x1, y1, x2, y2, s], the predicted box for role and 

                        # associated score for the action-role pair.

     } ]

  ```

Assuming your detections are stored in ```det_file=/path/to/detections/detections.pkl```, do

  ```Shell

  from vsrl_eval import VCOCOeval

  vcocoeval = VCOCOeval(vsrl_annot_file, coco_file, split_file)

    # e.g. vsrl_annot_file: data/vcoco/vcoco_val.json

    #      coco_file:       data/instances_vcoco_all_2014.json

    #      split_file:      data/splits/vcoco_val.ids

  vcocoeval._do_eval(det_file, ovr_thresh=0.5)

  ```

  We introduce two scenarios for ```role AP``` evaluation. 

    

  1. [Scenario 1] In this scenario, for the test cases with missing role annotations an agent role prediction is correct if the action is correct & the overlap between the person boxes is >0.5 & the corresponding role is empty e.g. ```[0,0,0,0]``` or ```[NaN,NaN,NaN,NaN]```. This scenario is fit for missing roles due to occlusion.

  2. [Scenario 2] In this scenario, for the test cases with missing role annotations an agent role prediction is correct if the action is correct & the overlap between the person boxes is >0.5 (the corresponding role is ignored). This scenario is fit for the cases with roles outside the COCO categories.