https://github.com/osu-nlp-group/cosmo
[CIKM'24] Reviving the Context: Camera Trap Species Classification as Link Prediction on Multimodal Knowledge Graphs
https://github.com/osu-nlp-group/cosmo
animal-ecology multimodal-knowledge-graph species-classification
Last synced: about 1 year ago
JSON representation
[CIKM'24] Reviving the Context: Camera Trap Species Classification as Link Prediction on Multimodal Knowledge Graphs
- Host: GitHub
- URL: https://github.com/osu-nlp-group/cosmo
- Owner: OSU-NLP-Group
- License: mit
- Created: 2024-01-05T03:53:39.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2025-04-02T15:24:30.000Z (about 1 year ago)
- Last Synced: 2025-04-02T16:29:34.270Z (about 1 year ago)
- Topics: animal-ecology, multimodal-knowledge-graph, species-classification
- Language: Python
- Homepage: https://osu-nlp-group.github.io/COSMO/
- Size: 4.85 MB
- Stars: 7
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# COSMO
[](https://paperswithcode.com/sota/image-classification-on-iwildcam2020-wilds?p=bringing-back-the-context-camera-trap-species)
## [CIKM'24] Reviving the Context: Camera Trap Species Classification as Link Prediction on Multimodal Knowledge Graphs
**Paper**: https://arxiv.org/pdf/2401.00608.pdf
**Project webpage**: https://osu-nlp-group.github.io/COSMO/
**Authors**: Vardaan Pahuja, Weidi Luo, Yu Gu, Cheng-Hao Tu, Hong-You Chen, Tanya Berger-Wolf, Charles Stewart, Song Gao, Wei-Lun Chao, and Yu Su
## Installation
```
pip install -r requirements.txt
```
## Data Preprocessing
### iWildCam2020-WILDS
```
bash preprocess_iwildcam.sh
```
Note: The dir. `data/iwildcam_v2.0/train/` contains images for all splits.
### Snapshot Mountain Zebra
1. Download snapshot_mountain_zebra.zip from [this link](https://buckeyemailosu-my.sharepoint.com/:u:/g/personal/pahuja_9_buckeyemail_osu_edu/EWI05mXQsopNskBo78a_l_ABSZJHl0uCsdNMu72aXmNNiA?e=LOtm5Q) and uncompress it into a directory `data/snapshot_mountain_zebra/`.
2. Download images using the command `gsutil -m cp -r "gs://public-datasets-lila/snapshot-safari/MTZ/MTZ_public" data/snapshot_mountain_zebra/`
2. Run `bash preprocess_mountain_zebra.sh`
## Training
Note: The below commands will use the DistMult model by default. Use the following hyperparameter configuration:
- For iWildCam2020-WILDS, set `DATA_DIR` to `data/iwildcam_v2.0/`, `IMG_DIR` to `data/iwildcam_v2.0/train/`, and `DATASET` to `iwildcam`
- For Snapshot Mountain Zebra, set `DATA_DIR` to `data/snapshot_mountain_zebra/` and `IMG_DIR` to `data/snapshot_mountain_zebra/`, and `DATASET` to `mountain_zebra`.
- For ConvE, use `--kg-embed-model conve --embedding-dim 200` in args.
### Image-only model (ERM baseline)
```
python -u run_image_only_model.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR > CKPT_DIR/log.txt
```
### COSMO, no-context baseline
```
python -u main.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR > CKPT_DIR/log.txt
```
### COSMO, taxonomy
```
python -u main.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR --add-id-id > CKPT_DIR/log.txt
```
### COSMO, location
```
python -u main.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR --add-image-location > CKPT_DIR/log.txt
```
### COSMO, time
```
python -u main.py --data-dir DATA_DIR --img-dir IMG_DIR/ --save-dir CKPT_DIR --add-image-time > CKPT_DIR/log.txt
```
### COSMO, taxonomy + location + time
```
python -u main.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR --add-id-id --add-image-time --add-image-location > CKPT_DIR/log.txt
```
### MLP-concat baseline
```
python -u run_kge_model_baseline.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR --embedding-dim 512 --use-subtree --only-hour --time_input_dim 1 --early-stopping-patience 10 > CKPT_DIR/log.txt
```
## Evaluation
### Evaluate a model (specify split)
```
python eval.py --ckpt-path --split test --data-dir DATA_DIR --img-dir IMG_DIR
```
## Error Analysis
### Taxonomy analysis
```
cd gen_utils/
python analyze_taxonomy_model.py --data-dir DATA_DIR --img-dir IMG_DIR --ckpt-1-path --ckpt-2-path
```
### Plot location correlation analysis
```
cd gen_utils/
python analyze_img_loc.py --data-dir DATA_DIR
```
### Plot time correlation analysis
```
cd gen_utils/
python analyze_img_time.py --data-dir DATA_DIR
```
### Under-represented Species Analysis
#### Dump predictions for baseline image-only model
```
cd gen_utils/
python dump_imageonly_pred_specie_wise.py --ckpt-path --split test --out-dir
```
#### Dump predictions for COSMO model
```
cd gen_utils/
python dump_kge_pred_specie_wise.py --ckpt-path --split test --out-dir
```
#### Compare performance for under-represented species
```
cd gen_utils/
python eval_kge_specie_wise.py --y-pred-path-1 --y-pred-path-2
```
## Citation
```
@inproceedings{10.1145/3627673.3679545,
author = {Pahuja, Vardaan and Luo, Weidi and Gu, Yu and Tu, Cheng-Hao and Chen, Hong-You and Berger-Wolf, Tanya and Stewart, Charles and Gao, Song and Chao, Wei-Lun and Su, Yu},
title = {Reviving the Context: Camera Trap Species Classification as Link Prediction on Multimodal Knowledge Graphs},
year = {2024},
isbn = {9798400704369},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3627673.3679545},
doi = {10.1145/3627673.3679545},
booktitle = {Proceedings of the 33rd ACM International Conference on Information and Knowledge Management},
pages = {1825–1835},
numpages = {11},
keywords = {KG link prediction, camera traps, multimodal knowledge graph, species classification},
location = {Boise, ID, USA},
series = {CIKM '24}
}
```