{"id":19520132,"url":"https://github.com/osu-nlp-group/cosmo","last_synced_at":"2025-04-26T07:31:30.725Z","repository":{"id":215571210,"uuid":"739222438","full_name":"OSU-NLP-Group/COSMO","owner":"OSU-NLP-Group","description":"[CIKM'24] Reviving the Context: Camera Trap Species Classification as Link Prediction on Multimodal Knowledge Graphs","archived":false,"fork":false,"pushed_at":"2025-04-02T15:24:30.000Z","size":5085,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-02T16:29:34.270Z","etag":null,"topics":["animal-ecology","multimodal-knowledge-graph","species-classification"],"latest_commit_sha":null,"homepage":"https://osu-nlp-group.github.io/COSMO/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OSU-NLP-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-05T03:53:39.000Z","updated_at":"2025-04-02T15:24:33.000Z","dependencies_parsed_at":"2025-04-02T16:35:56.969Z","dependency_job_id":null,"html_url":"https://github.com/OSU-NLP-Group/COSMO","commit_stats":null,"previous_names":["osu-nlp-group/cosmo"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FCOSMO","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FCOSMO/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FCOSMO/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FCOSMO/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OSU-NLP-Group","download_url":"https://codeload.github.com/OSU-NLP-Group/COSMO/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250953572,"owners_count":21513391,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["animal-ecology","multimodal-knowledge-graph","species-classification"],"created_at":"2024-11-11T00:23:57.635Z","updated_at":"2025-04-26T07:31:30.719Z","avatar_url":"https://github.com/OSU-NLP-Group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# COSMO\n\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/bringing-back-the-context-camera-trap-species/image-classification-on-iwildcam2020-wilds)](https://paperswithcode.com/sota/image-classification-on-iwildcam2020-wilds?p=bringing-back-the-context-camera-trap-species)\n\n## [CIKM'24] Reviving the Context: Camera Trap Species Classification as Link Prediction on Multimodal Knowledge Graphs\n\n**Paper**: https://arxiv.org/pdf/2401.00608.pdf\n\n**Project webpage**: https://osu-nlp-group.github.io/COSMO/\n\n**Authors**: Vardaan Pahuja, Weidi Luo, Yu Gu, Cheng-Hao Tu, Hong-You Chen, Tanya Berger-Wolf, Charles Stewart, Song Gao, Wei-Lun Chao, and Yu Su\n\n## Installation\n\n```\npip install -r requirements.txt\n```\n\n## Data Preprocessing\n\n### iWildCam2020-WILDS\n```\nbash preprocess_iwildcam.sh\n```\nNote: The dir. `data/iwildcam_v2.0/train/` contains images for all splits.\n\n### Snapshot Mountain Zebra\n1. Download snapshot_mountain_zebra.zip from [this link](https://buckeyemailosu-my.sharepoint.com/:u:/g/personal/pahuja_9_buckeyemail_osu_edu/EWI05mXQsopNskBo78a_l_ABSZJHl0uCsdNMu72aXmNNiA?e=LOtm5Q) and uncompress it into a directory `data/snapshot_mountain_zebra/`.\n2. Download images using the command `gsutil -m cp -r \"gs://public-datasets-lila/snapshot-safari/MTZ/MTZ_public\" data/snapshot_mountain_zebra/`\n2. Run `bash preprocess_mountain_zebra.sh`\n\n\n## Training\n\nNote: The below commands will use the DistMult model by default. Use the following hyperparameter configuration:\n\n- For iWildCam2020-WILDS, set `DATA_DIR` to `data/iwildcam_v2.0/`, `IMG_DIR` to `data/iwildcam_v2.0/train/`, and `DATASET` to `iwildcam`\n- For Snapshot Mountain Zebra, set `DATA_DIR` to `data/snapshot_mountain_zebra/` and `IMG_DIR` to `data/snapshot_mountain_zebra/`, and `DATASET` to `mountain_zebra`.\n- For ConvE, use `--kg-embed-model conve --embedding-dim 200` in args.\n\n\n### Image-only model (ERM baseline)\n```\npython -u run_image_only_model.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR \u003e CKPT_DIR/log.txt\n```\n\n### COSMO, no-context baseline\n```\npython -u main.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR \u003e CKPT_DIR/log.txt\n```\n\n### COSMO, taxonomy\n```\npython -u main.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR --add-id-id \u003e CKPT_DIR/log.txt\n```\n\n### COSMO, location\n```\npython -u main.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR --add-image-location \u003e CKPT_DIR/log.txt\n```\n\n### COSMO, time\n```\npython -u main.py --data-dir DATA_DIR --img-dir IMG_DIR/ --save-dir CKPT_DIR --add-image-time \u003e CKPT_DIR/log.txt\n```\n\n### COSMO, taxonomy + location + time\n```\npython -u main.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR --add-id-id --add-image-time --add-image-location \u003e CKPT_DIR/log.txt\n```\n\n### MLP-concat baseline\n```\npython -u run_kge_model_baseline.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR --embedding-dim 512 --use-subtree --only-hour --time_input_dim 1 --early-stopping-patience 10 \u003e CKPT_DIR/log.txt\n\n```\n\n## Evaluation\n\n### Evaluate a model (specify split)\n```\npython eval.py --ckpt-path \u003cPATH TO COSMO CKPT\u003e --split test --data-dir DATA_DIR --img-dir IMG_DIR\n```\n\n## Error Analysis\n\n### Taxonomy analysis\n```\ncd gen_utils/\npython analyze_taxonomy_model.py --data-dir DATA_DIR --img-dir IMG_DIR --ckpt-1-path \u003cPATH TO COSMO+TAXONOMY CKPT\u003e --ckpt-2-path \u003cPATH TO COSMO BASE CKPT\u003e\n```\n\n### Plot location correlation analysis\n```\ncd gen_utils/\npython analyze_img_loc.py --data-dir DATA_DIR\n```\n\n### Plot time correlation analysis\n```\ncd gen_utils/\npython analyze_img_time.py --data-dir DATA_DIR\n```\n\n\n### Under-represented Species Analysis\n\n#### Dump predictions for baseline image-only model\n```\ncd gen_utils/\npython dump_imageonly_pred_specie_wise.py --ckpt-path \u003cPATH TO IMAGE-ONLY MODEL\u003e --split test --out-dir \u003cOUT DIR\u003e\n```\n\n#### Dump predictions for COSMO model\n```\ncd gen_utils/\npython dump_kge_pred_specie_wise.py --ckpt-path \u003cPATH TO COSMO MODEL\u003e --split test --out-dir \u003cOUT DIR\u003e\n```\n\n#### Compare performance for under-represented species\n```\ncd gen_utils/\npython eval_kge_specie_wise.py --y-pred-path-1 \u003cPATH TO PREDICTIONS JSON FILE OF BASELINE MODEL\u003e --y-pred-path-2 \u003cPATH TO COSMO PREDICTIONS JSON FILE\u003e\n```\n\n## Citation\n```\n@inproceedings{10.1145/3627673.3679545,\nauthor = {Pahuja, Vardaan and Luo, Weidi and Gu, Yu and Tu, Cheng-Hao and Chen, Hong-You and Berger-Wolf, Tanya and Stewart, Charles and Gao, Song and Chao, Wei-Lun and Su, Yu},\ntitle = {Reviving the Context: Camera Trap Species Classification as Link Prediction on Multimodal Knowledge Graphs},\nyear = {2024},\nisbn = {9798400704369},\npublisher = {Association for Computing Machinery},\naddress = {New York, NY, USA},\nurl = {https://doi.org/10.1145/3627673.3679545},\ndoi = {10.1145/3627673.3679545},\nbooktitle = {Proceedings of the 33rd ACM International Conference on Information and Knowledge Management},\npages = {1825–1835},\nnumpages = {11},\nkeywords = {KG link prediction, camera traps, multimodal knowledge graph, species classification},\nlocation = {Boise, ID, USA},\nseries = {CIKM '24}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosu-nlp-group%2Fcosmo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fosu-nlp-group%2Fcosmo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosu-nlp-group%2Fcosmo/lists"}