https://github.com/phizaz/latent-set-prediction
Official implementation of Set Prediction in the Latent Space (LSP)
https://github.com/phizaz/latent-set-prediction
chest-xray deep-learning neurips-2021 report-generation set-prediction
Last synced: 6 months ago
JSON representation
Official implementation of Set Prediction in the Latent Space (LSP)
- Host: GitHub
- URL: https://github.com/phizaz/latent-set-prediction
- Owner: phizaz
- Created: 2021-10-25T14:18:27.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2022-09-04T03:04:14.000Z (about 3 years ago)
- Last Synced: 2025-03-25T19:21:28.186Z (6 months ago)
- Topics: chest-xray, deep-learning, neurips-2021, report-generation, set-prediction
- Language: Python
- Homepage:
- Size: 91.9 MB
- Stars: 5
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Official implementation of Set Prediction in the Latent Space (LSP)
A NeurIPS2021 paper ([site](https://konpat.me/lsp), [paper](https://openreview.net/forum?id=1ANcwXQuijU), [presentation video](https://slideslive.com/38969041)):
```
@inproceedings{NEURIPS2021_d61e9e58,
author = {Preechakul, Konpat and Piansaddhayanon, Chawan and Naowarat, Burin and Khandhawit, Tirasan and Sriswasdi, Sira and Chuangsuwanich, Ekapol},
booktitle = {Advances in Neural Information Processing Systems},
editor = {M. Ranzato and A. Beygelzimer and Y. Dauphin and P.S. Liang and J. Wortman Vaughan},
pages = {25516--25527},
publisher = {Curran Associates, Inc.},
title = {Set Prediction in the Latent Space},
url = {https://proceedings.neurips.cc/paper/2021/file/d61e9e58ae1058322bc169943b39f1d8-Paper.pdf},
volume = {34},
year = {2021}
}
```
# Usage
Ready to use LSP (latent set prediction) code is available in `lsp.py`.
It's self-contained and annotated with comments using the convention from the published paper.
You can see a simplified example (with synthetic data) in `try.ipynb`.
## Example use
```
from lsp import LSPLoss
...
# assume a set prediction model & encoder model
S = set_prediction(x)
G = encoder(gt)
# where
# S = set elements (n, c)
# len_S = cardinalities of sets in a batch
# G, len_G should be the same size as S
# LSP latent loss
# default params
loss_fn = LSPLoss('gcr', w_loss_gs=1, w_loss_sg=0.1, d=1e-3)
latent = loss_fn(S, len_S, G, len_G)
# return values contain
# - S_pi = ordered set elements, used for loss calculation to allow proper gradient flow
# - S_i = index of the ordering such that S[S_i] == S_pi
# - loss = latent loss
# feeding the ordered set elements to the prediction head
# while allowing the gradient to flow through LSP correctly
pred = prediction_head(latent.S_pi)
total_loss = task_loss(pred, gt) + latent.loss
total_loss.backward()
```
# Reproducibility
We included reproducible code for the main experiments including:
- [CLEVR object description prediction task](image_captioning)
- [MIMIC-CXR chest radiograph report generation task](image_captioning)
- [MNIST object detection task](object_detection)