https://github.com/superb-man/supclr-multilabel-classification
Multilabel classification using the technique of supervised contrastive learning
https://github.com/superb-man/supclr-multilabel-classification
Last synced: 6 months ago
JSON representation
Multilabel classification using the technique of supervised contrastive learning
- Host: GitHub
- URL: https://github.com/superb-man/supclr-multilabel-classification
- Owner: Superb-Man
- Created: 2024-11-01T15:11:35.000Z (11 months ago)
- Default Branch: master
- Last Pushed: 2025-01-16T06:30:03.000Z (9 months ago)
- Last Synced: 2025-02-07T03:22:14.879Z (8 months ago)
- Language: Python
- Homepage:
- Size: 1.05 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: READme.md
Awesome Lists containing this project
README
# Multi-Label Contrastive Loss (MLCL)
- Jaccard Similarity
Efficient computation of Jaccard similarity between labels $\mathbf{y}_i$ and $\mathbf{y}_j$:
$$\text{Jaccard}(\mathbf{y}_i,\mathbf{y}_j)=\frac{\mathbf{y}_i^\top\mathbf{y}_j}{\|\mathbf{y}_i\|_1+\|\mathbf{y}_j\|_1-\mathbf{y}_i^\top\mathbf{y}_j}$$
- Weighting
The weighting function applies the Jaccard similarity raised to a power $\beta$:
$$w_{i,j}=(\text{Jaccard}(\mathbf{y}_i,\mathbf{y}_j))^\beta$$
- **Positive Mask**
$$m_{i,j}^+=\begin{cases}
1,&\text{if } Jaccard(y_i,y_j)>c_{threshold}\\
0,& otherwise
\end{cases}$$
- **Negative Mask**
$$m_{i,j}^-=1-m_{i,j}^+$$
- Similarity Logits:
$$sim(i,j)=\frac{\mathbf{f}_i\cdot\mathbf{f}_j}{\tau}$$
- Log probabilities are computed as:
$$\log p_{i,j}=sim(i,j)-\log\sum_k e^{sim(i,k)}$$
## Loss Formulation
### Positive Contribution
The positive contribution to the loss is:
$$\text{positive log probability}=\frac{\sum_{j}m_{i,j}^+\cdot w_{i,j}\cdot\log p_{i,j}}{\sum_{j}m_{ij}^++\epsilon}$$
### Negative Contribution
The negative contribution to the loss is:
$$\text{negative log probability}=\frac{\sum_{j}m_{i,j}^-\cdot\log p_{i,j}}{\sum_{j}m_{i,j}^-+\epsilon}$$
### Total Loss
The total loss for the batch:
$$\mathcal{L}=-\frac{1}{N}\sum_{i=1}^N\left[\alpha\sum_{j\neq i}m_{ij}^+w_{ij}\log p_{ij}+(1-\alpha)\sum_{j\neq i}m_{ij}^-\log p_{ij}\right]+\lambda\mathcal{R}$$
Where:
- $\alpha$: Weight for positive pairs.
- $\mathcal{R}$: Regularization term, defined as:
$$\mathcal{R}=\frac{1}{N}\sum_{i}\|\mathbf{f}_i\|_2$$
---
# MultiSupConLoss
- Positive Mask:
$$m_{i,j}=\begin{cases}
1,&\text{if } Jaccard(y_i,y_j)>c_{threshold}\\
0,& otherwise
\end{cases}$$
- Weights are computed as:
$$w_{i,j} = \frac{Jaccard Similarity_{i,j}}{c_{\text{threshold}} + \epsilon}$$
- The similarity between contrastive features is computed as:
$${sim}_{i,j}=\frac{\text{f}_i\cdot\text{f}_j}{\text{temperature}}$$
- Probability:
$$p_{i,j}=\log\left(\frac{e^{sim(i,j)}}{\sum_{k=1}^{N}e^{sim(i,k)}}\right)$$
## Loss Formulation
The MultiSupConLoss is defined as:
$$\text{loss}=-\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{N}m_{i,j}\cdot w_{i,j}\cdot{p}_{i,j}$$