https://github.com/arkaung/multidiv-teachers
Research repo for Active Learning from Divergent Multiple Teachers
https://github.com/arkaung/multidiv-teachers
active-learning neuralnetworks pytorch
Last synced: 3 months ago
JSON representation
Research repo for Active Learning from Divergent Multiple Teachers
- Host: GitHub
- URL: https://github.com/arkaung/multidiv-teachers
- Owner: ArkAung
- Created: 2020-03-17T12:31:56.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2020-03-28T17:10:45.000Z (over 6 years ago)
- Last Synced: 2025-03-12T07:27:49.711Z (over 1 year ago)
- Topics: active-learning, neuralnetworks, pytorch
- Language: Python
- Size: 18.6 KB
- Stars: 1
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## Active Learning from Divergent Multiple Teachers
The idea behind "Active Learning from Divergent Multiple Teachers" is to construct multiple teacher neural networks with different architectures, each potentially producing varying softmax results for an input $X$. Then, KL divergence is applied to all possible combinations of these softmax outputs. After that, mean of the all the KL-divergences is calculated. The samples with highest mean KL-divergence is chosen as a prime candidate for being sent to oracle for being labelled. Those samples with low KL-divergence are considered as confident samples and multiple softmax probability distributions are treated as multiple soft-labels for the student network. The student network learns from this multi-label distribution. During inference time, the student network looks outputs with highest agreement.
### Pipeline
* Train $N$ neural networks on training data
** Add temperature layer right before softmax layer
* Use those networks to make predictions on unlabelled data
* Get the softmax values of unlabelled from those networks
* Use multi-teacher knowledge distillation to train a student network for faster inference time, using the softmax outputs of training data from teacher network works
* Calculate the mean divergences among all possible combinations of softmax pairs from $N$ networks. There will be $N_c_2$ combinations
* Get the top $100$ data points with highest divergence to be sent to oracle.
** This can be elaborated into better selection process of cutting the number of data points which are above certain divergence threshold.
*