https://github.com/farfarfun/funkeras

前沿算法，keras实现的搬运工
https://github.com/farfarfun/funkeras

attention bert decoder encoder keras transformer yolo

Last synced: 15 days ago
JSON representation

前沿算法，keras实现的搬运工

• Host: GitHub
• URL: https://github.com/farfarfun/funkeras
• Owner: farfarfun
• License: apache-2.0
• Created: 2019-11-22T07:13:04.000Z (almost 5 years ago)
• Default Branch: master
• Last Pushed: 2023-10-10T07:06:57.000Z (about 1 year ago)
• Last Synced: 2024-10-10T17:14:32.646Z (28 days ago)
• Topics: attention, bert, decoder, encoder, keras, transformer, yolo
• Language: Python
• Homepage:
• Size: 23.4 MB
• Stars: 2
• Watchers: 2
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# notekeras

[my blog](http://blog.notechats.cn/)

[attention](https://github.com/CyberZHG/keras-self-attention),

[transformer](https://github.com/CyberZHG/keras-transformer), 

[bert](https://github.com/CyberZHG/keras-bert) 等都是从大神[CyberZHG](https://github.com/CyberZHG/keras-transformer)学习来的，

主要是因为想学习源码并进行一些标注，所以进行了一些翻译、标注和整合

yolo 参考 [keras-yolo3](https://github.com/qqwweee/keras-yolo3), [keras-yolo3](https://github.com/SpikeKing/keras-yolo3-detection.git)

# Attention

## ScaledDotProductAttention

[参考论文](https://arxiv.org/pdf/1706.03762.pdf)

$$\text{Attention}(Q, K, V) = \text{softmax}(\frac{Q K^T}{\sqrt{d_k}}) V$$

## SeqSelfAttention

[参考论文](https://arxiv.org/pdf/1806.01264.pdf)

### multiplicative

$$e_{t, t'} = x_t^T W_a x_{t'} + b_a$$

$$a_{t} = \text{softmax}(e_t)$$

$$l_t = \sum_{t'} a_{t, t'} x_{t'}$$

### additive

$$h_{t, t'} = \tanh(x_t^T W_t + x_{t'}^T W_x + b_h)$$

$$e_{t, t'} = W_a h_{t, t'} + b_a$$

$$a_{t} = \text{softmax}(e_t)$$

$$l_t = \sum_{t'} a_{t, t'} x_{t'}$$

## SeqWeightedAttention

[参考论文](https://arxiv.org/pdf/1708.00524.pdf)

$$Y = \text{softmax}(XW + b) X$$

## MultiHeadAttention

[参考论文](https://arxiv.org/pdf/1706.03762.pdf)