https://github.com/k-h-ismail/torchortho
Code for Learnable polynomial, trigonometric, and tropical activations
https://github.com/k-h-ismail/torchortho
activations ai artificial-intelligence basis convolutional-neural-networks deep-learning fourier-series gpt-2 imagenet kolmogorov-arnold-networks machine-learning mapping neural-networks openwebtext orthogonal polynomials pytorch transformers trigonometric-functions tropical-geometry
Last synced: about 1 year ago
JSON representation
Code for Learnable polynomial, trigonometric, and tropical activations
- Host: GitHub
- URL: https://github.com/k-h-ismail/torchortho
- Owner: K-H-Ismail
- License: gpl-3.0
- Created: 2024-09-27T15:14:50.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2025-06-08T13:40:23.000Z (about 1 year ago)
- Last Synced: 2025-06-08T13:41:14.812Z (about 1 year ago)
- Topics: activations, ai, artificial-intelligence, basis, convolutional-neural-networks, deep-learning, fourier-series, gpt-2, imagenet, kolmogorov-arnold-networks, machine-learning, mapping, neural-networks, openwebtext, orthogonal, polynomials, pytorch, transformers, trigonometric-functions, tropical-geometry
- Language: Python
- Homepage: https://arxiv.org/abs/2502.01247
- Size: 42 KB
- Stars: 9
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# torchortho 📐
[](https://arxiv.org/abs/2502.01247)
[](https://pypi.org/project/torchortho/)
[](https://opensource.org/licenses/GPL-3.0)
[](https://pypi.org/project/torchortho/)
[](https://paperswithcode.com/sota/text-generation-on-openwebtext?p=learnable-polynomial-trigonometric-and)
`torchortho` is a **PyTorch library** for **learnable activation functions** based on:
- **Hermite Polynomials** 🧙♂️
- **Fourier Series** 〰
- **Tropical Polynomials & Rational Functions** 🌴
These **adaptive activations** dynamically adjust during training, offering improved expressivity, better gradient flow, and enhanced generalization for **vision and language models**.
---
## **📜 Paper Reference**
This library is based on the paper:
📄 **[Polynomial, Trigonometric, and Tropical Activations](https://arxiv.org/abs/2502.01247)** *(Khalfaoui-Hassani & Kesselheim, 2025)*.
For experimental results, check our repos:
- **Vision models** (ConvNeXt with `torchortho` activations): [🔗 GitHub](https://github.com/K-H-Ismail/ConvNeXt-ortho)
- **Language models** (GPT-2 with `torchortho` activations): [🔗 GitHub](https://github.com/K-H-Ismail/pytorch-language-models)
---
## **📦 Installation**
Install from PyPI:
```bash
pip install torchortho
```
or install directly from GitHub:
```bash
pip install git+https://github.com/K-H-Ismail/torchortho.git
```
---
## **📝 Usage**
You can use `torchortho` activations just like any other PyTorch activation:
### **Example: Using Hermite Activation**
```python
import torch
from torchortho import HermiteActivation
# Define a learnable Hermite activation
degree = 5
activation = HermiteActivation(degree)
# Forward pass
x = torch.rand(7, 4, 3, 2)
y = activation(x)
# Compute gradients
loss = y.sum()
loss.backward()
print("Gradients of activation coefficients:", activation.coefficients.grad)
print("Output:", y)
```
### **Example: Using Fourier Activation in a Neural Network**
```python
import torch
import torch.nn as nn
from torchortho import FourierActivation
class CustomMLP(nn.Module):
def __init__(self, input_dim, hidden_dim, output_dim):
super().__init__()
self.fc1 = nn.Linear(input_dim, hidden_dim)
self.activation = FourierActivation(degree=4) # Learnable Fourier activation
self.fc2 = nn.Linear(hidden_dim, output_dim)
def forward(self, x):
x = self.fc1(x)
x = self.activation(x)
return self.fc2(x)
# Initialize the model
model = CustomMLP(input_dim=10, hidden_dim=32, output_dim=1)
x = torch.randn(5, 10)
output = model(x)
print("Model Output:", output)
```
---
## ⚡ Why Use torchortho?
### 1️⃣ Adaptive and Learnable Activations
Unlike static activations (ReLU, GELU), `torchortho` functions **dynamically adapt** during training, allowing models to **learn optimal activation functions** for different tasks.
| Activation Type | Strengths |
|---------------------------------|-----------|
| **Hermite Activation** | Adaptive polynomial approximation, variance-preserving, smooth optimization |
| **Fourier Activation** | Captures periodic structures in data (useful for NLP, physics-based models, and time-series) |
| **Tropical Polynomial Activation** | Convex activation for structured learning (e.g., decision boundaries, optimization landscapes) |
| **Rational Activation** | Generalizes standard Tropical activation functions for non-convex function fitting |
### 2️⃣ Improved Expressivity and Gradient Flow
- **Better function approximation** → Increases expressivity for deep networks.
- **Variance-preserving initialization** → Ensures stable training, avoiding vanishing/exploding gradients.
- **More flexible than ReLU/SwiGLU** → Adapts activation behavior based on data.
### 3️⃣ Benchmarked on Real-World Models
The effectiveness of `torchortho` activations has been validated on large-scale deep learning benchmarks:
✅ **Image Classification (ConvNeXt-T on ImageNet-1K)**
- Replacing GELU with `torchortho` activations **improves top-1 accuracy**.
✅ **Language Modeling (GPT-2 on OpenWebText)**
- Learnable activations **reduce perplexity** compared to GELU-based models.
For full benchmarks, see:
- **[Vision repo](https://github.com/K-H-Ismail/ConvNeXt-ortho)**
- **[Language repo](https://github.com/K-H-Ismail/pytorch-language-models)**
## **📜 License**
This project is licensed under the **GPL-3.0 License**. See [LICENSE](./LICENSE) for details.
---
## **🙌 Contributing**
We welcome contributions! Feel free to **submit issues, open PRs, or suggest improvements**.
---
## **📬 Contact**
For questions or collaborations, reach out via **[GitHub Issues](https://github.com/K-H-Ismail/torchortho/issues)**.
## 📚 Citation
If you use `torchortho` in your research, please cite the following paper:
```bibtex
@article{khalfaoui2025polynomial,
title={Polynomial, trigonometric, and tropical activations},
author={Khalfaoui-Hassani, Ismail and Kesselheim, Stefan},
journal={arXiv preprint arXiv:2502.01247},
year={2025}
}
```