{"id":25119139,"url":"https://github.com/k-h-ismail/torchortho","last_synced_at":"2025-07-22T22:05:13.214Z","repository":{"id":275732878,"uuid":"864142924","full_name":"K-H-Ismail/torchortho","owner":"K-H-Ismail","description":"Code for Learnable polynomial, trigonometric, and tropical activations","archived":false,"fork":false,"pushed_at":"2025-06-08T13:40:23.000Z","size":43,"stargazers_count":9,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-08T13:41:14.812Z","etag":null,"topics":["activations","ai","artificial-intelligence","basis","convolutional-neural-networks","deep-learning","fourier-series","gpt-2","imagenet","kolmogorov-arnold-networks","machine-learning","mapping","neural-networks","openwebtext","orthogonal","polynomials","pytorch","transformers","trigonometric-functions","tropical-geometry"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2502.01247","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/K-H-Ismail.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-09-27T15:14:50.000Z","updated_at":"2025-06-08T13:40:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"ed539c27-b64f-4e4e-8f41-476af599583d","html_url":"https://github.com/K-H-Ismail/torchortho","commit_stats":null,"previous_names":["k-h-ismail/torchortho"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/K-H-Ismail/torchortho","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/K-H-Ismail%2Ftorchortho","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/K-H-Ismail%2Ftorchortho/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/K-H-Ismail%2Ftorchortho/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/K-H-Ismail%2Ftorchortho/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/K-H-Ismail","download_url":"https://codeload.github.com/K-H-Ismail/torchortho/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/K-H-Ismail%2Ftorchortho/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266580342,"owners_count":23951201,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["activations","ai","artificial-intelligence","basis","convolutional-neural-networks","deep-learning","fourier-series","gpt-2","imagenet","kolmogorov-arnold-networks","machine-learning","mapping","neural-networks","openwebtext","orthogonal","polynomials","pytorch","transformers","trigonometric-functions","tropical-geometry"],"created_at":"2025-02-08T04:35:26.424Z","updated_at":"2025-07-22T22:05:13.207Z","avatar_url":"https://github.com/K-H-Ismail.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# torchortho 📐  \n[![arXiv](https://img.shields.io/badge/arXiv-2502.01247-b31b1b.svg)](https://arxiv.org/abs/2502.01247)\n[![PyPI version](https://img.shields.io/pypi/v/torchortho)](https://pypi.org/project/torchortho/)\n[![License: GPL-3.0](https://img.shields.io/badge/License-GPL%203.0-blue.svg)](https://opensource.org/licenses/GPL-3.0)\n[![Python Versions](https://img.shields.io/pypi/pyversions/torchortho.svg)](https://pypi.org/project/torchortho/)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/learnable-polynomial-trigonometric-and/text-generation-on-openwebtext)](https://paperswithcode.com/sota/text-generation-on-openwebtext?p=learnable-polynomial-trigonometric-and)\n\n`torchortho` is a **PyTorch library** for **learnable activation functions** based on:  \n- **Hermite Polynomials** 🧙‍♂️\n- **Fourier Series** 〰\n- **Tropical Polynomials \u0026 Rational Functions** 🌴  \n\nThese **adaptive activations** dynamically adjust during training, offering improved expressivity, better gradient flow, and enhanced generalization for **vision and language models**.\n\n---\n\n## **📜 Paper Reference**\nThis library is based on the paper:  \n📄 **[Polynomial, Trigonometric, and Tropical Activations](https://arxiv.org/abs/2502.01247)** *(Khalfaoui-Hassani \u0026 Kesselheim, 2025)*.  \n\nFor experimental results, check our repos:  \n- **Vision models** (ConvNeXt with `torchortho` activations): [🔗 GitHub](https://github.com/K-H-Ismail/ConvNeXt-ortho)  \n- **Language models** (GPT-2 with `torchortho` activations): [🔗 GitHub](https://github.com/K-H-Ismail/pytorch-language-models)  \n\n---\n\n## **📦 Installation**\nInstall from PyPI:  \n```bash\npip install torchortho\n```\nor install directly from GitHub:\n```bash\npip install git+https://github.com/K-H-Ismail/torchortho.git\n```\n\n---\n\n## **📝 Usage**\nYou can use `torchortho` activations just like any other PyTorch activation:\n\n### **Example: Using Hermite Activation**\n```python\nimport torch\nfrom torchortho import HermiteActivation\n\n# Define a learnable Hermite activation\ndegree = 5\nactivation = HermiteActivation(degree)\n\n# Forward pass\nx = torch.rand(7, 4, 3, 2)\ny = activation(x)\n\n# Compute gradients\nloss = y.sum()\nloss.backward()\n\nprint(\"Gradients of activation coefficients:\", activation.coefficients.grad)\nprint(\"Output:\", y)\n```\n\n### **Example: Using Fourier Activation in a Neural Network**\n```python\nimport torch\nimport torch.nn as nn\nfrom torchortho import FourierActivation\n\nclass CustomMLP(nn.Module):\n    def __init__(self, input_dim, hidden_dim, output_dim):\n        super().__init__()\n        self.fc1 = nn.Linear(input_dim, hidden_dim)\n        self.activation = FourierActivation(degree=4)  # Learnable Fourier activation\n        self.fc2 = nn.Linear(hidden_dim, output_dim)\n\n    def forward(self, x):\n        x = self.fc1(x)\n        x = self.activation(x)\n        return self.fc2(x)\n\n# Initialize the model\nmodel = CustomMLP(input_dim=10, hidden_dim=32, output_dim=1)\nx = torch.randn(5, 10)\noutput = model(x)\nprint(\"Model Output:\", output)\n```\n\n---\n\n## ⚡ Why Use torchortho?\n\n### 1️⃣ Adaptive and Learnable Activations  \nUnlike static activations (ReLU, GELU), `torchortho` functions **dynamically adapt** during training, allowing models to **learn optimal activation functions** for different tasks.\n\n| Activation Type                 | Strengths |\n|---------------------------------|-----------|\n| **Hermite Activation**          | Adaptive polynomial approximation, variance-preserving, smooth optimization |\n| **Fourier Activation**          | Captures periodic structures in data (useful for NLP, physics-based models, and time-series) |\n| **Tropical Polynomial Activation** | Convex activation for structured learning (e.g., decision boundaries, optimization landscapes) |\n| **Rational Activation**         | Generalizes standard Tropical activation functions for non-convex function fitting |\n\n### 2️⃣ Improved Expressivity and Gradient Flow  \n- **Better function approximation** → Increases expressivity for deep networks.  \n- **Variance-preserving initialization** → Ensures stable training, avoiding vanishing/exploding gradients.  \n- **More flexible than ReLU/SwiGLU** → Adapts activation behavior based on data.\n\n### 3️⃣ Benchmarked on Real-World Models  \nThe effectiveness of `torchortho` activations has been validated on large-scale deep learning benchmarks:  \n\n✅ **Image Classification (ConvNeXt-T on ImageNet-1K)**  \n   - Replacing GELU with `torchortho` activations **improves top-1 accuracy**.  \n\n✅ **Language Modeling (GPT-2 on OpenWebText)**  \n   - Learnable activations **reduce perplexity** compared to GELU-based models.  \n\nFor full benchmarks, see:  \n- **[Vision repo](https://github.com/K-H-Ismail/ConvNeXt-ortho)**  \n- **[Language repo](https://github.com/K-H-Ismail/pytorch-language-models)**  \n\n\n## **📜 License**\nThis project is licensed under the **GPL-3.0 License**. See [LICENSE](./LICENSE) for details.\n\n---\n\n## **🙌 Contributing**\nWe welcome contributions! Feel free to **submit issues, open PRs, or suggest improvements**.  \n\n---\n\n## **📬 Contact**\nFor questions or collaborations, reach out via **[GitHub Issues](https://github.com/K-H-Ismail/torchortho/issues)**.  \n\n## 📚 Citation  \nIf you use `torchortho` in your research, please cite the following paper:\n\n```bibtex\n@article{khalfaoui2025polynomial,\n  title={Polynomial, trigonometric, and tropical activations},\n  author={Khalfaoui-Hassani, Ismail and Kesselheim, Stefan},\n  journal={arXiv preprint arXiv:2502.01247},\n  year={2025}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fk-h-ismail%2Ftorchortho","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fk-h-ismail%2Ftorchortho","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fk-h-ismail%2Ftorchortho/lists"}