{"id":27162868,"url":"https://github.com/notedance/adahessian","last_synced_at":"2026-04-09T20:03:59.941Z","repository":{"id":286831080,"uuid":"962707141","full_name":"NoteDance/Adahessian","owner":"NoteDance","description":"TensorFlow implementation for Adahessian optimizer","archived":false,"fork":false,"pushed_at":"2025-04-08T14:54:54.000Z","size":0,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-08T15:47:24.546Z","etag":null,"topics":["deep-learning","deep-reinforcement-learning","keras","machine-learning","optimizer","reinforcement-learning","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NoteDance.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-04-08T14:54:02.000Z","updated_at":"2025-04-08T14:56:08.000Z","dependencies_parsed_at":"2025-04-08T23:45:15.039Z","dependency_job_id":null,"html_url":"https://github.com/NoteDance/Adahessian","commit_stats":null,"previous_names":["notedance/adahessian"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NoteDance%2FAdahessian","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NoteDance%2FAdahessian/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NoteDance%2FAdahessian/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NoteDance%2FAdahessian/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NoteDance","download_url":"https://codeload.github.com/NoteDance/Adahessian/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247957878,"owners_count":21024774,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","deep-reinforcement-learning","keras","machine-learning","optimizer","reinforcement-learning","tensorflow"],"created_at":"2025-04-09T01:34:15.359Z","updated_at":"2026-04-09T20:03:59.922Z","avatar_url":"https://github.com/NoteDance.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Adahessian\n\n**Overview**:\n\nThe **Adahessian Optimizer** is an advanced second-order optimization algorithm that leverages the Hessian trace (approximated using Hutchinson's method) to adaptively scale learning rates for each parameter. Adahessian extends first-order optimization techniques like Adam by incorporating curvature information from the loss surface, which enables better adaptation to the optimization landscape, especially for highly non-convex problems.\n\n**Parameters**:\n\n- **`learning_rate`** *(float)*: Initial learning rate (default: `0.1`).\n- **`beta1`** *(float)*: Exponential decay rate for the first moment estimates (default: `0.9`).\n- **`beta2`** *(float)*: Exponential decay rate for the Hessian diagonal squared estimates (default: `0.999`).\n- **`epsilon`** *(float)*: Small value to prevent division by zero (default: `1e-8`).\n- **`weight_decay`** *(float)*: L2 regularization factor for weights (default: `0.0`).\n- **`hessian_power`** *(float)*: Scaling factor for the Hessian diagonal (default: `1.0`).\n- **`update_each`** *(int)*: Frequency (in steps) for Hessian trace updates (default: `1`).\n- **`n_samples`** *(int)*: Number of samples for Hutchinson’s approximation (default: `1`).\n- **`avg_conv_kernel`** *(bool)*: Whether to average Hessian diagonals over convolutional kernel dimensions (default: `False`).\n- **`clipnorm`** *(float, optional)*: Clips gradients by their norm.\n- **`clipvalue`** *(float, optional)*: Clips gradients by their value.\n- **`global_clipnorm`** *(float, optional)*: Clips gradients by their global norm.\n- **`use_ema`** *(bool, default=False)*: Enables Exponential Moving Average (EMA) for model weights.\n- **`ema_momentum`** *(float, default=0.99)*: Momentum for EMA updates.\n- **`ema_overwrite_frequency`** *(int, optional)*: Frequency for overwriting weights with EMA values.\n- **`loss_scale_factor`** *(float, optional)*: Scaling factor for loss values.\n- **`gradient_accumulation_steps`** *(int, optional)*: Number of steps for gradient accumulation.\n- **`name`** *(str, default=\"adahessian\")*: Name of the optimizer.\n\n---\n\n**Example Usage**:\n```python\nimport tensorflow as tf\nfrom adahessian import Adahessian\n\n# Define model and loss\nmodel = tf.keras.Sequential([...])\nloss_fn = tf.keras.losses.MeanSquaredError()\n\n# Initialize optimizer\noptimizer = Adahessian(\n    learning_rate=0.01, \n    beta1=0.9, \n    beta2=0.999, \n    weight_decay=0.01\n)\n\n# Training step\n@tf.function\ndef train_step(x, y, model, optimizer):\n    with tf.GradientTape(persistent=True) as tape:\n        predictions = model(x, training=True)\n        loss = loss_fn(y, predictions)\n        gradients = tape.gradient(loss, model.trainable_variables)\n    optimizer.apply_gradients(zip(gradients, model.trainable_variables), tape)\n\n# Training loop\nfor epoch in range(epochs):\n    for x_batch, y_batch in dataset:\n        train_step(x_batch, y_batch, model, optimizer)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnotedance%2Fadahessian","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnotedance%2Fadahessian","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnotedance%2Fadahessian/lists"}