{"id":25509189,"url":"https://github.com/tonyduan/glm-lib","last_synced_at":"2025-04-10T14:04:52.804Z","repository":{"id":57434956,"uuid":"231504717","full_name":"tonyduan/glm-lib","owner":"tonyduan","description":"Lightweight Python library for GLM regression.","archived":false,"fork":false,"pushed_at":"2022-12-20T04:41:25.000Z","size":75,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-24T12:47:34.363Z","etag":null,"topics":["generalized-linear-models","machine-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tonyduan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-01-03T03:25:14.000Z","updated_at":"2023-02-07T02:31:29.000Z","dependencies_parsed_at":"2023-01-29T23:45:15.653Z","dependency_job_id":null,"html_url":"https://github.com/tonyduan/glm-lib","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonyduan%2Fglm-lib","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonyduan%2Fglm-lib/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonyduan%2Fglm-lib/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonyduan%2Fglm-lib/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tonyduan","download_url":"https://codeload.github.com/tonyduan/glm-lib/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248027407,"owners_count":21035594,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["generalized-linear-models","machine-learning"],"created_at":"2025-02-19T08:54:13.828Z","updated_at":"2025-04-10T14:04:52.784Z","avatar_url":"https://github.com/tonyduan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"### Generalized Linear Models in Python\n\nLast update: January 2020.\n\n---\n\nA generalized linear model fits an exponential family distribution with a linear model. The resulting optimization problem is convex when the natural parameterization is used.\n\n**Exponential Family Distributions**\n\nWe assume the response is generated from a distribution parameterized by $\\eta$,\n```math\np_\\eta(y) = b(y)\\exp(\\eta^\\top T(y) - a(\\eta)).\n```\n\n\nHere $\\eta$ is the natural parameter, $T(y)$ is the sufficient statistic, and $a(\\eta)$ is the log partition function. Using a linear model with a canonical link function (i.e. natural parameterization), we suppose\n```math\n\\eta = \\theta x \\implies \\nabla_\\theta \\eta_i = x,\\ \\nabla^2_\\theta \\eta_i = xx^\\top.\n```\n\n\nNote that here $\\eta \\in\\mathbb{R}^p, \\theta\\in\\mathbb{R}^{p,n},x\\in\\mathbb{R}^n$.\n\n**Fisher Scoring**\n\nThe model is fit using maximum likelihood. We take a natural gradient step using the Fisher information in $\\theta$,\n\n```math\n\\theta \\leftarrow \\theta - \\mathcal{I}_\\theta^{-1}\\nabla_\\theta \\left(-\\log p_\\theta(y)\\right).\n```\n\n\nNotice that by the chain rule, we have the following score and Hessian of the log-likelihood.\n```math\n\\begin{align*}\n\\nabla_\\theta -\\log p_\\theta(y) \u0026 = \\nabla_\\eta-\\log p_\\eta(y) x^\\top\\\\\n\u0026 =(\\mathbb{E}_\\eta[T(Y)] - T(y))x^\\top \\\\\n\\nabla^2_{\\theta_i}-\\log p_\\theta(y) \u0026 = \\nabla_{\\eta_i}^2 -\\log p_\\eta(y) xx^\\top\\\\\n\u0026 = \\mathrm{Var}_\\eta[T_i(Y)]xx^\\top\n\\end{align*}\n```\n\n\nThe Fisher information matrix with respect to the $i$-th row of $\\theta$ is then the expected value of a constant (i.e. the Hessian has no dependence on observations $y$), so\n```math\n\\mathbb{E}_\\eta[\\nabla^2_{\\theta_i} -\\log p_\\theta(y)] = \\mathrm{Var}_\\eta[T_i(Y)]x x^\\top.\n```\nNotice this coincides with a Newton-Raphson step.\n\n#### References\n\n[1] Nelder, J.A., and Wedderburn, R.W.M. (1972). Generalized Linear Models. Journal of the Royal Statistical Society. Series A (General) *135*, 370–384.\n\n---\n\n#### Appendix\n\nHere we list useful results of exponential families.\n\n**Result 1.** $T(y)$ are sufficient statistics for the parameters $\\eta$.\n```math\np_\\eta(y_1,\\cdots,y_n) = b(y_1)\\cdots b(y_n)\\exp\\left(\\eta^\\top \\sum_{i=1}^n T(y_i)-na(\\eta)\\right)\n```\n\n\n**Result 2.** The gradients of the log partition function yield moments of the sufficient statistics. First recall that\n```math\na(\\eta) = \\log \\int_y b(y)\\exp \\left(\\eta^\\top T(y)\\right)dy.\n```\nNow observe that\n```math\n\\begin{align*}\n\\frac{\\partial}{\\partial \\eta_i} a(\\eta) \u0026 = \\frac{\\int_y b(y)\\exp \\left(\\eta^\\top T(y)\\right)T_i(y)dy}{\\int_y b(y)\\exp \\left(\\eta^\\top T(y)\\right)dy}\\\\\n\u0026 = \\int_y b(y)\\exp(\\eta^\\top T(y) - a(\\eta))T_i(y)dy\\\\\n\u0026 = \\mathbb{E}_\\eta[T_i(Y)].\n\\end{align*}\n```\nSimilarly it can be shown that\n```math\n\\frac{\\partial}{\\partial \\eta_i\\eta_j}a(\\eta) = \\mathrm{Cov}[T_i(Y), T_j(Y)].\n```\n\n\n**Result 3.** The negative log-likelihood of an exponential family distribution is always convex with respect to the natural parameters. This is because the Hessian is a constant positive semi-definite matrix in this case, coinciding with the variance of the sufficient statistics and with no dependence on observations.\n```math\n\\begin{align*}\n-\\log p_\\eta(y) \u0026 = a(\\eta) -\\eta^\\top T(y) -\\log b(y)\\\\\n\\nabla_\\eta -\\log p_\\eta(y) \u0026 = \\nabla_\\eta a(\\eta) - T(y)\\\\\n                            \u0026 = \\mathbb{E}_\\eta[T(Y)] - T(y)\\\\\n\\nabla^2_\\eta -\\log p_\\eta(y) \u0026 = \\nabla^2_\\eta a(\\eta) = \\text{Var}_\\eta[T(Y)] \\succeq 0.\n\\end{align*}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftonyduan%2Fglm-lib","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftonyduan%2Fglm-lib","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftonyduan%2Fglm-lib/lists"}