{"id":18308558,"url":"https://github.com/hyzhak/mle","last_synced_at":"2025-04-09T11:29:01.097Z","repository":{"id":143505871,"uuid":"91684810","full_name":"hyzhak/mle","owner":"hyzhak","description":"Machine Learning: Maximum Likelihood Estimation (MLE)","archived":false,"fork":false,"pushed_at":"2017-06-20T10:57:44.000Z","size":671,"stargazers_count":1,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-15T05:27:27.024Z","etag":null,"topics":["machine-learning","map","maximum-a-posteriori-estimation","maximum-likelihood-estimation","mle"],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hyzhak.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-18T11:20:44.000Z","updated_at":"2017-10-22T09:29:36.000Z","dependencies_parsed_at":"2023-04-26T23:01:05.739Z","dependency_job_id":null,"html_url":"https://github.com/hyzhak/mle","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyzhak%2Fmle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyzhak%2Fmle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyzhak%2Fmle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyzhak%2Fmle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hyzhak","download_url":"https://codeload.github.com/hyzhak/mle/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248030464,"owners_count":21036198,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["machine-learning","map","maximum-a-posteriori-estimation","maximum-likelihood-estimation","mle"],"created_at":"2024-11-05T16:08:26.977Z","updated_at":"2025-04-09T11:29:01.091Z","avatar_url":"https://github.com/hyzhak.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MLE and MAP\nMachine Learning: Maximum Likelihood Estimation (MLE) and Maximum a Posteri (MAP) Estimation\n\n## Subtleties\n\nML doesn't work good with sparse data because `P(X | Y)` might be zero.\n(for example, Xi = birthdate, Xi = Jan_25_1992)\n\n```\nP(Y=1 | X1...Xn) = (P(Y=1) * Mult P(Xi | Y=1) for i) / P (X1...Xn)\n```\n\nWe can solve it by using prior with MAP estimation.\n\n# MLE\n\n## Pros\n\n- invariant under reparameterization. So we can wrap\n![\\theta_{MLE}](http://www.sciweavers.org/tex2img.php?eq=%20%5Ctheta_%7BMLE%7D\u0026bc=Transparent\u0026fc=Black\u0026im=jpg\u0026fs=12\u0026ff=arev\u0026edit=0)\nin any function.\n\n\n# MAP\n\n## Pros\n\n- avoid overfitting (regularization / shrinkage)\n- tends to look like MLE asymptotically\n\n## Cons\n\n- point estimation (no representation of uncertainty in θ).\nBecause it could choose spike of θ because it has higher probability\n- not invariant under reparameterization\n- must assume prior on θ\n\n## Examples\n\n### Univariate Gaussian mean\n\n![\\theta_{MAP} =  \\overline{x} * \\frac{n}{n + \\sigma^2} + \\mu * \\frac{\\sigma^2}{n + \\sigma^2}](https://latex.codecogs.com/gif.latex?\\inline\u0026space;\\theta_{MAP}\u0026space;=\u0026space;\\overline{x}\u0026space;*\u0026space;\\frac{n}{n\u0026space;\u0026plus;\u0026space;\\sigma^{2}}\u0026space;\u0026plus;\u0026space;\\mu\u0026space;*\u0026space;\\frac{\\sigma^{2}}{n\u0026space;\u0026plus;\u0026space;\\sigma^{2}})\n\nin other words it is sample mean plus prior mean.\n\n![\\theta_{MLE} = \\overline{x}](https://latex.codecogs.com/gif.latex?\\theta_{MLE}%20=%20\\overline{x})\n\nso when n-\u003e0 we get \n\n![\\theta_{MAP} \\rightarrow \\mu](https://latex.codecogs.com/gif.latex?\\theta_{MAP}%20\\rightarrow%20\\mu)\n\nbut when n-\u003e∞ we get\n\n![\\theta_{MAP} \\rightarrow \\theta_{MLE}](https://latex.codecogs.com/gif.latex?\\theta_{MAP}%20\\rightarrow%20\\theta_{MLE})\n\n- [MLE symbolic example](https://github.com/hyzhak/mle/blob/master/mle.ipynb)\n- [MLE statsmodel example](https://github.com/hyzhak/mle/blob/master/mle-statsmodel.ipynb)\n\n\n## Related Topics\n- The Cramer-Rao Lower Bound\n  - [Central Limit Theorem](https://en.wikipedia.org/wiki/Central_limit_theorem)\n    sum independent random variables are tend toward a normal distribution\n- [Likelihood Ratio Test](https://en.wikipedia.org/wiki/Likelihood-ratio_test) (compare zero hypothesis with ml value)\n- [Wald Test](https://en.wikipedia.org/wiki/Wald_test)\n- etc\n\n## Videos\n\n### Jeff Miller (mathematicalmonk)\n- [(ML 6.1) Maximum a posteriori (MAP) estimation](https://www.youtube.com/watch?v=kkhdIriddSI)\n- [(ML 6.2) MAP for univariate Gaussian mean](https://www.youtube.com/watch?v=KogqeZ_88-g)\n- [(ML 6.3) Interpretation of MAP as convex combination](https://www.youtube.com/watch?v=SFQK57G5VF8)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhyzhak%2Fmle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhyzhak%2Fmle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhyzhak%2Fmle/lists"}