{"id":51716583,"url":"https://github.com/appleweiping/cs229-machine-learning","last_synced_at":"2026-07-17T03:36:30.259Z","repository":{"id":369026863,"uuid":"1287989306","full_name":"appleweiping/cs229-machine-learning","owner":"appleweiping","description":"Solutions to the Stanford CS229 Machine Learning problem-set coding assignments — regression/GLMs, GDA \u0026 Naive Bayes, SVM, k-means \u0026 EM, PCA, and RL — with measured results","archived":false,"fork":false,"pushed_at":"2026-07-03T07:41:17.000Z","size":3484,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-03T09:27:10.840Z","etag":null,"topics":["cs229","csdiy","em-algorithm","machine-learning","numpy","pca","svm"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/appleweiping.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-07-03T07:13:31.000Z","updated_at":"2026-07-03T07:42:29.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/appleweiping/cs229-machine-learning","commit_stats":null,"previous_names":["appleweiping/cs229-machine-learning"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/appleweiping/cs229-machine-learning","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/appleweiping%2Fcs229-machine-learning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/appleweiping%2Fcs229-machine-learning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/appleweiping%2Fcs229-machine-learning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/appleweiping%2Fcs229-machine-learning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/appleweiping","download_url":"https://codeload.github.com/appleweiping/cs229-machine-learning/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/appleweiping%2Fcs229-machine-learning/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35566571,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-17T02:00:06.162Z","response_time":116,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cs229","csdiy","em-algorithm","machine-learning","numpy","pca","svm"],"created_at":"2026-07-17T03:36:29.556Z","updated_at":"2026-07-17T03:36:30.248Z","avatar_url":"https://github.com/appleweiping.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CS229 Machine Learning — Problem-Set Coding Solutions\n\n\u003e Vectorised NumPy implementations of the coding portions of the **Stanford\n\u003e CS229 — Machine Learning** problem sets — an independent, from-skeleton\n\u003e implementation, part of a [csdiy.wiki](https://csdiy.wiki/) full-catalog build.\n\n![status](https://img.shields.io/badge/status-complete-brightgreen)\n![language](https://img.shields.io/badge/python-informational)\n![license](https://img.shields.io/badge/license-MIT-blue)\n\n## Overview\n\nCS229 (Andrew Ng; this build follows the widely-mirrored 2018 autumn offering)\nis Stanford's graduate machine-learning course. Its four problem sets ship\nNumPy starter skeletons with real datasets and mark the coding tasks with\n`*** START CODE HERE ***`. This repo implements **every coding problem** across\nPS1–PS4 from scratch in clean, vectorised NumPy — supervised learning and GLMs,\ngenerative and kernel methods, unsupervised learning, and reinforcement\nlearning — and reports the **measured** result of actually running each one.\n\n## Results (measured on CPU, `OMP_NUM_THREADS=3`, NumPy 2.4)\n\n| Problem set | Assignment | What it does | Result (measured) |\n|---|---|---|---|\n| PS1 | 1b Logistic regression (Newton) | binary classification | val acc **0.90** (ds1), **0.91** (ds2) |\n| PS1 | 1e GDA | generative classifier | val acc **0.83** (ds1), **0.91** (ds2) |\n| PS1 | 2 Positive-only labels | α-correction | test acc vs true label: naive **0.50** → corrected **0.95** (oracle 0.98) |\n| PS1 | 3d Poisson regression (GLM) | count regression | val corr(true, pred) **1.00** |\n| PS1 | 5b/5c Locally weighted regression | non-parametric fit + τ sweep | best τ = **0.05**, val MSE **0.0124**, test MSE **0.0170** |\n| PS2 | 5 Kernel perceptron | linear vs RBF kernel | test acc: dot **0.525** vs RBF **0.920** |\n| PS2 | 6 Naive Bayes spam | multinomial NB + Laplace | test acc **0.9785**; top words: `claim, won, prize, tone, urgent!` |\n| PS2 | 6 SVM spam (RBF, Pegasos) | radius tuning | best radius **0.1**, test acc **0.9695** |\n| PS3 | 4 EM for GMM | unsup. + semi-sup. | unsup. ~**117** iters (LL ≈ −1835); semi-sup. ~**19** iters |\n| PS3 | 5 k-means image compression | colour quantization | 16 colours → **4 bits/px**, **6× compression** |\n| PS4 | 4 ICA | blind source separation | 5 mixed → 5 separated `.wav`; unmixing `W` recovered |\n| PS4 | 6 Cart-pole control | model-based RL + value iteration | balance time **11 → ~100 steps**, converged after **113** failures |\n\n### A few figures (full set in [`results/figures/`](results/figures))\n\n| GDA decision boundary (PS1) | k-means 6× compression (PS3) | Cart-pole learning curve (PS4) |\n|---|---|---|\n| ![gda](results/figures/ps1_gda_ds1.png) | ![kmeans](results/figures/ps3_kmeans_compressed.png) | ![cartpole](results/figures/ps4_cartpole_learning_curve.png) |\n\n| Kernel perceptron — dot (fails) | Kernel perceptron — RBF (separates) | Positive-only: naive vs α-corrected |\n|---|---|---|\n| ![dot](results/figures/ps2_perceptron_dot.png) | ![rbf](results/figures/ps2_perceptron_rbf.png) | ![corr](results/figures/ps1_posonly_corrected.png) |\n\n## Implemented assignments\n\n- [x] **PS1 1b** — Logistic regression via Newton's method\n- [x] **PS1 1e** — Gaussian Discriminant Analysis\n- [x] **PS1 2** — Logistic regression with incomplete, positive-only labels (+ α correction)\n- [x] **PS1 3d** — Poisson regression (GLM, gradient ascent)\n- [x] **PS1 5b/5c** — Locally weighted linear regression + bandwidth tuning\n- [x] **PS2 5** — Kernelized perceptron (dot \u0026 RBF kernels)\n- [x] **PS2 6** — SMS spam classification: multinomial Naive Bayes + RBF-kernel SVM\n- [x] **PS3 4** — EM for a Gaussian Mixture Model (unsupervised \u0026 semi-supervised)\n- [x] **PS3 5** — k-means colour compression\n- [x] **PS4 4** — Independent Component Analysis (blind source separation)\n- [x] **PS4 6** — Cart-pole balancing with model-based RL / value iteration\n\n## Project structure\n\n```\ncs229-machine-learning/\n├── data/            # real CS229 datasets, split per problem set (ps1..ps4)\n├── ps1/src/         # regression \u0026 GLMs (util, linear_model, p01..p05, run.py)\n├── ps2/src/         # perceptron, Naive Bayes, SVM\n├── ps3/src/         # EM/GMM, k-means\n├── ps4/src/         # ICA, cart-pole RL (+ course-provided env.py physics)\n├── notes/           # derivations tying the math to the code\n├── results/         # run logs + curated figures/\n├── requirements.txt\n└── LICENSE\n```\n\n## How to run\n\n```bash\n# Python 3.11. Use the shared csdiy env or a fresh venv:\npython -m pip install -r requirements.txt\n\n# Each problem set has a self-contained driver; run from its src/ directory.\ncd ps1/src \u0026\u0026 python run.py      # all PS1 problems (or: python run.py 1)\ncd ../../ps2/src \u0026\u0026 python run.py\ncd ../../ps3/src \u0026\u0026 python run.py\ncd ../../ps4/src \u0026\u0026 python run.py\n```\n\nEach driver writes predictions and figures to its local `output/` directory and\nprints the measured metrics. On this machine every problem set runs on CPU in\nunder a couple of minutes (the kernel perceptron and cart-pole loop are the\nslowest).\n\n## Verification\n\nEvery number in the results table was produced by actually running the code on\nthis machine; the captured stdout for each problem set is in\n[`results/`](results) (`ps1_run.log` … `ps4_run.log`) and the figures in\n[`results/figures/`](results/figures). Highlights:\n\n- **PS1** `p02` prints `α ≈ 0.17` and recovers test accuracy 0.50 → 0.95 with the\n  correction; `p03d` reaches correlation 1.00 between predicted and true counts.\n- **PS2** `p06` reports Naive Bayes test accuracy **0.9785** and the SVM radius\n  sweep (`0.01→0.92, 0.1→0.95, 1→0.93, 10→0.88` on validation) selecting 0.1.\n- **PS3** `p04` logs the monotonically-increasing log-likelihood and the\n  iteration counts (unsupervised ~117 vs semi-supervised ~19); `p05` logs the\n  decreasing distortion and the 6× compression ratio.\n- **PS4** `p06` logs steps-to-failure climbing from 11 to ~100 and converging\n  after 113 failures — the learning curve figure shows the plateau clearly.\n\n## Tech stack\n\nPython 3.11, NumPy (all models), Matplotlib (figures), SciPy (ICA `.wav` I/O and\nthe cart-pole learning-curve smoothing). No deep-learning frameworks — every\nalgorithm is implemented directly.\n\n## Key ideas / what I learned\n\n- Second-order optimisation (Newton's method) vs first-order gradient ascent,\n  and when each is the natural fit (logistic regression vs the Poisson GLM).\n- Generative (GDA) vs discriminative (logistic) modelling of the same boundary,\n  and how the shared-covariance assumption shows up in the data.\n- The kernel trick made concrete: the same perceptron goes from unable to\n  separate the data (dot kernel) to 92% accuracy (RBF kernel).\n- EM as coordinate ascent on a log-likelihood, and how a little supervision\n  removes the label-permutation symmetry and accelerates convergence.\n- Model-based RL: estimating an MDP from experience and solving it with value\n  iteration to actually balance a pole.\n\n## Credits \u0026 license\n\nBased on the problem sets of **CS229: Machine Learning** by Andrew Ng and the\nCS229 teaching staff (Stanford University). This repository is an independent\neducational reimplementation; all course materials, problem statements, and\ndatasets belong to their original authors. The cart-pole physics simulator\n(`ps4/src/env.py`) is course-provided infrastructure, used unmodified and\ncredited in-file. Original code in this repo is released under the\n[MIT License](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fappleweiping%2Fcs229-machine-learning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fappleweiping%2Fcs229-machine-learning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fappleweiping%2Fcs229-machine-learning/lists"}