{"id":17897312,"url":"https://github.com/matteosox/nba","last_synced_at":"2025-03-23T06:31:19.224Z","repository":{"id":37842750,"uuid":"349892150","full_name":"matteosox/nba","owner":"matteosox","description":"NBA stats and analysis","archived":false,"fork":false,"pushed_at":"2023-02-26T23:25:49.000Z","size":20167,"stargazers_count":11,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-18T18:21:38.587Z","etag":null,"topics":["docker","jupyter-notebook-environment","nba","nba-stats","python"],"latest_commit_sha":null,"homepage":"https://nba.mattefay.com","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/matteosox.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-21T03:33:50.000Z","updated_at":"2023-10-19T02:26:38.000Z","dependencies_parsed_at":"2024-10-28T15:52:47.958Z","dependency_job_id":null,"html_url":"https://github.com/matteosox/nba","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matteosox%2Fnba","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matteosox%2Fnba/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matteosox%2Fnba/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matteosox%2Fnba/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/matteosox","download_url":"https://codeload.github.com/matteosox/nba/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245066496,"owners_count":20555402,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","jupyter-notebook-environment","nba","nba-stats","python"],"created_at":"2024-10-28T15:08:47.204Z","updated_at":"2025-03-23T06:31:17.394Z","avatar_url":"https://github.com/matteosox.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003ematteosox/nba\u003c/h1\u003e\n\n## [Badges? We ain't got no badges! We don't need no badges! I don't have to show you any stinking badges!](https://www.youtube.com/watch?v=VqomZQMZQCQ)\n\n\u003ch3 align=\"center\"\u003e\nStatus\u003cbr\u003e\n\u003ca href=\"https://github.com/matteosox/nba/actions/workflows/data_pipeline.yaml\"\u003e\u003cimg alt=\"Data Pipeline\" src=\"https://github.com/matteosox/nba/actions/workflows/data_pipeline.yaml/badge.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/matteosox/nba/actions/workflows/setup_test_push.yaml\"\u003e\u003cimg alt=\"Setup, Test, \u0026 Build\" src=\"https://github.com/matteosox/nba/actions/workflows/setup_test_push.yaml/badge.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://nba.mattefay.com\"\u003e\u003cimg alt=\"Website Status\" src=\"https://img.shields.io/website?down_color=red\u0026down_message=offline\u0026label=nba.mattefay.com\u0026up_color=brightgreen\u0026up_message=online\u0026url=https%3A%2F%2Fnba.mattefay.com\"\u003e\u003c/a\u003e\n\u003cbr\u003eLanguages\u003cbr\u003e\n\u003ca href=\"https://docs.python.org/3.10/\"\u003e\u003cimg alt=\"Python: 3.10\" src=\"https://img.shields.io/static/v1?label=Python\u0026message=3%2C10\u0026logo=python\u0026color=%233776AB\"\u003e\u003c/a\u003e\n\u003ca href=\"https://www.gnu.org/software/bash/\"\u003e\u003cimg alt=\"Bash: 5.1\" src=\"https://img.shields.io/static/v1?label=Bash\u0026message=5%2C1\u0026logo=GNU-bash\u0026color=%234EAA25\"\u003e\u003c/a\u003e\n\u003ca href=\"https://www.typescriptlang.org/\"\u003e\u003cimg alt=\"Typescript: \" src=\"https://img.shields.io/github/package-json/dependency-version/matteosox/nba/dev/typescript?filename=app%2Fpackage.json\u0026logo=typescript\u0026color=%233178C6\u0026label=Typescript\"\u003e\u003c/a\u003e\n\u003cbr\u003eServices\u003cbr\u003e\n\u003ca href=\"https://github.com/matteosox/nba\"\u003e\u003cimg alt=\"VCS: Github\" src=\"https://img.shields.io/static/v1?label=VCS\u0026message=Github\u0026logo=github\u0026color=%23181717\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/matteosox/nba/actions\"\u003e\u003cimg alt=\"CI: Github Actions\" src=\"https://img.shields.io/static/v1?label=CI\u0026message=Github%20Actions\u0026logo=Github-Actions\u0026color=%232088FF\"\u003e\u003c/a\u003e\n\u003ca href=\"https://hub.docker.com/repository/docker/matteosox/nba\"\u003e\u003cimg alt=\"Container Repo: Docker Hub\" src=\"https://img.shields.io/static/v1?label=Container%20Repo\u0026message=Docker%20Hub\u0026logo=Docker\u0026color=%232496ED\"\u003e\u003c/a\u003e\n\u003ca href=\"https://protonvpn.com/\"\u003e\u003cimg alt=\"VPN: ProtonVPN\" src=\"https://img.shields.io/static/v1?label=VPN\u0026message=ProtonVPN\u0026logo=ProtonVPN\u0026color=%2356B366\"\u003e\u003c/a\u003e\n\u003ca href=\"https://aws.amazon.com/s3/\"\u003e\u003cimg alt=\"Storage: AWS S3\" src=\"https://img.shields.io/static/v1?label=Storage\u0026message=AWS%20S3\u0026logo=Amazon-S3\u0026color=%23569A31\"\u003e\u003c/a\u003e\n\u003ca href=\"https://vercel.com/matteosox/nba\"\u003e\u003cimg alt=\"CD \u0026 Hosting: Vercel\" src=\"https://img.shields.io/static/v1?label=CD%20%26%20Hosting\u0026message=Vercel\u0026logo=Vercel\u0026color=%23000000\"\u003e\u003c/a\u003e\n\u003ca href=\"https://www.hover.com/\"\u003e\u003cimg alt=\"Domain: Hover\" src=\"https://img.shields.io/static/v1?label=Domain\u0026message=Hover\"\u003e\u003c/a\u003e\n\u003cbr\u003eDependencies\u003cbr\u003e\n\u003ca href=\"https://jupyter.org/try\"\u003e\u003cimg alt=\"Analysis environment: Jupyter\" src=\"https://img.shields.io/static/v1?label=Analysis%20environment\u0026message=Jupyter\u0026logo=Jupyter\u0026color=%23F37626\"\u003e\u003c/a\u003e\n\u003ca href=\"https://www.dynaconf.com/\"\u003e\u003cimg alt=\"Config: Dynaconf\" src=\"https://img.shields.io/static/v1?label=Config\u0026message=Dynaconf\"\u003e\u003c/a\u003e\n\u003ca href=\"https://nextjs.org/\"\u003e\u003cimg alt=\"Web framework: Next.js\" src=\"https://img.shields.io/static/v1?label=Web%20framework\u0026message=Next%2Cjs\u0026logo=Next.js\u0026color=%23000000\"\u003e\u003c/a\u003e\n\u003ca href=\"https://reactjs.org/\"\u003e\u003cimg alt=\"UI: React\" src=\"https://img.shields.io/static/v1?label=UI\u0026message=React\u0026logo=React\u0026color=%2361DAFB\"\u003e\u003c/a\u003e\n\u003ca href=\"https://pbpstats.readthedocs.io/en/latest/\"\u003e\u003cimg alt=\"NBA stats: pbpstats\" src=\"https://img.shields.io/static/v1?label=NBA%20stats\u0026message=pbpstats\"\u003e\u003c/a\u003e\n\u003ca href=\"https://docs.pymc.io/\"\u003e\u003cimg alt=\"Models: pymc3\" src=\"https://img.shields.io/static/v1?label=Models\u0026message=pymc3\"\u003e\u003c/a\u003e\n\u003cbr\u003eTesting\u003cbr\u003e\n\u003ca href=\"https://github.com/psf/black\"\u003e\u003cimg alt=\"Python style: Black\" src=\"https://img.shields.io/static/v1?label=Python%20style\u0026message=Black\u0026color=%23000000\"\u003e\u003c/a\u003e\n\u003ca href=\"https://www.pylint.org/\"\u003e\u003cimg alt=\"Python quality: Pylint\" src=\"https://img.shields.io/static/v1?label=Python%20quality\u0026message=Pylint\"\u003e\u003c/a\u003e\n\u003ca href=\"https://www.shellcheck.net/\"\u003e\u003cimg alt=\"Bash quality: ShellCheck\" src=\"https://img.shields.io/static/v1?label=Bash%20quality\u0026message=ShellCheck\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n## General Info\n\n### NBA Stats and Analysis\n\n- Author: Matt Fay\n- Email: matt.e.fay@gmail.com\n- [Repo](https://github.com/matteosox/nba)\n- [Site](https://nba.mattefay.com)\n\n### Description\n\nThis repo has three main parts:\n1) `pynba`: a Python package of stuff — utilities, data loaders/serializers, scripts — to analyze nba data.\n2) `notebooks`: a collection of Jupyter notebooks analyzing nba data using `pynba`.\n3) `app`: a Next.js web app hosted at [nba.mattefay.com](https://nba.mattefay.com), displaying the latest stats.\n\n## User Notes\n\n### Jupyter Notebook Environment\n\n_TL;DR: To start up the notebook environment, run `notebooks/run.sh`, which will open up a browser tab for you._\n\nWe use a Dockerized Jupyter notebook environment for data analysis. The `notebooks/run.sh` bash script starts this container and opens up a web browser to the Jupyter server for you, with the repo mounted to `/root/nba`. This allows you to edit the `pynba` package without needing to restart the container, since it is installed in [editable mode](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs). The Jupyter notebook directory is the repo's `notebooks` directory, which contains version controller notebooks.\n\n## Future Work\n\n- Analysis\n    - Travel and rest adjustments\n    - Re-evaluate priors\n    - Confirm reduction in home court advantage\n    - Fix 2020 bubble games\n    - Playoffs?!\n- App\n    - Theme/style\n    - Replace images with interactives\n    - Improve tables (sortable, hover for definition, colorize for z-scores)\n\n## Contributing\n\n### Getting started\n\nWe use Docker for a clean environment within which to build, test, analyze, and so on. The `setup.sh` script in the `cicd` directory will build the relevant images for you. Running things natively isn't a supported/maintained thing.\n\nIn addition to Docker, you'll need some developer secrets, in `build/notebook.local.env` and `build/app.local.env`. Those files are git ignored for obvious reasons, so you'll need to ask around to get those credentials.\n\nFinally, while you don't need it to do most workflows, it's probably a good idea to setup docker so you have push access to the Docker hub registry. That'll require a personal access token, which you can then use in combination with your Docker ID to login using `docker login`.\n\n### Code Style\n\nWe use PEP8 for Python, but don't trip, just run `test/black_lint.sh` to get all your spaces in a row.\n\n### Pull Requests\n\nThe `main` branch has [branch protections](https://help.github.com/en/github/administering-a-repository/about-protected-branches) turned on in Github, requiring one reviewer to approve a PR before merging. We also use the code owners feature to specify who can approve certain PRs. As well, merging a PR requires status checks to complete successfully.\n\nWhen naming a branch, please use the syntax `firstname/branch-name-here`. If you plan to collaborate with others on that branch, use `team/branch-name-here`.\n\n## Developer Workflows\n\n### Updating python requirements\n\n_TL;DR: Run `requirements/update_requirements_inside_docker.sh`._\n\nThere are two requirements files checked into this directory:\n1) `requirements.in`\n2) `requirements.txt`\n\nThe `.in` files are where we collect immediate dependencies, described in PyPI format (with versions pinned only as needed). The `.txt` files are generated by running the `requirements/update_requirements_in_docker.sh` script. This script runs the `requirements/update_requirements.sh` inside the `notebook` Docker container. We do this because `pip-compile` should be run from the same virtual environment as your project so conditional dependencies that require a specific Python version, or other environment markers, resolve relative to your project’s environment.\n\nThis gives us both a flexible way to describe dependencies while still achieving reproducible builds. Inspired by [this](https://hynek.me/articles/python-app-deps-2018/) and [this](https://pythonspeed.com/articles/pipenv-docker/).\n\n### Handling Config\n\nWhile the `constants.py` module contains values that don't change with each run, the `config.py` module makes configuration values available in the Python runtime that DO change. This uses [dynaconf](https://www.dynaconf.com/) to inject and load dynamic configuration from 1) `settings.toml` for defaults for each envionment, and 2) environment variables, prefixed with `PYNBA` and registered in `settings.toml`. The `meta_config.py` module provides a convenient syntax for creating a config dataclass with typed values that loads each parameter dynamically from `dynaconf`. You can see an example of this in the `config.py` module. The `dynaconf` environment is determined by the `ENV_FOR_DYNACONF` environment variable.\n\nTo pass environment variables into the Docker runtime, either for `dynaconf` or other purposes, you have two options:\n1) export them in your development environment, then register them in the `notebook.env`. For example, to select a `dynaconf` environment other than `default`, you'll need to export it as `ENV_FOR_DYNACONF`. Note that this variable is already registered in `notebook.env` to be passed in.\n2) add them to your `notebook.local.env` file, which is in the `.gitignore` so it won't be committed. This is where we keep developer credentials for example.\n\nInspired by the [12-factor application guide](https://12factor.net/config).\n\n### Developing the NextJS App\n\n_TL;DR: Run `app/run.sh`._\n\nTo ease developing the NextJS web app, we use `npm run dev` in a Docker container with the app mounted. This starts the app in [development mode](https://nextjs.org/docs/api-reference/cli#development), which takes advantage of NextJS's [fast refresh](https://nextjs.org/docs/basic-features/fast-refresh) functionality, which catches exceptions and loads code updates near-instantaneously.\n\n### Updating node packages\n\n_TL;DR: Run `app/run.sh npm update`._\n\nTo install a new npm package using `npm install new-package`, you can use the same script with an optional command, e.g. `app/run.sh YOUR CMD HERE`. To update all packages, run `app/run.sh npm update`.\n\n## Continuous Integration\n\nWe use Github actions to run our CI pipeline on every pull request. The configuration can be found in `.github/workflows/setup_test_push.yaml`. That said, every step of CI can also be run locally.\n\n### Settin' up\n\n_TL;DR: To run tests, run `cicd/setup.sh`._\n\nThis builds the two relevant docker images, `notebook`, and `app`.\n\nWe do a couple of neat cacheing tricks to speed things up. First off, in the `Dockerfile`s themselves, we use the `RUN --mount=type=cache` functionality of Docker BuildKit to cache Python packages stored in `~/.cache/pip`. This keeps you local machine from re-downloading new Python packages each time. We don't use this for OS-level packages, i.e. those installed using `apt`, to reduce the size of the images. I tried and failed to get this to work for `npm install` and the `node_modules` directory, with mysteriously useless results. This was inspired by [this blog post](https://pythonspeed.com/articles/docker-cache-pip-downloads/)\n\nSecond, we use the new `BUILDKIT_INLINE_CACHE` feature to cache our images using Docker Hub. This is configured in the `docker build` command, and is smart enough to only download the layers you need. This was inspired by [this blog post](https://pythonspeed.com/articles/speeding-up-docker-ci/). This DOES work in Github Actions, while the prior functionality does not.\n\n### Testin'\n\n_TL;DR: To run tests, run `cicd/test.sh`._\n\n#### Python Autoformatting\n\nWe use [`Black`](https://black.readthedocs.io/en/stable/index.html) to check our Python code for proper formatting. If it doesn't pass, you can autoformat your code using Black by running `test/black_lint.sh`. The settings for this can be found in `pyproject.toml`.\n\n#### Python Linting\n\nWe use [`Pylint`](https://www.pylint.org/) for more general linting of our Python code. `Pylint` has some crazy settings and defaults, so use the `pylintrc` config file generously to use `pylint` to your favor, and not to your detriment.\n\n#### Python Unit Tests\n\nWe use the built-in Python module `unittest`'s [test discovery](https://docs.python.org/3/library/unittest.html#test-discovery) functionality. This requires that all of the test files must be modules or packages importable from the root of the repo. As well, they must match the pattern `test*.py`. Our practice is to put tests for a module in a test folder in the same directory, which can then also contain data and other files needed to run those tests.\n\nThe package is installed using `setuptools`'s [`find_packages` function](https://setuptools.readthedocs.io/en/latest/setuptools.html#using-find-packages). We use the `exclude` feature to exclude all test code, i.e. `exclude=[\"*.tests\", \"*.tests.*\", \"tests.*\", \"tests\"]`.\n\nThus, to run tests, we mount the root of the repo to the location in the container it's been installed. All of this is handled nicely by running `test.sh`, which uses the `notebook` container.\n\n#### Bash Script Tests\n\nWe use [ShellCheck](https://github.com/koalaman/shellcheck) to test all the bash scripts in the repo. This helps us avoid some of the many sharp corners of bash, improving quality, readability, and style. This is run in `test/shellcheck.sh`.\n\n#### Web App Tests\n\n**TBD**\n\n### Pushin'\n\n_TL;DR: To push docker images to Docker Hub, run `cicd/push.sh`._\n\nNote that you'll need to be logged in to Docker be able to do this locally. We use the `docker/login-action@v1` build action to login for the automated CI, and it uses a personal access token named `github-actions` from my Docker Hub account to do that, with the username and token stored as secrets.\n\nIn Docker Hub, we have one repository per image:\n    - `notebook` -\u003e `matteosox/nba-notebook`\n    - `app` -\u003e `matteosox/nba-app`\n\nIn addition to pushing each image with a tag as the git SHA of the code producing it, we also push an untagged (i.e. tagged as `latest`) image when pushing from the `main` branch.\n\n### Buildin'\n\nWe don't build the Next.js app in the `setup_test_push.yaml` Github Actions workflow because Vercel is configured to do this for us. That said, as with all other CI workflows, we support running these locally. To build the app, use `app/run.sh npm run build`.\n\n## Data Pipeline\n\nWe run a nightly data pipeline job using Github Actions to update the data hosted on the site. This is configured in `.github/workflows/data_pipeline.yaml`. Again, each step of the process can be run locally.\n\n### VPN\n\nUnfortunately, the NBA blocks API traffic from many cloud hosting providers, e.g. AWS. In our case, [Github Actions runs on Azure](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#cloud-hosts-for-github-hosted-runners). To get around this issue, I've setup a free ProtonVPN account. The various secrets needed to get that to work (a base certificate, TLS authentication key, and openvpn username \u0026 password) are stored as secrets in Github and injected as environment variables in the relevant step. An openvpn config file — `config.ovpn` — can be found in the `vpn` directory, along with a `connect.sh` bash script used by the workflow to set things up.\n\n### Update Data\n\nThe first step of the data pipeline runs the `pynba_update` Python console script inside of the notebook Docker image using the `cicd/etl.sh` bash script. This queries for data for each league in the current year, saving any updates to the local data directory (mounted to the container). For seasons with new games found, we also calculate updated team ratings and generate updated plots.\n\n### Sync Data to S3\n\nThe second (and final) step of the data pipeline runs the `pynba_sync` Python console script using the same environment/mechanism as before. This local data — pbpstats files, season parquet files, incremental possessions parquet files, team ratings \u0026 plots — is then synced to s3, where it can be accessed by the site.\n\n### Github Actions Artifacts\n\nWe store an artifact of the local data directory at the completion of each run of the data pipeline, both for historical data, and ease of debugging.\n\n## Continuous Deployment\n\nThe Python package `pynba` is strictly for code refactoring in this repo's Jupyter notebook environment, so it isn't packaged up and released, e.g. to PyPI.org.\n\n### Vercel\n\nThe NextJS app is deployed to nba.mattefay.com by [Vercel](https://vercel.com/), the company behind NextJS. The deployment process is integrated with Github, so that any commit to the `main` branch results in a new deploy. Conveniently, Vercel also builds and deploys a \"staging\" site for every commit that changes the `app` directory, making them available through comments in your pull request for example.\n\nData and plots are stored in the `nba-mattefay` bucket on AWS S3. To access these files, we inject AWS credentials with environment variables. Unfortunately, Vercel [reserves the usual environment variables](https://vercel.com/docs/platform/limits#reserved-variables) for this, i.e. `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. To get around this, we store them as the non-standard `AccessKeyId` and `SecretAccessKey` environment variables, and manually load credentials in the `aws_s3.ts` javascript library, similar to [this approach](https://vercel.com/support/articles/how-can-i-use-aws-sdk-environment-variables-on-vercel) on Vercel's website. These credentials are from the `matteosox-nba-vercel` AWS IAM user, with read-only access to this one bucket.\n\n### DNS\n\nI own the domain mattefay.com through hover.com. I host my blog there, using format.com. This repo's site is hosted at the nba.mattefay.com subdomain. Since [Vercel](https://vercel.com/) is hosting this site, I have a CNAME DNS record in Hover to alias that subdomain to them, i.e. `CNAME nba cname.vercel-dns.com`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatteosox%2Fnba","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatteosox%2Fnba","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatteosox%2Fnba/lists"}