{"id":18422638,"url":"https://github.com/ndjenkins85/ndj_cookie","last_synced_at":"2025-08-23T15:42:51.146Z","repository":{"id":37195984,"uuid":"398887854","full_name":"ndjenkins85/ndj_cookie","owner":"ndjenkins85","description":"My personal python project quick start. Useful to jump into projects with favourite tooling, and to record best practices","archived":false,"fork":false,"pushed_at":"2022-12-09T04:42:54.000Z","size":7211,"stargazers_count":1,"open_issues_count":16,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-13T15:43:25.954Z","etag":null,"topics":["best-practices","python-packaging","quick-start"],"latest_commit_sha":null,"homepage":"https://www.ndjenkins.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ndjenkins85.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null}},"created_at":"2021-08-22T19:51:57.000Z","updated_at":"2021-12-22T11:02:09.000Z","dependencies_parsed_at":"2023-01-25T16:16:35.509Z","dependency_job_id":null,"html_url":"https://github.com/ndjenkins85/ndj_cookie","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/ndjenkins85/ndj_cookie","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndjenkins85%2Fndj_cookie","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndjenkins85%2Fndj_cookie/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndjenkins85%2Fndj_cookie/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndjenkins85%2Fndj_cookie/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ndjenkins85","download_url":"https://codeload.github.com/ndjenkins85/ndj_cookie/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndjenkins85%2Fndj_cookie/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271755044,"owners_count":24815331,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["best-practices","python-packaging","quick-start"],"created_at":"2024-11-06T04:32:03.444Z","updated_at":"2025-08-23T15:42:51.116Z","avatar_url":"https://github.com/ndjenkins85.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Python project quick start\n\nThis repo is my personal python project quick starter.\nIt contains my favourite tools and options for creating python projects for data science, web development, and adhoc projects.\nWhile this is intended to be a personal resource, this is open to public users.\n\nThe quick-start includes the following features:\n\n- Full set-up guide and checklist - *so you can quickly set up tooling and get into coding*\n- Choice of dependency and virtual environment management with full-featured poetry workflow, partial conda workflow, and docker with poetry - *so you can ensure code runs on different environments*\n- Pre-populated Git and Github assets including gitignore, codeowners, templates, issue labels - *to make your Github experience more enjoyable*\n- Full featured documentation. Includes README with prompts, opinionated CONTRIBUTORS guide, and LICENSE. Sphinx documentation generation with markdown and API reference support, and ability to convert schema tables into data documentation. Github actions to automatically generate Github pages - *to easily expose project documentation*\n- CI/CD framework including pre-commit for quick formatting, task automation with Nox, and Github Actions - *to ensure high quality releases*\n- Linting, type checking, and tests with minimum of tool config files and close nox and pyproject.toml integration - *to standardize code and minimize clutter*\n- Tools for release management including tagging and versioning process, Github actions for release notes, test-pypi and release actions - *to simplify the code release process*\n- `my_project` dummy project example with logging, imports, pytest, argparse CLI, poetry scripts, docstrings - *for a minimal non-tool codeset with warmed up examples*\n- `ndj_pipeline` machine learning pipeline and framework - *to separate data engineering concerns and build repeatable experiments*\n\nThis guide contains three major sections:\n\n* [About this repo](#python-project-quick-start). General info about this repo.\n* [Setup new repo](#instructions-for-copying-to-set-up-new-project). Instructions for copying this repo to create a new project.\n* [README Template](#my-project). Warmed up README template for new projects with writing prompts, instructions for usage and development.\n\nThe following sources have been inspiration for creating my own project quick starter.\n\n* [Hypermodern python](https://cjolowicz.github.io/posts/hypermodern-python-01-setup/)\n* [SQLmodel](https://github.com/tiangolo/sqlmodel)\n* [Project blueprint](https://github.com/johnthagen/python-blueprint)\n\n# Instructions for copying to set up new project\n\n## 1. Clone repo\n\nClone repo locally and delete the `.git` directory to start fresh.\n\n## 2. Pre-use editing\n\n- Edit the `my_project.__init__.py` to revert to version `0.1.0`.\n- Rename `my_project` package name and imports\n- Edit the `README.md` to include [my project](#my-project) onwards.\n- Check python and library versions in noxfile\n- Check python and library versions, and branch name in Github actions\n- Change [LICENSE](LICENSE) as required. Use [this guide](https://github.com/Lucas-C/pre-commit-hooks#removing-old-license-and-replacing-it-with-a-new-one) to redo licenses across project\n\nChange name from my_project to new name in:\n\n- `README.md`\n- `CONTRIBUTING.md`\n- `.flake8`\n- `pyproject.toml`\n- `docs/conf.py`\n- `setup.py`\n- `tests/test_utils.py`\n- `Dockerfile`\n- `docker-compose.yml`\n\n## 3. Choose any of poetry, conda (or docker-compose) for project.\n\n### Poetry\n\nSee the [Environment 1: Poetry](#environment-1-poetry) in the Developer Guide to set up your own environment first.\n\nRemove tools not required by poetry, but required for conda\n- Delete `setup.py`\n- Delete `docs/requirements.txt`\n- Edit Conda references in `README.md`\n- Dockerfile and docker-compose.yml\n\n### Conda\n\nSee the [Environment 2: Conda](#environment-2-conda) in the Developer Guide to set up your own environment first.\n\n*Note*: using conda will mean incompatability with some Nox, Github actions, and library publish functionality. Only the default Nox sessions are included (with light flake8 checks), plus black and docs.\n\nAdditional conda related setup:\n\n* Setup project details in `setup.py`.\n* Remove or update the following Github Actions:\n  * coverage\n  * release\n  * test-pypi\n  * tests\n* Update project README specify conda instructions\n\n### Docker-compose\n\n`Dockerfile` and `docker-compose` are supported using poetry for dependencies.\nSee above instructions for conda cleanup.\n\nSee the [Environment 3: Docker](#environment-3-docker) in the Developer Guide to set up your own environment first.\n\n## 4. Instantiate pre-commit, add log directory, create new git repo\n\n```bash\npre-commit install\ngit init\ngit add .\ngit add logs/.gitkeep --force\ngit commit -m \"initial commit\"\ngit tag 0.1.0\n```\n\nCreate a repo in github and follow instructions to push (including tags).\n\nCheck if the branch name is `main` or `master` - Github Actions are set to use `main`.\n\n## 5. Post push cleanup\n\n- Setup Codecov connection\n- Setup pypi and test-pypi secrets, uncomment test-pypi github action\n- In Github repo set up dependabot and Github pages\n\n# My Project\n\nWhat is it, at a high high level?\nWho is the audience or end users? Any requirements?\nWhat are the feature and benefits?\n\n* [Instructions for users](#instructions-for-users)\n  * [Installation](#installation)\n  * [Usage documentation](#usage-documentation)\n  * [Bug reports](#bug-reports)\n* [Instructions for developers](#instructions-for-developers)\n  * [Environment 1: Poetry](#environment-1-poetry)\n  * [Environment 2: Conda](#environment-2-conda)\n  * [Environment 3: Docker](#environment-3-docker)\n  * [Testing with Nox](#testing-with-nox)\n  * [Code formatting with Pre-commit](#code-formatting-with-pre-commit)\n* [Contributors](#contributors)\n\n## Instructions for users\n\nThe following are the quick start instructions for using the project as an end-user.\n\nFollow the [Instructions for developers](#instructions-for-developers) to set up the virtual environment and dependency management.\nWe recommend `poetry` for full functionality. An alternative `conda` environment has been prepared but will not work with `nox`.\n\n### Installation\n\nNote: Instructions marked with %% are not functioning and are for demo purposes only.\n\nInstall the project using pip %%:\n\n```bash\npip install my_project\n```\n\nInclude an example of running the program with expected outputs.\n\nTo replicate the data transformations and model results, run the following commands from the project root.\nThese should be run from the `poetry shell`, or `conda` environment, or with the `poetry run` prefix.\n```bash\npython -m ndj_pipeline.transform\npython -m ndj_pipeline.model -p data/doordash_pred.yaml\npython -m ndj_pipeline.final_prediction_clean\n\n```\n\nThis will produce a feature rich dataset in `data/processed`, model results and metrics under `data/doordash_pred`, and the formatted predictions file under `data_to_predict.csv`.\n\nExample of using poetry to create scripts.\n\n```bash\nmy_project -i1 environment.yml -i2 environment.yml -v\n```\n\nAlternatively run from Dockerfile or docker-compose.\nSee [Docker environment instructions](#environment-3-docker) for more details.\n\n### Usage documentation\n\nThe user guides can be found on [github pages](https://ndjenkins85.github.io/ndj_cookie).\nThis includes overview of features, discussion of `ndj_pipeline` framework, and API reference.\n\n### Bug reports\n\nPlease raise an [issue](https://github.com/ndjenkins85/ndj_cookie/issues) with `bug` label and I will look into it!\n\n## Instructions for developers\n\nThe following are the setup instructions for developers looking to improve this project.\nFor information on current contributors and guidelines see the [contributors](#contributors) section.\nFollow each step here and ensure tests are working.\n\n### Environment 1: Poetry\n\n[Poetry](https://python-poetry.org/docs/) handles virtual environment management, dev and optional extra libraries, library development, builds and publishing.\n\nCheck the poetry website for the latest instructions on how to install poetry.\nYou can use the following command on OS/linux to install poetry 1.1.9 used in this project.\n\n```bash\ncurl -sSL https://install.python-poetry.org | python - --version 1.1.9\n```\n\nIt is recommended to set virtual environment creation to within project using the following command.\nThis adds a `.venv` directory to project to handle cache and virtual environment.\n```bash\npoetry config virtualenvs.in-project true\n```\n\nYou can set up the virtual environment in the repo using the following command.\nMake sure that any other virtual environments (i.e. `conda deactivate`) are deactivated before running.\n\n```bash\npoetry install\n```\n\nTroubleshooting: You may need to point poetry to the correct python interpreter using the following command.\nIn another terminal and in conda, run `which python`.\n```bash\npoetry env use /path/to/python3\n```\n\nWhen the environment is correctly installed, you can enter the virtual environment using `poetry shell`. Library can be built using `poetry build`.\n\n### Environment 2: Conda\n\n[Conda](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html) is a lightweight solution for Anaconda python users to handle virtual environment management and basic library specification.\n\nFollowing commands will create the conda environment and setup the library in interactive development mode using setup.py.\n\n```bash\nconda env create -f environment.yml\nconda activate my_project\npip install -e .\n```\n\nLibrary can be built using\n\n```bash\npython setup.py bdist_wheel\n```\n\n### Environment 3: Docker\n\n[Docker](https://www.docker.com/) goes beyond virtual environment management to virtualize the operating system itself. The docker container is specified through a Dockerfile and can be run with docker commands or docker-compose. Dependeny management is handled through poetry.\n\nUse either of the following commands to setup and run the docker environment.\n\n``` bash\ndocker build -t ndj_cookie/my_project .\ndocker run --rm ndj_cookie/my_project\ndocker stop $(docker ps -a -q)\n```\n\nExample docker-compose also included.\n\n``` bash\ndocker-compose build\ndocker-compose up\ndocker-compose down\n```\n\n### Testing with Nox\n\n[Nox](https://nox.thea.codes/en/stable/index.html) is a command-line tool that automates testing in multiple Python environments, similar to tox, Makefiles or scripts. Unlike tox, Nox uses a standard Python file for configuration.\n\nHere it is used for code quality, testing, and generating documentation.\n\nThe following command can be used to run mypy, lint, and tests.\nIt is recommended to run these before pushing code, as this is run with Github Actions.\nSome checks such as black are run more frequently with [pre-commit](#code-formatting-with-pre-commit).\n\n```bash\npoetry run nox\n```\n\nLocal Sphinx documentation can be generated with the following command.\nDocumentation publishing using Github Actions to Github pages is enabled by default.\n\n```bash\npoetry run nox -s docs\n```\n\nOther available commands include:\n\n```bash\npoetry run nox -rs coverage\n```\n\n### Code formatting with Pre-commit\n\n[Pre-commit](https://pre-commit.com/) is a framework for managing and maintaining multi-language pre-commit hooks.\n\nIt intercepts the `git commit` command to run checks of staged code before the commit is finalized.\nThe checks are specified in `.pre-commit-config.yaml`.\nChecks in use are quick, pragmatic, and apply automatic formatting checks.\nIf checks fail, it is usually only a matter of re-staging the files (`git add`) and attempting to commit again.\n\nThe aim is to provide a lightweight way to keep some code standards automatically in line with standards.\nThis does not replace the need to run nox tests, although pre-commits will satisfy some of the nox test checks.\n\nOn first time use of the repository, pre-commit will need to be installed locally.\nYou will need to be in the `poetry shell` or `conda` environment.\nRun the following command to perform a first time install.\n\n```bash\npre-commit install\n```\n\nThis will cache several code assets used in the checks.\n\nWhen you have new code to commit, pre-commit will kick in and check the code.\nAlternatively, you can run the following command to run for all files in repo.\n\n``` bash\npre-commit run --all-files\n```\n\n## Contributors\n\n* [Nick Jenkins](https://www.nickjenkins.com.au) - Data Scientist, API \u0026 Web dev, Team lead, Writer\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) in Github repo for specific instructions on contributing to project.\n\nUsage rights governed by [LICENSE](LICENSE)  in Github repo or page footer.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fndjenkins85%2Fndj_cookie","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fndjenkins85%2Fndj_cookie","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fndjenkins85%2Fndj_cookie/lists"}