https://github.com/ejw-data/devops-setup
This repo shows the process of automating common repository tasks like formatting code, performing linting tasks before merging, applying unit testing, and other automations through git and github events.
https://github.com/ejw-data/devops-setup
github-actions pre-commit-hook pytest python
Last synced: 11 months ago
JSON representation
This repo shows the process of automating common repository tasks like formatting code, performing linting tasks before merging, applying unit testing, and other automations through git and github events.
- Host: GitHub
- URL: https://github.com/ejw-data/devops-setup
- Owner: ejw-data
- Created: 2023-09-16T06:21:47.000Z (over 2 years ago)
- Default Branch: prod
- Last Pushed: 2025-02-03T22:40:49.000Z (about 1 year ago)
- Last Synced: 2025-02-03T23:28:26.156Z (about 1 year ago)
- Topics: github-actions, pre-commit-hook, pytest, python
- Language: Python
- Homepage:
- Size: 292 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# dev-activity
Author: Erin James Wills, ejw.data@gmail.com
Photo by Campaign Creators on Unsplash
```
Status: Setup complete. Ready to create additional customizations.
```
## Overview
This repo is used to demonstrate work flow processes used in data pipelines. Here are some examples:
* Checking for formatting and file structure issues prior to committing with git hooks:`pre-commit`
* Testing code functionality by checking input and output consistency: `unittest` and `pytest`
* Creating custom git commands that automate common routines
## Purpose
The overall goal of this repo is very specific: I would like to automate common routines and create a workflow of best practices. Intially, I thought I would use two features to add workflows - git hooks by using `pre-commit` and github actions as my simple CI/CD tool. I immediately saw three problems. First, pre-commits are nice in concept but by default they run on all branches unless you manipulate the config file for each branch. I would prefer to have different procedures for `bug` and `feature` branches as well as a `test` branch and a `dev` branch. The final branch would be the `prod` branch. The second problem is that pre-commits are locally controlled and can be modified or simply bypassed if the user desires. The pre-commits should be used not as a policy but as a best practice before sending updates to be merged. The third problem is that the commit failures can become annoying and discourage frequent commits for version control and pushes for backup purposes. These realizatons helped me design the following plan as a compromise that encourages good habits and separates that from good policies:
1. Branches for `bug` and `feat` do not have required git pre-commit rules. Users can commit and push anything to their branch but they need to use the `git autopush` command which is a [custom command that I created for this particular scenario](./docs/custom-git.md). This command allows for bypassing the pre-commit rules and adds a prefix to the commit message that identifies it as a `RAW` branch or a `CLEANED` branch. The custom command simplifies the process. The user should also know that before wanting to push code for a merge that the pre-commit program should be run.
1. Since users will not be able to guarentee their use of pre-commits, the github actions will have a check to make sure that before a merge commit occurs to the `test` branch that the checks have been completed.
1. After a merge to `test` has occurred then several unit tests will be deployed specific to the changes. If those pass, then the branch is merged into `dev`
1. After a merge to `dev` a final set of broader functionality testing occurs.
1. The last step is to merge the `dev` into `prod` and at this phase the standard reliabilty testing occurs through synthetic and direct testing.
## GitHub Workflows
Right now these files are just proof-of-concept tests to solve initial design issues.
* `feature-push-checks` only runs on pushes of branches starting with `bug-*` or `feat-*`. This forces the `pre-commit ci` to run based on the setup found in `.pre-commit-config.yaml`
* `test-pull-request-checks.yaml` only runs when a merge is attempted on the `test` branch. This branch runs the pytest functionality tests.
* `prod-pull-request-checks.yaml` only runs when a merge is attempted on the `prod` branch. This is a template workflow from github and will be customized in the future.
## Other GitHub Configurations
* Protection from force-pushes and branch deletion was added to the `test`, `dev`, and `prod` branches.
* Added [instructions](./docs/commit-stds.md) about creating commit messages.
## Unit Testing Uses
* For batch data that may be applied to training on a machine learning pipeline:
* Check that all required columns are present for the pipeline
* Check for a minimum number of rows
* Check for data range (min/max)
* Check for date range
* Check for distribution
* Check for maximum execution time
## Future Work
- [x] Update names for github workflows
- [x] Use `test-pull-request-checks.yaml` as a template for checks on the `dev` and `prod` branch
- [x] Rename `main` branch to `prod`
- [x] Determine what type of configuration is needed with `pre-commit ci`. Using default settings currently.
- [x] Create github workflow that uses `.pre-commit-config.yaml` in the `push` or `pull_request` step of the `test` branch to fix formatting. This solves the issue if the user bypasses the `pre-commit` locally.
- [ ] Create a workflow that can automatically push changes from any branch to `test` > `dev` > `prod` assuming no errors.
- [ ] Create an environment variable on GitHub to use in scripts and apis
## Installation
- `pip install pre-commit`
- To confirm installation, type: `pre-commit —V`
## Intitial Setup
1. Create an initial yaml file. This step could be done manually.
* Type: `pre-commit sample-config > .pre-commit-config.yaml`
1. Add any additional pre-commits as needed. The hook can be created in this file or reference an external file url.
1. Before this file or any changes to this file can take effect, run this command: `pre-commit install`
## Usage
1. Functionality is now automatically added when running `git commit -m ""`
## Reference:
[pre-commit-hooks](https://github.com/pre-commit/pre-commit-hooks)
Popular hook projects:
| Title | Description |
|---------------|---------------------|
| removestart | |
| isort | formats code |
| black | formats python code |
| flake8 | formats code |
| mypy | static type checker |
| dodgy | checks for commit passwords, secret keys, etc |
| pylint | formats code |
| bandit | checks for security issues |