{"id":15906422,"url":"https://github.com/softchris/mlops-exercise","last_synced_at":"2025-03-21T11:32:33.704Z","repository":{"id":237553795,"uuid":"794724696","full_name":"softchris/mlops-exercise","owner":"softchris","description":"exericse for mlops","archived":false,"fork":false,"pushed_at":"2024-07-16T16:48:19.000Z","size":11,"stargazers_count":2,"open_issues_count":14,"forks_count":38,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-07T13:23:04.612Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/softchris.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-01T20:22:23.000Z","updated_at":"2024-07-16T16:48:22.000Z","dependencies_parsed_at":"2024-05-07T20:54:06.305Z","dependency_job_id":null,"html_url":"https://github.com/softchris/mlops-exercise","commit_stats":null,"previous_names":["softchris/mlops-exercise"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softchris%2Fmlops-exercise","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softchris%2Fmlops-exercise/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softchris%2Fmlops-exercise/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softchris%2Fmlops-exercise/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/softchris","download_url":"https://codeload.github.com/softchris/mlops-exercise/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221814961,"owners_count":16885086,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-06T13:23:13.200Z","updated_at":"2024-10-28T10:04:42.547Z","avatar_url":"https://github.com/softchris.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Exercise - understanding MLOps using GitHub Actions\n\n## Requirements\n\n- A GitHub account.\n- Basic knowledge of Python.\n- Python installed on your machine.\n\n## Problem Statement\n\nYou have a python script or Notebook and you want to ensure that any changes made to the script or notebook are automatically tested so that you can ensure that the changes do not break the code or worsens the performance of the model.\n\n## Theory\n\nWhat's a GitHub Action?\n\nGitHub Actions help you automate tasks within your software development life cycle. GitHub Actions are event-driven, meaning that you can run a series of commands after a specified event has occurred. For example, every time someone creates a pull request for a repository, you can automatically run a command that executes a software testing script.\n\nWhat's a workflow?\n\nA workflow is a configurable automated process made up of one or more jobs. You must create a YAML file to define your workflow configuration. The file must be stored in the `.github/workflows` directory of your repository.\n\nExample of a workflow file:\n\n```yaml\nname: Manually triggered workflow\non: \n  workflow_dispatch:\n  \njobs:\n  check-bats-version:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: actions/setup-node@v4\n        with:\n          node-version: '14'\n      - run: npm install -g bats\n      - run: bats -v\n```\n\nThe above workflow file is triggered manually (`workflow_dispatch`).\n\n- It defines a job called `check-bats-version` that runs on the latest version of Ubuntu.\n- It defines a series of steps that the job should take:\n  - Checks out the repository.\n  - Sets up Node.js version 14.\n  - Installs `bats` (Bash Automated Testing System) globally.\n  - Runs the `bats -v` command to check the version of `bats`.\n\n## Assignment\n\nIn this assignment, we'll set up a workflow that runs a test script whenever a push is made to the repository. The test script will test the performance of a model that predicts whether a credit card transaction is fraudulent or not.\n\n### Step 0 (optional, if you want to test the code locally)\n\nIn this steps, you'll try to run the code and the tests. It would therefore require you to have the following installed on your machine:\n\n- Python\n\n1. Clone/Fork this repository to your GitHub account.\n\n1. Start the project by creating a virtual environment and install dependencies:\n\n    ```bash\n    python -m venv venv\n    source venv/bin/activate\n    pip install -r requirements.txt\n    ```\n\n    For Windows users:\n\n    ```bash\n    python -m venv venv\n    venv\\Scripts\\activate\n    pip install -r requirements.txt\n    ```\n\n1. Generate a random dataset using the following code:\n\n    ```python\n    python util/generate.py\n    \n    ```\n\n    This command stores credit_card_records.csv in the data folder.\n\n1. Run the app using the following command:\n\n    ```bash\n    python app.py\n    ```\n\n    It should output the following:\n\n    ```bash\n    Model accuracy: \u003csome value between  and 1\u003e\n    ```\n\n1. Run the tests using the following command, using `pytest`:\n\n    ```bash\n    pytest\n    ```\n\n    It should output the following:\n\n    ```bash\n    .\n    ----------------------------------------------------------------------\n    Ran 2 tests in 0.000s\n\n    OK\n    ```\n\n### Step 1 - your first workflow\n\nThe object of this step is to get your first feel for what it's like to have GitHub Actions automate things for you.\n\n1. Create a workflow file that runs the test script whenever you choose to run it manually.\n\n    Copy the file `solution/workflows/manual.yml` to `.github/workflows/manual.yml`.\n\n    Run the workflow by going to your repo on GitHub, clicking on the `Actions` tab, and then clicking on the `Run workflow` button.\n\n    You should see the workflow running and bats version being printed in the logs.\n\n**What did you learn?**\n\nYou've learned how the GitHub Actions platform works and how to create a workflow that runs on demand. Next, we'll apply GitHub Actions to our specific use case.\n\n### Step 2 - run the tests on new pull request\n\nA common way of working is to create a new branch for a new feature or bug fix, make the changes, and then create a pull request to merge the changes into the main branch. This way of working gives your colleagues a chance to review your code before it's merged.\n\nHere's a great opportunity to let a GitHub Action run the tests for you whenever a new pull request is created. A failed test means you've introduced a bug or worsened the performance of the model.\n\n1. Copy the file `solution/workflows/pull_request.yml` to `.github/workflows/pull_request.yml`.\n\n    Let's examine why this works:\n\n    ```yml\n    on:\n      pull_request:\n        types: [opened, reopened]\n    ```\n\n    The above code listens for pull requests that are opened or reopened and runs the job `check_code` if so.\n\n    Inspecting the job, we see the following definition:\n\n    ```yml\n     runs-on: ubuntu-latest\n        steps:\n        - uses: actions/checkout@v4\n        - name: Set up Python\n          uses: actions/setup-python@v4\n          with:\n            python-version: '3.x'\n        - name: Install dependencies\n          run: |\n            python -m pip install --upgrade pip\n            pip install -r requirements.txt\n        - name: Test with pytest\n          run: |\n            pip install pytest pytest-cov\n            pytest tests.py --doctest-modules --junitxml=junit/test-results.xml --cov=com --cov-report=xml --cov-report=html\n    ```\n\n    Here's what the job does:\n\n    - It runs on the latest version of Ubuntu.\n    - It checks out the repository.\n    - It sets up Python 3.x.\n    - It installs dependencies.\n    - It runs the tests using `pytest`.\n\n**What did you learn?**\n\nIf the test fails, the pull request will indicate so, this is a great way to signal that this code should not be merged into the main branch as it would break the code or worsen the performance of the model.\n\n### Step 3 (optional) - compare the performance of the model\n\nSo far, our tests look like so:\n\n```python\ndef test_model_file_created():\n    app.main()  # Assuming the main function encapsulates the training logic\n    assert os.path.exists('models/model.pkl')\n\ndef test_model_score():\n    score = app.main()  # Assuming the main function returns the score\n    assert isinstance(score, float)\n    assert 0.0 \u003c= score \u003c= 1.0\n```\n\nThese tests are great for ensuring the model is created and that the score is within a certain range. However, they don't test the performance of the model versus earlier version of the model. So how can we introduce such a mechanism?\n\nOne way to do this is to:\n\n- Store the score of the model in a file. Such a file could look like so:\n\n    ```json\n    [{\n        \"version\": \"1.0\",\n        \"score\": 0.8\n    },\n    {\n        \"version\": \"1.1\",\n        \"score\": 0.04\n    }]\n    ```\n\n    Here we see that the model is improving for each version. In the context of testing the model to see if we've improved the model or worsened it is to train the model and compare the score with the previous score, in this case, compare the score of version 1.1 with the current version you're working on.\n\nHere's the changes we need to make to the tests:\n\n1. Create a file called `model_scores.json` in the root of the project.\n\n1. Add the following code to the test file, `tests.py`:\n\n    ```python\n    import json\n    import os\n\n    def test_model_score():\n        score = app.main()  # Assuming the main function returns the score\n        assert isinstance(score, float)\n        assert 0.0 \u003c= score \u003c= 1.0\n\n        # Load the model scores\n        with open('model_scores.json', 'r') as f:\n            model_scores = json.load(f)\n\n        # Get the latest model score\n        latest_score = model_scores[-1]['score']\n\n        # Compare the latest score with the current score\n        assert score \u003e= latest_score\n    ```\n\n    Now, we've added a test that compares the latest score with the current score. If the current score is worse than the latest score, the test will fail.\n\n    \u003e NOTE: if the tests pass, you will see that in the PR, add a new entry if so to the JSON file like so:\n\n    ```json\n    {\n        \"version\": \"1.2\",\n        \"score\": \u003cnew score\u003e\n    }\n    ```\n\n    \u003e TIP: it's a good idea if you're a developer to tag this commit with a version number, e.g., `v1.2` like so: `git tag v1.2` and then push the tag to GitHub like so: `git push origin v1.2`. This way, you can easily find where in your code the model was improved or worsened.\n\n**What did you learn?**\n\nYou've learned how to compare the performance of the model with the previous version of the model. This is a great way to ensure that the model is improving and not worsening.\n\n## Hand in\n\nSend a link to your repository on GitHub to your teacher.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftchris%2Fmlops-exercise","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoftchris%2Fmlops-exercise","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftchris%2Fmlops-exercise/lists"}