{"id":24123886,"url":"https://github.com/aslisabanci/algorithmia_github_issue_hook","last_synced_at":"2025-02-28T21:24:46.364Z","repository":{"id":65158971,"uuid":"279232001","full_name":"aslisabanci/algorithmia_github_issue_hook","owner":"aslisabanci","description":"Algorithmia Github Issue Hook","archived":false,"fork":false,"pushed_at":"2020-07-13T17:38:19.000Z","size":7,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-03-15T04:47:24.116Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aslisabanci.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-07-13T06:39:46.000Z","updated_at":"2022-09-30T18:03:10.000Z","dependencies_parsed_at":"2023-01-13T15:43:28.354Z","dependency_job_id":null,"html_url":"https://github.com/aslisabanci/algorithmia_github_issue_hook","commit_stats":{"total_commits":5,"total_committers":1,"mean_commits":5.0,"dds":0.0,"last_synced_commit":"8ad202cf878a300301242c111f8b19e4fce85016"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aslisabanci%2Falgorithmia_github_issue_hook","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aslisabanci%2Falgorithmia_github_issue_hook/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aslisabanci%2Falgorithmia_github_issue_hook/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aslisabanci%2Falgorithmia_github_issue_hook/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aslisabanci","download_url":"https://codeload.github.com/aslisabanci/algorithmia_github_issue_hook/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241243437,"owners_count":19933117,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-11T13:29:32.748Z","updated_at":"2025-02-28T21:24:46.342Z","avatar_url":"https://github.com/aslisabanci.png","language":"Python","readme":"# Algorithmia-Github Issue Hook\nInspired by [Algorithmia CI/CD](https://github.com/marketplace/actions/algorithmia-ci-cd), I created this Github action as a POC when playing with Github Actions and Algorithmia. \n\nIf you have a model that you evaluate periodically against your configured metric and a threshold, and you want to be notified of its performance drops; you can incorporate this action as a workflow. \n\nThrough this action, you can configure: \n- The path of the checkpoint weights that you want to restore your model with\n- Metric name \n- Threshold value to compare the test results\n\nBehind the scenes, your Github token, Github repo name and the latest commit SHA is also passed to your Algorithmia algorithm. Given these, your algorithm can use this hook as a feedback line and can report its problems back to you by creating a Github issue on your repo.\n\n\n# Workflow Input\n\n```\napi_key:\n    description: 'Algorithmia API key'\n    required: true\npath:\n    description: 'directory name for the local project stored in /github/workspace due to an \"actions/checkout\" command'\n    required: true\nmetric:\n    description: Evaluation metric of the model on the test set\n    required: true\nthreshold:\n    description: Threshold value for the test metric, under which the algorithm creates a Github issue\n    required: true\ncheckpoints:\n    description: The path of the checkpoints to restore your model from\n    required: true\n```\n\n\n# Example workflow\nThe workflow below shows a periodic run setup, scheduled to run at minute 0 past every 12th hour.\nCheckout my sample repository https://github.com/aslisabanci/algorithmia_issue_reporter to see this in action.\n\n```\nname: Algorithmia-Github Issue Hook\n\non:\n  schedule:\n    - cron: \"0 */12 * * *\n\njobs:\n  algorithmia-ci:\n    runs-on: ubuntu-latest\n\n    steps:\n      - name: Checkout\n        uses: actions/checkout@v2.0.0\n        with:\n          ref: ${{github.sha}}\n          path: algorithm\n      - name: Algorithmia-Github Issue Hook\n        uses: aslisabanci/algorithmia_github_issue_hook@v0.1-alpha\n        env:\n          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}\n          GITHUB_REPOSITORY: ${{ secrets.GITHUB_REPOSITORY }}\n        with:\n          api_key: ${{ secrets.api_key }}\n          path: algorithm\n          metric: accuracy\n          threshold: 0.85\n          checkpoints: ./checkpoints/fashion_mnist_checkpoint\n          \n```\n\n# Improvement ideas\nSince this is only a POC hack to play with and learn more about Github Actions and Algorithmia, a few things can be done differently for normal use cases. \n- Instead of using model checkpoints, you can restore your model directly from a saved model. Restoring a model from saved weights requires that you first create a model with the same architecture as the original model - so you may not prefer that. \n- You can update the workflow to make use of multiple performance metrics (and multiple thresholds), and not just a single one\n- You can apply this \"issue reporting\" logic to check your model's input data as well. If the distribution or the quality of your data changes, you can again get notified of this through your responsible algorithm's bug reports :]\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faslisabanci%2Falgorithmia_github_issue_hook","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faslisabanci%2Falgorithmia_github_issue_hook","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faslisabanci%2Falgorithmia_github_issue_hook/lists"}