{"id":30591806,"url":"https://github.com/github/ai-assessment-comment-labeler","last_synced_at":"2025-08-29T17:40:41.540Z","repository":{"id":311933623,"uuid":"1037413102","full_name":"github/ai-assessment-comment-labeler","owner":"github","description":"AI-powered issue intake tool. Action for retrieving AI assessments for supplied text body","archived":false,"fork":false,"pushed_at":"2025-08-25T14:37:09.000Z","size":303,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-27T23:39:06.273Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/github.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":"SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-13T14:33:41.000Z","updated_at":"2025-08-27T18:30:11.000Z","dependencies_parsed_at":"2025-08-27T23:39:10.305Z","dependency_job_id":"817153c4-e626-41d1-bff2-d723b6821c36","html_url":"https://github.com/github/ai-assessment-comment-labeler","commit_stats":null,"previous_names":["github/ai-assessment-comment-labeler"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/github/ai-assessment-comment-labeler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/github%2Fai-assessment-comment-labeler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/github%2Fai-assessment-comment-labeler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/github%2Fai-assessment-comment-labeler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/github%2Fai-assessment-comment-labeler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/github","download_url":"https://codeload.github.com/github/ai-assessment-comment-labeler/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/github%2Fai-assessment-comment-labeler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272733365,"owners_count":24984261,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-29T02:00:10.610Z","response_time":87,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-29T17:40:38.490Z","updated_at":"2025-08-29T17:40:41.502Z","avatar_url":"https://github.com/github.png","language":"TypeScript","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"readme":"# AI Issue Assessment Commenter\n\n\u003e GitHub Action to automatically assess issues with an AI model, post (or optionally suppress) a structured review comment, and apply standardized AI derived labels based on configurable prompt files.\n\n## Table of Contents\n- [Overview](#overview)\n- [How It Works](#how-it-works)\n- [Features](#features)\n- [Prompt File Schema](#prompt-file-schema)\n- [Inputs](#inputs)\n- [Label → Prompt Mapping](#label--prompt-mapping)\n- [Regex Customization](#regex-customization)\n- [Suppressing the Comment](#suppressing-the-comment)\n- [Example Workflow Setup](#example-workflow-setup)\n- [Outputs / Labels Added](#outputs--labels-added)\n- [Required Permissions](#required-permissions)\n- [Troubleshooting](#troubleshooting)\n- [Development](#development)\n- [Contributing](#contributing)\n- [Security / Reporting Issues](#security--reporting-issues)\n- [FAQ](#faq)\n- [License](#license)\n\n## Overview\nThis action evaluates newly labeled GitHub Issues using an AI model available through **GitHub Models** (or a compatible endpoint you provide) and:\n1. Selects one or more prompt configuration files based on existing issue labels.\n2. Runs inference with the chosen model \u0026 system prompt.\n3. Extracts an \"assessment\" value from the AI response (via a configurable regex) and converts it into a standardized label of the form:\n   `ai:\u003cprompt-stem\u003e:\u003cassessment\u003e` (lowercased, spaces preserved unless you modify regex or post processing).\n4. Optionally posts the full AI response as a comment (unless a suppression marker is detected).\n5. Removes the trigger label so the process is idempotent and can be retriggered manually by re‑adding it.\n\n## How It Works\nHigh level flow:\n1. Issue receives a trigger label (e.g. `request ai review`).\n2. Action runs and gathers all issue labels.\n3. Each label is checked against your `labels_to_prompts_mapping` list.\n4. For each matched prompt file:\n   - System prompt + model + max tokens are resolved (overrides from workflow inputs if provided).\n   - Inference is executed with the issue body as user content.\n   - Response is scanned:\n     - Optional \"no comment\" directive → skip comment.\n     - Assessment header line → derive label.\n   - Summary written to the job summary.\n5. All derived labels are added; trigger label is removed.\n\n## Features\n- Multiple prompt files per issue (supports multifaceted assessments).\n- Per prompt inference (each prompt gets its own AI run \u0026 resulting label).\n- Override model / max tokens at workflow level or rely on prompt file.\n- Configurable assessment extraction via regex.\n- Configurable comment suppression via regex.\n- Clear action summary with raw AI output + derived assessment.\n- Works with any model accessible via the GitHub Models API endpoint you specify.\n\n## Prompt File Schema\nExample `.prompt.yml` file:\n```yaml\nmessages:\n  - role: system\n    content: \u003e+\n      You are a world-class product manager that will help decide whether a particular bug report is completely filled out and able to start being worked on by a team member.\n      1. Given a bug report analyze it for the following key elements: a clear description of the problem, steps to reproduce, expected versus actual behavior, and any relevant visual proof. \n      2. Rate each element provided in the report as `complete`, `incomplete`, or `unable to determine` except for Screenshots if included. Justify the rating by explaining what is missing or unclear in each element.\n      3. The title of the response should be based on the overall completeness rating of all the provided elements. For example: \"### AI Assessment: Ready for Review\" if complete, \"### AI Assessment: Missing Details\" if incomplete, or \"### AI Assessment: Unsure\" if unable to determine.\n      4. When determining the overall completeness rating do not include the Screenshots or relevant visual proof section. This section is more of a \"nice to have\" versus \"hard requirement\" and it should be ignored. \n  - role: user\n    content: '{{input}}'\nmodel: openai/gpt-4o-mini\nmodelParameters:\n  max_tokens: 100\ntestData: []\nevaluators: []\n```\nRequired elements:\n- `messages`: Must include at least one `system` and one `user` with `{{input}}` placeholder.\n- `model`: A model identifier in `{publisher}/{model_name}` format compatible with GitHub Models.\n- `modelParameters.max_tokens` (optional) used if workflow input `max_tokens` not provided.\n\n## Inputs\nVarious inputs are defined in [`action.yml`](action.yml):\n\n| Name | Description | Required | Default |\n| :-- | :-- | :-- | :-- |\n| `token` | Token for API calls (usually `${{ secrets.GITHUB_TOKEN }}`) | true | `github.token` |\n| `ai_review_label` | Label that triggers AI processing | true | |\n| `issue_number` | Issue number | true | |\n| `issue_body` | Issue body to feed into AI | true | |\n| `prompts_directory` | Directory containing `.prompt.yml` files | true | |\n| `labels_to_prompts_mapping` | Mapping string `label,prompt.yml\\|label2,prompt2.yml` | true | |\n| `model` | Override model (falls back to prompt file) | false | |\n| `endpoint` | Inference endpoint | false | `https://models.github.ai/inference` |\n| `max_tokens` | Override max tokens (else prompt file else 200) | false | 200 |\n| `repo_name` | Repository name (auto detected) | false | |\n| `owner` | Repository owner (auto detected) | false | |\n| `assessment_regex_pattern` | Pattern to capture assessment line | false | `^###.*[aA]ssessment:\\s*(.+)$` |\n| `assessment_regex_flags` | Flags for assessment regex | false | |\n| `no_comment_regex_pattern` | Pattern to detect comment suppression | false | |\n| `no_comment_regex_flags` | Flags for suppress pattern | false | |\n\n## Label → Prompt Mapping\nProvide a single string where pairs are separated by `|` and each pair is `label,prompt-file-name`. Example:\n```\nlabels_to_prompts_mapping: 'bug,bug-review.prompt.yml|support request,request-intake.prompt.yml|security,security-assessment.prompt.yml'\n```\nRules:\n- Matching is case sensitive to how GitHub returns labels.\n- A label may map to only one prompt file (first match used if duplicates are present).\n- Multiple labels can each trigger their prompt; all selected prompts are processed.\n\n## Regex Customization\nDefault assessment extraction looks for any markdown header starting with `###` and containing `Assessment:` (case insensitive if you supply `i` flag) and captures the remainder of the line:\n```\n^###.*[aA]ssessment:\\s*(.+)$\n```\nExample variations:\n- Want stricter title: `^### AI Assessment:\\s*(.+)$`\n- Want to allow other synonyms: `^###.*(Assessment|Alignment):\\s*(.+)$` (then handle capture group 2 in post processing—current implementation expects one capture group, so keep a single `(.+)`).\n\nIf your header is like:\n```\n## Result: Passed\n```\nYou could set:\n```\nassessment_regex_pattern: '^## Result:\\s*(.+)$'\n```\n\n## Suppressing the Comment\nAdd an instruction in the system prompt to emit a marker when you only want labeling. Example system instruction snippet:\n```\nIf the overall assessment is fully ready, append: \u003c!-- no-comment --\u003e\n```\nThen configure:\n```\nno_comment_regex_pattern: '\u003c!--.*no.*comment.*--\u003e'\nno_comment_regex_flags: 'i'\n```\nWhen the pattern is found, the comment step is skipped; label + summary still generated.\n\n## Example Workflow Setup\nBelow is an example workflow file. It triggers whenever a label is added, checks for the trigger label, processes, then removes it.\n```yaml\nname: AI Issue Assessment\non:\n  issues:\n    types: [labeled]\njobs:\n  ai-assessment:\n    if: github.event.label.name == 'request ai review'\n    runs-on: ubuntu-latest\n    permissions:\n      issues: write\n      models: read\n      contents: read\n    steps:\n      - name: Checkout\n        uses: actions/checkout@v4\n\n      - name: Setup Node.js\n        uses: actions/setup-node@v4\n\n      - name: Run AI assessment\n        id: ai-assessment\n        uses: github/ai-assessment-comment-labeler@main\n        with:\n          token: ${{ secrets.GITHUB_TOKEN }}\n          issue_number: ${{ github.event.issue.number }}\n          issue_body: ${{ github.event.issue.body }}\n          repo_name: ${{ github.event.repository.name }}\n          owner: ${{ github.repository_owner }}\n          ai_review_label: 'request ai review'\n          prompts_directory: './Prompts'\n          labels_to_prompts_mapping: 'bug,bug-review.prompt.yml|support request,request-intake.prompt.yml'\n```\nMultiple prompts example with custom overrides:\n```yaml\nwith:\n  model: openai/gpt-4o-mini\n  max_tokens: 300\n  labels_to_prompts_mapping: 'bug,bug-review.prompt.yml|performance,perf-triage.prompt.yml|design,ux-assessment.prompt.yml'\n```\n\n## Outputs / Labels Added\nFor each prompt file used (e.g. `bug-review.prompt.yml`), the assessment line text (after `Assessment:`) is:\n1. Lowercased\n2. Prefixed with `ai:\u003cprompt-stem\u003e:` where `\u003cprompt-stem\u003e` is the file name without extension and trailing `-prompt` parts preserved.\n\nExamples:\n- `### AI Assessment: Ready for Review` → `ai:bug-review:ready for review`\n- `### AI Assessment: Missing Details` → `ai:bug-review:missing details`\n- No header found → `ai:bug-review:unsure`\n\nThese labels let you filter, search, or automate additional workflows.\n\n## Required Permissions\nRecommended minimal permissions block:\n```yaml\npermissions:\n  issues: write   # create comment \u0026 add/remove labels\n  models: read    # access GitHub Models inference\n  contents: read  # read prompt files\n```\n\n## Troubleshooting\n| Symptom | Likely Cause | Fix |\n| --- | --- | --- |\n| Action exits early: \"Required inputs are not set\" | Missing mandatory input | Ensure all required `with:` fields are present |\n| No labels added | Assessment regex failed | Adjust `assessment_regex_pattern` / flags |\n| Comment missing | Suppression regex matched | Remove or modify `no_comment_regex_pattern` |\n| Fallback label `unsure` | No header matched regex | Update system prompt to ensure header form |\n| Model error | Unsupported or misspelled model | Verify model exists in GitHub Models catalog |\n| 404 prompt file | Wrong `prompts_directory` path | Ensure path relative to repo root |\n\nEnable debug logs by setting in workflow:\n```\nenv:\n  ACTIONS_STEP_DEBUG: true\n```\n(Requires enabling debug logging in repository settings.)\n\n## Development\nLocal development steps:\n```bash\n# Install dependencies\nbun install\n\n# Lint\nbun run lint\n\n# Auto fix + build dist\nbun run build\n\n# Run locally (requires env vars if outside GitHub Actions)\nGITHUB_TOKEN=ghp_xxx ISSUE_NUMBER=1 bun run src/index.ts\n```\nKey scripts (`package.json`): `lint`, `lintFix`, `build`.\nThe action bundles to `dist/index.js` (Node 20 runtime).\n\n### Testing\nBasic tests live under `src/__tests__`. Add additional parsing / utility tests as needed.\n\n### Releasing\n- Update version tag or reference a commit SHA in downstream workflows.\n- Optionally create a Git tag \u0026 release notes summarizing changes.\n\n## Contributing\nSee [`CONTRIBUTING.md`](CONTRIBUTING.md) \u0026 follow the code of conduct in [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md).\n\n## Security / Reporting Issues\nFor vulnerability disclosures follow [`SECURITY.md`](SECURITY.md). Please do not open public issues for sensitive reports.\n\n## FAQ\n**Q: Can I run multiple prompts in one execution?**  Yes, any label in the mapping that matches the issue produces a separate inference \u0026 label.\n\n**Q: How do I force a re-run?**  Re-add the trigger label.\n\n**Q: Can I use a completely different header phrase?**  Yes, adjust `assessment_regex_pattern` to capture the desired line; the first capture group is used as the assessment value.\n\n**Q: Can I trim / normalize spaces?**  Current implementation lowercases assessment as is. You can post process by adding a follow up workflow reacting to `labeled` events.\n\n**Q: Will it modify existing AI labels?**  It only adds new labels; it does not remove prior `ai:` labels. Clean up logic can be added in a future enhancement.\n\n## License\nSee [`LICENSE.txt`](LICENSE.txt).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgithub%2Fai-assessment-comment-labeler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgithub%2Fai-assessment-comment-labeler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgithub%2Fai-assessment-comment-labeler/lists"}