{"id":51186559,"url":"https://github.com/tailscale/policybottest","last_synced_at":"2026-06-27T11:00:29.607Z","repository":{"id":367640002,"uuid":"1281525483","full_name":"tailscale/policybottest","owner":"tailscale","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-26T20:54:12.000Z","size":21,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-26T22:18:37.991Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tailscale.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-26T16:38:55.000Z","updated_at":"2026-06-26T20:54:16.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/tailscale/policybottest","commit_stats":null,"previous_names":["tailscale/policybottest"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/tailscale/policybottest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tailscale%2Fpolicybottest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tailscale%2Fpolicybottest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tailscale%2Fpolicybottest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tailscale%2Fpolicybottest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tailscale","download_url":"https://codeload.github.com/tailscale/policybottest/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tailscale%2Fpolicybottest/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34850575,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-27T02:00:06.362Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-27T11:00:26.432Z","updated_at":"2026-06-27T11:00:29.572Z","avatar_url":"https://github.com/tailscale.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# policybottest\n\n`policybottest` is a small CLI for unit-testing\n[palantir/policy-bot](https://github.com/palantir/policy-bot)\n`.policy.yml` files. Upstream's `policy-bot validate` checks\nsyntax; this tool additionally checks that the policy actually\n*evaluates* the way the author expected for a set of hypothetical\npull requests.\n\nIt works by loading the policy with the same Go evaluator\npolicy-bot itself uses and running it against a fake\n`pull.Context` populated from a companion `.policy-test.yml`\nfile.\n\n## Usage\n\n```\npolicybottest -policy .policy.yml -tests .policy-test.yml\n```\n\nExit code is `0` if every test's assertions pass and `1`\notherwise. `2` is reserved for setup errors (bad YAML, missing\nfiles, an unparseable policy).\n\n## Test file schema\n\nThe test file is YAML, parsed in strict mode (unknown fields\nfail the load). The canonical Go definition is in\n[`testconfig.go`](./testconfig.go); this section is the\nhuman-readable reference.\n\n### Top-level keys\n\n| Key             | Type                              | Required | Default                                                          |\n| --------------- | --------------------------------- | -------- | ---------------------------------------------------------------- |\n| `repository`    | object (`{owner, name}`)          | no       | `{owner: tailscale, name: test}`                                 |\n| `teams`         | map of `org/team` → list of users | no       | `{}`                                                             |\n| `organizations` | map of `org` → list of users      | no       | `{}`                                                             |\n| `tests`         | list of test cases                | **yes**  | —                                                                |\n\n`teams` and `organizations` populate the membership lookups the\npolicy evaluator uses for `requires.teams` /\n`requires.organizations` / `has_author_in.teams` /\n`has_author_in.organizations`. Any team or org the policy\nreferences must appear here, otherwise membership checks always\nreturn false. Team keys must include the org prefix\n(e.g. `myorg/admins`, not just `admins`).\n\n### `tests[]` — a single test case\n\n| Key            | Type                | Required | Notes                                       |\n| -------------- | ------------------- | -------- | ------------------------------------------- |\n| `name`         | string              | **yes**  | Must be unique across the file.             |\n| `pull_request` | object (see below)  | yes      | The hypothetical PR.                        |\n| `expect`       | object (see below)  | yes      | Assertions to check after evaluation.       |\n\n### `tests[].pull_request`\n\n| Key             | Type            | Default          | Notes                                                                 |\n| --------------- | --------------- | ---------------- | --------------------------------------------------------------------- |\n| `author`        | string          | `\"\"`             | GitHub login of the PR author.                                        |\n| `title`         | string          | `\"\"`             | Used by `title:` predicates.                                          |\n| `body`          | string          | `\"\"`             | Used by predicates that read the PR body.                             |\n| `base_branch`   | string          | `\"main\"`         | Used by `targets_branch:`.                                            |\n| `head_branch`   | string          | `\"feature\"`      | Used by `from_branch:`.                                               |\n| `labels`        | list of string  | `[]`             | Used by `has_labels:`.                                                |\n| `draft`         | bool            | `false`          |                                                                       |\n| `changed_files` | list of `File`  | `[]`             | See below.                                                            |\n| `comments`      | list of object  | `[]`             | Each: `{user, body}`.                                                 |\n| `reviews`       | list of object  | `[]`             | Each: `{user, state, body?}`.                                         |\n\n### `File` shorthand\n\nA changed file may be written either as a bare string (the path)\nor as an explicit object. Both forms are equivalent for path-only\nrules:\n\n```yaml\nchanged_files:\n  - tailcfg/tailcfg.go            # status defaults to \"modified\"\n  - filename: cmd/new/main.go\n    status: added                 # \"modified\" | \"added\" | \"deleted\"\n    additions: 42\n    deletions: 0\n```\n\n### `tests[].expect`\n\nEither assertion may be omitted; an empty `expect:` block is\nlegal and asserts nothing about per-rule behavior, but see the\nnote on top-level `skipped` below.\n\n| Key      | Type                          | Notes                                                                 |\n| -------- | ----------------------------- | --------------------------------------------------------------------- |\n| `status` | string (status enum)          | The overall status of `policy.approval` evaluation.                   |\n| `rules`  | map of `rule name` → status   | Per-rule assertions, by `approval_rule.name` from the policy file.    |\n\nWhen `rules` is set, the runner walks the policy-bot evaluation\nresult tree and looks up each rule by its `Name`. A name that is\nnot found in the tree fails the test and prints the list of\nnames that *were* found, so typos are caught immediately.\n\n#### Top-level `skipped` is treated as failure\n\nThe policy-bot evaluator's `Result.Status` has four values:\n`skipped`, `pending`, `approved`, `disapproved`. The server-side\nstatus-posting layer (`eval_context.go` in upstream) translates\na top-level `skipped` result into a **failing** GitHub status\ncheck with the description \"All rules were skipped. At least one\nrule must match.\" — this is something policy-bot fails the PR\non, not a no-op.\n\nTo make tests catch this rather than rubber-stamping it, the\nrunner reports a top-level `skipped` result as a failure whenever\n`expect.status` is not set. If you genuinely want to assert that\nthe top-level skipped state is intentional, opt in explicitly:\n\n```yaml\nexpect:\n  status: skipped\n```\n\nPer-rule (child) `skipped` is never auto-failed, since \"this rule\ndidn't apply\" is a normal evaluation outcome inside an `or` block\nor a path-gated rule.\n\n### Enums\n\nStatus values (used in `expect.status` and `expect.rules` values),\nmirroring policy-bot's `common.EvaluationStatus`:\n\n| Value          | Meaning                                                                  |\n| -------------- | ------------------------------------------------------------------------ |\n| `skipped`      | The rule's `if:` predicates didn't match; it's not part of the decision. |\n| `pending`      | The rule applies but its requirements aren't yet satisfied.              |\n| `approved`     | The rule applies and is satisfied.                                       |\n| `disapproved`  | The rule applies and the PR is actively blocked.                         |\n\nFile status values (used in `changed_files[].status`):\n\n| Value      | Aliases    |\n| ---------- | ---------- |\n| `modified` | (default)  |\n| `added`    |            |\n| `deleted`  | `removed`  |\n\nReview state values (used in `reviews[].state`), all\ncase-insensitive:\n\n| Value               | Aliases                            |\n| ------------------- | ---------------------------------- |\n| `approved`          | `approve`                          |\n| `changes_requested` | `changes-requested`, `rejected`    |\n| `commented`         | `comment` (also the implicit default) |\n| `dismissed`         |                                    |\n| `pending`           |                                    |\n\n## Worked example\n\n```yaml\nteams:\n  myorg/admins: [alice, bob]\n  myorg/devs:   [alice, bob, carol]\n\ntests:\n  - name: an admin's review approves an admin-only path\n    pull_request:\n      author: carol\n      changed_files:\n        - admin/secrets.go\n      reviews:\n        - user: alice\n          state: approved\n    expect:\n      status: approved\n      rules:\n        \"admin paths require an admin review\": approved\n\n  - name: the author cannot self-approve\n    pull_request:\n      author: alice\n      changed_files:\n        - admin/secrets.go\n      reviews:\n        - user: alice\n          state: approved\n    expect:\n      status: pending\n      rules:\n        \"admin paths require an admin review\": pending\n\n  - name: changes outside admin/ skip the admin rule\n    pull_request:\n      author: carol\n      changed_files:\n        - README.md\n    expect:\n      status: skipped\n      rules:\n        \"admin paths require an admin review\": skipped\n```\n\nSee [`testdata/`](./testdata/) for a real worked example: the\ntailscale.com policy with ten cases covering path-scoping,\nteam-gated review, and a comment-based override escape hatch.\n\n## Limitations\n\n* No support yet for `has_successful_status` /\n  `has_workflow_result` predicates (no fake CI statuses are\n  wired up); add fields to `PullRequest` in `testconfig.go` if\n  you need them.\n* The fake context does not model `RequestedReviewers`,\n  `RepositoryCollaborators`, or `Permissions`; rules that gate\n  on those will see empty sets.\n* Test cases are evaluated independently; there is no notion of\n  time-ordering between events on a single PR.\n\n## Development\n\n```\ngo test ./...\ngo run . -policy testdata/tailscale.com.policy.yml \\\n         -tests  testdata/tailscale.com.policy-test.yml\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftailscale%2Fpolicybottest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftailscale%2Fpolicybottest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftailscale%2Fpolicybottest/lists"}