{"id":51858575,"url":"https://github.com/stuttgart-things/machinery-catalog-locator","last_synced_at":"2026-07-24T03:31:06.652Z","repository":{"id":360165160,"uuid":"1248947753","full_name":"stuttgart-things/machinery-catalog-locator","owner":"stuttgart-things","description":"Go API to resolve Backstage Location entities and remove resources via Pull Request","archived":false,"fork":false,"pushed_at":"2026-07-17T20:02:06.000Z","size":116,"stargazers_count":0,"open_issues_count":14,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-17T22:05:55.171Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stuttgart-things.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-25T07:46:07.000Z","updated_at":"2026-06-01T10:14:15.000Z","dependencies_parsed_at":"2026-07-17T22:02:34.180Z","dependency_job_id":null,"html_url":"https://github.com/stuttgart-things/machinery-catalog-locator","commit_stats":null,"previous_names":["stuttgart-things/machinery-catalog-locator"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/stuttgart-things/machinery-catalog-locator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stuttgart-things%2Fmachinery-catalog-locator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stuttgart-things%2Fmachinery-catalog-locator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stuttgart-things%2Fmachinery-catalog-locator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stuttgart-things%2Fmachinery-catalog-locator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stuttgart-things","download_url":"https://codeload.github.com/stuttgart-things/machinery-catalog-locator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stuttgart-things%2Fmachinery-catalog-locator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35826032,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-24T02:00:07.870Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-24T03:31:05.101Z","updated_at":"2026-07-24T03:31:06.554Z","avatar_url":"https://github.com/stuttgart-things.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# machinery-catalog-locator\n\nThe **GitOps half** of the [machinery](https://github.com/stuttgart-things/machinery)\nstack: machinery shows the live status of resources in the cluster;\nmachinery-catalog-locator shows the same resources from the Git side\n(the catalog declarations that produced them) and removes them via\nPull Request when they should go away.\n\nTwo surfaces, sharing one implementation:\n\n- **gRPC API** \u0026mdash; for the umbrella tool that fronts both services\n- **HTMX dashboard** \u0026mdash; for humans\n\nBoth go through the same in-process `CatalogService` so behaviour and\nerrors are identical regardless of the caller.\n\nThe resolver is **platform-agnostic** — it depends only on the\n`catalog.FileReader` interface. The GitHub adapter (reads via the\nContents API, PRs via go-git + go-github) is isolated in\n`internal/github`; a GitLab adapter can be added without touching the\nresolver.\n\n## Layout\n\n```\ncatalogservice/         catalog_service.proto + generated Go (gRPC contract)\ncmd/server/             main.go — gRPC server + HTMX HTTP server, graceful shutdown\ncmd/local-resolve/      main.go — offline CLI that resolves testdata via LocalReader\ninternal/config/        environment-based configuration\ninternal/catalog/       platform-agnostic resolver\n  types.go              domain types (SourceRef, Entity, Node)\n  reader.go             FileReader interface + LocalReader (StripPathPrefix)\n  resolver.go           recursive Location resolution with cycle detection\n  locationedit.go       YAML editing (remove target/document, preserves comments)\n  crossplane.go         CrossplaneRefs + FindByCrossplaneSource — catalog ↔ claim\ninternal/github/        GitHub forge adapter\n  client.go             App- or token-based authentication\n  reader.go             FileReader implementation\n  pr.go                 PR engine (clone/commit/push via go-git)\ninternal/grpcserver/    CatalogServiceServer implementation\n  server.go             unary RPCs (ResolveTree, ListResources, RemoveTarget,\n                        DeleteResource, GetEntityManifest, ListEntitiesByCrossplaneSource)\n  watch.go              WatchTree — poll-based snapshot diff streamer\n  convert.go            catalog domain ↔ proto translation\ninternal/web/           HTMX frontend\n  web.go                HTTP handlers calling the gRPC server in-process\n  templates/*.html      embedded templates (index, tree, action_result)\ntestdata/catalog/       fixtures: catalog tree + linked Crossplane manifests\n  crossplane-claims/    real Claim/XR manifests referenced from catalog entities\n```\n\n## Concept\n\nBackstage walks from a few entry points through `Location` entities into\nmore files — a directed graph. machinery-catalog-locator rebuilds that\ngraph: each resolved node knows the file it came from (`Source`), the\nreferring location (`Parent`), and the exact target string\n(`ViaTarget`). Those three pieces are exactly what deletion needs:\nremoving a resource and its referring target in a **single PR** keeps\nBackstage from logging an error for the orphaned target.\n\n## gRPC API\n\nDefined in [`catalogservice/catalog_service.proto`](catalogservice/catalog_service.proto).\n\n| RPC               | Type          | Purpose                                                                 |\n|-------------------|---------------|-------------------------------------------------------------------------|\n| `ResolveTree`     | unary         | Full catalog graph rooted at a blob URL                                 |\n| `ListResources`   | unary         | Flat list of all non-Location entities (optional kind filter)           |\n| `RemoveTarget`    | unary         | Open a PR removing one target from a `location.yaml`                    |\n| `DeleteResource`  | unary         | Open a PR removing a resource and the referring parent target          |\n| `WatchTree`       | server-stream | Replay snapshot as `ADDED`, then stream poll diffs (`ADDED`/`MODIFIED`/`DELETED`) |\n| `GetEntityManifest` | unary       | Fetch the Crossplane Claim/XR manifests linked from a catalog entity    |\n| `ListEntitiesByCrossplaneSource` | unary | Reverse lookup: which catalog entities reference this Crossplane manifest URL |\n\nHealth: standard `grpc.health.v1.Health`.\nKeepalive: 2 min ping / 20 s timeout — long-lived `WatchTree` streams\nsurvive idle gateway timeouts.\n\nGenerate Go code after editing the proto:\n\n```bash\ntask proto\n```\n\n(Requires `protoc`, `protoc-gen-go`, `protoc-gen-go-grpc` on `PATH`.)\n\n## HTMX UI\n\nA small dark-themed dashboard at `HTTP_PORT` (default `:8080`):\n\n- Enter a root blob URL → resolves and shows the full tree\n- Each `Location` row exposes `✕ \u003ctarget\u003e` buttons that POST to\n  `RemoveTarget`\n- Each resource exposes `Delete` which POSTs to `DeleteResource`, and\n  `View claim` which GETs `GetEntityManifest` and swaps the linked\n  Crossplane YAML in below the node\n- The action panel above the tree shows the resulting PR URL (or the\n  error)\n\nEvery HTMX form calls the same `grpcserver.Server` methods that remote\ngRPC callers hit — there is no separate REST surface to maintain.\n\n## Setup\n\nCreate a GitHub App (recommended) with repository permissions *Contents:\nRead \u0026 write* and *Pull requests: Read \u0026 write*. Download the private\nkey as PEM. For local development a `GITHUB_TOKEN` is sufficient. The\nfull App walkthrough (org-owned, exact permissions, install, Vault/ESO\nwiring) is in [`GITHUB_APP.md`](GITHUB_APP.md).\n\n```bash\ncp .env.example .env      # fill in the values\ntask tidy                 # resolve dependencies (network required)\ntask test\ntask run                  # gRPC on :50051, HTMX on :8080\n```\n\n## Local development (no GitHub)\n\n`cmd/local-resolve` drives the resolver against a directory of fixtures\nthrough `catalog.LocalReader`, so you can iterate on YAML without\ncredentials or network access. The `testdata/catalog` tree (see its\n[README](testdata/catalog/README.md)) covers the multi-target root,\nmulti-document files, nested locations, cycles, and broken targets.\n\n```bash\ntask local                                         # gum picker over every *.yaml\ntask local FILE=edge-cases/singular-target.yaml    # skip the picker\ngo run ./cmd/local-resolve -root testdata/catalog -file all-locations.yaml\n```\n\nOutput is an indented tree showing each entity's kind/name/namespace,\nthe file it came from, the target it was reached by, and any\nannotations.\n\n### Linking entities to Crossplane claims / XRs\n\nCatalog entities reference their Crossplane source manifest in Git via\n`metadata.annotations`. The convention used in `testdata/catalog`:\n\n| Annotation | Purpose |\n|------------|---------|\n| `machinery.stuttgart-things.com/crossplane-kind`        | XR / Claim `kind`         |\n| `machinery.stuttgart-things.com/crossplane-api-version` | XR / Claim `apiVersion`   |\n| `machinery.stuttgart-things.com/crossplane-claim`       | Git blob URL of the Claim |\n| `machinery.stuttgart-things.com/crossplane-xr`          | Git blob URL of the XR    |\n\n```yaml\nmetadata:\n  name: claim-router\n  namespace: insurance\n  annotations:\n    machinery.stuttgart-things.com/crossplane-kind: ClaimRouter\n    machinery.stuttgart-things.com/crossplane-api-version: insurance.stuttgart-things.com/v1alpha1\n    machinery.stuttgart-things.com/crossplane-claim: https://github.com/stuttgart-things/catalog/blob/main/crossplane-claims/insurance/claim-router.yaml\n```\n\nAnnotations ride on the existing `EntityMetadata` field — they flow\nthrough the gRPC `Entity` message and the HTMX templates unchanged, so\ndownstream tools (the umbrella dashboard, `machinery`) can read them\nwithout any schema work.\n\nThe helper `catalog.CrossplaneRefs(entity)` returns the parsed\nreferences (kind=`\"claim\"|\"xr\"`, URL, `SourceRef`) so callers can fetch\nthe linked manifest with the same `FileReader` they already use for the\ncatalog. Because `ParseBlobURL` populates `SourceRef.Path` from the\nblob URL's path component, and `LocalReader` ignores Owner/Repo/Ref,\nthe **same annotation URL works against both GitHub and the local\nfixtures** — as long as the URL's path matches the fixture layout under\n`testdata/catalog/`. That's how `task local` round-trips from a catalog\nentity to its Crossplane manifest with no remote calls.\n\n`cmd/local-resolve` follows the annotations by default:\n\n```bash\ntask local                                  # prints summary of each linked manifest\ngo run ./cmd/local-resolve -claims=false    # skip the manifest fetch\ngo run ./cmd/local-resolve -claims-content  # also dump the full manifest YAML\n```\n\n### Resolving just one entity's claim\n\nTo go straight from a catalog entity name to its Crossplane manifest:\n\n```bash\ntask claim NAME=claim-router                # YAML to stdout\ntask claim NAME=claim-router@insurance      # namespace-qualified lookup\ntask claim NAME=payment-api-db \u003e /tmp/claim.yaml | kubectl apply -f -\n\n# Equivalent direct invocation:\ngo run ./cmd/local-resolve -name claim-router -claim-only\n```\n\n`-name` alone (without `-claim-only`) keeps the tree render but limits\nit to the matching entity, which is handy for inspecting one resource's\nannotations + linked manifest in one shot.\n\nProgrammatic equivalent for callers embedding the package:\n\n```go\nroots, _ := resolver.Resolve(ctx, root)\nnode, _ := catalog.Find(roots, \"Component\", \"claim-router\", \"insurance\")\nfor _, ref := range catalog.CrossplaneRefs(node.Entity) {\n    body, _ := reader.Read(ctx, ref.Source) // same reader, no new auth\n    // ref.Kind ∈ {\"claim\",\"xr\"}; body is the raw manifest YAML\n}\n```\n\n### Reverse lookup: claim → catalog entity\n\nWhen you point `local-resolve` (or `task local` via the gum picker) at\na Crossplane manifest, it also scans the catalog tree (default\n`all-locations.yaml`, override with `-catalog`) and reports the catalog\nentity referencing it:\n\n```\n$ go run ./cmd/local-resolve -file crossplane-claims/insurance/claim-router.yaml\n…\nresolved 1 non-Location resource(s)\n\nreferenced by (via catalog all-locations.yaml):\n  Component/claim-router @ insurance\n    catalog file: components/claim-router/catalog-info.yaml\n```\n\nThe underlying helper is `catalog.FindByCrossplaneSource(roots, path)`\n— takes a path, returns every catalog node whose Crossplane annotation\nresolves to it.\n\n### Running the full server locally (no GitHub)\n\n`cmd/server -local-root \u003cdir\u003e` boots the same gRPC + HTMX server the\nproduction binary runs, but with `catalog.LocalReader` instead of the\nGitHub adapter — no credentials needed. Read-only RPCs (`ResolveTree`,\n`ListResources`, `GetEntityManifest`, `ListEntitiesByCrossplaneSource`)\nwork normally; the PR-opening ones (`RemoveTarget`, `DeleteResource`)\nreturn `Unimplemented`.\n\n```bash\ntask run-local\n# gRPC on :50051, HTMX on :8080\n\n# Open http://localhost:8080 in a browser and paste any GitHub blob URL\n# pointing at testdata/catalog/all-locations.yaml — even the realistic\n# https://github.com/stuttgart-things/machinery-catalog-locator/blob/main/testdata/catalog/all-locations.yaml\n# works (see the prefix-stripping note below).\n```\n\nURLs whose path includes the local-root prefix\n(`testdata/catalog/all-locations.yaml`) would otherwise produce the\ndoubled path `testdata/catalog/testdata/catalog/all-locations.yaml`.\nThe server defaults `LocalReader.StripPathPrefix` to the local root so\nthe prefix is removed before joining — annotation URLs (which use the\nshorter `crossplane-claims/...` paths) are untouched and resolve\ndirectly.\n\n## Calling from another tool\n\nA third tool (e.g. a unified machinery + catalog-locator dashboard)\nimports the generated client and dials the gRPC server:\n\n```go\nimport (\n    \"google.golang.org/grpc\"\n    \"google.golang.org/grpc/credentials/insecure\"\n    \"github.com/stuttgart-things/machinery-catalog-locator/catalogservice\"\n)\n\nconn, _ := grpc.Dial(\"catalog-locator:50051\", grpc.WithTransportCredentials(insecure.NewCredentials()))\nclient := catalogservice.NewCatalogServiceClient(conn)\n\ntree, _ := client.ResolveTree(ctx, \u0026catalogservice.ResolveTreeRequest{\n    RootUrl: \"https://github.com/stuttgart-things/catalog/blob/main/all-locations.yaml\",\n})\n\n// \"Show this entity's claim\" — one RPC, server handles the GitHub fetch.\nmanifests, _ := client.GetEntityManifest(ctx, \u0026catalogservice.GetEntityManifestRequest{\n    RootUrl:   \"https://github.com/stuttgart-things/catalog/blob/main/all-locations.yaml\",\n    Kind:      \"Component\",\n    Name:      \"claim-router\",\n    Namespace: \"insurance\",\n})\nfor _, m := range manifests.Manifests {\n    // m.LinkKind ∈ {\"claim\",\"xr\"}, m.Source identifies the file, m.Body is raw YAML.\n}\n```\n\nFor ad-hoc probing use [`grpcurl`](https://github.com/fullstorydev/grpcurl).\nReflection is enabled, so no proto file is needed:\n\n```bash\ngrpcurl -plaintext localhost:50051 list\ngrpcurl -plaintext localhost:50051 list catalogservice.CatalogService\n\ngrpcurl -plaintext \\\n  -d '{\"root_url\":\"https://github.com/stuttgart-things/catalog/blob/main/all-locations.yaml\"}' \\\n  localhost:50051 catalogservice.CatalogService/ResolveTree\n\n# Fetch the Crossplane Claim linked from a single catalog entity:\ngrpcurl -plaintext \\\n  -d '{\n    \"root_url\":\"https://github.com/stuttgart-things/catalog/blob/main/all-locations.yaml\",\n    \"kind\":\"Component\",\"name\":\"claim-router\",\"namespace\":\"insurance\"\n  }' \\\n  localhost:50051 catalogservice.CatalogService/GetEntityManifest\n\n# Reverse: which catalog entity references this Crossplane manifest?\ngrpcurl -plaintext \\\n  -d '{\n    \"root_url\":\"https://github.com/stuttgart-things/catalog/blob/main/all-locations.yaml\",\n    \"manifest_url\":\"https://github.com/stuttgart-things/catalog/blob/main/crossplane-claims/insurance/claim-router.yaml\"\n  }' \\\n  localhost:50051 catalogservice.CatalogService/ListEntitiesByCrossplaneSource\n```\n\n## Deployment \u0026 PR previews\n\nThe service ships as a container image **built with [ko](https://ko.build)**\n(no Dockerfile — see [`.ko.yaml`](.ko.yaml), built from `./cmd/server`) and a\nset of Kubernetes manifests defined in **[`kcl/`](kcl/)** (Deployment, Service,\nConfigMap, optional Gateway API HTTP/GRPCRoute — no cluster-watch config or\nRBAC, since catalog-locator reads Git, not the cluster API). The KCL base is\npublished as a kustomize OCI artifact, so a deployer overlays the per-env image\ntag + namespace without rebuilding it.\n\nTwo ghcr packages are produced:\n\n| Package | Built by | Contents |\n|---------|----------|----------|\n| `ghcr.io/stuttgart-things/machinery-catalog-locator` | ko | runtime image |\n| `ghcr.io/stuttgart-things/machinery-catalog-locator-kustomize` | KCL → kustomize OCI | manifests base |\n\n### CI (`.github/workflows/`)\n\nThin wrappers over [`stuttgart-things/github-workflow-templates`](https://github.com/stuttgart-things/github-workflow-templates):\n\n| Workflow | Trigger | Does |\n|----------|---------|------|\n| `build-scan-image` | push / PR | ko build + scan + push the image |\n| `push-kustomize-pr` | PR | publish the kustomize OCI tagged `pr-\u003cn\u003e-\u003csha\u003e` |\n| `comment-preview-url` | PR labelled `preview` | post the preview URL on the PR |\n| `cleanup-pr-artifacts` | PR closed | delete the `pr-\u003cn\u003e-*` ghcr tags |\n\n### Per-PR preview environments\n\nLabel a PR **`preview`** and a full per-PR environment is deployed on\n`homerun2-dev`, reachable at\n`https://machinery-catalog-locator-pr-\u003cn\u003e.homerun2-dev.sthings-vsphere.labul.sva.de`\n(gRPC at the `-grpc` host). Removing the label or closing the PR tears it down.\n\nThe platform itself lives in\n[`stuttgart-things/argocd`](https://github.com/stuttgart-things/argocd) under\n`platforms/machinery-catalog-locator-pr-preview/` — an `ApplicationSet` that\nfans out over (clusters labelled `machinery-catalog-locator-pr-preview`) ×\n(PRs labelled `preview`), consuming the `pr-\u003cn\u003e-\u003csha\u003e` image + kustomize tags\nabove. The workload authenticates as the GitHub App; its private key is pulled\nfrom Vault into each preview namespace by the External Secrets Operator (see\n[`GITHUB_APP.md`](GITHUB_APP.md) for the App + Vault/ESO setup).\n\n## Notes \u0026 limits\n\n- If a file contains several `---`-separated entities, deletion extracts\n  only the affected document instead of removing the whole file.\n- Cross-repo: when a location points into another repo via an absolute\n  blob URL, the parent target is **not** removed automatically (a\n  separate PR is required). `DeleteResourceResponse.parent_touched`\n  reports this.\n- `WatchTree` is poll-based — there is no native push for Git contents.\n  The interval is clamped to `[5s, 1h]`; two consecutive resolve\n  failures end the stream so a transient GitHub outage doesn't tear\n  down every watcher silently.\n- The `LocalReader` in `internal/catalog/reader.go` enables offline\n  validation and is used by the tests.\n\n## Roadmap\n\n- GitLab adapter: implement `catalog.FileReader` plus a `PRService`\n  equivalent — the resolver stays untouched.\n- Optional mode that uses the Backstage Catalog API\n  (`/api/catalog/entities`, annotation `backstage.io/managed-by-location`)\n  instead of resolving on its own.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstuttgart-things%2Fmachinery-catalog-locator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstuttgart-things%2Fmachinery-catalog-locator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstuttgart-things%2Fmachinery-catalog-locator/lists"}