{"id":20376667,"url":"https://github.com/andrebriggs/service-introspection","last_synced_at":"2026-02-07T19:01:23.176Z","repository":{"id":150973783,"uuid":"215383124","full_name":"andrebriggs/service-introspection","owner":"andrebriggs","description":null,"archived":false,"fork":false,"pushed_at":"2019-10-19T01:09:32.000Z","size":886,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-06T19:46:27.613Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andrebriggs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-15T19:44:53.000Z","updated_at":"2019-10-19T01:09:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"b9736efe-613e-4e7c-a28f-0fc3e06e59dd","html_url":"https://github.com/andrebriggs/service-introspection","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/andrebriggs/service-introspection","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrebriggs%2Fservice-introspection","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrebriggs%2Fservice-introspection/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrebriggs%2Fservice-introspection/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrebriggs%2Fservice-introspection/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andrebriggs","download_url":"https://codeload.github.com/andrebriggs/service-introspection/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrebriggs%2Fservice-introspection/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29204940,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-07T17:44:10.191Z","status":"ssl_error","status_checked_at":"2026-02-07T17:44:07.936Z","response_time":63,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T01:39:05.859Z","updated_at":"2026-02-07T19:01:23.157Z","avatar_url":"https://github.com/andrebriggs.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# AzDO Service Hooks As A Changelog To Enable Service Introspection In Bedrock\n\nThis repository covers an alternative approach to decorating Azure Devops Pipelines with metadata to achieve knowledge of the status of a GitOps flow using [Bedrock](aka.ms/bedrock) patterns.\n\n## What is Spektate?\n[Spektate](https://github.com/Microsoft/spektate) is a dashboard tool to allows a holistic view of container based applications as they flow from source code to container registries to _high level definition_ repositories, _manifest_ repositories and finally Kubernetes cluster deployment. \n\n![spektate.png](spektate.png)\n\n## What is SPK?\n\n[SPK](https://github.com/CatalystCode/spk) is a CLI tool that helps automate cloud infrastructure and service management. Moreover, SPK provides _service introspection_. Spektate and SPK overlap in the _service introspection_ area and will eventually merge.\n\n## Why do we want an alternative method to retrieve CI/CD metadata?\nSpektate requires a client must modify their existing `azure-pipelines.yaml` file in order to decorate telemetry information. This telemetry is recorded at the beginning and end of Azure pipeline runs. The recorded data is sent to indexed storage. The key point is that a user must add this telemetry explicity modify their production configuration. \n\nThe also must do this for each new pipeline they wish to add. That means if a client has 12 microservices across 12 pipelines each `azure-pipelines.yaml` file must be tediously decorated! 😬\n\n## How would we achieve less coupling and a better onboarding experience?\n\n![service-introspection.png](service-introspection.png)\n\nThe diagram above has can be read through the following steps:\n\n1. Developers make changes to various Git repositories\n2. Git repository changes trigger associated Azure Pipelines\n3. Azure DevOps is configured via service hooks to post JSON messages in an Azure Queue\n4. An Azure Function is configured to be queue triggered and listen for enqueued messages\n5. The Azure Functions processes the messages against the indexed storage and determines whether to add or update records.\n6. The SPK CLI or the Spektate Dashboard can consume information about where applications are in the CI/CD pipeline. \n\n**TLDR;** Essentially we use the service hooks feature of Azure DevOps as a changelog publisher. We subscribe and process to changes to pipeline state and metadata that are logged.\n\nThis leads to several benefits:\n- Better Spektate onboarding user experience 🍻\n- Less coupling to custom solutions. 🙅🏾‍♂️\n- All pipelines funnel into one place. ⛳️\n- Allow SPK service introspection to evolve separately from \"client code\" 🗿\n- Easier testing 🥳\n\n## How would we deploy this approach?\n\nAll cloud infra (Azure DevOps Project, Azure Storage pieces, Azure Function) can be deployed by SPK. The Azure DevOps project and the Azure Queue are decoupled from the rest of the solution. This means service introspection can easily be added to existing Azure DevOps projects.\n\nWe can use the Azure DevOps API to [programmatically](https://docs.microsoft.com/en-us/rest/api/azure/devops/hooks/subscriptions/create%20subscriptions%20query?view=azure-devops-rest-5.1) create service hooks that map to the Azure Storage:\n![service-hooks.png](service-hooks.png)\n\n## What type of management needs to be maintained for this approach?\n\nThe introduction of queues means one must deal with all the issues of queues (expiration, dead letter queues, etc). The processor (Az Functions) must be idempotent and must use defensive coding techniqiues when updating and inserting against the indexed storage layer. The indexed storage layer must be cleaned up after a while.\n\n## Caveats with this approach\nThe main issue in my investigation is that the existing Spektate is able to retrieve the latest Git commit id of the _manifest repo_. Technically the manifest repo isn't connected to a pipeline, but rather is an imported artifact of _HLD_ repository.\n\nSee Step 4 for [Spektate Onboarding](https://github.com/Microsoft/spektate#onboard-a-bedrock-project-to-use-spektate):\n\u003cpre\u003e\n- script: |\n    cd \"$HOME\"\n    cd hello-bedrock-manifest\n    \u003cb\u003elatest_commit=$(git rev-parse --short HEAD)\u003c/b\u003e\n    cd ../spektate/pipeline-scripts\n    source venv/bin/activate\n    echo \"python update_pipeline.py $(ACCOUNT_NAME) $(ACCOUNT_KEY) $(TABLE_NAME) $(PARTITION_KEY) p3 $(Build.BuildId) manifestCommitId $latest_commit\"\n    python update_pipeline.py $(ACCOUNT_NAME) $(ACCOUNT_KEY) $(TABLE_NAME) $(PARTITION_KEY) p3 $(Build.BuildId) manifestCommitId $latest_commit\n  displayName: Update commit id in database\n\u003c/pre\u003e\n\nWhat this means is that **latest_commit** above will not be in any service hook JSON payload since it is a custom point in time variable. The above snippet is appended to a client's _azure-pipelines.yaml_ file.\n\nThere are ways to get around this but they involve creating [artifacts](https://docs.microsoft.com/en-us/azure/devops/artifacts/overview?view=azure-devops\u0026viewFallbackFrom=vsts) or modifiying a customer's existing YAML file. We can also find ways to get the commit id of the manifest repository in a more passive way (outside of the Azure pipeline process). \n\nAnother caveat is that one can argue more complexity is introduced with the  moving pieces with Azure Functions and Queues. One must wieght the benefits of decoupling with more components. \n\n## Examples of Service Hook Event Payloads\nBelow are examples of the JSON payloads that Azure DevOps sends to the queue. We us the `jq` tool to parse the payloads and extracts relevant data that would be stored in indexed storage. One could imagine how an Azure Function would extract data from the JSON queue messages.\n\nThese examples use JSON files that are in this repository.\n\n### Build Started\n\n`$ cat example-json/started_build.json | jq '.eventType, .createdDate, .message.text, .resource.run.pipeline.id,.resource.run.name, .resource.run.id'`\n```\n\"ms.vss-pipelines.run-state-changed-event\"\n\"2019-10-02T19:54:00.3307136Z\"\n\"Run 20191002.2 in progress.\"\n7\n\"20191002.2\"\n479\n```\n\n### Build Ended\n\n`$ cat example-json/completed_build.json | jq '.eventType, .createdDate, .message.text, .resource.definition.id, .resource.buildNumber, .resource.id'`\n\n```\n\"build.complete\"\n\"2019-10-02T19:57:48.7584186Z\"\n\"Build 20191002.2 succeeded\"\n7\n\"20191002.2\"\n479\n```\n\n### Stage State Change\n\n`$ cat example-json/stage_state_change.json | jq '.eventType, .createdDate, .message.text, .resource.pipeline.id,.resource.stage.name, .resource.run.id'`\n\n```\n\"ms.vss-pipelines.stage-state-changed-event\"\n\"2019-10-02T20:10:39.1376851Z\"\n\"Run 20191002.3 stage build running.\"\n7\n\"build\"\n480\n```\n\n\n\n### Example of Get Build Id API Result JSON\n\n`$ cat example-json/get_build_id_api_result.json | jq '.status, .buildNumber, .result,.startTime,.finishTime'`\n\n```\n\"completed\"\n\"20191002.4\"\n\"succeeded\"\n\"2019-10-02T21:56:04.844615Z\"\n\"2019-10-02T22:00:04.8462149Z\"\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandrebriggs%2Fservice-introspection","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandrebriggs%2Fservice-introspection","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandrebriggs%2Fservice-introspection/lists"}