{"id":20903314,"url":"https://github.com/akyriako/cloudtrace-exporter","last_synced_at":"2025-12-25T15:45:17.248Z","repository":{"id":224005940,"uuid":"762141505","full_name":"akyriako/cloudtrace-exporter","owner":"akyriako","description":"A Knative Eventing custom source to stream events from Open Telekom Cloud CloudTrace service and load them as graph in a Neo4j database.","archived":false,"fork":false,"pushed_at":"2024-05-03T10:20:43.000Z","size":809,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-19T13:49:15.814Z","etag":null,"topics":["cloudtrace","knative","knative-eventing","neo4j","opentelekomcloud"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/akyriako.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-23T07:01:00.000Z","updated_at":"2024-06-03T15:16:21.000Z","dependencies_parsed_at":"2024-02-23T08:27:14.443Z","dependency_job_id":"023e3d23-b651-4043-b9a9-a4af2c3a024c","html_url":"https://github.com/akyriako/cloudtrace-exporter","commit_stats":null,"previous_names":["akyriako/cloudtrace-exporter"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akyriako%2Fcloudtrace-exporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akyriako%2Fcloudtrace-exporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akyriako%2Fcloudtrace-exporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akyriako%2Fcloudtrace-exporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/akyriako","download_url":"https://codeload.github.com/akyriako/cloudtrace-exporter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243293795,"owners_count":20268142,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudtrace","knative","knative-eventing","neo4j","opentelekomcloud"],"created_at":"2024-11-18T13:12:28.497Z","updated_at":"2025-12-25T15:45:17.218Z","avatar_url":"https://github.com/akyriako.png","language":"Go","readme":"# cloudtrace-exporter\n\nA custom exporter that collects traces from [Open Telekom Cloud](https://www.open-telekom-cloud.com/en) CloudTrace service and loads them as graph in a\n[Neo4j](https://neo4j.com/) database.\n\n[Cloud Trace Service (CTS)](https://www.open-telekom-cloud.com/en/products-services/core-services/cloud-trace) is an \neffective monitoring tool that allows users to analyze their cloud resources using traces. A tracker is automatically \ngenerated when the service is started and monitors access to all the respective user’s cloud resources using the generated traces. \nThe monitoring logs can be saved long-term and cost-effectively in the [Object Storage Service (OBS)](https://www.open-telekom-cloud.com/en/products-services/core-services/object-storage-service). \nThe CTS can also be used in conjunction with [Simple Message Notification (SMN)](https://www.open-telekom-cloud.com/en/products-services/core-services/simple-message-notification), \nallowing the user to receive a message when certain events occur.\n\nThis custom exporter is taking a different route. It's utilizing [Knative Eventing](https://knative.dev/docs/eventing/) \nto create a custom source (**cts_exporter**) that collects traces from CTS and forwards them, as [Cloud Events](https://cloudevents.io/) \nto an agnostic _sink_, defined by an environment variable called `K_SINK`, as is required by Knative Eventing specifications \nfor interconnecting microservices. In addition to cts_exporter, a custom sink (**neo4j_sink**) that listens for those \nCloud Events is provided, which loads these events in a Neo4j database as graphs. You could positively bind the cts_exporter\nto **any** other sink that conforms to Knative specifications. You can find an example in the repo that uses\n_gcr.io/knative-releases/knative.dev/eventing/cmd/event_display_ as a target sink. That is a demo Knative Eventing Service that\nsimply logs the events in the `os.Stdout`. \n\n![graph.png](assets%2Fimg%2Fgraph.png)\n\n[Neo4j](https://neo4j.com/) is a highly acclaimed graph database management system developed by Neo4j, Inc. Unlike \ntraditional relational databases that store data in tables, Neo4j is designed around the concept of storing and managing \ndata as nodes and relationships. This structure is particularly well-suited for handling complex and interconnected data, \nmaking it easier to model, store, and query relationships directly.\n\nGraph databases like Neo4j are based on graph theory and use graph structures with nodes, edges, and properties to \nrepresent and store data. In this context:\n\n- **Nodes** represent entities (such as subjects, actions, resources, tenants \u0026 regions in the context of CloudTrace domain).\n- **Relationships** provide directed, named connections between nodes. These relationships can also have properties that \nprovide more context about the connection (such as who performed an action, on which resource this action was performed, \nin which tenant is this resource is member, in which region is this tenant located)\n- **Properties** are key-value pairs attached to nodes and relationships, allowing for the storage of additional \ninformation about those elements (such as unique identifiers for nodes, tenant and domain identifiers, subjects name etc)\n\nThe graph generated for every CloudTrace record can be summarized by the following domain object:\n\n![graph-mock.png](assets%2Fimg%2Fgraph-mock.png)\n\nAn **ACTION** (login, logout, start an ECS instance etc) is _PERFORMED_BY_ a **SUBJECT** (user, agent etc) and is _APPLIED_ON_ \na **RESOURCE** (ECS instance, CCE cluster etc) resulting _WITH_STATUS_ either **NORMAL**, **WARNING** or **INCIDENT** depending on \nthe outcome of this **ACTION**. The **RESOURCE** is _MEMBER_OF_ a **TENANT** which is _LOCATED_AT_ a specific **REGION**. \nThe central element of this domain model is the **ACTION**.\n\nTerms in **BOLD** signify a **Node** and those in _ITALICS_ signify a **Relationship**.\n\nNeo4j is widely used in various applications that require efficient analysis and querying of complex networks of data. \nExamples include social networks, recommendation engines, fraud detection, network and IT operations, and more. It \noffers a powerful query language called [Cypher](https://neo4j.com/product/cypher-graph-query-language/), specifically \ndesigned for working with graph data, enabling users to intuitively and efficiently retrieve and manipulate data within a graph structure.\n\n## Usage\n\nUse the `clouds.tpl` as a template, and fill in a `clouds.yaml` that contains all the relevant auth information for your connecting\nto your Open Telekom Cloud Tenant. **cts_exporter** requires the presence of this file.\n\n```yaml\n\nclouds:\n  otc:\n    profile: otc\n    auth:\n      username: '\u003cUSER_NAME\u003e'\n      password: '\u003cPASSWORD\u003e'\n      ak: '\u003cACCESS_KEY\u003e'\n      sk: '\u003cSECRET_KEY\u003e'\n      project_name: 'eu-de_\u003cPROJECT_NAME\u003e\n      user_domain_name: 'OTC0000000000xxxxxxxxxx'\n      auth_url: 'https://iam.eu-de.otc.t-systems.com:443/v3'\n    interface: 'public'\n    identity_api_version: 3\n\n```\n\n\u003e [!CAUTION]\n\u003e **clouds.yaml** is already added to **.gitignore**, so there is no danger leaking its sensitive contents in public!\n\nAdditionally, you need to set the following environment variables for **cts_exporter**: \n\n- `OS_CLOUD` the cloud profile you want to choose from your **cloud.yaml** file\n- `OS_DEBUG` whether you want to swap to debug mode, defaults to `false`\n- `CTS_TRACKER` the CTS tracker you want to hook on, default to `system`\n- `CTS_FROM` an integer value in minutes, that signifies how long in the past to look for traces and the interval between two consecutive queries, defaults to `5`\n- `CTS_X_PNP` whether you want to push the collected traces to a sink, defaults to `true` \n\n\u003e [!IMPORTANT]\n\u003e There are two additional environment variables, that need to be addressed separately, and those are: \n\u003e\n\u003e - `K_SINK` the URL of the resolved sink\n\u003e - `K_CE_OVERRIDES` a JSON object that specifies overrides to the outbound event\n\u003e \n\u003e If you choose to deploy **cts_exporter** as a plain Kubernetes `Deployment`, for test reasons, using `deploy/manifests/cloudtrace-exporter-deployment.yaml` \n\u003e you need to explicitly set the value of `K_SINK` yourself.\n\u003e This will not unfold the whole functionality, because the resource will be deployed outside of the realm of responsibility \n\u003e of Knative reconcilers. As mentioned again, this is **exclusively** for quick test purposes.\n\u003e \n\u003e If you deploy **cts_exporter** as a `ContainerSource` or `SinkBinding`, Knative will take care of the rest and inject in \n\u003e your container an environment variable named `K_SINK` by itself.\n\nFor **neo4j_sink** you need to set the following environment variables:\n\n- `NEO4J_URI` the Neo4j connection uri for your instance, defaults to `neo4j://localhost:7687`\n- `NEO4J_USER` the username to use for authentication\n- `NEO4J_PASSWORD` the password to use for authentication\n\n\u003e [!NOTE]\n\u003e At the moment, the client wrapper around Neo4j driver, built in **neo4j_sink**, is supporting only Basic Auth.\n\n## Deployment\n\nThe project is coming with a `Makefile` that takes care of everything for you, from building (using [ko](https://ko.build/);\nneither a `Dockerfile` is needed nor docker registries to push the generated container images) to deployment on a \nKubernetes cluster. Only thing you need, if you not working inside the provided Dev Container, is to have a Kubernetes \ncluster in place, already employed with **Knative Serving \u0026 Eventing** artifacts and a Neo4j database instance, whose \nendpoints are reachable from your Kubernetes pods.\n\nBefore deploying anything, you need to define:\n\n- the values of **cts_exporter** environment variables in `deploy/manifests/cloudtrace-exporter-configmap.yaml` e.g:\n\n    ```yaml\n    apiVersion: v1\n    kind: ConfigMap\n    metadata:\n      name: cloudtrace-exporter-config\n      namespace: default\n    data:\n      OS_CLOUD: \"otc\"\n      OS_DEBUG: \"false\"\n      CTS_X_PNP: \"true\"\n      CTS_FROM: \"1\"\n    ```\n\n- the values of **neo4j_sink** environment variables in `deploy/manifests/cloudtrace-neo4j-sink-secrets.yaml` e.g:\n\n    ```yaml\n    apiVersion: v1\n    kind: ConfigMap\n    metadata:\n      name: cloudtrace-exporter-config\n      namespace: default\n    data:\n      OS_CLOUD: \"otc\"\n      OS_DEBUG: \"false\"\n      CTS_X_PNP: \"true\"\n      CTS_FROM: \"1\"\n    ```\n\n### Local\n\n#### Install\n\n##### Configuration\n\nYou can (re)deploy the configuration (`ConfigMaps` and `Secrets`) of all workloads using one target:\n\n```shell\nmake install-configuration\n```\n\n##### Binaries\n\n\u003e [!NOTE]\n\u003e The targets below will rebuild all the container images from code, redeploy configuration and then deploy\n\u003e our custom exporter and sink.\n\nAs mentioned earlier, you are given two options as how to deploy **cts_exporter** as a Knative workload; either as a \n`ContainerSource`:\n\n```shell\nmake install-containersource\n```\n\nor as a `SinkBinding`:\n\n```shell\nmake install-sinkbinding\n```\n\n\u003e [!IMPORTANT]\n\u003e **neo4j_sink** will be deployed as a `Knative Service`, and its endpoint will serve as the value of `K_SINK` environment\n\u003e variable that **cts_explorer** will push the collected CloudEvents to.\n\n\n#### Uninstall\n\n```shell\nmake uninstall\n```\n\n## Development\n\nDevelopment comes as well with \"batteries included\". You can either go ahead and start debugging straight on your local\nmachine, or take advantage of the `.devcontainer.json` file that can be found in the repo, that instructs any IDE that \nsupports Dev Containers, to set up an isolated containerized environment for you with a Neo4j database included.\n\n### Local\n\nWorking on your plain local host machine (no remote containers), requires the following:\n\n- Assign values to the environment variables for both binaries, as mentioned earlier in this document\n- Provide a Neo4j database instance. You can choose among a simple container, a Kubernetes workload or even the new [Neo4j Desktop](https://neo4j.com/docs/desktop-manual/current/)\n- Have a Kubernetes cluster, already set up for Knative Serving \u0026 Eventing.\n\n### Dev Container\n\n#### Extensions \u0026 Features\n\nA Dev Container will be created, with all the necessary prerequisites to get you started developing immediately. A\ncontainer, based on `mcr.microsoft.com/devcontainers/base:jammy` will be spawned with the following features pre-installed:\n\n- Resource Monitor\n- Git, Git Graph\n- Docker in Docker\n- Kubectl, Helm, Helmfile, K9s, KinD, Dive\n- [Bridge to Kubernetes](https://learn.microsoft.com/en-us/visualstudio/bridge/overview-bridge-to-kubernetes) Visual Studio Code Extension\n- Latest version of Golang\n\nA `postCreateCommand` (**.devcontainer/setup.sh**) will provision:\n\n- A containerized **Kubernetes cluster** with 1 control and 3 worker nodes **and** a private registry, using KinD (cluster manifest is in **.devcontainer/cluster.yaml**) \n- A standalone **Neo4j cluster** (you can change that and get a HA cluster by increasing the value of `minimumClusterSize` in **.devcontainer/overrides.yaml**)\n- the necessary resources for the **Knative Serving \u0026 Eventing infrastructure**\n\n#### Ports \u0026 Services\n\nYou can access Neo4j either internally within the cluster or externally from your container or from your local host.\n\n##### Internally\n\nIf you want to access Neo4j internally from another pod of the cluster, you just need to consume the Kubernetes `Service` \nendpoint which in our setup would be `neo4j://n4j-cluster.n4j-lb-neo4j.service.cluster.local`\n\n##### Externally\n\nYou need, as long as you are working with Visual Studio Code,  \nto forward the 3 ports (`7473`, `7474` and `7687`) exposed from the **n4j-cluster-lb-neo4j** Service, so your Neo4j \ndatabase is accessible from your Dev Container environment. \n\n\u003e [!TIP]\n\u003e You can just port-forward the Kubernetes Service ports straight from K9s, in an integrated Visual Studio Code terminal,\n\u003e and Visual Studio Code will pick up automatically those ports and forward them to your local machine.\n\n![devcontainer.png](assets%2Fimg%2Fdevcontainer.png)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakyriako%2Fcloudtrace-exporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fakyriako%2Fcloudtrace-exporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakyriako%2Fcloudtrace-exporter/lists"}