{"id":30280287,"url":"https://github.com/konnektr-io/http-query-operator","last_synced_at":"2025-08-16T15:03:21.213Z","repository":{"id":309685741,"uuid":"1037177001","full_name":"konnektr-io/http-query-operator","owner":"konnektr-io","description":"A Kubernetes Database Operator that allows to generate resources based on an http request.","archived":false,"fork":false,"pushed_at":"2025-08-13T08:55:57.000Z","size":26420,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-13T09:21:23.893Z","etag":null,"topics":["controller","http","kubernetes","operator"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/konnektr-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-13T07:17:53.000Z","updated_at":"2025-08-13T08:56:00.000Z","dependencies_parsed_at":"2025-08-13T09:32:40.119Z","dependency_job_id":null,"html_url":"https://github.com/konnektr-io/http-query-operator","commit_stats":null,"previous_names":["konnektr-io/http-query-operator"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/konnektr-io/http-query-operator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/konnektr-io%2Fhttp-query-operator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/konnektr-io%2Fhttp-query-operator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/konnektr-io%2Fhttp-query-operator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/konnektr-io%2Fhttp-query-operator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/konnektr-io","download_url":"https://codeload.github.com/konnektr-io/http-query-operator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/konnektr-io%2Fhttp-query-operator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270728047,"owners_count":24635132,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-16T02:00:11.002Z","response_time":91,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["controller","http","kubernetes","operator"],"created_at":"2025-08-16T15:01:39.955Z","updated_at":"2025-08-16T15:03:21.189Z","avatar_url":"https://github.com/konnektr-io.png","language":"Go","readme":"# HTTP Query Operator\n\n[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n[![Build Status](https://github.com/konnektr-io/http-query-operator/actions/workflows/build-push.yaml/badge.svg)](https://github.com/konnektr-io/http-query-operator/actions/workflows/build-push.yaml)\n\n## Overview\n\nThe HTTP Query Operator is a Kubernetes operator designed to manage Kubernetes resources based on the results of HTTP API calls. It periodically polls specified HTTP endpoints, processes the JSON responses, and uses Go templates to render Kubernetes manifests for each item returned by the API.\n\nThe operator handles the reconciliation loop, ensuring that the resources in the cluster match the desired state defined by the HTTP API responses and the template. This allows for dynamic configuration and resource management driven directly by external APIs and services.\n\n## Features\n\n* **CRD Driven:** Configuration is managed via an `HTTPQueryResource` Custom Resource Definition.\n* **HTTP API Polling:** Periodically queries HTTP/HTTPS endpoints at a configurable interval.\n* **Multiple Authentication:** Supports Basic Auth, Bearer Token, API Key, and OAuth2 Client Credentials authentication.\n* **JSONPath Support:** Extract specific data from JSON responses using JSONPath expressions.\n* **Go Templating:** Define Kubernetes resource manifests using Go templates with Sprig functions.\n* **Item-to-Resource Mapping:** Each item in the API response typically generates one Kubernetes resource.\n* **Status Updates:** Optionally send HTTP callbacks with resource status after reconciliation.\n* **Secret Management:** Securely fetches authentication credentials from Kubernetes Secrets.\n* **Reconciliation:** Creates, updates, and (optionally) deletes Kubernetes resources to match the API results.\n* **Pruning:** Automatically cleans up resources previously created by the operator if they no longer correspond to an item in the API response (configurable).\n* **Ownership:** Sets Owner References on created resources for automatic garbage collection by Kubernetes when the `HTTPQueryResource` is deleted.\n* **Labeling:** Labels created resources for easy identification and potential pruning.\n\n## Prerequisites\n\n* **kubectl:** For interacting with the Kubernetes cluster.\n* **Helm:** For installing the operator.\n* **Kubernetes Cluster:** Access to a Kubernetes cluster (e.g., kind, Minikube, EKS, GKE, AKS).\n* **HTTP API Endpoint:** A running HTTP/HTTPS API endpoint accessible from the Kubernetes cluster.\n\n## Getting Started\n\n### 1. Install the Operator using Helm\n\nYou can deploy the operator using Helm from the official chart repository:\n\n```bash\nhelm repo add konnektr https://charts.konnektr.io\nhelm repo update\nhelm install http-query-operator konnektr/http-query-operator \\\n  --namespace \u003cnamespace\u003e \\\n  --create-namespace \\\n  --set image.tag=\u003cversion\u003e \\\n  --set gvkPattern=\"v1/ConfigMap;apps/v1/Deployment\" \\\n  --set installCRDs=true\n```\n\n* By default, the image tag will match the Helm chart's `appVersion`.\n* You can override any value in `values.yaml` using `--set` or a custom `values.yaml`.\n* The `gvkPattern` parameter allows you to specify which Kubernetes resources the operator should manage.\n* The CRDs are not installed by default; install with the installCRDs parameter or manually as described below.\n\n#### Install the CRDs (required)\n\n```bash\nkubectl apply -f https://github.com/konnektr-io/http-query-operator/releases/latest/download/crds.yaml\n```\n\n#### Uninstall\n\n```bash\nhelm uninstall http-query-operator -n \u003cnamespace\u003e\n```\n\n### 2. Prepare the HTTP API\n\nEnsure your HTTP API endpoint is running and accessible from your cluster. The API should return JSON responses that can be processed by the operator.\n\nExample API response format:\n\n```json\n{\n  \"users\": [\n    {\n      \"user_id\": 1,\n      \"username\": \"alice\",\n      \"email\": \"alice@example.com\",\n      \"status\": \"active\"\n    },\n    {\n      \"user_id\": 2,\n      \"username\": \"bob\", \n      \"email\": \"bob@example.com\",\n      \"status\": \"active\"\n    }\n  ]\n}\n```\n\n### 3. Verify the Operator Pod\n\nCheck that the operator pod is running:\n\n```bash\nkubectl get pods -n \u003cnamespace\u003e\n# Look for a pod named like controller-manager-...\n\n# View logs\nkubectl logs -n \u003cnamespace\u003e -l control-plane=controller-manager -f\n```\n\n### 4. Create Authentication Credentials Secret (Optional)\n\nIf your HTTP API requires authentication, create a Kubernetes Secret containing the authentication details. The operator will read credentials from this Secret.\n\n**Example `api-credentials.yaml`:**\n\n```yaml\napiVersion: v1\nkind: Secret\nmetadata:\n  name: api-credentials\n  # IMPORTANT: Deploy this secret in the same namespace as your HTTPQueryResource CR,\n  # or specify the secret's namespace in the CR spec.\n  namespace: default\ntype: Opaque\nstringData:\n  # For Basic Authentication\n  username: \"your_api_user\"\n  password: \"your_api_password\"\n  \n  # For Bearer Token Authentication\n  token: \"your_bearer_token\"\n  \n  # For API Key Authentication\n  apikey: \"your_api_key\"\n  \n  # For OAuth2 Client Credentials Authentication\n  clientId: \"your_oauth2_client_id\"\n  clientSecret: \"your_oauth2_client_secret\"\n```\n\nApply the secret:\n\n```bash\nkubectl apply -f api-credentials.yaml\n```\n\n## Usage\n\nCreate an `HTTPQueryResource` custom resource to tell the operator which HTTP endpoint to query and how to generate resources.\n\n**Example `config/samples/http_v1alpha1_httpqueryresource.yaml`:**\n\n```yaml\napiVersion: konnektr.io/v1alpha1\nkind: HTTPQueryResource\nmetadata:\n  name: user-configmaps-example\n  namespace: default # Namespace where this CR is deployed and where resources will be created by default\nspec:\n  # How often to query the HTTP endpoint and reconcile\n  pollInterval: \"1m\"\n  # Whether to delete resources if their corresponding API item disappears (default: true)\n  prune: true\n  http:\n    # The HTTP endpoint URL\n    url: \"https://api.example.com/users\"\n    # HTTP method (default: GET)\n    method: \"GET\"\n    # Optional headers\n    headers:\n      Accept: \"application/json\"\n      User-Agent: \"http-query-operator\"\n    # Optional: JSONPath to extract array data from response\n    responsePath: \"users\"\n    # Optional: Authentication\n    authenticationRef:\n      name: api-credentials\n      type: basic # or \"bearer\" or \"apikey\" or \"oauth2\"\n      # Optional: Custom key names in the secret\n      # usernameKey: \"username\"\n      # passwordKey: \"password\"\n      # tokenKey: \"token\"\n      # apikeyKey: \"apikey\"\n      # apikeyHeader: \"X-API-Key\"\n  # Go template for the Kubernetes resource(s)\n  template: |\n    apiVersion: v1\n    kind: ConfigMap\n    metadata:\n      # Name must be unique per item. Use data from the API response.\n      # Ensure the resulting name is DNS-compatible!\n      name: user-{{ .Item.username | lower }}-config\n      namespace: default\n      labels:\n        # Use API data in labels/annotations\n        user_id: \"{{ .Item.user_id }}\"\n        # This label is automatically added by the controller:\n        # konnektr.io/managed-by: http-query-operator-controller\n    data:\n      email: \"{{ .Item.email }}\"\n      status: \"{{ .Item.status }}\"\n      username: \"{{ .Item.username }}\"\n      # Example using Go template functions (time)\n      managedTimestamp: \"{{ now | date \"2006-01-02T15:04:05Z07:00\" }}\"\n  # Optional: HTTP callback for status updates\n  statusUpdate:\n    url: \"https://api.example.com/users/{{ .Item.user_id }}/status\"\n    method: \"PATCH\"\n    headers:\n      Content-Type: \"application/json\"\n    bodyTemplate: |\n      {\n        \"kubernetes_status\": \"{{ .Resource.status | toJson }}\",\n        \"updated_at\": \"{{ now | date \"2006-01-02T15:04:05Z07:00\" }}\"\n      }\n    authenticationRef:\n      name: api-credentials\n      type: bearer\n```\n\n**Apply the sample CR:**\n\n```bash\nkubectl apply -f config/samples/http_v1alpha1_httpqueryresource.yaml -n default\n```\n\n**Check the results:**\nAfter the `pollInterval` duration, the operator should query the HTTP endpoint and create resources based on the template.\n\n```bash\n# Check the status of the HTTPQueryResource\nkubectl get httpqueryresource user-configmaps-example -n default -o yaml\n\n# Check for created resources (ConfigMaps in this example)\nkubectl get configmaps -n default -l konnektr.io/managed-by=http-query-operator-controller\nkubectl get configmap user-alice-config -n default -o yaml # Example for user 'alice'\n```\n\n### Example with Deployment Management\n\nHere is an example `HTTPQueryResource` Custom Resource that creates Kubernetes `Deployments` based on API data and sends status updates back to the API:\n\n```yaml\napiVersion: konnektr.io/v1alpha1\nkind: HTTPQueryResource\nmetadata:\n  name: deployment-example\n  namespace: default\nspec:\n  pollInterval: \"2m\"\n  prune: true\n  http:\n    url: \"https://api.example.com/applications\"\n    method: \"GET\"\n    headers:\n      Authorization: \"Bearer ${TOKEN}\"\n    responsePath: \"applications\"\n    authenticationRef:\n      name: api-credentials\n      type: bearer\n  template: |\n    apiVersion: apps/v1\n    kind: Deployment\n    metadata:\n      name: {{ .Item.name }}\n      namespace: default\n      labels:\n        app_id: \"{{ .Item.id }}\"\n    spec:\n      replicas: {{ .Item.replicas | default 1 }}\n      selector:\n        matchLabels:\n          app: {{ .Item.name }}\n      template:\n        metadata:\n          labels:\n            app: {{ .Item.name }}\n        spec:\n          containers:\n          - name: {{ .Item.name }}\n            image: {{ .Item.image }}\n            ports:\n            - containerPort: {{ .Item.port | default 8080 }}\n  statusUpdate:\n    url: \"https://api.example.com/applications/{{ .Item.id }}/status\"\n    method: \"PUT\"\n    headers:\n      Content-Type: \"application/json\"\n    bodyTemplate: |\n      {\n        \"deployment_status\": {\n          \"replicas\": {{ .Resource.status.replicas | default 0 }},\n          \"available_replicas\": {{ .Resource.status.availableReplicas | default 0 }},\n          \"ready_replicas\": {{ .Resource.status.readyReplicas | default 0 }}\n        },\n        \"last_updated\": \"{{ now | date \"2006-01-02T15:04:05Z07:00\" }}\"\n      }\n    authenticationRef:\n      name: api-credentials\n      type: bearer\n```\n\nIn this example:\n\n* The HTTP request fetches application data from an API endpoint.\n* The `template` generates a Kubernetes `Deployment` for each application.\n* The `statusUpdate` sends deployment status back to the API after reconciliation.\n\n### Example with OAuth2 Authentication\n\nThis example shows how to use OAuth2 Client Credentials flow to authenticate with your API:\n\n```yaml\napiVersion: konnektr.io/v1alpha1\nkind: HTTPQueryResource\nmetadata:\n  name: oauth2-api-example\n  namespace: default\nspec:\n  pollInterval: \"5m\"\n  prune: true\n  http:\n    url: \"https://api.oauth2example.com/v1/resources\"\n    method: \"GET\"\n    headers:\n      Accept: \"application/json\"\n    responsePath: \"data\"  # Extract from {\"data\": [...]} response\n    authenticationRef:\n      name: oauth2-credentials\n      type: oauth2\n      tokenUrl: \"https://auth.oauth2example.com/oauth2/token\"\n      scopes: \"read:resources write:status\"\n  template: |\n    apiVersion: v1\n    kind: ConfigMap\n    metadata:\n      name: resource-{{ .Item.id }}\n      namespace: default\n    data:\n      id: \"{{ .Item.id }}\"\n      name: \"{{ .Item.name }}\"\n      description: \"{{ .Item.description }}\"\n      updated_at: \"{{ now | date \"2006-01-02T15:04:05Z07:00\" }}\"\n```\n\nThe corresponding OAuth2 credentials secret:\n\n```yaml\napiVersion: v1\nkind: Secret\nmetadata:\n  name: oauth2-credentials\n  namespace: default\ntype: Opaque\nstringData:\n  clientId: \"your_oauth2_client_id\"\n  clientSecret: \"your_oauth2_client_secret\"\n```\n\nIn this example:\n\n* The operator uses OAuth2 Client Credentials flow to get an access token from `tokenUrl`.\n* The access token is automatically added to requests as `Authorization: Bearer \u003ctoken\u003e`.\n* Tokens are automatically refreshed when they expire.\n* Multiple scopes can be requested by separating them with spaces.\n\n## CRD Specification (`HTTPQueryResourceSpec`)\n\n* `pollInterval` (string, required): Duration string specifying how often to poll the HTTP endpoint (e.g., `\"30s\"`, `\"5m\"`, `\"1h\"`).\n* `prune` (boolean, optional, default: `true`): If `true`, resources previously managed by this CR that no longer correspond to an item in the latest API response will be deleted.\n* `http` (object, required):\n  * `url` (string, required): The HTTP/HTTPS endpoint URL to query.\n  * `method` (string, optional, default: `\"GET\"`): HTTP method (GET, POST, PUT, PATCH, DELETE).\n  * `headers` (map, optional): HTTP headers to include in the request.\n  * `body` (string, optional): Request body for POST/PUT/PATCH requests. Can be a Go template.\n  * `responsePath` (string, optional, default: `\"$\"`): JSONPath expression to extract array data from response.\n  * `authenticationRef` (object, optional): Reference to authentication configuration.\n    * `name` (string, required): Name of the Secret containing authentication details.\n    * `namespace` (string, optional): Namespace of the Secret. Defaults to the `HTTPQueryResource`'s namespace.\n    * `type` (string, required, enum: `\"basic\"`, `\"bearer\"`, `\"apikey\"`, `\"oauth2\"`): Type of authentication.\n    * `usernameKey` (string, optional): Key in the Secret for the username (basic auth). Defaults to `\"username\"`.\n    * `passwordKey` (string, optional): Key in the Secret for the password (basic auth). Defaults to `\"password\"`.\n    * `tokenKey` (string, optional): Key in the Secret for the token (bearer auth). Defaults to `\"token\"`.\n    * `apikeyKey` (string, optional): Key in the Secret for the API key. Defaults to `\"apikey\"`.\n    * `apikeyHeader` (string, optional): Header name for API key authentication. Defaults to `\"X-API-Key\"`.\n    * `clientIdKey` (string, optional): Key in the Secret for OAuth2 client ID. Defaults to `\"clientId\"`.\n    * `clientSecretKey` (string, optional): Key in the Secret for OAuth2 client secret. Defaults to `\"clientSecret\"`.\n    * `tokenUrl` (string, optional): OAuth2 token endpoint URL for client credentials flow. Required for `oauth2` type.\n    * `scopes` (string, optional): OAuth2 scopes to request (space-separated). Optional for `oauth2` type.\n* `template` (string, required): A Go template string that renders a valid Kubernetes resource manifest (YAML or JSON).\n  * **Template Context:** The template receives a map with the following structure:\n\n  ```go\n  {\n      \"Item\": {\n          \"field1\": value1,\n          \"field2\": value2,\n          // ... other fields from the API response item\n      },\n      \"Index\": 0 // Index of the item in the response array\n  }\n  ```\n* `statusUpdate` (object, optional): Configuration for HTTP status update callbacks.\n  * `url` (string, required): The HTTP/HTTPS endpoint URL for status updates. Can be a Go template.\n  * `method` (string, optional, default: `\"PATCH\"`): HTTP method for status updates.\n  * `headers` (map, optional): HTTP headers to include in the status update request.\n  * `bodyTemplate` (string, required): Go template for the request body. Receives the resource data.\n  * `authenticationRef` (object, optional): Authentication details for status updates (same structure as above).\n\n  * **Template Context:** The template receives a map with the following structure for status updates:\n\n  ```go\n  {\n      \"Resource\": { // The individual child resource being updated\n          \"apiVersion\": \"apps/v1\",\n          \"kind\": \"Deployment\",\n          \"metadata\": {\n              \"name\": \"app-example\",\n              \"namespace\": \"default\",\n              // ... other metadata fields\n          },\n          \"status\": {\n              \"replicas\": 1,\n              \"availableReplicas\": 1,\n              // ... other status fields\n          }\n          // ... other resource fields\n      },\n      \"Item\": { // The original API response item that generated this resource\n          \"id\": \"123\",\n          \"name\": \"app-example\",\n          // ... other API fields\n      }\n  }\n  ```\n\n  * You can use standard Go template functions and Sprig functions. Access item data via `.Item.field_name` and resource data via `.Resource.status.field_name`.\n\n## Cascading Deletion and Finalizer Logic\n\nBy default, deleting an `HTTPQueryResource` will **not** delete the resources it manages (such as ConfigMaps, Deployments, etc).\n\nIf you want the operator to delete all managed resources when the `HTTPQueryResource` is deleted, you must explicitly add the following finalizer to the resource:\n\n```yaml\nmetadata:\n  finalizers:\n    - konnektr.io/httpqueryresource-finalizer\n```\n\nWhen this finalizer is present, the operator will:\n\n1. On deletion (when you run `kubectl delete httpqueryresource ...`), the operator will first delete all managed resources (those labeled with `konnektr.io/managed-by: \u003ccontroller-name\u003e`).\n2. Once all managed resources are deleted, the operator will remove the finalizer, allowing the `HTTPQueryResource` to be deleted.\n\n**How to use:**\n- To enable cascading deletion, patch your resource before deleting:\n\n  ```bash\n  kubectl patch httpqueryresource \u003cname\u003e -n \u003cnamespace\u003e --type='json' -p='[{\"op\": \"add\", \"path\": \"/metadata/finalizers/-\", \"value\": \"konnektr.io/httpqueryresource-finalizer\"}]'\n  ```\n- Then delete as usual:\n\n  ```bash\n  kubectl delete httpqueryresource \u003cname\u003e -n \u003cnamespace\u003e\n  ```\n\nIf the finalizer is not present, deleting the `HTTPQueryResource` will **not** delete any managed resources.\n\n## Development\n\n1. **Prerequisites:** Ensure Go, Docker, `kubectl`, `controller-gen`, and access to a Kubernetes cluster are set up.\n2. **Clone:** `git clone \u003crepository-url\u003e`\n3. **Modify Code:** Make changes to the API (`api/v1alpha1/`) or controller (`internal/controller/`).\n4. **Regenerate Code:** After modifying API types or RBAC/CRD markers, run:\n\n    ```bash\n    # Regenerate deepcopy methods for API types\n    controller-gen object paths=./api/v1alpha1\n\n    # Regenerate CRD and RBAC manifests\n    # Adjust paths if needed, especially on Windows: paths=./api/v1alpha1,./internal/controller\n    controller-gen rbac:roleName=manager-role crd webhook paths=./api/v1alpha1,./internal/controller output:crd:artifacts:config=config/crd/bases output:rbac:artifacts:config=config/rbac\n    ```\n\n5. **Build:**\n\n    ```bash\n    go build ./...\n    # Or build the container image (see step 4 in Getting Started)\n    ```\n\n6. **Deploy:** Re-deploy the operator using the steps in \"Getting Started\".\n\n## Contributing\n\nContributions are welcome! Please follow standard GitHub practices: fork the repository, create a feature branch, make your changes, and submit a pull request. Ensure your code builds, passes any tests, and includes updates to documentation if necessary.\n\n## License\n\nThis project is licensed under the Apache License 2.0. See the [LICENSE](LICENSE) file for details.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkonnektr-io%2Fhttp-query-operator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkonnektr-io%2Fhttp-query-operator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkonnektr-io%2Fhttp-query-operator/lists"}