{"id":16753738,"url":"https://github.com/awakari/core","last_synced_at":"2025-04-11T22:30:52.983Z","repository":{"id":163088779,"uuid":"615596059","full_name":"awakari/core","owner":"awakari","description":"Core Awakari system deployment and tests","archived":false,"fork":false,"pushed_at":"2024-08-27T17:53:25.000Z","size":4893,"stargazers_count":9,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-25T19:21:43.032Z","etag":null,"topics":["grpc","grpc-api","grpc-services","helm-chart","jetstream","mongodb","nats","real-time-search","realtime-search-engine","search-alerts","self-hosted"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/awakari.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-18T05:14:26.000Z","updated_at":"2024-10-11T09:23:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"7fe0f0a9-450a-49f0-83e2-cae683aa51b7","html_url":"https://github.com/awakari/core","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awakari%2Fcore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awakari%2Fcore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awakari%2Fcore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awakari%2Fcore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/awakari","download_url":"https://codeload.github.com/awakari/core/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248489491,"owners_count":21112584,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["grpc","grpc-api","grpc-services","helm-chart","jetstream","mongodb","nats","real-time-search","realtime-search-engine","search-alerts","self-hosted"],"created_at":"2024-10-13T03:00:21.794Z","updated_at":"2025-04-11T22:30:47.931Z","avatar_url":"https://github.com/awakari.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# Contents\n\n1. [Overview](#1-overview)\u003cbr/\u003e\n2. [Configuration](#2-configuration)\u003cbr/\u003e\n3. [Deployment](#3-deployment)\u003cbr/\u003e\n   3.1. [Cluster Preparation](#31-cluster-preparation)\u003cbr/\u003e\n   3.2. [Connect Subscriptions](#32-connect-subscriptions)\u003cbr/\u003e\n   3.3. [Core Installation](#33-core-installation)\u003cbr/\u003e\n   3.4. [External DB](#34-external-db)\u003cbr/\u003e\n   3.5. [Tracing](#35-tracing)\u003cbr/\u003e\n4. [Usage](#4-usage)\u003cbr/\u003e\n   4.1. [Client SDK](#41-client-sdk)\u003cbr/\u003e\n   4.2. [API](#42-api)\u003cbr/\u003e\n   \u0026nbsp;\u0026nbsp;\u0026nbsp;4.2.1. [Preparation](#421-preparation)\u003cbr/\u003e\n   \u0026nbsp;\u0026nbsp;\u0026nbsp;4.2.2. [Preparation](#422-subscriptions)\u003cbr/\u003e\n   \u0026nbsp;\u0026nbsp;\u0026nbsp;4.2.3. [Read Events](#423-read-events)\u003cbr/\u003e\n   \u0026nbsp;\u0026nbsp;\u0026nbsp;4.2.4. [Write Events](#424-write-events)\u003cbr/\u003e\n5. [Design](#5-design)\u003cbr/\u003e\n6. [Contributing](#6-contributing)\u003cbr/\u003e\n   6.1. [Versioning](#61-versioning)\u003cbr/\u003e\n   6.2. [Issue Reporting](#62-issue-reporting)\u003cbr/\u003e\n   6.3. [Building](#63-building)\u003cbr/\u003e\n   6.4. [Testing](#64-testing)\u003cbr/\u003e\n   \u0026nbsp;\u0026nbsp;\u0026nbsp;6.4.1. [Functional](#641-functional)\u003cbr/\u003e\n   \u0026nbsp;\u0026nbsp;\u0026nbsp;6.4.2. [Performance](#642-performance)\u003cbr/\u003e\n   6.5. [Releasing](#65-releasing)\u003cbr/\u003e\n\n\n# 1. Overview\n\nThis repo contains the Helm chart for the Awakari core system deployment.\nThe core doesn't include subscriptions storage. \nTo run the core system on own premises, request access to the cloud instance of subscriptions storage.\n\n# 2. Configuration\n\nFor a component-specific options see the corresponding sub-chart configuration. Here follow own configuration options: \n\n| Variable               | Default | Description                                                                                                              |\n|------------------------|---------|--------------------------------------------------------------------------------------------------------------------------|\n| mongodb.internal       | `true`  | Defines whether to deploy the MongoDB internally or use external one.                                                    | \n| queue.backend.nats     | `true`  | Enables the NATS JetStream queue wrapper service. Exclusive, can not be used together with other queue backends.         |\n| semaphore.backend.nats | `true`  | Enables the NATS-based distributed semaphore service. Exclusive, can not be used together with other semaphore backends. |\n| tracing.enabled        | `false` | Enables the distributed tracing, internal Jaeger and Cassandra deployment as well to collect the spans.                  | \n\n# 3. Deployment\n\nThere are the following resources required:\n1. Own K8s cluster\n2. Cloud subscriptions service access\n\n## 3.1. Cluster Preparation\n\nCreate the target namespace:\n```shell\nkubectl create namespace awakari\n```\n\nRequest and use the public GitHub registry access token to pull Awakari images:\n```shell\ndocker login ghcr.io -u akurilov -p \u003cACCESS_TOKEN\u003e\n```\n\nCreate the image pull secret:\n```shell\nkubectl create secret generic github-registry \\\n    -n awakari \\\n    --from-file=.dockerconfigjson=\u003chome/.docker/config.json\u003e \\\n    --type=kubernetes.io/dockerconfigjson\n```\n\n## 3.2. Connect Subscriptions\n\nUsing cloud subscriptions requires mutual TLS authentication and encryption to secure the client subscriptions data.\nTo access the cloud subscriptions it's necessary to have a client certificate.\n\n\u003e [!IMPORTANT]\n\u003e Cloud subscriptions doesn't have any access to events data being processed by the core system.\n\nFor the demo purposes, there is the cloud instance `demo.subscriptions.awakari.cloud` available.\nReady-to-use demo client certificates are in the [certs/demo](certs/demo) directory.\n\nFor production usage, prepare own client certificate request:\n```shell\nopenssl req -new -newkey rsa:4096 -nodes \\\n  -keyout client.key \\\n  -out client.csr \\\n  -addext \"subjectAltName=DNS:subscriptions.awakari.cloud\" \\\n  -subj '/CN=group0.company1.com'\n```\n\n\u003e **Warning**\n\u003e Never specify additional certificate attributes like \"O\", \"OU\", etc.\n\u003e The resulting DN should not contain commas.\n\nThen request the client certificate (currently by [email](mailto:awakari@awakari.com)).\n\nAfter the client certificate (`client.crt`) is received, create a pair of cluster secrets:\n```shell\nkubectl create secret generic -n awakari secret-subscriptions-tls-client-key --from-file=client.key\nkubectl create secret generic -n awakari secret-subscriptions-tls-client-crt --from-file=client.crt\n```\n\n## 3.3. Core Installation\n\nInstall the package:\n```shell\nhelm repo add awakari-core https://awakari.github.io/core\nhelm install core core-0.0.0.tgz -n awakari\n```\n\n\u003e **Warning**\n\u003e Do not change the \"core\" release name\n\nTo connect the core system to the demo cloud subscriptions, override the address:\n```shell\nhelm install core core-0.0.0.tgz -n awakari \\\n  --set subscriptionsproxy.api.subscriptions.uri=demo.subscriptions.awakari.cloud:443\n```\n\n## 3.4. External DB\n\n\u003e [!NOTE]\n\u003e This step is optional, by default the core system comes with internal MongoDB sharded cluster.\n\nTo use external MongoDB, use the values file [values-mongodb-ext.yaml](helm/core/values-mongodb-ext.yaml) for the \nreference and substitute these with own values.\n\n## 3.5. Tracing\n\n\u003e [!NOTE]\n\u003e This step is optional and may be useful for the performance testing purposes.\n\nIt's possible to enable the [Open Telemetry](https://opentelemetry.io/) tracing collection. Core system uses \n[Jaeger](https://www.jaegertracing.io/) collector API for this. Core system can deploy own Jaeger instance or use \nexternal one.\n\nTo enable tracing during the deployment the [values-tracing.yaml](helm/core/values-tracing.yaml) file may be used: \n```shell\nhelm install core core-0.0.0.tgz -n awakari \\\n  --set subscriptionsproxy.api.subscriptions.uri=demo.subscriptions.awakari.cloud:443 \\\n  --values helm/core/values-tracing.yaml\n```\n\nTo use external Jaeger, set the root value `tracing.enabled` to `false` and leave it enabled for the components with \nsetting the custom Jaeger collector URI:\n```shell\nhelm install core core-0.0.0.tgz -n awakari \\\n  --set subscriptionsproxy.api.subscriptions.uri=demo.subscriptions.awakari.cloud:443 \\\n  --values helm/core/values-tracing.yaml \\\n  --set tracing.enabled=false \\\n  --set evaluator.tracing.collector.uri=http://external-jaeger-collector:14268/api/traces \\\n  --set matches.tracing.collector.uri=http://external-jaeger-collector:14268/api/traces \\\n  --set reader.tracing.collector.uri=http://external-jaeger-collector:14268/api/traces \\\n  --set resolver.tracing.collector.uri=http://external-jaeger-collector:14268/api/traces \\\n  --set writer.tracing.collector.uri=http://external-jaeger-collector:14268/api/traces\n```\n\nTo observe the traces:\n1. run some workload, e.g. [e2e-test](test/e2e_test.go) or [perf-e2e-test](test/perf_e2e_test.go)\n2. port-forward the jaeger-query port 16686 to local and open it in browser.\n\n# 4. Usage\n\n## 4.1. Client SDK\n\nRefer to [Client SDK Usage](https://github.com/awakari/client-sdk-go#3-usage).\n\n\u003e [!NOTE]\n\u003e Usage Limits and Permits APIs are not available in the Core.\n\n## 4.2. API\n\n### 4.2.1. Preparation\n\n1. Install [grpcurl](https://github.com/fullstorydev/grpcurl)\n2. Download the necessary proto files and save to the current directory:\n   1. [Cloud Events](https://awakari.com/proto/cloudevent.proto)\n   2. [Reader](https://awakari.com/proto/reader.proto)\n   3. [Resolver](https://awakari.com/proto/resolver.proto)\n3. Port-forward services to local:\n   1. `core-reader` -\u003e 50051\n   2. `core-resolver` -\u003e 50052\n   3. `core-subscriptionsproxy` -\u003e 50053\n\n### 4.2.2. Subscriptions\n\nCreate:\n```shell\ngrpcurl \\\n  -plaintext \\\n  -H 'X-Awakari-User-Id: john.doe@company1.com' \\\n  -d @ \\\n  localhost:50053 \\\n  awakari.subscriptions.proxy.Service/Create\n```\n\nExample payload:\n```json\n{\n   \"description\": \"Tesla model S updates\",\n   \"enabled\": true,\n   \"cond\": {\n      \"not\": false,\n      \"tc\": {\n        \"key\": \"\",\n        \"term\": \"Tesla Model S\",\n        \"exact\": false\n      }\n    }\n}\n```\n\nA successful response contains the created subscription id:\n```json\n{\n  \"id\": \"547857e3-adfc-48a5-a49e-110cfdedbaab\"\n}\n```\n\nNote the created subscription id and use it further to read the messages.\nLearn more about the [Subscriptions API](https://github.com/awakari/client-sdk-go/blob/master/api/grpc/subscriptions/service.proto).\n\n### 4.2.3. Read Events\n\n```shell\ngrpcurl \\\n  -plaintext \\\n  -proto reader.proto \\\n  -max-time 86400 \\\n  -H 'X-Awakari-User-Id: john.doe@company1.com' \\\n  -d @ \\\n  localhost:50051 \\\n  awakari.reader.Service/Read\n```\n\nSpecify the subscription id in the payload:\n```json\n{\"start\": {\"batchSize\": 1, \"subId\": \"547857e3-adfc-48a5-a49e-110cfdedbaab\"}}\n```\n\nThis starts a reader stream. A new event appear in the response once system receives anything matching the subscription. \nLeave this shell/window open and switch to another. Later switch back and check for new events received.\n\nIt's necessary to acknowledge every received message:\n```json\n{\"ack\": { \"count\": 1}}\n```\n\n### 4.2.4. Write Events\n\n```shell\ngrpcurl \\\n  -plaintext \\\n  -proto writer.proto \\\n  -H 'X-Awakari-User-Id: john.doe@company1.com' \\\n  -d @ \\\n  localhost:50052 \\\n  awakari.resolver.Service/SubmitMessages\n```\n\nSpecify the events to write in the payload:\n```json\n{\n   \"msgs\": [\n      {\n         \"id\": \"3426d090-1b8a-4a09-ac9c-41f2de24d5ac\",\n         \"type\": \"example.type\",\n         \"source\": \"example/uri\",\n         \"spec_version\": \"1.0\",\n         \"attributes\": {\n            \"subject\": {\n               \"ce_string\": \"Tesla price updates\"\n            },\n            \"time\": {\n               \"ce_timestamp\": \"2023-07-03T23:20:50.52Z\"\n            }\n         },\n         \"text_data\": \"Tesla model S is now available at lower price\"\n      }\n   ]\n}\n```\n\nA successful response looks like:\n```json\n{\n  \"ackCount\": 1\n}\n```\n\nAfter this it's possible to submit more messages.\n\nWhen finished, close the writer stream by pressing ^C or leave it open to publish any other messages later.\n\n# 5. Design\n\nThe core of Awakari consist of:\n* Stateful components\n  * Conditions, e.g. [Text](https://github.com/awakari/conditions-text)\n  * [Matches](https://github.com/awakari/matches)\n  * [Messages](https://github.com/awakari/messages)\n* Stateless components\n  * [Subscriptions-Proxy](https://github.com/awakari/subscriptions-proxy)\n  * [Reader](https://github.com/awakari/reader)\n  * [Evaluator](https://github.com/awakari/evaluator)\n  * [Writer](https://github.com/awakari/writer)\n  * [Resolver](https://github.com/awakari/resolver)\n* 3-rd part components\n  * Mongodb (sharded)\n  * Redis in-memory cache\n  * NATS message bus\n\n![components](doc/components-2023-07-27.png)\n\n# 6. Contributing\n\n## 6.1. Versioning\n\nThe service uses the [semantic versioning](http://semver.org/).\nThe single source of the version info is the git tag:\n```shell\ngit describe --tags --abbrev=0\n```\n\n## 6.2. Issue Reporting\n\nTODO\n\n## 6.3. Building\n\nBuild a helm package:\n```shell\nfor i in core conditions-number conditions-text evaluator matches messages queue-nats reader resolver subscriptions-proxy semaphore-nats writer; do git clone git@github.com:awakari/$i.git; done\ncd core\nhelm dependency update helm/core\nhelm package helm/core\n```\n\n## 6.4. Testing\n\n### 6.4.1. Functional\n\nThe repo contains core functional end-to-end tests.\n\nTo run the tests locally:\n\n1. Port-forward the reader API to local port 50051\n2. Port-forward the resolver API to local port 50052\n3. Port-forward the subscriptions API to local port 50053\n4. \n```shell \nmake test\n```\n\nTo run the tests in K8s cluster:\n```shell\nhelm test core -n awakari --filter name=core-test\n```\n\n### 6.4.2. Performance\n\n![perf-e2e-test-report-2023-07-11](doc/perf-e2e-test-report-2023-07-11.png)\n\n![perf-e2e-test-report-2023-08-18](doc/perf-e2e-test-report-2023-08-18.png)\n\n## 6.5. Releasing\n\nTo release a new version (e.g. `1.2.3`) it's enough to put a git tag:\n```shell\ngit tag -v1.2.3\ngit push --tags\n```\n\nThe corresponding CI job is started to build a helm chart and publish it with the specified tag (+latest).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fawakari%2Fcore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fawakari%2Fcore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fawakari%2Fcore/lists"}