{"id":22615457,"url":"https://github.com/watchdogcloud/terrier","last_synced_at":"2025-04-11T12:12:39.430Z","repository":{"id":248919854,"uuid":"825197275","full_name":"watchdogcloud/terrier","owner":"watchdogcloud","description":"Watchdog Configuration and Management Server ","archived":false,"fork":false,"pushed_at":"2025-03-24T06:32:55.000Z","size":1286,"stargazers_count":4,"open_issues_count":5,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-25T08:38:15.670Z","etag":null,"topics":["docker-compose","kafka","monitoring-tool","redis","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/watchdogcloud.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-07T04:54:31.000Z","updated_at":"2025-02-23T12:14:15.000Z","dependencies_parsed_at":"2024-07-17T23:17:24.372Z","dependency_job_id":"d494be6d-7dc6-4c3f-818c-b636f03fc2a2","html_url":"https://github.com/watchdogcloud/terrier","commit_stats":null,"previous_names":["watchdogcloud/terrier"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/watchdogcloud%2Fterrier","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/watchdogcloud%2Fterrier/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/watchdogcloud%2Fterrier/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/watchdogcloud%2Fterrier/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/watchdogcloud","download_url":"https://codeload.github.com/watchdogcloud/terrier/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248398560,"owners_count":21097292,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker-compose","kafka","monitoring-tool","redis","typescript"],"created_at":"2024-12-08T19:07:06.099Z","updated_at":"2025-04-11T12:12:39.401Z","avatar_url":"https://github.com/watchdogcloud.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align='center'\u003e\n\n# Watchdog - Terrier Config and Management Server\n\n\u003cimg src=\".github/logo.png\" alt=\"canario.png\" width='300'\u003e\u003c/img\u003e\n\u003chr\u003e\n\n![example workflow](https://github.com/watchdogcloud/terrier/actions/workflows/ghcr-deploy.yml/badge.svg)\n\n\u003c/div\u003e\n\n\nTerrier is the core configuration and management server that forms the backbone of the Watchdog ecosystem,from handling requests from all Watchdog Client SDKs to maintaining communication and management of Kafka Brokers across distributed systems.\n\n### Core \n\nTerrier Sends timely alerts to stakeholders based on predefined thresholds and conditions. Alerts notify stakeholders about critical issues affecting system health and performance.Currently the server uses SMTP Mailing to send notifications regarding hardware issues like Resource Spikes and Cumulative Threshold overflows.\n\n\n### Spike Triggers\nThese are triggers that detect sudden and significant changes in the metric values, indicating potential issues that need immediate attention.\n\n\u003e Example : If the CPU usage suddenly spikes from 30% to 90% within a minute, a spike trigger will detect this abrupt change and send an alert to notify stakeholders of a potential issue requiring immediate investigation.\n\n\n### Cumulative Triggers\nThese triggers detect gradual changes in metric values over time, indicating potential issues that may arise if the trend continues.\n\n\u003e Example : If memory usage steadily increases by 5% each hour over a day, a cumulative trigger will recognize this trend and alert stakeholders ,indicating that there might be a memory leak or inefficient resource usage that could cause problems if not addressed.\n\n\nTerrier continuously streams real-time metrics concerning disk usage, RAM, and memory utilization (for now). These metrics are important for monitoring application health and performance, giving insights into resource utilization and potential bottlenecks. Over this data , workers perform computational tasks - It aggregates, analyzes, and computes metrics to detect anomalies, trends, and patterns indicative of system health or operational issues.\n\nInternally, Terrier manages Kafka clusters, brokers, producers, and consumers. It ensures the operation of Kafka messaging infrastructure \u0026 ensures reliable data ingestion and stream processing.\n\nIt also Persistently stores historical metrics and metadata,project and user authentication information , configuration settings, and operational data in a NoSQL DB.\n\n\n## Installation\n\n\u003e [!WARNING]\n\u003e Terrier relies on Kafka, Zookeeper, and Redis to facilitate communication between services and data computation. Terrier won't work without it's dependencies.\n\n### Prerequisites\n\n1. **Kafka**: A distributed event streaming platform capable of handling trillions of events a day,if configured properly.\n2. **Zookeeper**: A centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.\n3. **Redis**: An in-memory data structure store, used as a database, cache, and message broker.\n\n### Configuration\n\nTo configure Terrier, edit the `config/default.json` file. Below are the attributes and their descriptions:\n\n\n| Attribute | Description |\n|-----------|-------------|\n| `host` | The hostname or IP address where the Terrier server will run. |\n| `port` | The port number on which the Terrier server will listen. |\n| `public` | The path to the public directory for serving static files. |\n| `paginate.default` | The default number of items per page for pagination. |\n| `paginate.max` | The maximum number of items per page for pagination. |\n| `authentication.entity` | The entity to authenticate (e.g., user). |\n| `authentication.service` | The service used for authentication (e.g., users). |\n| `authentication.secret` | The secret key for JWT authentication. |\n| `authentication.authStrategies` | The authentication strategies to use (e.g., jwt, local). |\n| `authentication.jwtOptions.header` | Custom headers for the JWT, such as the type (`typ`). |\n| `authentication.jwtOptions.audience` | The audience that the JWT is intended for. This helps to ensure the token is being sent to the correct recipient. |\n| `authentication.jwtOptions.issuer` | The issuer of the JWT. This helps to ensure the token was issued by a trusted source. |\n| `authentication.jwtOptions.algorithm` | The algorithm used to sign the JWT, such as `HS256`. |\n| `authentication.jwtOptions.expiresIn` | The duration after which the JWT will expire, for example, `'365d'` for 365 days. |\n| `authentication.local.usernameField` | The field for the username in local authentication. |\n| `authentication.local.passwordField` | The field for the password in local authentication. |\n| `mongodb` | The MongoDB connection string. |\n| `otpless.clientId` | The client ID for OTP-less authentication.Get one from `https://otpless.com/` |\n| `otpless.clientSecret` | The client secret for OTP-less authentication. Get one from `https://otpless.com/`|\n| `otpless.expiry` | The expiration time for OTPs in seconds. |\n| `otpless.otpLength` | The length of the OTP. |\n| `otpless.hash` | The hashing algorithm used for OTPs. |\n| `kafkaConf.bootstrap_servers` | The list of Kafka bootstrap servers. |\n| `kafkaConf.topics.name` | The name of the Kafka topic. |\n| `kafkaConf.topics.numPartitions` | The number of partitions for the Kafka topic. A common practice is to set the number of partitions to a multiple of the number of consumers. This ensures an even distribution of workload among consumers.|\n| `kafkaConf.topics.replicationFactor` | The replication factor for the Kafka topic. |\n| `keygen.length` | The length of the generated keys. |\n| `smtp.email` | The SMTP email address used for sending emails. |\n| `smtp.password` | The SMTP password used for sending emails. |\n| `smtp.host` | The SMTP server host. |\n| `smtp.port` | The SMTP server port. |\n| `smtp.secure` | Whether to use a secure connection for SMTP. |\n| `alerts.spike.cpu` | The CPU usage threshold for spike alerts. |\n| `alerts.spike.mem` | The memory usage threshold for spike alerts. |\n| `alerts.spike.disk` | The disk usage threshold for spike alerts. |\n| `alerts.cumulative.cpu` | The CPU usage threshold for cumulative alerts. |\n| `alerts.cumulative.mem` | The memory usage threshold for cumulative alerts. |\n| `alerts.cumulative.disk` | The disk usage threshold for cumulative alerts. |\n| `alerts.cumulative.durationInMin` | The duration in minutes for cumulative alerts. |\n\nA sample configuration file is present at `config/test.json`\n\n### JWT Options\n\nThe `authentication.jwtOptions` configuration allows you to customize the properties of the JWT (JSON Web Token) used for authentication. Here's a detailed explanation of each option:\n\n- **header**: An object specifying custom headers for the JWT. The `typ` field typically indicates the type of token, usually set to \"access\".\n\n\u003e [!TIP]\n\u003e Ensure the `typ` is set correctly to differentiate between access and refresh tokens.\n- **audience**: The audience that the JWT is intended for. This is usually the URL of your application or service. It ensures that the token is being sent to the correct recipient.\n\n\u003e [!TIP]\n\u003e Use a consistent audience string across your services to simplify token validation.\n- **issuer**: The issuer of the JWT, usually your application's name or URL. This ensures that the token was issued by a trusted source.\n\n\u003e [!TIP] \n\u003e Match the issuer with the domain of your application to avoid token spoofing.\n- **algorithm**: The algorithm used to sign the JWT. Common algorithms include `HS256` (HMAC using SHA-256).\n\n\u003e [!TIP]\n\u003e Consider using stronger algorithms like `RS256` for enhanced security, especially in production environments.\n- **expiresIn**: The duration after which the JWT will expire. It can be specified in seconds or as a string describing a time span (e.g., `'365d'` for 365 days).\n\n\u003e [!TIP]\n\u003e Set an appropriate expiry time to balance security and user convenience. Shorter expiry times enhance security but may require more frequent re-authentication.\n\n### Running the Server\n\nUse Docker Compose to start the Terrier server along with its dependencies. This command will start the services defined in your `docker-compose.yml` file.\n\n```sh\ndocker-compose -f docker/docker-compose.yml up\n```\n\n\u003e [!TIP]\n\u003e Use the `-d` flag to run the services in the background\n\n### Kafka UI\n\nTerrier is integrated with Kafka UI, an OSS graphical user interface to manage and view the state of Kafka. This includes monitoring current messages, topics, consumer groups, and more.\n\n- Access Kafka UI: Kafka UI runs on port 8080. You can access it by navigating to `http://localhost:8080` in your web browser.\n\n\n### Architectural Reasons\n\nTerrier supports the dynamic scaling of Kafka clusters and other components to handle varying workloads and data volumes efficiently across multiple projects.\n\nProject is aimed at offering flexibility through APIs and interfaces for integration with Third-party tools and custom applications. This allows organizations to extend and customize Terrier's functionality according to specific operational requirements.\n\n### Future Development\n\nTerrier's upcoming development roadmap:\n\n- Dashboard for Real-Time Stream Monitoring\n- Advanced Alerting Mechanisms\n\nFor updates, documentation, and community support, visit the [Terrier GitHub repository](https://github.com/watchdogcloud/terrier). \n\n\u003cdiv align='center'\u003e\n  \u003cimg src=\"https://skillicons.dev/icons?i=kafka,ts,redis,docker,\"\u003e\u003c/img\u003e\n\u003c/div\u003e\n\u003chr\u003e\n\u003cdiv align='center'\u003e\n  Enjoy monitoring with Terrier! 🐕\n\u003c/div\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwatchdogcloud%2Fterrier","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwatchdogcloud%2Fterrier","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwatchdogcloud%2Fterrier/lists"}