https://github.com/redborder/rb-aioutliers
RedBorder Outliers Detector with Keras
https://github.com/redborder/rb-aioutliers
autodelivery deep-learning outlier-detection redborder-ng
Last synced: 2 months ago
JSON representation
RedBorder Outliers Detector with Keras
- Host: GitHub
- URL: https://github.com/redborder/rb-aioutliers
- Owner: redBorder
- License: agpl-3.0
- Created: 2023-07-10T14:53:09.000Z (almost 3 years ago)
- Default Branch: master
- Last Pushed: 2026-03-20T15:13:06.000Z (3 months ago)
- Last Synced: 2026-03-21T06:53:49.488Z (3 months ago)
- Topics: autodelivery, deep-learning, outlier-detection, redborder-ng
- Language: Python
- Homepage: https://redborder.github.io/rb-aioutliers/
- Size: 2.19 MB
- Stars: 4
- Watchers: 0
- Forks: 3
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# rb-aioutliers Main package
[](https://www.gnu.org/licenses/agpl-3.0)
[](https://codecov.io/gh/redBorder/rb-aioutliers)
[](https://app.fossa.com/projects/git%2Bgithub.com%2FredBorder%2Frb-aioutliers?ref=badge_shield)
[](https://app.fossa.com/projects/git%2Bgithub.com%2FredBorder%2Frb-aioutliers?ref=badge_large)
Main package to install redborder AI outliers in Rocky Linux 9
#### Platforms
- Rocky Linux 9
## Running the example
This code shows runs the outlier detection on a mock dataset. Its recomended to use pipenv or similar to avoid overwritting dependencies.
```bash
git clone git@github.com:redBorder/rb-aioutliers.git
cd rb-aioutliers
pip install -r resources/src/requirements.txt
bash resources/src/example/run_example.sh
```
## Installation
1. yum install epel-release && rpm -ivh http://repo.redborder.com/redborder-repo-0.0.3-1.el7.rb.noarch.rpm
2. yum install rb-aioutliers
## Model design
Initially, data is extracted from a designated druid datasource in timeseries format, with configurable metrics and settings. After rescaling from zero to one and segmentation, an autoencoder reconstructs the data, enabling anomaly detection through k-sigma thresholding.
The anomalies are outputed in Json format together with the data reconstructed by the autoencoder.
## Model execution
rb-aioutliers utilizes the Flask framework to create an HTTP server. Users can send Druid queries via POST requests to the /calculate endpoint. When rb-aioutliers receives the Druid query, it sends a request to the Druid broker, retrieves the necessary data, and then proceeds to execute the anomaly detection model.
After executing the model, the server can respond with one of two status messages:
`HTTP OK 200 Success`: In this case, the response body will be structured as follows:
```
{
"status": "success",
"anomalies": [Array_of_anomalies],
"prediction": [Array_of_predictions]
}
```
`HTTP 500 Internal Server Error`: If there is an issue during the process, the response will contain an error message:
```
{
"status": "error",
"msg": "Error_description_message"
}
```
## Model training
The rb-aioutliers service generates a custom Druid query and sends it to the Redborder cluster Druid broker. After sending the query, it retrieves the data and attempts to train a model with custom parameters such as epochs or batch size. Once the model training is complete, it outputs a backup file and the generated Keras model.
## Clusterization with Chef, Consul, AWS S3 & Redis
The rb-aioutliers service can operate in cluster mode! This is achieved by dividing the service into two components: the executor and the trainer.
# Executor
The executor service is registered in HashiCorp Consul. When the Redborder WebUI sends a request to api/v1/outliers, it will be directed to the rb-aioutliers.service, which, in turn, will redirect the request to any of the nodes running the rb-aioutliers REST server. This server will download the relevant model from S3, based on the sensor specified in the Druid query, and execute it.
# Trainer
For the training service, the Chef client will create individual configuration files for each node, which are generated based on the cookbook and templates. These configuration files specify which sensor should be trained for. The rb-aioutliers-train service will then proceed to (**take a look at Trainer jobs below**) download the appropriate model from S3 for each node. Once the training is completed, the rb-aioutliers-train service will upload the resulting trained model to S3. It's important to note that each model is unique and specific to the corresponding sensor.
# Trainer Jobs
The trainer nodes send job data to a Redis server (this is done by the ```Trainer service```). Each node also requests the job queue. The RQ worker processes these jobs in the background. If any node goes down, there will be no problem, as the sensor information for training is stored in Redis, and another node can take over its tasks.
For more info about deploy with Chef Server take a look at [Outliers Cookbook](https://github.com/redBorder/cookbook-rb-aioutliers)
## Docker support
If you want to run the app inside a docker container run the following commands
```bash
cd ./resources/src
docker-compose up --build -d
```
now if you list your docker container you will see the following container running.
| Container ID | Image | Command | Created | Status | Exposed Ports |
|-----------------|--------------------------|------------------------|-----------------|-----------------|---------------------------------------------|
| cb18a72ab60e | src_rb_aioutliers_rest | python __main__.py | 3 minutes ago | Up 3 minutes | 0.0.0.0:39091->39091/tcp, :::39091->39091/tcp |
## API Endpoints
### `/calculate`
- **HTTP Method:** POST
- **Description:** Initiates anomaly detection (model execution) with a Druid query.
- **Request Body:** JSON data containing the Druid query in base64 string.
**Example Request:**
```application-x-www-form-urlencoded
POST /calculate (application-x-www-form-urlencoded)
query=base64_string
```
**Example Druid Query:**
```
{
"dataSource": "rb_flow",
"granularity": {
"type": "period",
"period": "pt1m",
"origin": "2023-09-21T09:00:00Z"
},
"intervals": [
"2023-09-21T09:00:00+00:00/2023-09-21T10:00:00+00:00"
],
"filter": {
"type": "selector",
"dimension": "sensor_name",
"value": "FlowSensor"
},
"queryType": "timeseries",
"context": {
"timeout": 90000,
"skipEmptyBuckets": "true"
},
"limitSpec": {
"type": "default",
"limit": 100,
"columns": []
},
"aggregations": [
{
"type": "longSum",
"name": "bytes",
"fieldName": "sum_bytes"
}
],
"postAggregations": []
}
```
**Example Respone:**
```
{
"anomalies": [
{
"expected": 36453984.6858499,
"timestamp": "2023-09-28T07:00:00.000Z"
},
{
"expected": 36453984.6858499,
"timestamp": "2023-09-28T07:00:00.000Z"
},
{
"expected": 36453984.6858499,
"timestamp": "2023-09-28T07:00:00.000Z"
},
{
"expected": 36453984.6858499,
"timestamp": "2023-09-28T07:00:00.000Z"
},
{
"expected": 36453984.6858499,
"timestamp": "2023-09-28T07:00:00.000Z"
},
{
"expected": 36453984.6858499,
"timestamp": "2023-09-28T07:00:00.000Z"
},
{
"expected": 36453984.6858499,
"timestamp": "2023-09-28T07:00:00.000Z"
},
{
"expected": 36453984.6858499,
"timestamp": "2023-09-28T07:00:00.000Z"
},
{
"expected": 36453984.6858499,
"timestamp": "2023-09-28T07:00:00.000Z"
},
{
"expected": 36453984.6858499,
"timestamp": "2023-09-28T07:00:00.000Z"
},
{
"expected": 45825798.264862545,
"timestamp": "2023-09-28T07:01:00.000Z"
}
],
"status": "success"
}
```
## Contributing
1. Fork the repository on Github
2. Create a named feature branch (like `add_component_x`)
3. Write your change
4. Write tests for your change (if applicable)
5. Run the tests, ensuring they all pass
6. Submit a Pull Request using Github
## License and Authors
- Miguel Álvarez Adsuara
- Pablo Rodriguez Flores
LICENSE: AFFERO GENERAL PUBLIC LICENSE, Version 3, 19 November 2007