Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/minvws/nl-kat-mula
Repo nl-kat-mula for openkat
https://github.com/minvws/nl-kat-mula
Last synced: about 1 month ago
JSON representation
Repo nl-kat-mula for openkat
- Host: GitHub
- URL: https://github.com/minvws/nl-kat-mula
- Owner: minvws
- License: eupl-1.2
- Created: 2022-06-27T12:51:43.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-03-03T10:43:15.000Z (almost 2 years ago)
- Last Synced: 2023-03-04T04:56:57.143Z (almost 2 years ago)
- Language: Python
- Homepage: https://openkat.nl
- Size: 1.64 MB
- Stars: 5
- Watchers: 18
- Forks: 2
- Open Issues: 8
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
**This repository has been archived because all repositories of the individual
OpenKAT modules have been merged in the main OpenKAT repository at
https://github.com/minvws/nl-kat-coordination**
# Scheduler
## Purpose
The scheduler is responsible for scheduling the execution of tasks. The
execution of those tasks are being prioritized / scored by a ranker. The tasks
are then pushed onto a priority queue.
Within the project of KAT, the scheduler is tasked with scheduling boefje and
normalizer tasks.
## Architecture
See [design](docs/design.md) document for the architecture and the
[extending](docs/extending.md) document for the extending the scheduler with
your own custom schedulers, and rankers.
### Stack, packages and libraries
| Name | Version | Description |
|----------------|----------|----------------------------------------------------|
| Python | 3.8 | |
| FastAPI | 0.73.0 | Used for api server |
| Celery | 5.2.3 | Used for event listening, and dispatching of tasks |
### External services
The scheduler interfaces with the following services:
| Service | Usage |
|---------|-------|
| [Octopoes](https://github.com/minvws/nl-kat-octopoes) | Retrieving random OOI's of organizations |
| [Katalogus](https://github.com/minvws/nl-kat-boefjes/tree/develop/katalogus) | Used for referencing available plugins and organizations |
| [Bytes](https://github.com/minvws/nl-kat-bytes) | Retrieve last run boefje for organization and OOI |
| [Boefjes](https://github.com/minvws/nl-kat-boefjes) | Sending boefje, and normalizer tasks to Celery |
| [RabbitMQ]() | Used for retrieving scan profile changes, and created raw data in bytes |
### Project structure
```
$ tree -L 3 --dirsfirst
.
├── docs/ # additional documentation
├── scheduler/ # scheduler python module
│ ├── config # application settings configuration
│ ├── connectors # external service connectors
│ │ ├── listeners # channel/socket listeners
│ │ ├── services # rest api connectors
│ │ └── __init__.py
│ ├── context/ # shared application context
│ ├── models/ # internal model definitions
│ ├── queues/ # priority queue
│ ├── rankers/ # priority/score calculations
│ ├── repositories/ # data abstraction layer
│ ├── schedulers/ # schedulers
│ ├── server/ # scheduler rest api interface
│ ├── utils/ # common utility functions
│ ├── __init__.py
│ ├── __main__.py
│ ├── app.py # kat scheduler app implementation
│ └── version.py # version information
└─── tests/
├── factories/
├── integration/
├── simulation/
├── unit/
└── __init__.py
```
## Running / Developing
Typically the scheduler will be run from the overarching
[nl-kat-coordination](https://github.com/minvws/nl-kat-coordination) project. When
you want to run and the scheduler individually you can use the following setup.
We are using docker to setup our development environment, but you are free
to use whatever you want.
### Prerequisites
By the use of environment variables we load in the configuration of the
scheduler. Look at the [.env-dist](.env-dist) file for the application
configuration settings, to build a `.env` file. Refer to the
[`configuration.md`](docs/configuration.md) file for more information on the
individual settings.
### Running
```
# Build and run the scheduler in the background
$ docker-compose up --build -d scheduler
```
## Testing
```
# Run integration tests
$ make itest
# Run unit tests
$ make utest
# Individually test a file
$ make file=test_file.py utest
# Individually test a function
$ make file=test_file.py function='-k test_function' utest
```