Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/hitsz-ids/duetector
duetector🔍: Data Usage Extensible Detector for data usage observability.
https://github.com/hitsz-ids/duetector
bcc data-usage ebpf kata-containers observability
Last synced: about 1 month ago
JSON representation
duetector🔍: Data Usage Extensible Detector for data usage observability.
- Host: GitHub
- URL: https://github.com/hitsz-ids/duetector
- Owner: hitsz-ids
- License: apache-2.0
- Created: 2023-08-10T03:08:35.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-11-04T14:23:50.000Z (about 2 months ago)
- Last Synced: 2024-11-04T15:30:37.263Z (about 2 months ago)
- Topics: bcc, data-usage, ebpf, kata-containers, observability
- Language: Python
- Homepage: https://dataucon.idslab.io/
- Size: 1.98 MB
- Stars: 10
- Watchers: 2
- Forks: 8
- Open Issues: 12
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
duetector🔍: Data Usage Extensible detector(eBPF Support)
## Introduction
> duetector is one of the components in the DataUCON project, which is designed to provide support for data usage control. [Intro DataUCON](https://dataucon.idslab.io/).
duetector🔍 is an extensible data usage control detector that provides support for data usage control by probing for data usage behavior in the Linux kernel(based on eBPF).
**🐛🐞🧪 The project is under heavy development, looking forward to any bug reports, feature requests, pull requests!**
In the [ABAUC control model](https://github.com/hitsz-ids/dataucon), duetector can be used as a PIP (Policy Information Point) to obtain data usage behavior, so as to provide information about data usage behavior for PDP (Policy Decision Point). Provide information on data usage behavior to PDP (Policy Decision Point).
Try simple user case: [Simplest Open Count](./docs/usercases/simplest-open-count/README.md).
Join our [slack channel](https://join.slack.com/t/hitsz-ids/shared_invite/zt-2395mt6x2-dwf0j_423QkAgGvlNA5E1g).
## Table of Contents
- [Features](#Features)
- [Installation](#Installation)
- [Quick Start](#quick-start)
- [API](#API-documentation)
- [Maintainers](#Maintainers)
- [Contributors](#Contributors)
- [How to contribute](#How-to-contribute)
- [License](#License)
## Feature
- Plug-in system support, see [examples](./examples/README.md) for more details
- [x] Custom `Tracer` and `TracerManager`
- [x] Custom `Filters` and `FilterManager`
- [x] Custom `Collector` and `CollectorManager`
- [x] Custom `Analyzer` and `AnalyzerManager`
- Configuration Management
- [x] Configuration using a single configuration file
- [x] Generate Plugin Configuration
- [ ] Support for dynamically loading configurations
- `Tracer` Support
- [x] eBPF-based tracer
- [x] Shell command tracer
- [x] Subprocess tracer
- `Filter` Support
- [x] Pattern matching, based on regular expressions
- Data Collection and Analysis
- [x] `Analyzer` Support SQL database
- [x] `Collector` Support SQL database and *OpenTelemetry(Experimental)*
- User Interface
- [x] CLI Tools
- [x] PIP Service
- [ ] Control Panel
- Enhancements
- [ ] `RunC` containers identification
The eBPF program requires kernel support, see [Kernel Support](./docs/kernel_config.md)
## Installation
The code is distributed via Pypi, and you can install it with the following command
```bash
pip install duetector
```
Currently, the code relies on [BCC](https://github.com/iovisor/bcc) for on-the-fly compilation of eBPF code, we recommend [installing the latest BCC compiler](https://github.com/iovisor/bcc/blob/master/INSTALL.md)
Or use the Docker image that we provide, which uses [JupyterLab](https://github.com/jupyterlab/jupyterlab) as the **example** user application, or you can modify the [Dockerfile](./docker/Dockerfile) and [startup script](./docker/start.sh) to customize the user application.
```bash
docker pull dataucon/duetector:latest
```
Pre-releases will not be updated to `latest`, you can specify the tag to pull, e.g. `v0.0.1a`
```bash
docker pull dataucon/duetector:v0.0.1a
```
For more details on running with docker images see [here](./docs/how-to/run-with-docker.md)
## Quick start
> More documentation and examples can be found [here](./docs/).
### Start detector
Start monitor using the command line, since bcc requires root privileges, we use the `sudo` command, which will start all probes and collect the probes into the `duetector-dbcollector.sqlite3` file in the current directory
```bash
sudo duectl start
```
Press `CRTL+C` to exit monitoring and you will see a summary output on the screen
```
{'DBCollector': {'OpenTracer': {'count': 31, 'first at': 249920233249912, 'last': Tracking(tracer='OpenTracer', pid=641616, uid=1000, gid= 1000, comm='node', cwd=None, fname='SOME-FILE', timestamp=249923762308577, extended={})}}}
```
Enable `DEBUG` log
```bash
sudo DUETECTOR_LOG_LEVEL=DEBUG duectl start
```
At startup, the configuration file will be automatically generated at `~/.config/duetector`, and you can specify the configuration file to use with `--config`.
```bash
sudo duectl start --config
```
Configuration using environment variables is also supported:
```bash
Usage: duectl start [OPTIONS]
Start A bcc monitor and wait for KeyboardInterrupt
Options:
...
--load_env BOOLEAN Weather load env variables,Prefix: DUETECTOR_,
Separator:__, e.g. DUETECTOR_config__a means
config.a, default: True
...
```
When using a plugin, the default configuration file will not contain the plugin's configuration, use the dynamically-generated configuration directive to generate a configuration file with the plugin's configuration, this directive also supports merging existing configuration files and environment variables.
```bash
duectl generate-dynamic-config --help
```
Use `generate-config` to restore the default state in case of configuration file errors.
```bash
duectl generate-config
```
Going a step further, running in the background you can use the `duectl-daemon start` command, which will run a daemon in the background, which you can stop using `duectl-daemon stop`
Use `duectl-daemon --help` for more details:
```bash
Usage: duectl-daemon [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
start Start a background process of command `duectl start`.
status Show status of process.
stop Stop the process.
```
### Analyzing with analyzer
We provide an [Analyzer](https://duetector.readthedocs.io/en/latest/analyzer/index.html) that can query the data in storage, try it in [user case](./docs/usercases/simplest-open-count/README.md)
### Using duetector server
We provide a Duetector Server as an external PIP service and control interface
A Duetector Server can be started using `duectl-server` and will listen on `0.0.0.0:8120` by default, you can modify it using `--host` and `--port`.
```bash
$ duectl-server start --help
Usage: duectl-server start [OPTIONS]
Start duetector server
Options:
--config TEXT Config file path, default:
``~/.config/duetector/config.toml``.
--load_env BOOLEAN Weather load env variables, Prefix: ``DUETECTOR_``,
Separator:``__``, e.g. ``DUETECTOR_config__a`` means
``config.a``, default: True
--workdir TEXT Working directory, default: ``.``.
--host TEXT Host to listen, default: ``0.0.0.0``.
--port INTEGER Port to listen, default: ``8120``.
--workers INTEGER Number of worker processes, default: ``1``.
--help Show this message and exit.
```
After the service has started, visit `http://{ip}:{port}/docs` to see the API documentation.
Similarly, using `duectl-server-daemon start` you can run a Duetector Server in the background, and you can stop it using `duectl-server-daemon stop`
```bash
$ duectl-server-daemon
Usage: duectl-server-daemon [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
start Start a background process of command ``duectl-server start``.
status Show status of process.
stop Stop the process.
```
## API documentation
See [docs of duetector](https://duetector.readthedocs.io/)
## Maintainers
This project is initiated by **Institute of Data Security, Harbin Institute of Technology (Shen Zhen)**, if you are interested in this project and [DataUCON](https://dataucon.idslab.io/) project and willing to work together to improve it, welcome to join our open source community.
## Contributors
wh1isper
💻
MayDown
💻
tsdsnk
📖
zhemulin
📖
Mortal
📖
mingzhedream
📖
## How to contribute
Starting with the [good first issue](https://github.com/hitsz-ids/duetector/issues/70) and reading our [contributing guidelines](./CONTRIBUTING.md).
Learn about the designing and architecture of this project here: [docs/design](./docs/design/README.md).
## License
This project uses Apache-2.0 license, please refer to [LICENSE](./LICENSE).