https://github.com/mneedham/analysing-log-files
https://github.com/mneedham/analysing-log-files
Last synced: about 1 year ago
JSON representation
- Host: GitHub
- URL: https://github.com/mneedham/analysing-log-files
- Owner: mneedham
- Created: 2022-08-15T11:10:32.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2022-09-28T13:39:16.000Z (over 3 years ago)
- Last Synced: 2025-02-14T13:50:24.424Z (over 1 year ago)
- Language: Python
- Size: 27.3 KB
- Stars: 2
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Real-Time Analytics on HTTP Access Logs
Clone repository:
```
git clone git@github.com:mneedham/analysing-log-files.git
cd analysing-log-files
```
Launch Docker Compose:
```
docker-compose up
```
Generate log files:
```bash
python apache-fake-log-gen.py
```
```bash
faust -A app worker -l info
```
Query the access logs stream:
```bash
kcat -C -t access -b localhost:9092
```
Add Pinot Table
```bash
docker run -v $PWD/pinot/config:/config \
--network analysing-log-files_default \
apachepinot/pinot:0.11.0 \
AddTable \
-schemaFile /config/schema.json \
-tableConfigFile /config/table.json \
-controllerHost pinot-controller \
-exec
```