https://github.com/pykit3/k3logcollector
Scan log files on local machine, collector all interested logs, and send to somewhere for display.
https://github.com/pykit3/k3logcollector
log python
Last synced: 4 months ago
JSON representation
Scan log files on local machine, collector all interested logs, and send to somewhere for display.
- Host: GitHub
- URL: https://github.com/pykit3/k3logcollector
- Owner: pykit3
- License: mit
- Created: 2021-08-17T07:20:45.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2025-08-28T12:37:18.000Z (9 months ago)
- Last Synced: 2025-10-25T14:34:50.468Z (7 months ago)
- Topics: log, python
- Language: Python
- Homepage: https://blog.openacid.com
- Size: 36.1 KB
- Stars: 0
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# k3logcollector
[](https://github.com/pykit3/k3logcollector/actions/workflows/python-package.yml)
[](https://k3logcollector.readthedocs.io/en/stable/?badge=stable)
[](https://pypi.org/project/k3logcollector)
Scan log files on local machine, collector all interested logs, and send to somewhere for display.
k3logcollector is a component of [pykit3] project: a python3 toolkit set.
# Name
logcollector
Scan log files on local machine, collector all interested logs, and send
to somewhere for display.
# Description
We may want to see all error logs on all machines, so we need to collector
logs, and save it in somewhere. This module is used to collector logs on one
machine.
Normally, same error info will be loged repeatedly, we do not want
to save duplicated log info, so logs produced by same source file at
same line number in one second will be combined.
# Conf
configuration for log files. It is a dict, the key is the log name, the value
is the configuration for the log.
## file_path
the path of the log file.
## is_first_line
is a callback function.
The argument to this function is a line in log file, if the line is the
first line of a log, then return `True`, otherwise return `False`.
## get_level
is a callback function.
The argument to this function is the complete log string, which may contains
multiple lines. It should return the level of the log, which is a string.
## parse
is a callback function.
The argument to this function is the complete log string, which may contains
multiple lines. It should return a dict contains following fields.
- log_ts:
the timestamp of this log, such as `1523936052`, is a integer.
- level:
the level of this log, such as 'info'.
- source_file:
the source file in which the log was produced.
- line_number:
the number of the line at which the log was produced.
## level
is a list, used to specify the interested log levels.
# Install
```
pip install k3logcollector
```
# Synopsis
```python
import k3logcollector
def send_log(log_entry):
print("send the log entry to database or other place.")
def is_first_line(line):
print("return True if the line is the first line of a log,")
print("otherwise return False.")
def get_level(log_str):
print("return the level of the log.")
def parse(log_str):
print("parse the log.")
conf = {
'front_error_log': {
'file_path': 'path/to/log/file/xxx.error.log',
'level': ['warn', 'error'],
'is_first_line': is_first_line,
'get_level': get_level,
'parse': parse,
},
}
kwargs = {
'node_id': '123abc',
'node_ip': '1.2.3.4',
'send_log': send_log,
'conf': conf,
}
k3logcollector.run(**kwargs)
```
# Author
Zhang Yanpo (张炎泼)
# Copyright and License
The MIT License (MIT)
Copyright (c) 2015 Zhang Yanpo (张炎泼)
[pykit3]: https://github.com/pykit3