Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/MonaxGT/parsefields

Tools for parse JSON-like logs for collecting unique fields and events
https://github.com/MonaxGT/parsefields

event fields go golang json log logging mapping schema

Last synced: 13 days ago
JSON representation

Tools for parse JSON-like logs for collecting unique fields and events

Host: GitHub
URL: https://github.com/MonaxGT/parsefields
Owner: MonaxGT
License: apache-2.0
Created: 2019-04-12T22:15:10.000Z (over 5 years ago)
Default Branch: master
Last Pushed: 2019-05-05T18:55:53.000Z (over 5 years ago)
Last Synced: 2024-07-31T20:48:38.951Z (3 months ago)
Topics: event, fields, go, golang, json, log, logging, mapping, schema
Language: Go
Homepage:
Size: 32.2 KB
Stars: 8
Watchers: 2
Forks: 1
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

awesome-go - parsefields - Tools for parse JSON-like logs for collecting unique fields and events. (Data Structures and Algorithms / Text Analysis)
awesome-go - parsefields - Tools for parse JSON-like logs for collecting unique fields and events. (Data Structures and Algorithms / Text Analysis)
awesome-go-extra - parsefields - like logs for collecting unique fields and events|6|1|0|2019-04-12T22:15:10Z|2019-05-05T18:55:53Z| (Generators / Text Analysis)

README

[![Codacy Badge](https://api.codacy.com/project/badge/Grade/dc9f7432a97a4d5296045487e1db44f1)](https://app.codacy.com/app/MonaxGT/parsefields?utm_source=github.com&utm_medium=referral&utm_content=MonaxGT/parsefields&utm_campaign=Badge_Grade_Dashboard)
[![GoDoc](https://godoc.org/github.com/MonaxGT/parsefields?status.png)](https://godoc.org/github.com/MonaxGT/parsefields)
[![Build Status](https://travis-ci.com/MonaxGT/gomalshare.svg?branch=master)](https://travis-ci.com/MonaxGT/gomalshare)
[![Go Report Card](https://goreportcard.com/badge/github.com/MonaxGT/parsefields)](https://goreportcard.com/report/github.com/MonaxGT/parsefields)
[![Say Thanks!](https://img.shields.io/badge/Say%20Thanks-!-1EAEDB.svg)](https://saythanks.io/to/MonaxGT)

# parsefields

Tool for parse JSON-like logs for collecting unique fields. Main purpose to collect JSON-data with typical events and fields it is useful when you want to create mapping schema for database and you want to reduce the risks of forgotten fields.
By default separator between to nested structs is "->", but you can change it with environment.

API consist:

```
- POST /v1/json/
- POST /v1/mjson/
- GET /v1/fileds/
- GET /v1/events/
- GET /v1/events/:logname/:eventid
- DELETE /v1/events/:logname/:eventid
- DELETE /v1/fields/:field
```

P.S. additionally info about all new events/fields will be show in stdout.

## Deploy

```sh
docker build . -t parsefield
docker run -d -p 8000:8000 parsefield
```

```sh
docker-compose -p 8000:8000 -d up
```

## Usage

### Push new log for parse

Single message per request

```sh
curl -X POST -d '{"process_name": "calc.exe", "process_path":"C:\\windows\\system32"}' 127.0.0.1:8000/v1/json/
```

Multiple message per request

```sh
curl -X POST -d '[{"process_name": "calc.exe", "process_path":"C:\\windows\\system32"},{"process_image": "calc.exe", "process_path":"C:\\windows\\system32"},{"pid":"1"}]' 127.0.0.1:8000/v1/mjson/
```

### All unique fields

```sh
curl 127.0.0.1:8000/v1/fields/
```

### All unique events

```sh
curl 127.0.0.1:8000/v1/events/
```

### Show body of event

```sh
curl 127.0.0.1:8000/v1/events/Sysmon/999
```

### Delete events, fields

```sh
curl -X DELETE 127.0.0.1:8000/v1/events/Sysmon/999 - delete events with logname Sysmon and eventId 999
curl -X DELETE 127.0.0.1:8000/v1/fields/key - delete field with name key
```