Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/up9inc/basenine
Schema-free, document-oriented streaming database that optimized for monitoring network traffic in real-time
https://github.com/up9inc/basenine
database document-oriented json network streaming traffic
Last synced: about 1 month ago
JSON representation
Schema-free, document-oriented streaming database that optimized for monitoring network traffic in real-time
- Host: GitHub
- URL: https://github.com/up9inc/basenine
- Owner: up9inc
- License: apache-2.0
- Created: 2021-10-17T16:47:44.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2022-06-12T11:28:24.000Z (over 2 years ago)
- Last Synced: 2024-06-19T02:11:38.559Z (7 months ago)
- Topics: database, document-oriented, json, network, streaming, traffic
- Language: Go
- Homepage:
- Size: 467 KB
- Stars: 38
- Watchers: 4
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# Basenine
Schema-free, document-oriented streaming database that optimized for monitoring network traffic in real-time.
### Featured Aspects
- Has the fastest possible write speed.
- Has a read speed that scales linearly.
- Schema-free.
- Only allows create and read.
- Accepts JSON as the record format.
- Let's you query based on JSONPath.
- Has a rich filtering syntax for querying.
- Defines a TCP-based protocol.
- Has long lasting TCP connections.
- Watches the database and streams back the new records.
## Server
Run the server:
`make && ./basenine -port 9099`
### Protocol
The database server has these connection modes:
- **Insert mode** is a long lasting TCP connection to insert data into the `data_*.db` binary files on server's directory.
A client can elevate itself to insert mode by sending `/insert` command.
- **Insertion filter mode** is a short lasting TCP connection that lets you set an insertion filter which is executed
right before the insertion of each individual record. The default value of insertion filter is an empty string.
- **Query mode** lets you filter the records in the database based on a [filtering syntax named BFL](https://github.com/up9inc/basenine/wiki/BFL-Syntax-Reference).
Query mode streams the results to the client and is able to keep up where it left off even if the database have millions of records.
The TCP connection in this mode is long lasting as well. The filter cannot be changed without establishing a new connnection.
The server also streams the query progress through `/metadata` command to the client.
- **Single mode** is a short lasting TCP connection that returns a single record from the database based on the provided index value.
- **Fetch mode** is a short lasting TCP connection mode for fetching N number of records from the database,
starting from a certain offset, supporting both directions.
- **Validate mode** checks the query against syntax errors. Returns the error if it's syntactically invalid otherwise returns `OK`.
- **Macro mode** lets you define a macro for the query language like `http~proto.name == "http"`.
- **Limit mode** allows you to set a hard-limit for the database size in bytes like `100000000` (100MB).
The disk usage ranges between `50000000` (50MB) and `100000000` (100MB).
So the actual effective limit is the half of this value.
- **Flush mode** is a short lasting TCP connection mode that removes all the records in the database.
- **Reset mode** is a short lasting TCP connection mode that removes all the records in the database
and resets the core into its initial state.
### Query
Querying achieved through a filter syntax named **Basenine Filter Language (BFL)**. It enables the user to query the traffic logs efficiently and precisely.
```python
http and request.method == "GET" and request.path != "/example" and (request.query.a > 42 or request.headers["x"] == "y")
```
Please [see the syntax reference](https://github.com/up9inc/basenine/wiki/BFL-Syntax-Reference) for more info.
## Client
### Go
#### Insert
```go
// Establish a new connection to a Basenine server at localhost:9099
c, err := NewConnection("localhost", "9099")
if err != nil {
panic(err)
}
// Elevate to INSERT mode
c.InsertMode()
// There can be many Send and SendText calls
c.SendText(`{"brand":{"name":"Chevrolet"},"model":"Camaro","year":2019}`)
c.Send([]byte(`{"brand":{"name":"Chevrolet"},"model":"Camaro","year":2020}`))
c.SendText(`{"brand":{"name":"Chevrolet"},"model":"Camaro","year":2021}`)
// Close
c.Close()
```
#### Single
```go
// Retrieve the record with ID equals to 42 with an empty query
// The 4th argument query, is only effective in case of
// record altering helpers like `redact` are used.
// Please refer the BFL syntax reference for more info.
data, err := Single("localhost", "9099", fmt.Sprintf("%024d", 42), "")
if err != nil {
panic(err)
}
```
#### Fetch
```go
// Retrieve up to 20 records starting from offset 100, in reverse direction (-1),
// with query `brand.name == "Chevrolet"` and with a 5 seconds timeout.
// Returns a slice of records, first meta and the latest meta state.
data, firstMeta, lastMeta, err := Fetch("localhost", "9099", 100, -1 `brand.name == "Chevrolet"`, 20, 5*time.Second)
if err != nil {
panic(err)
}
```
#### Query
```go
// Establish a new connection to a Basenine server at localhost:9099
c, err := NewConnection("localhost", "9099")
if err != nil {
panic(err)
}
// Make []byte channels to recieve the data and the meta
data := make(chan []byte)
meta := make(chan []byte)
// Clean up
defer func() {
data <- []byte(CloseChannel)
meta <- []byte(CloseChannel)
c.Close()
}()
// Define a function to handle the data stream
handleDataChannel := func(wg *sync.WaitGroup, c *Connection, data chan []byte) {
defer wg.Done()
for {
bytes := <-data
if string(bytes) == CloseChannel {
return
}
// Do something with bytes
}
}
// Define a function to handle the meta stream
handleMetaChannel := func(c *Connection, meta chan []byte) {
for {
bytes := <-meta
if string(bytes) == CloseChannel {
return
}
// Do something with bytes
}
}
var wg sync.WaitGroup
go handleDataChannel(&wg, c, data)
go handleMetaChannel(c, meta)
wg.Add(1)
// The first argument can be an id, an empty string or "latest"
// to start the streaming from the latest record.
c.Query("", `brand.name == "Chevrolet"`, data, meta)
wg.Wait()
```
#### Validate
```go
err := Validate("localhost", "9099", `brand.name == "Chevrolet"`)
if err != nil {
// err should be nil, otherwise a connection error or a syntax error
}
```
#### Macro
```go
// Define a macro `chevy` expands into `brand.name == "Chevrolet"`
err := Macro("localhost", "9099", "chevy", `brand.name == "Chevrolet"`)
if err != nil {
// err can only be a connection error
}
```
#### Insertion Filter
```go
// Set the insertion filter to `brand.name == "Chevrolet" and redact("year")`
err := InsertionFilter("localhost", "9099", `brand.name == "Chevrolet" and redact("year")`)
if err != nil {
// err can only be a connection error
}
```
#### Limit
```go
// Set the database size limit to 100MB
err := Limit("localhost", "9099", 100000000)
if err != nil {
// err can only be a connection error
}
```
#### Flush
```go
// Remove all the records
err := Flush("localhost", "9099")
if err != nil {
// err can only be a connection error
}
```
#### Reset
```go
// Reset the database into its initial state
err := Reset("localhost", "9099")
if err != nil {
// err can only be a connection error
}
```