{"id":22841075,"url":"https://github.com/up9inc/basenine","last_synced_at":"2025-09-12T03:13:58.541Z","repository":{"id":38328864,"uuid":"418194955","full_name":"up9inc/basenine","owner":"up9inc","description":"Schema-free, document-oriented streaming database that optimized for monitoring network traffic in real-time","archived":false,"fork":false,"pushed_at":"2022-06-12T11:28:24.000Z","size":478,"stargazers_count":39,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-28T11:15:00.168Z","etag":null,"topics":["database","document-oriented","json","network","streaming","traffic"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/up9inc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-10-17T16:47:44.000Z","updated_at":"2025-04-14T20:52:40.000Z","dependencies_parsed_at":"2022-09-17T04:30:32.951Z","dependency_job_id":null,"html_url":"https://github.com/up9inc/basenine","commit_stats":null,"previous_names":[],"tags_count":65,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/up9inc%2Fbasenine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/up9inc%2Fbasenine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/up9inc%2Fbasenine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/up9inc%2Fbasenine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/up9inc","download_url":"https://codeload.github.com/up9inc/basenine/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251302782,"owners_count":21567601,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","document-oriented","json","network","streaming","traffic"],"created_at":"2024-12-13T01:14:43.666Z","updated_at":"2025-04-28T11:15:15.983Z","avatar_url":"https://github.com/up9inc.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Basenine\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/up9inc/basenine/releases/latest\"\u003e\n        \u003cimg alt=\"GitHub Latest Release\" src=\"https://img.shields.io/github/v/release/up9inc/basenine?logo=GitHub\u0026style=flat-square\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/up9inc/basenine/blob/master/LICENSE\"\u003e\n        \u003cimg alt=\"GitHub License\" src=\"https://img.shields.io/github/license/up9inc/basenine?logo=GitHub\u0026style=flat-square\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/up9inc/basenine/actions?query=workflow%3ATest\"\u003e\n        \u003cimg alt=\"GitHub Workflow Tests\" src=\"https://img.shields.io/github/workflow/status/up9inc/basenine/Test?logo=GitHub\u0026label=tests\u0026style=flat-square\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://codecov.io/gh/up9inc/basenine\"\u003e\n        \u003cimg alt=\"Code Coverage (Codecov)\" src=\"https://img.shields.io/codecov/c/github/up9inc/basenine?logo=Codecov\u0026style=flat-square\"\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\nSchema-free, document-oriented streaming database that optimized for monitoring network traffic in real-time.\n\n### Featured Aspects\n\n- Has the fastest possible write speed.\n- Has a read speed that scales linearly.\n- Schema-free.\n- Only allows create and read.\n- Accepts JSON as the record format.\n- Let's you query based on JSONPath.\n- Has a rich filtering syntax for querying.\n- Defines a TCP-based protocol.\n- Has long lasting TCP connections.\n- Watches the database and streams back the new records.\n\n## Server\n\nRun the server:\n\n`make \u0026\u0026 ./basenine -port 9099`\n\n### Protocol\n\nThe database server has these connection modes:\n\n- **Insert mode** is a long lasting TCP connection to insert data into the `data_*.db` binary files on server's directory.\nA client can elevate itself to insert mode by sending `/insert` command.\n\n- **Insertion filter mode** is a short lasting TCP connection that lets you set an insertion filter which is executed\nright before the insertion of each individual record. The default value of insertion filter is an empty string.\n\n- **Query mode** lets you filter the records in the database based on a [filtering syntax named BFL](https://github.com/up9inc/basenine/wiki/BFL-Syntax-Reference).\nQuery mode streams the results to the client and is able to keep up where it left off even if the database have millions of records.\nThe TCP connection in this mode is long lasting as well. The filter cannot be changed without establishing a new connnection.\nThe server also streams the query progress through `/metadata` command to the client.\n\n- **Single mode** is a short lasting TCP connection that returns a single record from the database based on the provided index value.\n\n- **Fetch mode** is a short lasting TCP connection mode for fetching N number of records from the database,\nstarting from a certain offset, supporting both directions.\n\n- **Validate mode** checks the query against syntax errors. Returns the error if it's syntactically invalid otherwise returns `OK`.\n\n- **Macro mode** lets you define a macro for the query language like `http~proto.name == \"http\"`.\n\n- **Limit mode** allows you to set a hard-limit for the database size in bytes like `100000000` (100MB).\nThe disk usage ranges between `50000000` (50MB) and `100000000` (100MB).\nSo the actual effective limit is the half of this value.\n\n- **Flush mode** is a short lasting TCP connection mode that removes all the records in the database.\n\n- **Reset mode** is a short lasting TCP connection mode that removes all the records in the database\nand resets the core into its initial state.\n\n### Query\n\nQuerying achieved through a filter syntax named **Basenine Filter Language (BFL)**. It enables the user to query the traffic logs efficiently and precisely.\n\n```python\nhttp and request.method == \"GET\" and request.path != \"/example\" and (request.query.a \u003e 42 or request.headers[\"x\"] == \"y\")\n```\n\nPlease [see the syntax reference](https://github.com/up9inc/basenine/wiki/BFL-Syntax-Reference) for more info.\n\n## Client\n\n### Go\n\n#### Insert\n\n```go\n// Establish a new connection to a Basenine server at localhost:9099\nc, err := NewConnection(\"localhost\", \"9099\")\nif err != nil {\n    panic(err)\n}\n\n// Elevate to INSERT mode\nc.InsertMode()\n\n// There can be many Send and SendText calls\nc.SendText(`{\"brand\":{\"name\":\"Chevrolet\"},\"model\":\"Camaro\",\"year\":2019}`)\nc.Send([]byte(`{\"brand\":{\"name\":\"Chevrolet\"},\"model\":\"Camaro\",\"year\":2020}`))\nc.SendText(`{\"brand\":{\"name\":\"Chevrolet\"},\"model\":\"Camaro\",\"year\":2021}`)\n\n// Close\nc.Close()\n```\n\n#### Single\n\n```go\n// Retrieve the record with ID equals to 42 with an empty query\n// The 4th argument query, is only effective in case of\n// record altering helpers like `redact` are used.\n// Please refer the BFL syntax reference for more info.\ndata, err := Single(\"localhost\", \"9099\", fmt.Sprintf(\"%024d\", 42), \"\")\nif err != nil {\n    panic(err)\n}\n```\n\n#### Fetch\n\n```go\n// Retrieve up to 20 records starting from offset 100, in reverse direction (-1),\n// with query `brand.name == \"Chevrolet\"` and with a 5 seconds timeout.\n// Returns a slice of records, first meta and the latest meta state.\ndata, firstMeta, lastMeta, err := Fetch(\"localhost\", \"9099\", 100, -1 `brand.name == \"Chevrolet\"`, 20, 5*time.Second)\nif err != nil {\n    panic(err)\n}\n```\n\n#### Query\n\n```go\n// Establish a new connection to a Basenine server at localhost:9099\nc, err := NewConnection(\"localhost\", \"9099\")\nif err != nil {\n    panic(err)\n}\n\n// Make []byte channels to recieve the data and the meta\ndata := make(chan []byte)\nmeta := make(chan []byte)\n\n// Clean up\ndefer func() {\n    data \u003c- []byte(CloseChannel)\n    meta \u003c- []byte(CloseChannel)\n    c.Close()\n}()\n\n// Define a function to handle the data stream\nhandleDataChannel := func(wg *sync.WaitGroup, c *Connection, data chan []byte) {\n    defer wg.Done()\n    for {\n        bytes := \u003c-data\n\n        if string(bytes) == CloseChannel {\n            return\n        }\n\n        // Do something with bytes\n    }\n}\n\n// Define a function to handle the meta stream\nhandleMetaChannel := func(c *Connection, meta chan []byte) {\n    for {\n        bytes := \u003c-meta\n\n        if string(bytes) == CloseChannel {\n            return\n        }\n\n        // Do something with bytes\n    }\n}\n\nvar wg sync.WaitGroup\ngo handleDataChannel(\u0026wg, c, data)\ngo handleMetaChannel(c, meta)\nwg.Add(1)\n\n// The first argument can be an id, an empty string or \"latest\"\n// to start the streaming from the latest record.\nc.Query(\"\", `brand.name == \"Chevrolet\"`, data, meta)\n\nwg.Wait()\n```\n\n#### Validate\n\n```go\nerr := Validate(\"localhost\", \"9099\", `brand.name == \"Chevrolet\"`)\nif err != nil {\n    // err should be nil, otherwise a connection error or a syntax error\n}\n```\n\n#### Macro\n\n```go\n// Define a macro `chevy` expands into `brand.name == \"Chevrolet\"`\nerr := Macro(\"localhost\", \"9099\", \"chevy\", `brand.name == \"Chevrolet\"`)\nif err != nil {\n    // err can only be a connection error\n}\n```\n\n#### Insertion Filter\n\n```go\n// Set the insertion filter to `brand.name == \"Chevrolet\" and redact(\"year\")`\nerr := InsertionFilter(\"localhost\", \"9099\", `brand.name == \"Chevrolet\" and redact(\"year\")`)\nif err != nil {\n    // err can only be a connection error\n}\n```\n\n#### Limit\n\n```go\n// Set the database size limit to 100MB\nerr := Limit(\"localhost\", \"9099\", 100000000)\nif err != nil {\n    // err can only be a connection error\n}\n```\n\n#### Flush\n\n```go\n// Remove all the records\nerr := Flush(\"localhost\", \"9099\")\nif err != nil {\n    // err can only be a connection error\n}\n```\n\n#### Reset\n\n```go\n// Reset the database into its initial state\nerr := Reset(\"localhost\", \"9099\")\nif err != nil {\n    // err can only be a connection error\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fup9inc%2Fbasenine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fup9inc%2Fbasenine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fup9inc%2Fbasenine/lists"}