https://github.com/theasp/lapidary
Search engine for logs stored in PostgreSQL
https://github.com/theasp/lapidary
clojurescript fluentd logging macchiato postgres postgresql
Last synced: 3 months ago
JSON representation
Search engine for logs stored in PostgreSQL
- Host: GitHub
- URL: https://github.com/theasp/lapidary
- Owner: theasp
- License: gpl-3.0
- Created: 2018-03-14T01:35:38.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2019-02-07T15:01:28.000Z (over 6 years ago)
- Last Synced: 2025-01-07T19:42:07.489Z (5 months ago)
- Topics: clojurescript, fluentd, logging, macchiato, postgres, postgresql
- Language: Clojure
- Size: 570 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Lapidary
A search engine for structured data using Postgres, with a similar syntax to Apache Lucene. This is intended for small to medium sized centralized logging configurations.
Pull requests welcome, especially documentation!
[](https://circleci.com/gh/theasp/lapidary)
## Features
* Data is stored as JSON in PostgreSQL
* Lucene like search syntax
* Full text search
## Screenshots
### Tables
### Query
## Installation
### Docker
Using the [docker-compose.yml](docker-compose.yml) file from the repository, run:
```
docker-compose up
```
This will start PostgreSQL, Lapidary and fluentd. You can log into Lapidary at http://localhost:8081 using the username admin and the password "ChangeMe!". Fluentd is listening on 24230 for forward connections, and 5145 for syslog.
You can log a message using:
```
logger -n localhost -P 5145 "Hello Lapidary!"
```
## Configuration
## Reference
### Query String
Querying the database is loosely based on Lucene syntax, though grouping is done in a more lisp like style.
The query consists fields to search, and the values to search for. If your value contains whitespace, you can wrap it in quotation marks. For example, to find all records which contain the word `fail` in the `message` field, you could use:
```
message:fail
```
By default, each field you search is joined together logically using `or`, so the following will find records which have `fail` or `ssh` in the `message field:
```
message:fail message:ssh
```
This could also be written using a list of acceptable values:
```
message:(fail ssh)
```
You can build more complex searches using `and` and `or`, with `or` being the default, so that the above search will logically expand to be:
```
(or message:fail message:ssh)
```
If you were looking for records that match both `fail` and `ssh`:
```
(and message:fail message:ssh)
```
A term can be negated by prepending it with `!`:, so to find records with `fail` and `ssh`, but not `mysql` in `message`, you could use:
```
(and message:fail message:ssh !message:mysql)
```
You can specify a comparison operator, using one of `<`, `<=`, `>`, `>=`, `=`, `!=`. For example, to find records with `OriginStatus` between 400 and 499, you could use:
```
(and OriginStatus:>=400 OriginStatus:<=499)
```
For `=` and `!=` the value will be matched exactly. For example, to match records with a `RequestMethod` of `POST` but not `POSTAL`, you can use:
```
RequestMethod:=POST
```
Addtionally, there are regex operators of `~` for a case sensitve regex, `~*` for a case insensntive regex, and the negative form of both using `!~` or `!~*`. If you want to match the regex `/sockjs/.*/websocket$` in a `RequestPath` field, you could use:
```
RequestPath:~"sockjs/.*/websocket$"
```
The comparison operators can also be used in a list:
```
RequestMethod:(=POST =HEAD)
```
As an alternative to using `and`, you can use a range of values with `[]` used to signify both ends of the range being inclusive, `{}` for exclusive, and `[}` or `{]` as a mix of both. The values are seperated using either `..` or ` to `. For `OriginStatus` between 400 and 499, you could use:
```
OriginStatus:[400 to 499]
```
Or, for the same effect:
```
OriginStatus:[400..500}
```
### API
TODO: Currently not available
## Project Roadmap