Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/jehiah/sortdb

Make a sorted tsv or csv datafile available over HTTP
https://github.com/jehiah/sortdb

Last synced: 9 days ago
JSON representation

Make a sorted tsv or csv datafile available over HTTP

Awesome Lists containing this project

README

        

sortdb
======

Sortdb makes a sorted tab (tsv) or comma (csv) delimitated file accessible via HTTP using mmap.

![Build Status](https://github.com/jehiah/sortdb/actions/workflows/ci.yaml/badge.svg?branch=master)
[![GitHub release](https://img.shields.io/github/release/jehiah/sortdb.svg)](https://github.com/jehiah/sortdb/releases/latest)

There is also an exported `sorteddb` Go package [![Docs](https://pkg.go.dev/badge/github.com/jehiah/sortdb.svg)](http://pkg.go.dev/github.com/jehiah/sortdb)

### Installing

Pre-built binaries for linux and darwin are available for [download](https://github.com/jehiah/sortdb/releases):

**Current Stable Release: `v1.2.0`**

### Usage

Usage of ./sortdb:
-db-file="": db file
-enable-logging=false: request logging
-field-separator="\t": field separator (eg: comma, tab, pipe)
-http-address=":8080": http address to listen on
-version=false: print version string

### API Endpoints:

* `/ping` Responds with HTTP 200 `OK`

* `/get?key=...` Response is `text/plain` with the full record that matched
(excluding the key), or a HTTP 404 if no match.

* `/mget?key=...&key=...` Response is `text/plain` with all records that match
(including the key), or an empty HTTP 200 if no matches

* `/fwmatch?key=...` Also known as prefix match. Response is `text/plain` with
the full records that have keys that start with the given key as a prefix,
or a HTTP 404 if no such records exist.

* `/range?start=...&end=...` Response is `text/plain` with the full records
that have keys lexically greater than or equal to the start key and less
than or equal to the end key, or a HTTP 404 if no such records exist. The end key
must be lexically greater than or equal to the start key.

* `/stats` Response is `application/json` with the following payload

```json
{
"total_requests": 2,
"total_seeks": 24,
"get_requests": 3,
"get_hits": 3,
"get_misses": 0,
"get_average_request": 448,
"get_95": 1323,
"get_99": 1323,
"mget_requests": 0,
"mget_hits": 0,
"mget_misses": 0,
"mget_average_request": 0,
"mget_95": 0,
"mget_99": 0,
"fwmatch_requests": 1,
"fwmatch_hits": 1,
"fwmatch_misses": 0,
"fwmatch_average_request": 10,
"fwmatch_95": 10,
"fwmatch_99": 10,
"range_requests": 2,
"range_hits": 1,
"range_misses": 1,
"range_average_request": 18,
"range_95": 24,
"range_99": 24,
"db_size": 767557632,
"db_mtime": 1435463934
}
```

* `/reload` re-mmap the db file

* `/exit` cause the current process to exit

* `/debug/pprof` the [net/http/pprof](http://golang.org/pkg/net/http/pprof/) debugging endpoints

a HUP signal will also cause sortdb to reload/remap the db file

--

### Sorting Files

The easiest way to sort an existing datafile is with the unix [sort](http://unixhelp.ed.ac.uk/CGI/man-cgi?sort) utility.

```bash
LC_COLLATE=C sort data.csv > sorted_data.csv
```

Note: The locale specified by the environment affects sort order. Set `LC_ALL=C` or `LC_COLLATE=C` to get the traditional sort order that uses native byte values.

--

Sortdb was originally developed by [@jayridge](https://github.com/jayridge) as part of the [simplehttp project](https://github.com/bitly/simplehttp/tree/master/sortdb) and was ported to Go by [Jehiah Czebotar](https://jehiah.cz/)