Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/akhenakh/geo-benthos
Benthos plugins to transform geographic coordinates from a stream.
https://github.com/akhenakh/geo-benthos
benthos geographic gis h3 pip point-in-polygon s2 stream-processing
Last synced: 3 months ago
JSON representation
Benthos plugins to transform geographic coordinates from a stream.
- Host: GitHub
- URL: https://github.com/akhenakh/geo-benthos
- Owner: akhenakh
- License: mit
- Created: 2023-05-12T01:41:52.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-02-27T05:18:28.000Z (11 months ago)
- Last Synced: 2024-06-21T20:07:04.195Z (7 months ago)
- Topics: benthos, geographic, gis, h3, pip, point-in-polygon, s2, stream-processing
- Language: Go
- Homepage:
- Size: 436 KB
- Stars: 4
- Watchers: 2
- Forks: 0
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Geo-Benthos
[Benthos](benthos.dev/) plugins to transform geographic coordinates from a stream.
Benthos is the swiss army of stream processing: Benthos solves common data engineering tasks such as transformations, integrations, and multiplexing with declarative and unit testable configuration.
This repo contains multiple Benthos plugins as Go modules, that you can build on demand (see `cmd/geo-benthos`).
Note that the h3 plugin is using a [CGO free version](https://github.com/akhenakh/goh3).
## Get the country for a latitude and longitude
Use `country` with the following parameters: `latitude`, `longitude`.
An example `position.json`:
```js
{"id":42, "lat": 48.86, "lng": 2.34}
```
A `country.yaml` pipeline.
```yaml
input:
file:
paths: ["testdata/position.json"]
codec: all-bytes
pipeline:
threads: 1
processors:
- mapping: |
#!blobl
root = this
root.country = country(this.lat, this.lng)
output:
label: "out"
stdout:
codec: lines
```
Enrich the input with the timezone:
```sh
go build -o geo-benthos ./cmd/geo-benthos
./geo-benthos -c testdata/country.yaml
{"country":["France"],"id":42,"lat":48.86,"lng":2.34}
```
country module is using [coord2country](https://github.com/akhenakh/coord2country).
## Transform latitude and longitude into an Uber h3 cell
Use `h3` with the following parameters: `latitude`, `longitude`, `resolution`.
An example `position.json`:
```js
{"id":42, "lat": 48.86, "lng": 2.34}
```
A `h3.yaml` pipeline.
```yaml
input:
file:
paths: ["testdata/position.json"]
codec: all-bytes
pipeline:
threads: 1
processors:
- mapping: |
#!blobl
root = this
root.h3 = h3(this.lat, this.lng, 5)
output:
label: "out"
stdout:
codec: lines
```
Enrich the input with the h3 cell:
```sh
go build -o geo-benthos ./cmd/geo-benthos
./geo-benthos -c testdata/h3.yaml
{"h3":"851fb467fffffff","id":42,"lat":48.86,"lng":2.34}
```
## Transform latitude and longitude into a Google s2 cell
Use `s2` with the following parameters: `latitude`, `longitude`, `resolution`.
An example `position.json`:
```js
{"id":42, "lat": 48.86, "lng": 2.34}
```
A `s2.yaml` pipeline.
```yaml
input:
file:
paths: ["testdata/position.json"]
codec: all-bytes
pipeline:
threads: 1
processors:
- mapping: |
#!blobl
root = this
root.s2 = s2_object(this.lat, this.lng, 15)
output:
label: "out"
stdout:
codec: lines
```
Enrich the input with the s2 cell:
```sh
go build -o geo-benthos ./cmd/geo-benthos
./geo-benthos -c testdata/s2.yaml
{"id":42,"lat":48.86,"lng":2.34,"s2":"2/033303031301002"}
```
## Get the timezone for a given latitude and longitude
Use `tz` with the following parameters: `latitude`, `longitude`.
An example `position.json`:
```js
{"id":42, "lat": 48.86, "lng": 2.34}
```
A `tz.yaml` pipeline.
```yaml
input:
file:
paths: ["testdata/position.json"]
codec: all-bytes
pipeline:
threads: 1
processors:
- mapping: |
#!blobl
root = this
root.tz = tz(this.lat, this.lng)
output:
label: "out"
stdout:
codec: lines
```
Enrich the input with the timezone:
```sh
go build -o geo-benthos ./cmd/geo-benthos
./geo-benthos -c testdata/tz.yaml
{"tz":"Europe/Paris","id":42,"lat":48.86,"lng":2.34}
```
tz module is using [tzf](https://github.com/ringsaturn/tzf).
## Generate random position in a range (mainly for debug)
This is an input plugin that will generate random coordinates in your range.
```yaml
input:
randpos:
min_lat: 46.0
max_lat: 48.0
min_lng: 2.0
max_lng: 2.3
output:
label: "out"
stdout:
codec: lines
```
## Live Testing
Run this command and point your browser to http://localhost:4195/
```sh
./geo-benthos blobl server --no-open --host 0.0.0.0 --input-file ./testdata/position.json -m testdata/s2_mapping.txt
```
## TODO
- [ ] s2 shape index to perform PIP
- [ ] spatialite lookup to perform PIP
- [ ] random points in a rect
- [X] lat lng to h3
- [X] lat lng to s2
- [X] lat lng to tz
- [X] lat lng to country