Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/astroband/astrologer
Index Stellar historical data with ElasticSearch
https://github.com/astroband/astrologer
astrograph blockchain elasticsearch go golang stellar
Last synced: about 2 months ago
JSON representation
Index Stellar historical data with ElasticSearch
- Host: GitHub
- URL: https://github.com/astroband/astrologer
- Owner: astroband
- License: mit
- Created: 2019-03-28T15:35:23.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2020-10-16T10:28:42.000Z (almost 4 years ago)
- Last Synced: 2024-06-20T11:59:14.197Z (3 months ago)
- Topics: astrograph, blockchain, elasticsearch, go, golang, stellar
- Language: Go
- Homepage: https://astrograph.io
- Size: 344 KB
- Stars: 8
- Watchers: 3
- Forks: 1
- Open Issues: 9
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://circleci.com/gh/astroband/astrologer) [](https://goreportcard.com/report/astroband/astrologer) []((https://github.com/astroband/astrologer/blob/master/LICENSE))
# Astrologer
Exports historical data to ElasticSearch storage (some data are still WIP).
# Installation
```
go get [email protected]/astroband/astrologer
```
# Creating indexes
```
./astrologer create-index
```
Use `--force` flag to force recreate from scratch.
# Export from scratch
```
./astrologer export # Export everything
./astrologer export 23269090 # Start at ledger 23269090
./astrologer export 23269090 100 # 100 ledgers starting with 23269090
./astrologer export +1000 # Skip first 1000 ledgers
./astrologer export +1000 1000 # Skip first 1000 ledgers, limit to 1000
./astrologer export -- -1000 # Last 1000 ledgers
./astrologer export -- -1000 500 # 500 ledgers, offset -1000 from last
```
You may use starting ledger number as second argument and ledger count as third. Note that real ledger count will be related to `--batch` parameter value, eg. if you specify start 0, count 150 and batch 100, 200 ledgers will be exported.
There are also `--verbose` and `--dry-run` flags for debug purposes.
# Ingest
```
./astrologer ingest
```
Will start live ingest from lastest ledger. You may use starting ledger number as second argument or specify some starting ledger in near past (useful for deployment):
```
./astrologer ingest -- -100
```
Will start ingestion from current ledger -100
# Postman
There are some example queries (aggregations mostly) in PostMan format.
https://www.getpostman.com/downloads
See `es.postman_collection.json`
# Check cluster storage size
```curl localhost:9200/_cluster/stats?human\&pretty | more```
# Stats
Reports ledger segments existing in database.
```
./astrologer stats
+----------+----------+--------+
| MIN | MAX | COUNT |
+----------+----------+--------+
| 10 | 10 | 1 |
| 22 | 22 | 1 |
| 23268991 | 23368992 | 100002 |
+----------+----------+--------+
| TOTAL | 100004 |
+----------+----------+--------+
```
# ES Stats
Reports ledger segments existing elastic database.
```
./astrologer es-stats
```