https://github.com/kevinmichaelchen/kronos
Analytics / Events stored in Google Cloud Bigtable
https://github.com/kevinmichaelchen/kronos
Last synced: 3 months ago
JSON representation
Analytics / Events stored in Google Cloud Bigtable
- Host: GitHub
- URL: https://github.com/kevinmichaelchen/kronos
- Owner: kevinmichaelchen
- Created: 2019-05-31T21:24:19.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2019-09-20T14:21:52.000Z (over 5 years ago)
- Last Synced: 2025-02-23T22:43:39.556Z (3 months ago)
- Language: Go
- Homepage:
- Size: 89.8 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# kronos
This project provides a gRPC API for analytics.
## FAQ
### Where does the name come from?
Its name is a cooler misspelling of the [Greek god of time](https://en.wikipedia.org/wiki/Chronos).
### Why might we use this?
- Analytics are a big part of our workflow.
- We have a [JIRA ticket](https://irisvr.atlassian.net/browse/PROS-441) for implementing events on the Quest.
- Segment is for customer data.
- Bigtable is probably cheaper than a 3rd party offering.
### Why Bigtable?
- [Bigtable](https://en.wikipedia.org/wiki/Bigtable) is a sparsely populated table
that can scale to billions of rows, supports high read and write throughput at low latency,
and is an ideal source for MapReduce operations.
- Using [Cloud Bigtable](https://cloud.google.com/bigtable/docs/overview) means Google handles
upgrades, restarts, durability, auto-scaling, and cross-regional replication for us.
- Bigtable pairs well with Dataflow to form a [streaming analytics pipeline](https://cloud.google.com/solutions/big-data/stream-analytics/).
### What queries have been tested so far?
### Concerns
- How do we stop malicious actors from sending fabricated events? We'd probably use an API key similar to what Segment does.
- How do we migrate existing data from Segment into Cloud Bigtable?
- How do we port old Prospect builds to forward analytics events to this service instead of Segment?
## Running
### Start the emulator
We use the [gcloud emulator](https://cloud.google.com/sdk/gcloud/reference/beta/emulators/bigtable/)
to run locally.
```
# In one tab
gcloud beta emulators bigtable start
```
### Run tests
```
make testv
```
### Run the app
```
make
```
### Hitting the gRPC API
#### Sending a login event
```
grpcurl -v -plaintext \
-d '{"userID": "f78002f4-873d-4e79-bf13-0453c4951312", "timeMs": 1559520445749, "properties": {"a": "b"}}' \
:8080 proto.EventService/SendLoginEvent
grpcurl -v -plaintext \
-d '{"userID": "f78002f4-873d-4e79-bf13-0453c4951312", "timeMs": 1559520445750, "properties": {"a": "b"}}' \
:8080 proto.EventService/SendLoginEvent
```
#### Counting logins for user
```
grpcurl -v -plaintext \
-d '{"userID": "f78002f4-873d-4e79-bf13-0453c4951312", "start": 1559520445749, "end": 1559520640198}' \
:8080 proto.EventService/GetNumberOfLogins
```
#### Reading all events (debugging only)
```
grpcurl -v -plaintext -d '{}' :8080 proto.EventService/ReadEvents
```