https://github.com/marshalx/golang-test-task
My the first project in Go
https://github.com/marshalx/golang-test-task
analytics clickhouse go golang high-performance
Last synced: 6 months ago
JSON representation
My the first project in Go
- Host: GitHub
- URL: https://github.com/marshalx/golang-test-task
- Owner: MarshalX
- License: mit
- Created: 2023-02-20T08:39:40.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-02-20T09:28:14.000Z (over 2 years ago)
- Last Synced: 2024-10-19T07:15:35.629Z (12 months ago)
- Topics: analytics, clickhouse, go, golang, high-performance
- Language: Go
- Homepage:
- Size: 7.81 KB
- Stars: 0
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## Golang test task
Vacancy: backend developer of high load infrastructure.
### Task
Write a high-performance server that will accept a batch of events, enrich it with data, and write to ClickHouse.
Average number of events per batch: 30. Minimal RPS: 200. Events are separated by `\n` symbol in the batch. Data to
enrich: client IP, server time.
Example of **formatted** payload:
```json
{
"client_time": "2022-12-01 23:59:00",
"device_id": "8273D9AA-4ADF-4B37-A60F-3E9E645C821E",
"device_os": "iOS 16.1.0",
"session": "bmwRi8mAUypxjpaH",
"sequence": 1,
"event": "app_start",
"param_int": 0,
"param_str": "some text"
}
```
### Solution
Endpoint: POST `/submit`.
> We recommend inserting data in packets of at least 1000 rows, or no more than a single request per second.
Source: https://clickhouse.com/docs/en/about-us/performance/#performance-when-inserting-data
This recommendation is one of the reasons why `inMemoryStorage` was created.
The library _chconn_ (ClickHouse low-level Driver) was picked for high-performance bulk inserts.
[Benchmarks](https://github.com/vahid-sohrabloo/chconn#benchmarks)
`zap.Logger` is used for logs instead of `zap.SugaredLogger` to get even faster logging and fewer allocations.
Run ClickHouse with default configuration without password and expose ports:
```shell
docker run -d -p8123:8123 -p9000:9000 --name analytics-clickhouse-server --ulimit nofile=262144:262144 clickhouse/clickhouse-server
```
Curl for local tests:
```shell
curl --request POST \
--url http://localhost:3000/submit \
--header 'Content-Type: application/json' \
--data '{"client_time": "2022-12-01 23:59:00", "device_id": "8273D9AA-4ADF-4B37-A60F-3E9E645C821E", "device_os": "iOS 16.1.0","session": "bmwRi8mAUypxjpaH", "sequence": 1, "event": "app_start", "param_int": 0, "param_str": "some text"}'
```