https://github.com/vrann/leaderboard
Platform to collect and analyze partners performance with the contributions and public activity
https://github.com/vrann/leaderboard
Last synced: 3 months ago
JSON representation
Platform to collect and analyze partners performance with the contributions and public activity
- Host: GitHub
- URL: https://github.com/vrann/leaderboard
- Owner: vrann
- Created: 2017-08-07T12:33:30.000Z (almost 8 years ago)
- Default Branch: master
- Last Pushed: 2017-12-18T19:34:31.000Z (over 7 years ago)
- Last Synced: 2025-01-08T01:52:48.763Z (5 months ago)
- Language: JavaScript
- Size: 16.1 MB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Partners Statistic Platform
## Architecture
System is designed using Server-less architecture approach, where no dedicated server is used to run the analytics and data storage. System uses services only: Lambda, SQS, Elasticsearch Service, ECS to run the data retrieval and analytics.
Terraform is used to deploy the system to the AWS and connect all services to each other.
Frontend is designed as a static web page with rich javascript application which loads data from the API. It is hosted in the S3 bucket.
### Webhooks
System is subscribed to all GitHub webhooks and provides and API endpoint to accept the webhooks data and store it to the Elasticsearch Service. Endpoints is implemented using AWS API Gateway which sends all data to the AWS Lambda.
Lambda functions are implemented in NodeJS. For the GitHubWebhook see backend/index.js, exports.acceptWebHook.
The Webhooks allows real-time update of the data for available teams, team members, pull requests.
### Data Sync
Despite usage of the WebHooks, sometimes data still need to be synhronised. That can happen in two cases: 1) at the very beginning, when old data still was not loaded to the repository and 2) when data was lost in transition for whatever reason. In order to address both cases system uses batch jobs scheduled on the EC2 Container Service.
CheckDataIntegrity Lambda (see backend/index.js, exports.checkIntegrity) is running on the certain intervals of time and checks is there any missing data in ElasticSearch service. It checks: count of PRs on GitHuib, count of teams on GitHub, consistency in data between PRs, teams and memebers on ElasticSerach. If any problem was found it sends message to the SQS
Separate Lambda ProcessSQSQueue (see backend/index.js, exports.scheduleBatchJobs) reads messages from the queue and schedules batch jobs on the EC2 Container Service. Batch jobs are designed as NodeJS applications running in docker containers which syncronises teams and pull requests with the ElasticSearch.
## Deployment
Deployment is implemented using Infrastructure as Code (IAC) approach using Terraform
For the deployment scripts see `/deployment`
### 0. Configure Keys
Copy `deployment/terraform.tfvars.dist` to the `deployment/terraform.tfvars` and update it with correct account id, region and GitHub token
### 1. Build Artifacts
```
cd deployment
./create-lambda-zip.sh
```
### 2. Deploy environment
```
cd deployment
terraform apply
```
Save output of the command:
```
aws_ecr_repository = 417782006694.dkr.ecr.us-east-1.amazonaws.com/github-batch-repo
webhook_invoke_url = https://xhs658ppag.execute-api.us-east-1.amazonaws.com/production
```
### 4. Build and deploy container image
```
cd backend
docker build .
docker tag
$(aws ecr get-login)
docker push
```
### 5. Install GitHub webhooks
To set up a repository webhook on GitHub, head over to the Settings page of your repository, and click on Webhooks & services. After that, click on Add webhook.
Payload url:
Content type: `application/json`
Events: Send me all
### 6. Compile and deploy Frontend
```
cd frontend
webpack
cd dist
aws s3 sync . s3://partners.magento.com/
```