https://github.com/mayankagrawal94/influencer-tracker
This repository contains the Proof of Concept (PoC) for an Influencer Dashboard service. The service is designed to extract and store data from the Mockstagram API, dynamically distribute the workload among multiple worker instances, and maintain a time-series of influencer follower counts with a one-minute resolution.
https://github.com/mayankagrawal94/influencer-tracker
axios load-balancer mongodb nodejs relible scaleable worker-pool worker-service
Last synced: 8 months ago
JSON representation
This repository contains the Proof of Concept (PoC) for an Influencer Dashboard service. The service is designed to extract and store data from the Mockstagram API, dynamically distribute the workload among multiple worker instances, and maintain a time-series of influencer follower counts with a one-minute resolution.
- Host: GitHub
- URL: https://github.com/mayankagrawal94/influencer-tracker
- Owner: MayankAgrawal94
- Created: 2024-05-27T11:37:56.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-06-12T14:47:58.000Z (almost 2 years ago)
- Last Synced: 2025-07-25T22:44:42.769Z (10 months ago)
- Topics: axios, load-balancer, mongodb, nodejs, relible, scaleable, worker-pool, worker-service
- Language: JavaScript
- Homepage:
- Size: 38.1 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Influencer Tracker PoC
## Overview
This project is a Proof of Concept (PoC) for an Influencer Dashboard that extracts and stores data from the Mockstagram API. The system is designed to be horizontally scalable, allowing for dynamic load distribution among multiple worker instances.
## Features
- Automatic load distribution among workers.
- Dynamic range calculation for workers based on the number of active workers.
- Periodic update of influencer data with one-minute resolution timeline.
- Robust handling of worker registration and deregistration.
## Technology Stack
- Node.js
- Express.js
- MongoDB
- Axios
## Setup Instructions
### Prerequisites
- Node.js installed on your machine.
- MongoDB installed and running.
### Installation
1. **Clone the Repository**
```bash
git clone https://github.com/MayankAgrawal94/Influencer-Tracker.git
cd Influencer-Tracker/Tracker
```
2. **Install Dependencies**
```bash
npm install
```
3. **Setup Environment Variables**
Create a `.env` file in the root directory of 'Tracker' and add the following environment variables:
```
1.DB_CONNECTION_STRING=mongodb://localhost:27017
2.DB_NAME=influencerDB
3.BASE_URL=http://localhost:3000 /*(Mockstagram Api Base Url) **/
```
4. **Start the Mockstagram API**
Follow the instructions in the Mockstagram API repository to start the server locally.
5. **Run Worker Instances**
Open multiple terminal windows to run worker instances. Each instance will automatically register itself and distribute the load.
```bash
node worker.js
```
## Project Structure
```
.
├── app
│ ├── models
│ │ ├── Influencer.model.js
│ │ └── Worker.model.js
│ └── services
│ ├── Influencer.service.js
│ └── Worker.service.js
├── config
│ ├── db.js
│ └── env.constant.js
├── package.json
├── package-lock.json
├── README.md
└── worker.js
```
## How It Works
1. **Worker Registration and Heartbeat**
- Each worker registers itself in the MongoDB `Worker` collection with a unique `instanceId` and `workerId`.
- Workers periodically update their timestamp to indicate they are alive.
2. **Dynamic Range Calculation**
- Workers calculate the range of `pk` values they are responsible for based on the number of active workers.
- The range is recalculated whenever a worker joins or leaves.
3. **Periodic Data Update**
- Workers fetch influencer data from the Mockstagram API every minute and update the MongoDB `Influencer` collection.
- The follower count timeline and average follower count are updated with each new data point.
4. **Graceful Shutdown**
- Workers deregister themselves from the MongoDB collection upon receiving shutdown signals (SIGINT, SIGTERM).
## Contributing
If you have any suggestions or improvements, feel free to submit a pull request or open an issue.
## Contact
If you have any feedback, questions, or suggestions, feel free to reach out.
connect me at `jobs@mayankagrawal.co.in` or can DM me on [LinkedIn](https://www.linkedin.com/in/mayank-agrawal-59192940/).