https://github.com/narayanbhat166/leetcode_datahouse

This project is to scrape leetcode database and gain useful insights
https://github.com/narayanbhat166/leetcode_datahouse

leetcode rust rust-lang

Last synced: 9 days ago
JSON representation

This project is to scrape leetcode database and gain useful insights

• Host: GitHub
• URL: https://github.com/narayanbhat166/leetcode_datahouse
• Owner: Narayanbhat166
• Created: 2022-11-01T17:20:39.000Z (about 2 years ago)
• Default Branch: main
• Last Pushed: 2024-01-09T16:56:03.000Z (10 months ago)
• Last Synced: 2024-10-11T08:17:02.309Z (25 days ago)
• Topics: leetcode, rust, rust-lang
• Language: Rust
• Homepage:
• Size: 276 KB
• Stars: 6
• Watchers: 2
• Forks: 0
• Open Issues: 3
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# The Leetcode datahouse

A highly distributed, robust, modular, scalable web data extractor. Bring your own machine to start contributing to the data.

This project is aimed to extract the data from leetcode for data analysis. A project for beginners in rust to get started to learn and contribute.

Services

- Producer: This service is what generates the data by extracting it from the web. It coordinates with the `Controller` extract the right data and push it to the controller. There can be multiple producers running at any given time. These are run on the client machines and are light weight.

- Consumer: The service which pushes the data from a temporary storage or a message queue to a persistent storage. There will be only one consumer running at any given time.

- Controller: Coordinates the producers to produce the right data. It stores the produced data in a temporary storage like a message queue or redis. 

The communication between `Controller` and `Producer` happens by grpc.

# Architecture

![alt architecture diagram][diagram]

[diagram]: https://github.com/Narayanbhat166/leetcode_datahouse/assets/48803246/d0224a74-25a1-49df-b4fa-8fb37d6e8cc1