https://github.com/as2811-project/monolith-pipeline
A scale replica of the pipeline used for training TikTok's monolith recommender system
https://github.com/as2811-project/monolith-pipeline
etl-pipeline monolith recommender-system tiktok
Last synced: 11 months ago
JSON representation
A scale replica of the pipeline used for training TikTok's monolith recommender system
- Host: GitHub
- URL: https://github.com/as2811-project/monolith-pipeline
- Owner: as2811-project
- Created: 2024-11-19T12:37:55.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-11-25T13:19:37.000Z (over 1 year ago)
- Last Synced: 2025-05-25T15:48:14.226Z (about 1 year ago)
- Topics: etl-pipeline, monolith, recommender-system, tiktok
- Language: Jupyter Notebook
- Homepage:
- Size: 14.6 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
### Scale Replica of TikTok's Monolith Recommender System's Pipeline
I recently started reading TikTok's Monolith research paper and thought I'd replicate the ETL pipeline they built for the system. In this repo, you will find Terraform files to create an EC2 instance
and install the necessary dependencies (i.e., Kafka, Java) on launch. But you can ignore these as I've shelved IaC and AWS as a whole temporarily as I highly doubt I'll stay within the free tier limits.
You will find a notebook with some code to periodically send a row of data from the 'ratings.dat' file to the Kafka
topic. A separate Spark job runs to join the ratings data with the movies metadata (movies.dat) which is preloaded into memory.
To try this out:
* Clone the repo
* Create a `.env` file in the `spark` directory and add your Redshift credentials
* Run `docker compose up`
* Once everything's up and running, run the `payload.ipynb` notebook
* If there are no errors, you will be able to see `MicroBatchExecution` logs on your terminal
Data Source:
https://grouplens.org/datasets/movielens/10m/ (ensure this folder is in the project root)
Monolith:
https://arxiv.org/pdf/2209.07663
I might figure out a way to deploy this on AWS whilst remaining within the free tier limits later.