{"id":15097541,"url":"https://github.com/nadahamdy217/movies-data-etl-using-python-gcp","last_synced_at":"2026-01-06T21:50:45.293Z","repository":{"id":253157495,"uuid":"842646790","full_name":"nadahamdy217/Movies-Data-ETL-using-Python-GCP","owner":"nadahamdy217","description":"Developed a comprehensive ETL pipeline for movie data using Python, Docker, and a GCP Pub/Sub emulator. Successfully processed and published the data in a local Docker environment, showcasing advanced data engineering skills.","archived":false,"fork":false,"pushed_at":"2024-08-24T19:25:36.000Z","size":1231,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-01T17:44:26.655Z","etag":null,"topics":["analytics","data","data-engineering","data-ingestion","data-preparation","data-preprocessing","data-processing","data-project","docker","etl","etl-pipeline","gcp","matplotlib","matplotlib-pyplot","numpy","pandas","pubsub","python","scipy","seaborn"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nadahamdy217.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-14T19:18:03.000Z","updated_at":"2024-09-07T13:26:57.000Z","dependencies_parsed_at":"2024-08-14T21:34:38.599Z","dependency_job_id":"4b065033-1ed6-4340-976d-b53591e1912c","html_url":"https://github.com/nadahamdy217/Movies-Data-ETL-using-Python-GCP","commit_stats":{"total_commits":29,"total_committers":1,"mean_commits":29.0,"dds":0.0,"last_synced_commit":"b7aa6d30ca0b89104bb5d8227042274368e3819a"},"previous_names":["nadahamdy217/movies-data-etl-using-python-gcp"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nadahamdy217%2FMovies-Data-ETL-using-Python-GCP","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nadahamdy217%2FMovies-Data-ETL-using-Python-GCP/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nadahamdy217%2FMovies-Data-ETL-using-Python-GCP/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nadahamdy217%2FMovies-Data-ETL-using-Python-GCP/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nadahamdy217","download_url":"https://codeload.github.com/nadahamdy217/Movies-Data-ETL-using-Python-GCP/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245853769,"owners_count":20683245,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","data","data-engineering","data-ingestion","data-preparation","data-preprocessing","data-processing","data-project","docker","etl","etl-pipeline","gcp","matplotlib","matplotlib-pyplot","numpy","pandas","pubsub","python","scipy","seaborn"],"created_at":"2024-09-25T16:23:18.377Z","updated_at":"2026-01-06T21:50:45.263Z","avatar_url":"https://github.com/nadahamdy217.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🎥 ETL Movies Data Project\n\n## **🚀 Project Overview**\n\nWelcome to the ETL Movies Data Project! 🌟 This project is a deep dive into building an end-to-end ETL (Extract, Transform, Load) pipeline using Python, Docker, and a Google Cloud Pub/Sub emulator. We’re working with a dataset of movie ratings, transforming it into a format that's ready for analysis, and loading it into a Docker container for easy access and management. This project is perfect for those looking to simulate a real-world ETL process in a local environment. 💻\n\n---\n\n## **🗂 Project Structure**\n\nHere’s how our project is organized:\n\n### `ETL_MOVIES/`\n- **`data/`**                           # Where all the magic data lives! 🎩\n  - **`ratings.csv`**                 # Contains rated movie data 📊\n  - **`movies.csv`**                  # Contains information about movies 🎥\n  - **`full_data.csv`**               # The preprocessed movie data file 🗃\n\n- **`Dockerfile`**                    # Our recipe for the Docker environment 📦\n\n- **`requirements.txt`**              # All the ingredients (dependencies) 🛠\n\n- **`README.md`**                     # This very guide you’re reading! 📚\n\n- **`Scripts/`**                      # Scripts to automate various tasks 🎛\n  - **`setup_env.bat`**               # Environment setup script ⚙️\n  - **`download_data.bat`**           # Data download script ⬇️\n  - **`start_emulator.py`**           # Start the Pub/Sub emulator 🚀\n  - **`create_topic_subscription.py`** # Create Pub/Sub topic and subscription 📝\n  - **`publish_test_message.py`**     # Test data ingestion 🧪\n  - **`process_data.py`**             # Extract CSV files 📂\n  - **`preprocessing_data.py`**       # Clean up the data 🧼\n  - **`publish_data.py`**             # Publish data to the container 🚚\n\nFeel free to explore each part of the project to understand its role and how everything fits together. Happy coding! 👩‍💻👨‍💻\n\n\n## **🔧 Tools and Technologies**\n\n- **Python** 🐍: The core language for our scripts.\n- **Docker** 🐳: To containerize and run everything smoothly.\n- **Google Cloud Pub/Sub** ☁️: For simulating real-time data streaming.\n- **Pandas** 🐼: Our go-to for data manipulation.\n- **Windows Batch Scripting** 🖥: Automating the setup and downloads.\n- **Google Cloud SDK** 🌐: For interacting with Google Cloud services.\n\n---\n\n## **📋 Environment Setup**\n\nReady to get started? Here’s what you need to do:\n\n1. **Install Required Tools:**\n   - Docker Desktop\n   - Python 3.12.5\n   - Google Cloud SDK\n\n2. **Clone the Repository:**\n   - Use Git to clone the repository and dive into the project directory.\n\n3. **Install Python Dependencies:**\n   - Create a virtual environment and install dependencies with a flick of a command.\n\n---\n\n## **📚 Project Steps**\n\nHere’s a step-by-step guide to get you through the project:\n\n### **1️⃣ Setup Environment**\n\n**Goal:** Set up the project environment with all the necessary dependencies.\n\n**How:** Run the `setup_env.bat` script, and let the automation magic happen! ✨\n\n---\n\n### **2️⃣ Download Data**\n\n**Goal:** Get the movie ratings data.\n\n**How:** Simply run `download_data.bat`, and the data will be at your service! 📥\n\n---\n\n### **3️⃣ Set Up Pub/Sub Emulator**\n\n**Goal:** Simulate the Pub/Sub environment locally.\n\n**How:** Kickstart the emulator with `start_emulator.py`. 🚀\n\n---\n\n### **4️⃣ Build Docker Image**\n\n**Goal:** Package everything into a Docker image.\n\n**How:** Build the image using Docker, and watch it come to life! 🛠\n\n---\n\n### **5️⃣ Run Docker Container**\n\n**Goal:** Spin up the Docker container.\n\n**How:** Use the Docker run command, and let the container do its thing. 🏃‍♂️\n\n---\n\n### **6️⃣ Create Pub/Sub Topic and Subscription**\n\n**Goal:** Create a Pub/Sub topic and subscription.\n\n**How:** Execute `create_topic_subscription.py`, and set the stage for data flow. 🌐\n\n---\n\n### **7️⃣ Test Data Ingestion**\n\n**Goal:** Ensure data ingestion works smoothly.\n\n**How:** Run `publish_test_message.py` and see the messages flow! 🎯\n\n---\n\n### **8️⃣ Extract CSV Files**\n\n**Goal:** Extract and prepare the data.\n\n**How:** Run `process_data.py` and get your CSVs ready for action! 📑\n\n---\n\n### **9️⃣ Preprocess Data**\n\n**Goal:** Clean and prep the data.\n\n**How:** Execute `preprocessing_data.py`, and your data will be spotless! 🧼\n- inside this Python file:\n  - null values have been filled\n  - datatype correction\n  - merge data depending on the item_id\n\n---\n\n### **🔟 Create Folder in Docker Container**\n\n**Goal:** Create a place in the container for our data.\n\n**How:** Access the Docker terminal and create the `/data` folder. 📂\n\n---\n\n### **1️⃣1️⃣ Publish Data to Container**\n\n**Goal:** Send the data into the Docker container.\n\n**How:** Run `publish_data.py` and watch the data transfer! 🚚\n\n---\n\n## **💡 Understanding Pub/Sub and Its Role**\n\nGoogle Cloud Pub/Sub is all about handling data in real-time, and this project, allows us to simulate how large-scale data processing would work in the cloud. The Pub/Sub emulator lets us develop and test everything locally, so we’re ready for the real cloud when the time comes. ☁️\n\n---\n\n## **🚧 Challenges Faced**\n\n- **Handling Big Data:** Processing 100,000 rows was a challenge, but we conquered it! 💪\n- **Local Cloud Simulation:** Setting up the Pub/Sub emulator wasn’t easy, but it was worth it. 🎓\n- **Data Cleaning:** Ensuring clean and reliable data required some serious attention to detail. 🧹\n\n---\n\n## ** ⭐ Result**\n\n![image](https://github.com/user-attachments/assets/736f8cd2-bab3-4306-8e24-a3d266961f41)\n\n\n---\n\n## **🎉 Conclusion**\n\nThis project fully demonstrates how to build a robust ETL pipeline, complete with Dockerization and cloud simulations. Whether you’re here to learn or to build, this project has all the tools and guidance you need. Happy coding! 👩‍💻👨‍💻\n\n\n\n---\n\n## **📂 Repository**\n\nFind everything you need in our [GitHub repository](https://github.com/nadahamdy217/Movies-Data-ETL-using-Python-GCP/tree/main). Dive in, explore, and feel free to contribute! 🎁\n\n---\n\n## **Contributing**\n\nContributions are welcomed to this project! If you’d like to contribute or have any questions, please contact:\n\n- **Author:** Nada Hamdy Fatehy\n- **Email:** nadahamdy2172002@gmail.com\n- **LinkedIn:** [LinkedIn](https://www.linkedin.com/in/nada-hamdy-2265692a3/)\n- **GitHub:** [GitHub](https://github.com/nadahamdy217)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnadahamdy217%2Fmovies-data-etl-using-python-gcp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnadahamdy217%2Fmovies-data-etl-using-python-gcp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnadahamdy217%2Fmovies-data-etl-using-python-gcp/lists"}