{"id":22332539,"url":"https://github.com/theoddysey/yts-pipeline-postgres","last_synced_at":"2026-04-11T13:10:16.533Z","repository":{"id":241620250,"uuid":"804319153","full_name":"TheODDYSEY/YTS-Pipeline-Postgres","owner":"TheODDYSEY","description":"ETL pipeline  🪈 for scraping, transforming, and loading YTS movie data 🎞️ into PostgreSQL 🛢️ Container using Docker🐳","archived":false,"fork":false,"pushed_at":"2024-05-28T18:03:46.000Z","size":68,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-31T08:43:30.126Z","etag":null,"topics":["bs4-requests","docker","docker-compose","etl-pipeline","pandas","pipeline-dock-tech","postresql","python3","scraping-web","yaml"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TheODDYSEY.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-22T11:23:26.000Z","updated_at":"2024-05-28T18:07:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"01961f6a-f115-42a6-8faf-ce5d41b2cabd","html_url":"https://github.com/TheODDYSEY/YTS-Pipeline-Postgres","commit_stats":null,"previous_names":["theoddysey/yts-pipeline-postgres"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheODDYSEY%2FYTS-Pipeline-Postgres","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheODDYSEY%2FYTS-Pipeline-Postgres/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheODDYSEY%2FYTS-Pipeline-Postgres/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheODDYSEY%2FYTS-Pipeline-Postgres/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TheODDYSEY","download_url":"https://codeload.github.com/TheODDYSEY/YTS-Pipeline-Postgres/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245605907,"owners_count":20643068,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bs4-requests","docker","docker-compose","etl-pipeline","pandas","pipeline-dock-tech","postresql","python3","scraping-web","yaml"],"created_at":"2024-12-04T04:18:37.913Z","updated_at":"2025-12-30T23:24:49.388Z","avatar_url":"https://github.com/TheODDYSEY.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# YTS MovieData ETL Pipeline\n\n![Python](https://img.shields.io/badge/Python-3.8-blue)\n![PostgreSQL](https://img.shields.io/badge/PostgreSQL-16.2-blue)\n![Docker](https://img.shields.io/badge/Docker-20.10-blue)\n![Pandas](https://img.shields.io/badge/Pandas-1.3.3-blue)\n![BeautifulSoup](https://img.shields.io/badge/BeautifulSoup-4.10.0-blue)\n\n\n## Overview\n\nThe **MovieData ETL Pipeline** project is a comprehensive solution for extracting, transforming, and loading movie data from web sources into a PostgreSQL database. This project is designed with scalability and efficiency in mind, leveraging Docker for containerization, Python for scripting, and PostgreSQL for data storage.\n\n## Project Workflow\n\n\n\n## Table of Contents\n\n- [Project Structure](#project-structure)\n- [Prerequisites](#prerequisites)\n- [Setup and Installation](#setup-and-installation)\n- [Usage](#usage)\n- [ETL Process Details](#etl-process-details)\n- [Configuration](#configuration)\n- [Future Improvements](#future-improvements)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Project Structure\n\n```plaintext\nMovieData-ETL\n├── elt\n│   ├── Dockerfile\n│   ├── elt_pipeline.py\n│   └── requirements.txt\n├── source_db\n│   ├── Dockerfile\n│   ├── create_movies.sql\n│   └── insert_movies.sql\n├── worker.py\n├── output.csv\n├── output.xlsx\n├── report.md\n├── yts.txt\n├── docker-compose.yaml\n└── README.md\n```\n\n### `elt` Directory\n\n- **Dockerfile**: Defines the environment for running the ETL script.\n- **elt_pipeline.py**: Contains the ETL logic to load data into PostgreSQL.\n- **requirements.txt**: Lists the Python dependencies.\n\n### `source_db` Directory\n\n- **Dockerfile**: Defines the environment for the PostgreSQL database.\n- **create_movies.sql**: SQL script to create the movies table.\n- **insert_movies.sql**: SQL script to insert movie data.\n\n### Root Directory\n\n- **worker.py**: Script to scrape movie data from the web and generate SQL insert statements.\n- **output.csv**: CSV file containing scraped movie data.\n- **output.xlsx**: Excel file containing scraped movie data.\n- **report.md**: Markdown file for logging the ETL process.\n- **yts.txt**: Raw HTML content of the scraped web page.\n- **docker-compose.yaml**: Docker Compose configuration to orchestrate the ETL pipeline.\n\n## Prerequisites\n\nEnsure you have the following installed:\n\n- [Docker](https://docs.docker.com/get-docker/): For containerization.\n- [Docker Compose](https://docs.docker.com/compose/install/): To manage multi-container applications.\n- [Python 3.8](https://www.python.org/downloads/): For running the worker script.\n\n## Setup and Installation\n\n1. **Clone the Repository:**\n\n   ```bash\n   git clone https://github.com/TheODDYSEY/YTS-Pipeline-Postgres\n   cd MovieData-ETL\n   ```\n\n2. **Build and Run the Docker Containers:**\n\n   ```bash\n   docker-compose up --build\n   ```\n\n   This command builds the Docker images and starts the containers for both the PostgreSQL database and the ETL script.\n\n## Usage\n\n1. **Scrape Movie Data:**\n\n   The `worker.py` script scrapes the latest movie data from YTS and generates SQL insert statements.\n\n   ```bash\n   python worker.py\n   ```\n\n   This will generate `output.csv`, `output.xlsx`, and `insert_movies.sql` files with the scraped movie data.\n\n2. **Run the ETL Pipeline:**\n\n   The `elt_pipeline.py` script will wait for the PostgreSQL database to be ready, then load the movie data from the SQL script into the database.\n\n   ```bash\n   docker-compose up elt_script\n   ```\n\n## ETL Process Details\n\n### Extract\n\nThe extraction step involves scraping movie data from the [YTS website](https://yts.mx/browse-movies/0/all/all/0/featured/0/all) using the `worker.py` script. BeautifulSoup is used to parse the HTML and extract relevant movie information.\n\n### Transform\n\nThe extracted data is transformed into a structured format using Pandas. The data is saved into CSV and Excel files for reference and further processing.\n\n### Load\n\nThe transformed data is loaded into the PostgreSQL database. The `elt_pipeline.py` script uses the `psycopg2` library to connect to the database and load the data using SQL insert statements generated by the `worker.py` script.\n\n### Script Breakdown\n\n#### `worker.py`\n\n- **Step 1**: Sends a GET request to the specified URL to fetch movie data.\n- **Step 2**: Parses the HTML content using BeautifulSoup.\n- **Step 3**: Extracts movie details and stores them in a list.\n- **Step 4**: Creates a Pandas DataFrame from the extracted data.\n- **Step 5**: Saves the DataFrame to CSV and Excel files.\n- **Step 6**: Generates SQL insert statements and saves them to a `.sql` file.\n\n#### `elt_pipeline.py`\n\n- **Database Connection**: Connects to the PostgreSQL database using `psycopg2`.\n- **Data Loading**: Loads the movie data from the SQL script into the database.\n- **Retry Logic**: Implements retry logic to handle transient connection issues.\n\n## Configuration\n\nThe PostgreSQL database configuration and credentials are set in the `docker-compose.yaml` file and passed as environment variables to the Docker containers. Ensure the following values are correctly set:\n\n```yaml\nservices:\n  movies_postgres:\n    environment:\n      POSTGRES_DB: movies_db\n      POSTGRES_USER: postgres\n      POSTGRES_PASSWORD: secret\n```\n\n## Future Improvements\n\n- **Automated Scheduling**: Integrate a scheduler like `cron` or `Airflow` to automate the ETL process at regular intervals.\n- **Data Validation**: Implement data validation and cleaning steps to ensure data quality.\n- **Enhanced Logging**: Add more detailed logging to track the ETL process and handle errors more effectively.\n\n## Contributing\n\nContributions are welcome! Please read the [contributing guidelines](CONTRIBUTING.md) for more details on how to get started.\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n\n---\n\nFor any inquiries or support, please open an issue on our [GitHub repository](https://github.com/TheODDYSEY/YTS-Pipeline-Postgres/issues).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftheoddysey%2Fyts-pipeline-postgres","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftheoddysey%2Fyts-pipeline-postgres","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftheoddysey%2Fyts-pipeline-postgres/lists"}