{"id":20433349,"url":"https://github.com/jpcadena/car-sales-etl","last_synced_at":"2026-05-04T00:33:14.519Z","repository":{"id":68396372,"uuid":"598660403","full_name":"jpcadena/car-sales-etl","owner":"jpcadena","description":"ETL process for a Car Sales project.","archived":false,"fork":false,"pushed_at":"2023-02-28T21:18:57.000Z","size":1366,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-15T18:47:07.306Z","etag":null,"topics":["asyncpg","car-sales","data-analysis","data-engineering","data-visualization","database","etl","etl-pipeline","postgresql","python","sqlalchemy"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jpcadena.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-07T15:04:52.000Z","updated_at":"2023-07-20T09:46:02.000Z","dependencies_parsed_at":null,"dependency_job_id":"ff38ea7e-aebe-406c-9b62-ff360e6884cf","html_url":"https://github.com/jpcadena/car-sales-etl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jpcadena%2Fcar-sales-etl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jpcadena%2Fcar-sales-etl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jpcadena%2Fcar-sales-etl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jpcadena%2Fcar-sales-etl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jpcadena","download_url":"https://codeload.github.com/jpcadena/car-sales-etl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241975826,"owners_count":20051536,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asyncpg","car-sales","data-analysis","data-engineering","data-visualization","database","etl","etl-pipeline","postgresql","python","sqlalchemy"],"created_at":"2024-11-15T08:18:43.818Z","updated_at":"2025-10-29T16:48:38.906Z","avatar_url":"https://github.com/jpcadena.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# car-sales-etl\n\n\u003c!-- Improved compatibility of back to top link: See: https://github.com/othneildrew/Best-README-Template/pull/73 --\u003e\n\u003ca name=\"readme-top\"\u003e\u003c/a\u003e\n\n\u003c!-- PROJECT SHIELDS --\u003e\n\u003c!--\n*** Markdown \"reference style\" links for readability.\n*** Reference links are enclosed in brackets [ ] instead of parentheses ( ).\n--\u003e\n\n\n\u003c!-- PROJECT LOGO --\u003e\n\u003cbr /\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://github.com/othneildrew/Best-README-Template\"\u003e\n    \u003cimg src=\"assets/static/logo.png\" alt=\"Logo\" width=\"80\" height=\"80\"\u003e\n  \u003c/a\u003e\n\n\u003ch3 align=\"center\"\u003ecar-sales-etl\u003c/h3\u003e\n\n  \u003cp align=\"center\"\u003e\n    Description for car-sales-etl project\n    \u003cbr /\u003e\n    \u003ca href=\"https://github.com/jpcadena/car-sales-etl\"\u003e\u003cstrong\u003eExplore the docs »\u003c/strong\u003e\u003c/a\u003e\n    \u003cbr /\u003e\n  \u003c/p\u003e\n\u003c/div\u003e\n\n\n\n\u003c!-- TABLE OF CONTENTS --\u003e\n\u003cdetails\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\n      \u003ca href=\"#about-the-project\"\u003eAbout The Project\u003c/a\u003e\n      \u003cul\u003e\n        \u003cli\u003e\u003ca href=\"#built-with\"\u003eBuilt With\u003c/a\u003e\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#getting-started\"\u003eGetting Started\u003c/a\u003e\n      \u003cul\u003e\n        \u003cli\u003e\u003ca href=\"#prerequisites\"\u003ePrerequisites\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#installation\"\u003eInstallation\u003c/a\u003e\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#usage\"\u003eUsage\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#contributing\"\u003eContributing\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#license\"\u003eLicense\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#contact\"\u003eContact\u003c/a\u003e\u003c/li\u003e\n  \u003c/ol\u003e\n\u003c/details\u003e\n\n\n\n\u003c!-- ABOUT THE PROJECT --\u003e\n\n## About the project\n\n[![Project][project-screenshot]](https://example.com)\n\nThis project is about building a data pipeline to extract, transform, and\nload (ETL) data from a source to a target. The data source is a CSV file\ncontaining information about car sales. The target is a PostgreSQL database\ntable.\n\nPostgreSQL was preferred for the richer data handling with multiple data types,\ntransaction management and its highly scalability to provide great performance\nat CRUD operations.\\\nThe project followed SQLAlchemy models scheme based on OOP concepts that\nprovide an excellent abstraction when working with multiple datasets for a\nfuture process. This high level abstraction provides a greater control over the\ndata being inserted as the table structure can be defined with multiple\nconstraints and relationships.\\\nFor more advanced requirements, transactions, migrations and more complex\noperations can be performed through the ORM so managing large amounts of data\nwon't be an issue.\\\nThe project also works with PEP8 style that is tested with Pylint and this\nincludes type hinting for variables, functions arguments and more.\n\nIf performance is critical, consider using Python 3.11 in terms of handling\nexceptions that can be thrown and re-raised in shorter execution times.\\\nAssets are also included with future consideration for HTML and CSS files.\\\nTesting could be done using unittests (to be implemented in a future release).\n\n### Transformations\n\n- Remove any rows with missing values.\n- Convert the date columns to a standard format.\n- Create a new column to store the year of the sale.\n- Replace the categorical values in the \"Car Model\" column with numerical\n  values.\n\n### Requirements\n\n- The target database should be either PostgreSQL or MySQL.\n- The pipeline should be runnable using a command-line interface.\n- The pipeline should have error handling and logging capabilities.\n- The pipeline should be modular and easily extendable to handle additional\n  data sources and transformations.\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n## Built with\n\n* [![Python][Python.org]][Python-url]\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\n\n\u003c!-- GETTING STARTED --\u003e\n\n## Getting started\n\n### Prerequisites\n\n* [Python 3.10+][Python-docs-url]\n\n### Installation\n\n1. Clone the **repository**\n    ```\n    git clone https://github.com/jpcadena/car-sales-etl.git\n    ```\n2. Change the directory to **root project**\n    ```\n    cd car-sales-etl\n    ```\n3. Create a **virtual environment** *venv*\n    ```\n    python3 -m venv venv\n    ```\n4. Activate **environment** in Windows\n    ```\n    .\\venv\\Scripts\\activate\n    ```\n5. Or with Unix/Mac OS X\n    ```\n    source venv/bin/activate\n    ```\n6. Install requirements with PIP\n    ```\n    pip install -r requirements.txt\n    ```\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\n\n\u003c!-- USAGE EXAMPLES --\u003e\n\n### Usage\n\n1. Rename file **sample.env** to **.env**.\n2. Replace your **credentials** into the *.env* file.\n3. Execute with console.\n    ```\n    python main.py\n    ```\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\n\n\u003c!-- CONTRIBUTING --\u003e\n\n## Contributing\n\nIf you have a suggestion that would make this better, please fork the repo and\ncreate a pull request.\n\n1. Fork the Project\n2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)\n3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)\n4. Push to the Branch (`git push origin feature/AmazingFeature`)\n5. Open a Pull Request\n\nUse docstrings with **reStructuredText** format by adding triple double quotes\n**\"\"\"** after function definition.\\\nAdd a brief function description, also for the parameters including the return\nvalue and its corresponding data type.\\\nPlease use **linting** to check your code quality\nfollowing [PEP 8](https://peps.python.org/pep-0008/).\\\nCheck documentation\nfor [Visual Studio Code](https://code.visualstudio.com/docs/python/linting#_run-linting)\nor [Jetbrains Pycharm](https://github.com/leinardi/pylint-pycharm/blob/master/README.md).\\\n\nRecommended plugin for\nautocompletion: [Tabnine](https://www.tabnine.com/install)\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\n\n\u003c!-- LICENSE --\u003e\n\n## License\n\nDistributed under the MIT License.\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\n\n\u003c!-- CONTACT --\u003e\n\n## Contact\n\nLinkedIn: [Juan Pablo Cadena Aguilar][linkedin-url]\n\nE-mail: [Juan Pablo Cadena Aguilar](mailto:jpcadena@espol.edu.ec?subject=[GitHub]car-sales-etl)\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\n\n\u003c!-- MARKDOWN LINKS \u0026 IMAGES --\u003e\n\u003c!-- https://www.markdownguide.org/basic-syntax/#reference-style-links --\u003e\n\n[linkedin-url]: https://linkedin.com/in/juanpablocadenaaguilar\n\n[project-screenshot]: assets/static/project.png\n\n[Python.org]: https://img.shields.io/badge/python-3670A0?style=for-the-badge\u0026logo=python\u0026logoColor=ffdd54\n\n[Python-url]: https://www.python.org/\n\n[Python-docs-url]: https://docs.python.org/3.10/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjpcadena%2Fcar-sales-etl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjpcadena%2Fcar-sales-etl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjpcadena%2Fcar-sales-etl/lists"}