{"id":22547408,"url":"https://github.com/shaadclt/data-etl-pipeline","last_synced_at":"2026-05-17T06:45:37.536Z","repository":{"id":234801340,"uuid":"656762800","full_name":"shaadclt/Data-ETL-Pipeline","owner":"shaadclt","description":"This code demonstrates how to load the Fashion MNIST dataset using TensorFlow's Keras API, preprocess the data, and store it in a SQLite database. The Fashion MNIST dataset consists of grayscale images of clothing items with corresponding labels.","archived":false,"fork":false,"pushed_at":"2023-06-21T15:34:41.000Z","size":4,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-28T09:11:22.098Z","etag":null,"topics":["sqllite","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shaadclt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-06-21T15:29:12.000Z","updated_at":"2024-10-10T05:02:21.000Z","dependencies_parsed_at":"2024-04-20T20:54:07.375Z","dependency_job_id":"498e5239-13e6-4981-8dd1-fd1bfebcac69","html_url":"https://github.com/shaadclt/Data-ETL-Pipeline","commit_stats":null,"previous_names":["shaadclt/data-etl-pipeline"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/shaadclt/Data-ETL-Pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shaadclt%2FData-ETL-Pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shaadclt%2FData-ETL-Pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shaadclt%2FData-ETL-Pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shaadclt%2FData-ETL-Pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shaadclt","download_url":"https://codeload.github.com/shaadclt/Data-ETL-Pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shaadclt%2FData-ETL-Pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279002970,"owners_count":26083489,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-10T02:00:06.843Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["sqllite","tensorflow"],"created_at":"2024-12-07T15:11:46.473Z","updated_at":"2025-10-10T06:38:09.195Z","avatar_url":"https://github.com/shaadclt.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Data ETL Pipeline Using Python\n\nThis code demonstrates how to load the Fashion MNIST dataset using TensorFlow's Keras API, preprocess the data, and store it in a SQLite database. The Fashion MNIST dataset consists of grayscale images of clothing items with corresponding labels.\n\n### Loading and Preprocessing the Data\n\n- The code begins by importing the necessary libraries, including TensorFlow Keras and SQLite.\n- The Fashion MNIST dataset is loaded using `keras.datasets.fashion_mnist.load_data()`, and it is split into training and test sets.\n- The code then prints the shapes of the loaded data to verify the dimensions.\n- Next, the pixel values of the images are normalized by dividing by 255 to scale them between 0 and 1.\n- The `xtrain` and `xtest` arrays are reshaped to include a channel dimension for compatibility with convolutional neural networks.\n\n### Storing the Data in SQLite Database\n\n- A connection is established with an SQLite database file named `fashion_mnist.db`.\n- The code creates a table named `images` in the database if it doesn't exist already.\n- A loop iterates over the training set, inserting each image and its corresponding label into the `images` table using binary data conversion.\n- The connection is committed to save the changes.\n- Another loop iterates over the test set, inserting the images and labels into the `images` table.\n- The connection is committed again to finalize the changes.\n- Finally, the connection to the database is closed.\n\n### Retrieving Data from SQLite Database\n\n- The code re-establishes a connection with the `fashion_mnist.db` file.\n- A cursor is created to execute SQL queries.\n- The cursor selects all rows from the `images` table.\n- The selected rows are fetched and stored in the `rows` variable.\n- Additionally, the `pandas` library is used to read the entire `images` table into a DataFrame named `data`.\n\n### Usage\n\nTo use this code, follow these steps:\n\n1. Make sure you have TensorFlow and SQLite installed.\n2. Download or clone the code from this GitHub repository.\n3. Run the code in a Python environment.\n4. After execution, you will find a file named `fashion_mnist.db` containing the dataset stored in an SQLite database.\n\nAcknowledgments\n\nThis code is adapted from the Fashion MNIST example provided in the TensorFlow documentation.\n\nReferences\n\n- [Fashion MNIST Dataset](https://github.com/zalandoresearch/fashion-mnist)\n- [TensorFlow Keras](https://www.tensorflow.org/api_docs/python/tf/keras)\n- [SQLite](https://www.sqlite.org/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshaadclt%2Fdata-etl-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshaadclt%2Fdata-etl-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshaadclt%2Fdata-etl-pipeline/lists"}