{"id":22824481,"url":"https://github.com/meepks/etl-pipeline-using-python","last_synced_at":"2026-02-04T03:02:09.996Z","repository":{"id":243824719,"uuid":"813553651","full_name":"MeePks/ETL-PipeLine-Using-Python","owner":"MeePks","description":"This project implements an ETL (Extract, Transform, Load) pipeline that retrieves weather data from the OpenWeatherMap API, processes it, and stores it in a SQL Server database.","archived":false,"fork":false,"pushed_at":"2024-12-05T18:20:23.000Z","size":64,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-31T00:19:34.480Z","etag":null,"topics":["api","etl","python","sql","sqlserver"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MeePks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-11T09:46:43.000Z","updated_at":"2024-12-05T18:20:26.000Z","dependencies_parsed_at":"2024-12-05T19:34:24.572Z","dependency_job_id":null,"html_url":"https://github.com/MeePks/ETL-PipeLine-Using-Python","commit_stats":null,"previous_names":["meepks/etl-pipeline-using-python"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeePks%2FETL-PipeLine-Using-Python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeePks%2FETL-PipeLine-Using-Python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeePks%2FETL-PipeLine-Using-Python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeePks%2FETL-PipeLine-Using-Python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MeePks","download_url":"https://codeload.github.com/MeePks/ETL-PipeLine-Using-Python/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252586211,"owners_count":21772269,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","etl","python","sql","sqlserver"],"created_at":"2024-12-12T17:07:17.678Z","updated_at":"2026-02-04T03:02:09.929Z","avatar_url":"https://github.com/MeePks.png","language":"Python","readme":"This project implements an ETL (Extract, Transform, Load) pipeline that retrieves weather data from the OpenWeatherMap API, processes it, and stores it in a SQL Server database. The stored data is then used for visualization in Tableau.\n\nFeatures\n\nData Extraction: Fetches current weather and 5-day/3-hour forecast data for multiple cities from the OpenWeatherMap API.\n\nData Transformation: Converts temperature from Kelvin to Celsius and adjusts datetime from UTC to Nepal Time (NPT).\n\nData Loading: Inserts the transformed data into SQL Server tables using SQLAlchemy.\n\nEnvironment Variables: Securely manages API keys using environment variables to avoid exposure in the repository.\n\nVisualization: Prepares data for visualization in Tableau for comprehensive weather analysis.\n\n\n\n\nTechnologies Used\n\nPython: For scripting and data processing.\n\nSQLAlchemy: For database interactions.\n\nPandas: For data manipulation and transformation.\n\nOpenWeatherMap API: For fetching weather data.\n\nSQL Server: For data storage.\n\nTableau: For data visualization.\n\n\n\n\nGetting Started\n\nSet up Config File to store your API key. To create API key you can visit 'https://home.openweathermap.org/users/sign_up' and signup to generate your API key.\n\nConfigure the SQL Server connection in the Config file.\n\nRun the Python script to fetch, transform, and load data into the SQL Server database.\n\nConnect Tableau to the SQL Server database to visualize the data.\n\n\n\nPrerequisites\n\nPython 3.x\n\nSQL Server\n\nTableau\n\n\n\nUsage\n\nClone the repository.\n\nSet your OpenWeatherMap API keyinto the config file. I have setup config_sample.ini which you need to rename into config.ini file and seup necessary variable.\n\nConfigure your SQL Server connection string.\n\nRun the ETL script.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeepks%2Fetl-pipeline-using-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmeepks%2Fetl-pipeline-using-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeepks%2Fetl-pipeline-using-python/lists"}