{"id":24021220,"url":"https://github.com/zipcodecore/dataengineering.labs.noaadailysummaries","last_synced_at":"2025-04-15T21:17:10.180Z","repository":{"id":50579161,"uuid":"226411587","full_name":"ZipCodeCore/DataEngineering.Labs.NOAADailySummaries","owner":"ZipCodeCore","description":null,"archived":false,"fork":false,"pushed_at":"2020-03-19T20:36:54.000Z","size":399,"stargazers_count":0,"open_issues_count":0,"forks_count":22,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-15T21:17:05.232Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ZipCodeCore.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-06T21:11:43.000Z","updated_at":"2022-02-03T20:10:27.000Z","dependencies_parsed_at":"2022-08-24T01:40:18.578Z","dependency_job_id":null,"html_url":"https://github.com/ZipCodeCore/DataEngineering.Labs.NOAADailySummaries","commit_stats":null,"previous_names":[],"tags_count":0,"template":true,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZipCodeCore%2FDataEngineering.Labs.NOAADailySummaries","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZipCodeCore%2FDataEngineering.Labs.NOAADailySummaries/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZipCodeCore%2FDataEngineering.Labs.NOAADailySummaries/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZipCodeCore%2FDataEngineering.Labs.NOAADailySummaries/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ZipCodeCore","download_url":"https://codeload.github.com/ZipCodeCore/DataEngineering.Labs.NOAADailySummaries/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249153953,"owners_count":21221330,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-08T12:40:03.541Z","updated_at":"2025-04-15T21:17:10.164Z","avatar_url":"https://github.com/ZipCodeCore.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NOAA Daily Summaries\n\n### Prerequisite \n\n[NOAALocations](https://github.com/Zipcoder/DataEngineering.Labs.NOAALocations)\n\n### Exercise 1 \n\nLet's continue to explore the data provided by NOAA's climate data api. \nBy using the [web services documentation](https://www.ncdc.noaa.gov/cdo-web/webservices/v2#datasets), \nwe can see that they provide an endpoint which allows us to learn more about the datasets that are available.\n\n**Important:** Remember to pass the token header in order to access the service.\n\n![postman_noaa_get_datasets](imgs/postman_noaa_get_datasets.png)\n\nWe can see from the response that there are 11 different data sets we can query through this api. \nThe first result has an id of \"GHCND\" and a name of \"Daily Summaries\". \nThis data seems interesting. Let's dive a bit deeper into this dataset. \n\nThe web services documentation shows that we can specify an id to get information about a particular dataset. \nLet's try it.\n\n![postman_noaa_get_dataset_by_id](imgs/postman_noaa_get_dataset_by_id.png)\n\nHmm... that didn't provide us with any additional information than what we already had. \nIf we navigate the site and search through the web page, we can find the \n[Dataset Discovery](https://www.ncdc.noaa.gov/cdo-web/datasets) page with more details about this dataset. \n\nIf you expand this item and click on the link that says \"More\", it will take us to the \n[Dataset Overview](https://www.ncei.noaa.gov/metadata/geoportal/rest/metadata/item/gov.noaa.ncdc:C00861/html) page for \nthis dataset. \n\nFrom there, we can click on Documentation and choose \n[GHCN-Daily README File](https://www1.ncdc.noaa.gov/pub/data/ghcn/daily/readme.txt) to acquire more details about the \ndata.\n\nBy studying the README file we can see that this dataset provides a couple of interesting fields.\n* TMAX = Maximum temperature (tenths of degrees C)\n* TMIN = Minimum temperature (tenths of degrees C)\n\nOn the next exercise, we will fetch some of the data for this dataset and perform some basic exploration of said data \nfocusing on the fields TMIN and TMAX.\n\n### Exercise 2\n\nLet's get our hands on some daily summary data for each day in December 2018 for all of New Castle County, Delaware. \n\nBetween the previous exercises and the \n[web services documentation page for data](https://www.ncdc.noaa.gov/cdo-web/webservices/v2#data), \nyou should have all you need to query the service. You may need to make more than one rest request to acquire all of the\n data.\n\nWere you able to figure it out? Check your parameters against the table below (spoilers ahead)\n\n\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\n\n| parameter  | value      | \n| ---------- | ---------- |\n| datasetid  |  GHCND     |\n| locationid | FIPS:10003 |\n| startdate  | 2018-01-01 |\n| enddate    | 2018-01-31 |\n| limit      | 1000       | \n| offset     | *          | \n\n* The first rest call will have an offset of 1. The second rest call will have an offset of 1001.\n\nThe URL for the second rest request should look like the following:\n https://www.ncdc.noaa.gov/cdo-web/api/v2/data?datasetid=GHCND\u0026locationid=FIPS:10003\u0026startdate=2018-01-01\u0026enddate=2018-01-31\u0026limit=1000\u0026offset=1001\n\nSave the ouput to files in the data/daily_summaries directory:\n* daily_summaries_FIPS10003_jan_2018_0.json\n* daily_summaries_FIPS10003_jan_2018_1.json\n\nTake a few minutes to familiarize yourself with this data. \n\n### Exercise 3\n\nYet again we have the need to read json data and convert it to dataframe. Rather than keep repeating ourselves, \nlet's adjust the work we did on the NOAALocations lab and create a *json_helper.py* module.\n\n\n### Exercise 4\n\n* Open the notebook called *loading_and_graphing_daily_summaries.ipynb*\n* Follow the directions provided in the notebook and complete the cells as directed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzipcodecore%2Fdataengineering.labs.noaadailysummaries","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzipcodecore%2Fdataengineering.labs.noaadailysummaries","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzipcodecore%2Fdataengineering.labs.noaadailysummaries/lists"}