{"id":20201001,"url":"https://github.com/openclimatefix/gfs-downloader","last_synced_at":"2025-10-06T16:30:32.332Z","repository":{"id":157785521,"uuid":"624514043","full_name":"openclimatefix/gfs-downloader","owner":"openclimatefix","description":"NCEP GFS 0.25 Degree Global Forecast Grids Historical Archive: https://rda.ucar.edu/datasets/ds084.1/","archived":false,"fork":false,"pushed_at":"2023-06-22T16:13:54.000Z","size":149,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-01-20T23:58:16.025Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openclimatefix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null},"funding":{"github":["openclimatefix"],"patreon":null,"open_collective":"openclimatefix","ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2023-04-06T16:29:59.000Z","updated_at":"2024-05-07T14:40:10.000Z","dependencies_parsed_at":null,"dependency_job_id":"9671fd18-fd47-4b41-811c-f53c0f2d977e","html_url":"https://github.com/openclimatefix/gfs-downloader","commit_stats":{"total_commits":76,"total_committers":2,"mean_commits":38.0,"dds":"0.013157894736842146","last_synced_commit":"f3cb6452bdfdfbf47d7942b5185551c1d2a87ccf"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fgfs-downloader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fgfs-downloader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fgfs-downloader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fgfs-downloader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openclimatefix","download_url":"https://codeload.github.com/openclimatefix/gfs-downloader/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235534414,"owners_count":19005469,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T04:46:27.626Z","updated_at":"2025-10-06T16:30:27.027Z","avatar_url":"https://github.com/openclimatefix.png","language":"Python","funding_links":["https://github.com/sponsors/openclimatefix","https://opencollective.com/openclimatefix"],"categories":[],"sub_categories":[],"readme":"# gfs-downloader\n\nNCEP GFS 0.25 Degree Global Forecast Grids Historical Archive: https://rda.ucar.edu/datasets/ds084.1/\n\nRegister and make an account here: https://rda.ucar.edu/login/\n\nCreate a .env file with this in (make sure to name the file .env and leave it in the root directory):\n\n\nTo create a .env file do:\n\n```\nnano .env\n```\n\nThen add your credentials to the file\n\n```\nUCAR_EMAIL = \"INSERT_EMAIL\"\nUCAR_PASS = \"INESERT_PASSWORD\"\n```\n\n### Setting up the environment\n\n```\npip install -r requirements.txt\n```\n\n### Instal the ecCodes package\n\n```\nconda install -c conda-forge python-eccodes\n```\n\nAdd others (dask, etc...)\n\n### Download Operation\n\nTo see download script inputs: `poetry run Island_nwp_ext_latlon.py --help`\n\n**Example [DEMO] (if using poetry):**\n\n`poetry run python Island_nwp_ext_latlon.py Start_Date End_Date Output_Path LATMIN LATMAX LONGMIN LONGMAX`\n\nor (for normal venv)\n\n```\npython Island_nwp_ext_latlon.py Start_Date End_Date Output_Path LATMIN LATMAX LONGMIN LONGMAX\n```\n\nDate format YYYY-MM-DD\n\n\nTo keep operation running in the background use nohup (a nohup.out file will be created to keep a log):\n\n```\nnohup python Island_nwp_ext_latlon.py Start_Date End_Date Output_Path LATMIN LATMAX LONGMIN LONGMAX \u0026\n```\n\n\nSet the location of the output log, this example will create the nwp_run_v1.log at the location where the command was executed:\n\n```\nnohup python Island_nwp_ext_latlon.py Start_Date End_Date Output_Path LATMIN LATMAX LONGMIN LONGMAX \u003e\u003e nwp_run_v1.log 2\u003e\u00261 \u0026\n```\n\n\n\nWatch the log in the CLI live:\n\n```\ntail -f nwp_run_v1.log\n```\n\nLook at all the full logs with:\n\n```\nnano nwp_run_v1.log\n```\n\nMonitor scripts executing and uptime (This is also where you can find the PID in case the need to Kill a script) [Kill PID]:\n\n```\nps aux | grep Island_nwp_ext_latlon.py\n```\n\n\n\n\n### Multitasks using parallel:\n\nFirst instal parallel:\n\n```\nconda install parallel\n```\n\n##### Generating dates:\n\nUse the script `date_generator.py` to generated a \"date_chunks\" variable.\n\n```\npython date_generator.py Start_Date End_Date Num_Parallel\n```\n\nor manually set running:\n\n```\ndate_chunks=$(cat \u003c\u003cEOF\n2021-04-11 2021-06-19\n2021-06-19 2021-08-27\n2021-08-27 2021-11-04\n2021-11-04 2022-01-18\n2022-01-18 2022-03-22\n2022-03-22 2022-04-07\n2022-04-07 2022-05-30\n2022-05-30 2022-07-15\n2022-07-15 2022-08-07\n2022-08-07 2022-10-15\nEOF\n)\n```\n\ncheck if either method set it correctly:\n\n```\necho \"$date_chunks\"\n```\n\n\nRun this to create Num_Parallel tasks (Google Cloud Computing has a max of 10), stored at location Output_Path/task_{Number corresponding to execution order}.\n\nAdjust depending on your file structure\n\nThe {1} and {2} load in the start and end date of the script.\n\n```\necho \"$date_chunks\" | nohup parallel -j Num_Parallel --colsep ' ' 'python //gfs-downloader/scripts/data_collection/Island_nwp_ext_latlon.py {1} {2} //Output_Path/task_{#} LATMIN LATMAX LONGMIN LONGMAX' \u003e\u003e nwp_run_p1.log 2\u003e\u00261 \u0026\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fgfs-downloader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenclimatefix%2Fgfs-downloader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fgfs-downloader/lists"}