{"id":20915610,"url":"https://github.com/matthewmckenna/ca682-data-visualisation","last_synced_at":"2025-08-02T09:35:28.547Z","repository":{"id":44865679,"uuid":"161379603","full_name":"matthewmckenna/ca682-data-visualisation","owner":"matthewmckenna","description":"Code and analyses for CA682: Data Management and Visualisation project","archived":false,"fork":false,"pushed_at":"2022-12-08T01:28:33.000Z","size":1085,"stargazers_count":0,"open_issues_count":5,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-12T23:15:22.657Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/matthewmckenna.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-11T18:55:57.000Z","updated_at":"2019-06-11T09:39:09.000Z","dependencies_parsed_at":"2023-01-24T21:30:28.253Z","dependency_job_id":null,"html_url":"https://github.com/matthewmckenna/ca682-data-visualisation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/matthewmckenna/ca682-data-visualisation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matthewmckenna%2Fca682-data-visualisation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matthewmckenna%2Fca682-data-visualisation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matthewmckenna%2Fca682-data-visualisation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matthewmckenna%2Fca682-data-visualisation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/matthewmckenna","download_url":"https://codeload.github.com/matthewmckenna/ca682-data-visualisation/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matthewmckenna%2Fca682-data-visualisation/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268362998,"owners_count":24238544,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-02T02:00:12.353Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-18T16:16:30.715Z","updated_at":"2025-08-02T09:35:28.522Z","avatar_url":"https://github.com/matthewmckenna.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CA682: Data Management and Visualisation\n\nCode and analyses for CA682 module.\n\n## Data\n\n* Tarball containing 498 CSV files: [csvs.tar.gz](https://drive.google.com/open?id=18hNkat9vaakwkwfsJoylyVINF45yQr6t) [31 MB]\n\n## Finding weather station data\n\nMet Éireann provides access to historical data via their [website](https://www.met.ie/climate/available-data/historical-data).\nThe interface provided allows a user to download a dataset for one weather station at a time.\nHowever, for this project we require much more data, and would like all data for all weather stations.\n\nUsing the inspection tools in Chrome I found a `stations.csv` file that was being loaded.\nSome fields of interest here were the station name, county, latitude, longitude, and most importantly, the assigned station number.\n\nI then found a base url where the historical data was hosted.\nBy manually downloading a single dataset and examining the URL, I was able to determine the form of the URLs for the hosted data:\n\n`\"$BASE_URL\"/dly\"$STATION_NUMBER\".zip`\n\n## Cleaning up `stations.csv`\n\nThere is a lot of information that we don't need in `stations.csv`.\nThis file can be tidied up to extract only the fields of interest by running `clean_stations_csv.sh`.\nThis is a very simple shell script.\nThe contents are shown below:\n\n```bash\n$ cat clean_stations_csv.sh\n#!/bin/bash\n# extract fields of interest from `stations.csv`\n\n# directory containing the input file `stations.csv`\ndata_dir=$1\n\n# the name of the input file\ninput_file=\"$1\"/stations.csv\n# the name of the output file\noutput_file=\"$1\"/stations_clean.csv\n\n# add the header to the output file\necho \"station_number,station_name,county,latitude,longitude\" \u003e\"$output_file\"\n\n# extract the first five fields of the input file, skip the original header\n# and append the rest to the output file\ncut -d',' -f1-5 \"$input_file\" |tail -n+2 \u003e\u003e\"$output_file\"\n```\n\n## Getting all weather station data\n\nRun the Python application to download the data:\n\n```bash\n$ python get_data.py data/daily/zips\n```\n\nNote that the path passed (`data/daily/zips`) above, **must** exist.\nWith the example above, `518` zip archives were downloaded to `data/daily/zips`.\nThis took several minutes, running with two threads.\n\n## Unzipping the archives\n\nThe zip archives were downloaded to `data/daily/zips`.\nFrom the parent directory (`data/daily`), the following commands were run to extract the archives:\n\n```bash\n$ mkdir csvs\n$ for f in $(ls zips); do unzip -nd csv zips/$f; done \u003elog 2\u003e\u00261\n```\n\nSome errors were noted during the `unzip` operation (as noted in the `log` file generated).\nIt appears that a number of the constructed URLs were not valid.\nThere are `20` files (listed below) for which the archive size is \u003c 1 KB, and simply contain the source of a `404` error page.\n\n```bash\n$ find . -type f -size -1024c\n./dly3422.zip\n./dly9938.zip\n./dly2931.zip\n./dly538.zip\n./dly9323.zip\n./dly4413.zip\n./dly3037.zip\n./dly1807.zip\n./dly4702.zip\n./dly2604.zip\n./dly199.zip\n./dly5819.zip\n./dly5602.zip\n./dly9206.zip\n./dly8912.zip\n./dly2218.zip\n./dly8123.zip\n./dly5729.zip\n./dly9106.zip\n./dly907.zip\n```\n\nThis leaves us with valid weather data for `498` weather stations across Ireland.\n\n\n## Finding how many rows to skip\n\nThe CSV files are not standardised.\nIn order to load the CSVs into a `DataFrame`, we need to provide the `skiprows` argument.\nIn order to find out how many rows to skip, we can recursively `grep` for the header line.\nIn the `data` directory, run:\n\n```bash\ngrep -nr \"date,\" daily/csvs/dly*.csv |cut -d'/' -f3 |cut -d':' -f1,2 \u003eheader_rows.txt\n```\n\nWe can inspect the first `10` lines with:\n\n```bash\n$ head header_rows.txt\ndly10023.csv:10\ndly1024.csv:10\ndly1033.csv:10\ndly1042.csv:10\ndly1043.csv:10\ndly1075.csv:25\ndly108.csv:10\ndly1103.csv:10\ndly1107.csv:10\ndly1108.csv:10\n```\n\nFor most of the files, the header is on line `10`.\nHowever, for `dly1075.csv`, the header is on line `25`.\nWe can extract this information, and create a mapping using Python.\nWe'll also make use of the `stations_clean.csv` file generated earlier.\n\n## Data shift in `stations.csv`\n\nThere is an extra comma in the `county` field in the original `stations.csv` which caused the data to be shifted when running the script to clean this data.\nThis can be corrected by running the following code on macOS:\n\n```bash\n$ sed -i.bak \"460s/.*/5331,DELVIN CASTLE G.C.,Westmeath,53.608,-7.104/\" stations_clean.csv\n$ mv stations_clean.csv{.bak,}\n```\n\nor on Linux/Windows\n\n```bash\n$ sed -i \"460s/.*/5331,DELVIN CASTLE G.C.,Westmeath,53.608,-7.104/\" stations_clean.csv\n```\n\n\n## Creating a `DataFrame` with all station details\n\nWe want to gather all information about the stations in an easy-to-use format.\nWe'll use `pandas` to create `DataFrame` objects to work with.\nInitial cleaning and transformation is done in [`data-cleaning_2018-12-14.ipynb`](data-cleaning_2018-12-14.ipynb).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatthewmckenna%2Fca682-data-visualisation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatthewmckenna%2Fca682-data-visualisation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatthewmckenna%2Fca682-data-visualisation/lists"}