{"id":15904589,"url":"https://github.com/mepland/chance_of_showers","last_synced_at":"2026-02-07T05:01:03.151Z","repository":{"id":174583061,"uuid":"594942054","full_name":"mepland/chance_of_showers","owner":"mepland","description":"This project provides live water pressure measurements via a web dashboard running on a Raspberry Pi, logs the data, and creates time series forecasts of future water pressure.","archived":false,"fork":false,"pushed_at":"2025-01-11T04:54:03.000Z","size":51049,"stargazers_count":0,"open_issues_count":2,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-23T06:44:51.263Z","etag":null,"topics":["dashboard","data-aquisition","data-science","raspberry-pi","time-series-analysis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mepland.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-01-30T03:37:38.000Z","updated_at":"2024-05-27T01:21:52.000Z","dependencies_parsed_at":"2025-07-23T06:44:19.171Z","dependency_job_id":null,"html_url":"https://github.com/mepland/chance_of_showers","commit_stats":null,"previous_names":["mepland/chance_of_showers"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/mepland/chance_of_showers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mepland%2Fchance_of_showers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mepland%2Fchance_of_showers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mepland%2Fchance_of_showers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mepland%2Fchance_of_showers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mepland","download_url":"https://codeload.github.com/mepland/chance_of_showers/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mepland%2Fchance_of_showers/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29186742,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-07T03:35:06.566Z","status":"ssl_error","status_checked_at":"2026-02-07T03:34:57.604Z","response_time":63,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dashboard","data-aquisition","data-science","raspberry-pi","time-series-analysis"],"created_at":"2024-10-06T12:40:47.385Z","updated_at":"2026-02-07T05:01:03.134Z","avatar_url":"https://github.com/mepland.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# chance\\_of\\_showers\nMatthew Epland, PhD\n\nThis project provides live water pressure measurements\nvia a web dashboard running on a Raspberry Pi,\nlogs the data, and creates time series forecasts of future water pressure.\n\n[![Prophet](https://img.shields.io/badge/Prophet-3b5998.svg?style=flat\u0026logo=data:image/svg%2bxml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPHN2ZyBlbmFibGUtYmFja2dyb3VuZD0ibmV3IDAgMCA3Mi42IDcyIiB2ZXJzaW9uPSIxLjEiIHZpZXdCb3g9IjAgMCA3Mi42IDcyIiB4bWw6c3BhY2U9InByZXNlcnZlIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPgo8c3R5bGUgdHlwZT0idGV4dC9jc3MiPgoJLnN0MHtmaWxsOiNGRkZGRkY7fQo8L3N0eWxlPgo8cGF0aCBjbGFzcz0ic3QwIiBkPSJtMTAuMiA0OS4xdi0wLjFjMC03LjYgNS40LTEzLjggMTMuMy0xMy44IDEuMiAwIDIuMyAwLjIgMy40IDAuNGw2LjItNy42Yy0yLjgtMS4yLTYtMS44LTkuNS0xLjgtMTMuNy0wLjEtMjMuNiAxMC4zLTIzLjYgMjIuOXYwLjFjMCA1LjIgMS43IDEwLjEgNC43IDEzLjlsNi43LTguM2MtMC44LTEuNy0xLjItMy43LTEuMi01Ljd6Ii8+CjxwYXRoIGNsYXNzPSJzdDAiIGQ9Im0zNi42IDQ2LjFjMC4yIDEgMC4zIDIgMC4zIDN2MC4xYzAgNy42LTUuNCAxMy44LTEzLjMgMTMuOC0yLjMgMC00LjMtMC41LTYuMS0xLjVsLTcuOCA2LjNjMy44IDIuNiA4LjUgNC4yIDEzLjggNC4yIDEzLjcgMCAyMy42LTEwLjMgMjMuNi0yMi45di0wLjFjMC0zLjUtMC44LTYuNy0yLjEtOS42bC04LjQgNi43eiIvPgo8Y2lyY2xlIGNsYXNzPSJzdDAiIGN4PSI1Mi40IiBjeT0iMjAuMiIgcj0iNi45Ii8+CjxjaXJjbGUgY2xhc3M9InN0MCIgY3g9IjY4LjciIGN5PSIzLjkiIHI9IjMuOSIvPgo8L3N2Zz4K)](https://github.com/facebook/prophet)\n[![Darts](https://img.shields.io/badge/Darts-0023f7?style=flat\u0026logo=data:image/svg%2bxml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPHN2ZyB3aWR0aD0iMzQ4IiBoZWlnaHQ9IjM0OCIgdmVyc2lvbj0iMS4xIiB2aWV3Qm94PSIwIDAgMzQ4IDM0OCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KIDxnIHRyYW5zZm9ybT0idHJhbnNsYXRlKDE3NCwxNzQpIiBmaWxsPSJub25lIiBzdHJva2U9IiNmZmZmZmYiIHN0cm9rZS13aWR0aD0iOCI+CiAgPGNpcmNsZSByPSIxNzAiLz4KICA8Y2lyY2xlIGN4PSIyOCIgcj0iMTI0Ii8+CiAgPGNpcmNsZSBjeD0iNTciIHI9Ijc3Ii8+CiAgPGNpcmNsZSBjeD0iODUiIHI9IjMwIi8+CiA8L2c+Cjwvc3ZnPgo=)](https://github.com/unit8co/darts)\n[![PyTorch](https://img.shields.io/badge/PyTorch-%23EE4C2C.svg?style=flat\u0026logo=PyTorch\u0026logoColor=white)](https://pytorch.org)\n\n[![Plotly](https://img.shields.io/badge/Plotly-%233F4F75.svg?style=flat\u0026logo=plotly\u0026logoColor=white)](https://plotly.com)\n[![Matplotlib](https://img.shields.io/badge/Matplotlib-%23ffffff.svg?style=flat\u0026logo=data:image/svg%2bxml;base64,PHN2ZyB3aWR0aD0iMjQiIGhlaWdodD0iMjQiIHJvbGU9ImltZyIgdmVyc2lvbj0iMS4xIiB2aWV3Qm94PSIwIDAgMjQgMjQiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CiA8cGF0aCBkPSJtMTIgMGMtNi42MjMgMC0xMiA1LjM3Ny0xMiAxMnM1LjM3NyAxMiAxMiAxMiAxMi01LjM3NyAxMi0xMi01LjM3Ny0xMi0xMi0xMnptMCAwLjI3YzYuNDc0IDAgMTEuNzMgNS4yNTYgMTEuNzMgMTEuNzNzLTUuMjU2IDExLjczLTExLjczIDExLjczLTExLjczLTUuMjU2LTExLjczLTExLjczIDUuMjU2LTExLjczIDExLjczLTExLjczem0wLjAxNiAxMS40OTMtMi4xMjItOS41NTEtMC4wNjYgMC4wMTRhMTAuMDU2IDEwLjA1NiAwIDAgMC00LjIwOCAyLjAzNmwtMC4wNTIgMC4wNDMgNi4xODQgNy41MDEtNy42MDUtMy42NzItMC4wMjggMC4wNjJhOC44MzMgOC44MzMgMCAwIDAtMC42OCA1LjI5OWwwLjAxMiAwLjA2NiA4LjA4Ny0xLjQxMi00LjA2OSAxLjk3MyAwLjAxNyAwLjA1NmMwLjA3NSAwLjIzNCAwLjE2NiAwLjQ2MiAwLjI3NCAwLjY4M2wwLjAzNCAwLjA3IDQuMTAzLTIuNzgtMS4zMjYgNS45NjkgMC4wNjcgMC4wMTRjMC44MSAwLjE2MSAxLjY0NCAwLjE2MSAyLjQ1NCAwbDAuMDY1LTAuMDEzLTEuMDQ0LTUuODcgNi4wNzggNy42NjQgMC4wNTMtMC4wNDJhMTAuMDQgMTAuMDQgMCAwIDAgMS42MjktMS42MjlsMC4wNDItMC4wNTMtNy43MjItNi4xMjRoMi4yOTNsOGUtMyAtMC4wNThhMi41MTMgMi41MTMgMCAwIDAgMC0wLjY5M2wtMC4wMTEtMC4wNzYtMi4zMDIgMC42NCA0LjQ5OS01LjY5LTAuMDU1LTAuMDQyYTcuNDk1IDcuNDk1IDAgMCAwLTIuMDQxLTEuMDg4bC0wLjA2Mi0wLjAyMnoiIGZpbGw9ImJsYWNrIi8+Cjwvc3ZnPgo=)](https://matplotlib.org)\n[![Polars](https://img.shields.io/badge/Polars-cd792c.svg?style=flat\u0026logo=Polars\u0026logoColor=white)](https://github.com/pola-rs/polars)\n[![Pandas](https://img.shields.io/badge/Pandas-%150458.svg?style=flat\u0026logo=pandas\u0026logoColor=white)](https://github.com/pandas-dev/pandas)\n[![Flask](https://img.shields.io/badge/Flask-%23000.svg?style=flat\u0026logo=flask\u0026logoColor=white)](https://flask.palletsprojects.com)\n[![Bootstrap](https://img.shields.io/badge/Bootstrap-%238511FA.svg?style=flat\u0026logo=bootstrap\u0026logoColor=white)](https://getbootstrap.com)\n[![Socket.io](https://img.shields.io/badge/Socket.io-black?style=flat\u0026logo=socket.io\u0026badgeColor=010101)](https://socket.io)\n[![Raspberry Pi](https://img.shields.io/badge/-RaspberryPi-C51A4A?style=flat\u0026logo=Raspberry-Pi)](https://www.raspberrypi.com)\n[![KiCad](https://img.shields.io/badge/KiCad-314CB0.svg?style=flat\u0026logo=KiCad\u0026logoColor=white)](https://www.kicad.org)\n\n[![Poetry](https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json?style=flat)](https://python-poetry.org)\n[![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg?style=flat)](https://github.com/psf/black)\n[![linting: pylint](https://img.shields.io/badge/linting-pylint-yellowgreen?style=flat)](https://github.com/pylint-dev/pylint)\n[![linting: flake8](https://img.shields.io/badge/flake8-checked-blueviolet?style=flat)](https://github.com/PyCQA/flake8)\n[![checked with mypy](https://www.mypy-lang.org/static/mypy_badge.svg)](https://github.com/python/mypy)\n[![imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat)](https://github.com/PyCQA/isort)\n[![security: bandit](https://img.shields.io/badge/security-bandit-yellow.svg?style=flat)](https://github.com/PyCQA/bandit)\n[![linting: markdownlint](https://img.shields.io/badge/linting-markdownlint-blue?style=flat)](https://github.com/DavidAnson/markdownlint)\n[![linting: html5validator](https://img.shields.io/badge/linting-html5validator-blue?style=flat)](https://github.com/svenkreiss/html5validator)\n[![linting: StandardJS](https://img.shields.io/badge/StandardJS-222222.svg?style=flat\u0026logo=StandardJS\u0026logoColor=f3df49)](https://github.com/standard/standard)\n[![linting: yamllint](https://img.shields.io/badge/linting-yamllint-blue?style=flat)](https://github.com/adrienverge/yamllint)\n[![code style: Prettier](https://img.shields.io/badge/Prettier-222222.svg?style=flat\u0026logo=Prettier\u0026logoColor=f7b93e)](https://prettier.io)\n[![linting: checkmake](https://img.shields.io/badge/linting-checkmake-blue?style=flat)](https://github.com/mrtazz/checkmake)\n[![linting: shellcheck](https://img.shields.io/badge/linting-shellcheck-blue?style=flat)](https://github.com/koalaman/shellcheck)\n[![linting: shfmt](https://img.shields.io/badge/linting-shfmt-blue?style=flat)](https://github.com/mvdan/sh)\n[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?style=flat\u0026logo=pre-commit)](https://github.com/pre-commit/pre-commit)\n\n[![tests](https://github.com/mepland/chance_of_showers/actions/workflows/tests.yml/badge.svg?style=flat)](https://github.com/mepland/chance_of_showers/actions/workflows/tests.yml)\n[![healthchecks.io](https://healthchecks.io/badge/63dd8297-b724-4e7d-988b-7eeeca/0nnc0EMy.svg?style=flat)](https://healthchecks.io)\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg?style=flat)](LICENSE.md)\n\n## Introduction\nLiving in a 5th floor walk up in NYC can save you on rent and gym memberships,\nbut runs the risk of leaving you high and dry when your water pressure gives out!\nThe pressure delivered from the city's water mains is\n_[typically](https://cooperatornews.com/article/dispelling-myths-about-poor-water-pressure)_\nsufficient to reach the 6th floor,\nwith higher buildings needing a booster pump and one of NYC's iconic rooftop\n[water towers](https://www.amny.com/real-estate/water-towers-nyc-s-misunderstood-icons-1-6982696).\nMy building lacks a pump and water tower,\nleaving my top floor apartment with _just barely_ satisfactory pressure,\nas long as no other units are using water!\nAs you can see in the data below,\nmy daytime water pressure is all over the place.\nAfter being stranded soapy and cold halfway through a shower one too many times,\nI decided to use my data science and electronics skills to record\nthe time series of my apartment's hot water pressure\nwith the goal of forecasting future availability,\nand hence `chance_of_showers` was born!\n\n\u003cdiv align=\"center\"\u003e\n  \u003cvideo src=\"https://github.com/mepland/chance_of_showers/assets/4729931/f3b94d00-fa40-4b0b-8b95-1105d11e7acd\"\u003e\u003c/video\u003e\n\u003c/div\u003e\n\n## Data Analysis Results\nWIP\n\n### Time Series Plots\n\nBelow is a sample of the pressure data collected in November 2023.\nClicking the links will open interactive plotly plots, please explore!\n\n#### [Raw analog to digital converter (ADC) values](https://mepland.github.io/chance_of_showers/media/ana_outputs/mean_pressure_value_selected_data.html)\n\nThe data acquisition (DAQ) system saves the raw pressure data\nfrom the analog to digital converter (ADC) as an integer between 0 and 65472.\nNote that occasionally a water hammer will increase the pressure above its steady state value,\nmarked by the orange 100% reference line,\nwith a subsequent decay on the order of 10 minutes.\nWhen water is flowing at the pressure sensor,\nthe data is shown with an open purple marker.\nUsing water reduces the pressure slightly under normal conditions,\nand abruptly ends overpressure events.\n\n#### [Normalized values](https://mepland.github.io/chance_of_showers/media/ana_outputs/mean_pressure_value_normalized_selected_data.html)\n\nTo clean the data before fitting any models,\nI rescale the values to 0 and 1 between the steady state extrema.\nAny values that are outside the normalization range are capped.\n\n### Overall Pressure Distributions\n| ![Mean Pressure Value Density](media/ana_outputs/mean_pressure_value_density.png) | ![Mean Pressure Value Normalized vs Time of Week](media/ana_outputs/mean_pressure_value_normalized_vs_time_of_week.png) | ![Mean Pressure Value Normalized vs Time of Day](media/ana_outputs/mean_pressure_value_normalized_vs_time_of_day.png) |\n| :---: | :---: | :---: |\n\n### Prophet Results\n| ![Prophet Predict](media/ana_outputs/prophet/prophet_predict.png) | ![Prophet Components](media/ana_outputs/prophet/prophet_components.png) |\n| :---: | :---: |\n| ![Prophet Components Weekly](media/ana_outputs/prophet/prophet_component_weekly.png) | ![Prophet Components Daily](media/ana_outputs/prophet/prophet_component_daily.png) |\n\n## Hardware\n\n### Bill of Materials\nHere is a list of the components I used in my build.\nWith suitable alterations, the project could definitely be carried out with a wide array of other\nsensors, single board computers or microcontrollers, plumbing supplies, etc.\n\n#### Electronics\n\n* [Raspberry Pi 4 Model B 2 GB](https://www.raspberrypi.com/products/raspberry-pi-4-model-b)\n  * [USB C Power Supply](https://www.raspberrypi.com/products/type-c-power-supply)\n  * [Micro SD Card](https://www.amazon.com/gp/product/B09TQS634Y)\n* [8-Channel 10-Bit ADC with SPI Interface - MCP3008](https://www.digikey.com/en/products/detail/microchip-technology/MCP3008-I-P/319422)\n* [DFRobot Gravity Water Pressure Sensor - SEN0257](https://wiki.dfrobot.com/Gravity__Water_Pressure_Sensor_SKU__SEN0257)\n* [Water Flow Hall Effect Sensor Switch - YWBL-WH](https://www.amazon.com/Interface-Electromagnetic-Flowmeter-Industrial-Accessory/dp/B08B1NG4FZ)\n* [1 kΩ and 10 kΩ Resistors](https://www.amazon.com/gp/product/B072BL2VX1)\n* 830 Point Breadboard and Dupont Jumper Wires - [Included in GPIO Kit](https://www.amazon.com/gp/product/B08B4SHS18)\n\n#### Plumbing\n\n* [1/2\" NPT 3 Way Tee Connector](https://www.amazon.com/Stainless-Diverter-Movable-Flexible-Connector/dp/B09MT39487)\n* [Faucet Connector Hose, 3/8\" Female Compression Thread to 1/2\" Female NPT Thread - B1F09](https://www.amazon.com/gp/product/B000BQWNP8)\n* [Adapter, 3/8\" Male NPT to 1/2\" Female NPT](https://www.amazon.com/gp/product/B07LD3GN4X/ref=ppx_od_dt_b_asin_title_s01)\n* [Adapter, 1/2\" Male NPT to G1/4\" Female - ADT-N12M-G14F](https://koolance.com/threading-adapter-npt-1-2-male-to-g-1-4-female-adt-n12m-g14f)\n* [PTFE (Teflon) Thread Seal Tape](https://www.amazon.com/DOPKUSS-Plumbers-Sealant-Waterproof-Inches/dp/B095YCMHNX)\n\n#### Optional Components\n\n* [I2C OLED Display](https://www.amazon.com/dp/B01MRR4LVE)\n* [Geekworm Baseplate](https://www.amazon.com/gp/product/B07WCBXFD3)\n* Wiring\n  * [GPIO Extension Cable Kit](https://www.amazon.com/gp/product/B08B4SHS18)\n  * [Breadboard Jumper Wires](https://www.amazon.com/gp/product/B07CJYSL2T)\n  * [Clip to Dupont Jumper Wires](https://www.amazon.com/gp/product/B08M5GNY47)\n* Cooling\n  * [Heatsink - Geekworm P165-B](https://www.amazon.com/gp/product/B08N5VZN8R)\n  * [Fan - Noctua NF-A4x20 5V PWM 4-Pin 40x20mm](https://www.amazon.com/gp/product/B071FNHVXN)\n  * [2x20 Pin Header Kit](https://www.amazon.com/gp/product/B08GC18NMK) to clear heatsink\n  * One [M3 Screw](https://www.amazon.com/gp/product/B01I74TTWU) to attach fan to heatsink\n  * Four [M2.5 Screws](https://www.amazon.com/HELIFOUNER-Screws-Washers-Kit-Threaded/dp/B0BKSGC86F)\nto attach heatsink to Pi and baseplate\n\n### Circuit Diagram\nThe circuit diagram for this implementation\nis provided as a [KiCad](https://www.kicad.org) schematic\n[here](circuit_diagram/circuit_diagram.kicad_sch).\n\n![Circuit Diagram](circuit_diagram/circuit_diagram.svg)\n\n### Photos\n| ![Bottom](media/1_bottom.jpg) | ![Left](media/2_left.jpg) | ![Top](media/3_top.jpg) | ![Right](media/4_right.jpg) |\n| :---: | :---: | :---: | :---: |\n| ![Overhead](media/5_overhead.jpg) | ![Overhead Bottom OLED](media/6_overhead_bottom_oled.jpg) | ![Overhead Middle](media/7_overhead_middle.jpg) | ![Overhead Top GPIO](media/8_overhead_top_gpio.jpg) |\n| ![Left Bottom](media/9_left_bottom.jpg) | ![Left Top](media/10_left_top.jpg) | ![Right Top](media/11_right_top.jpg) | ![In Situ](media/12_insitu.jpg) |\n| ![In Situ OLED](media/13_insitu_oled.jpg) | ![In Situe OLED (Flash)](media/14_insitu_oled_flash.jpg) | ![Plumbing Front](media/15_plumbing_front.jpg) | ![Plumbing Back](media/16_plumbing_back.jpg) |\n\n## Data Acquisition (DAQ)\n\nThe DAQ system recorded 95.4% of possible data points overall,\nand 99.870% since implementing the cron job heartbeat monitoring.\n\n### Launching the DAQ Script\nThe provided [`start_daq`](daq/start_daq) bash script\nwill start the [`daq.py`](daq/daq.py) and [`fan_control.py`](fan_control/fan_control.py)\nscripts in new `tmux` windows.\nYou will need to update the `pkg_path` variable in `start_daq` per your installation location.\n\n```bash\nsource daq/start_daq\n```\n\n### Opening the Web Dashboard\nIf `daq: {display_web: true}` is set in [`config.yaml`](config.yaml),\nthe local IP address and port of the dashboard will be logged on DAQ startup.\nOpen this link in your browser to see the live dashboard, as shown in the introduction.\n\n### Setting up cron Jobs\nJobs to restart the DAQ on boot and every 30 minutes,\nas well as send heartbeat API calls - see below,\nare provided in the [`cron_jobs.txt`](daq/cron_jobs.txt) file.\nNote that loading this file with `crontab` will overwrite **any** current cron jobs,\nso check your existing settings first with `crontab -l`!\n\n```bash\ncrontab -l\n\ncrontab daq/cron_jobs.txt\n```\n\nYou can verify the cron jobs are running as expected with:\n\n```bash\ngrep CRON /var/log/syslog | grep $LOGNAME\n```\n\n### Heartbeat Monitoring\nYou can use the provided [`heartbeat`](daq/heartbeat) bash script to send heartbeat API calls\nfor the DAQ script to [healthchecks.io](https://healthchecks.io) for monitoring and alerting.\n[Configure your alert online at healthchecks.io](https://healthchecks.io/docs/configuring_checks),\nand then run the below commands to setup a `secrets.json` file with your alert's `uuid`.\nYou will need to update the `pkg_path` variable in `heartbeat` per your installation location.\nThe provided `cron_jobs.txt` will setup a cron job\nto send the heartbeat on the 15 and 45 minute of each hour.\n\n```bash\nsudo apt install -y jq\necho -e \"{\\n\\t\\\"chance_of_showers_heartbeat_uuid\\\": \\\"YOUR_UUID_HERE\\\"\\n}\" \u003e secrets.json\nsource daq/heartbeat\n```\n\nThe `heartbeat` script has also been setup to backup\nthe `daq/raw_data` and `daq/saved_data` directories to\n`backup_path=\"/media/usb_drive/daq_backup\"`.\nPlease configure `backup_path` to fit your path, or comment out the `rsync` lines to turn them off.\nRegular backups of the data to a separate drive are helpful as Raspberry Pis\nhave been known to corrupt their SD cards due to power loss or excessive writes.\n\n### Combining Raw DAQ Files\n\nRaw CSV files can be combined into convenient Parquet files\nprior to analysis with the [`etl.py`](daq/etl.py) script.\nIf the script crashes, you may need to manually repair\nany lines in the CSV files corrupted due to power losses.\nPolars should generate error messages indicating\nthe corrupt datetime to help you locate the problematic file and line.\n\n```bash\npython daq/etl.py\n```\n\n## Bayesian Optimization\n\nTo optimize the many hyperparameters present in this project,\nboth of the individual forecasting models themselves as well as how the data is prepared,\n[Bayesian optimization](https://github.com/mepland/data_science_notes)\nwas used to efficiently sample the parameter space.\nThe functions needed to run Bayesian optimization\nare located in [`bayesian_opt.py`](utils/bayesian_opt.py),\nand make use of the\n[`bayesian-optimization`](https://github.com/bayesian-optimization/BayesianOptimization) library.\n\nUnfortunately, actually running the optimization over GPU accelerated models\nis not as simple as calling the `run_bayesian_opt()` function.\nI have been unable to successfully detach the training of one GPU accelerated model\nfrom the next when training multiple models in a loop.\nThe second training session will still have access to the tensors of the first,\nleading to out of GPU memory errors, even when\n[using commands like `gc.collect()` and `torch.cuda.empty_cache()`](https://stackoverflow.com/questions/70508960/how-to-free-gpu-memory-in-pytorch).\nThe `torch` models created by `darts` are very convenient,\nbut do not provide as much configurability as building your own `torch` model from scratch,\nleaving me unable to fix this issue in a clean way.\n\nTo work around the GPU memory issues, a shell script,\n[`start_bayesian_opt`](ana/start_bayesian_opt), is used to repeatedly call `run_bayesian_opt()`\nvia the [`bayesian_opt_runner.py`](ana/bayesian_opt_runner.py) script.\nIn this way each model is trained in its own Python session,\ntotally clearing memory between training iterations.\nA signed pickle file is used to quickly load the necessary data and settings on each iteration.\nInstructions for running the whole Bayesian optimization workflow are provided below.\n\nSome hyperparameter points chosen for testing by the optimizer result in crashes during training,\neither due to memory limitations, or invalid parameter combinations that slip by pre-run checks.\nIn most cases these exceptions can be caught within Python,\nallowing the point to be automatically logged as having the worst possible `BAD_TARGET = -999.0`.\nHowever, the Python process itself is occasionally killed\nby the operating system with an uncatchable `SIGKILL` signal,\nlikely due to a request for too much memory.\nResuming the run will only result in repeated crashes on the same point as it is never written to disk.\nRather than rework the `bayesian-optimization` library to pre-register points before optimization,\na [`manual_bad_point.py`](ana/manual_bad_point.py) script is included\nto easily log the few such points by hand.\n\n### Running Bayesian Optimization\n\n1. Create the input `parent_wrapper.pickle` file for `bayesian_opt_runner.py`\nvia the `exploratory_ana.py` notebook.\n2. Configure the run in `start_bayesian_opt` and `bayesian_opt_runner.py`.\n3. Run the shell script, logging outputs to disk via:\n    * Log any bad points that are killed by the operating system\nwith `manual_bad_point.py` and resume the search.\n\n```bash\n./ana/start_bayesian_opt 2\u003e\u00261 | tee ana/models/bayesian_optimization/bayesian_opt.log\n```\n\n## Dev Notes\n\n### Data Analysis Setup - Installing CUDA and PyTorch\n\n1. Find the supported CUDA version (`11.8.0`) for the current release of PyTorch (`2.0.1`) [here](https://pytorch.org/get-started/locally).\n2. Install CUDA following the steps for the proper version and target platform [here](https://developer.nvidia.com/cuda-toolkit-archive).\n3. Update the poetry `pytorch-gpu-src` source to point to the correct PyTorch version in [`pyproject.toml`](pyproject.toml).\n    * This is in place of `pip install --index-url=...` as provided by the [PyTorch installation instructions](https://pytorch.org/get-started/locally).\n4. Install the poetry `ana` group with `make setupANA`.\n    * This will install `pytorch`, along with the other necessary packages.\n5. Check that PyTorch and CUDA are correctly configured with the following `python` commands:\n\n```python\nimport torch\n\nif torch.cuda.is_available():\n    print(\"CUDA is available\")\n    print(f\"Device name: {torch.cuda.get_device_name(torch.cuda.current_device())}\")\nelse:\n    print(\"CUDA IS NOT AVAILABLE!\")\n```\n\n### DAQ Setup - Installing Python 3.11 on Raspbian\nIf `python 3.11` is not available in your release of Raspbian,\nyou can compile it from source following the instructions [here](https://aruljohn.com/blog/python-raspberrypi),\nbut will also need to [install the sqlite extensions](https://stackoverflow.com/a/24449632):\n\n\u003c!-- markdownlint-disable MD013 --\u003e\n```bash\ncd /usr/src/\nsudo wget https://www.python.org/ftp/python/3.11.4/Python-3.11.4.tgz\nsudo tar -xzvf Python-3.11.4.tgz\ncd Python-3.11.4/\nsudo apt update \u0026\u0026 sudo apt full-upgrade -y\nsudo apt install -y build-essential libbz2-dev libc6-dev libexpat1-dev libffi-dev libgdbm-dev liblzma-dev libncurses5-dev libnss3-dev libsqlite3-dev libssl-dev lzma pkg-config zlib1g-dev\nsudo apt autoremove -y\nsudo apt update \u0026\u0026 sudo apt full-upgrade -y\n./configure --enable-optimizations --enable-loadable-sqlite-extensions\nsudo make altinstall\n\n# Should be Python 3.11.4 with your compile info\n/usr/local/bin/python3.11 -VV\n\n# Link binary\nsudo rm /usr/bin/python\nsudo rm /usr/bin/python3\nsudo ln -s /usr/local/bin/python3.11 /usr/bin/python\nsudo ln -s /usr/local/bin/python3.11 /usr/bin/python3\n\n# Should match /usr/local/bin/python3.11 -VV\npython -VV\n```\n\n#### Additional DAQ Dependencies\nTo finish setting up the DAQ system you must also:\n\n* Install `tmux`, which is not included in Raspbian by default.\n`tmux` is used to control multiple terminal sessions in [`start_daq`](daq/start_daq).\n* Install `pigpio`, which is not included in Raspbian Lite, i.e. headless, installations.\n`pigpio` is necessary to interface with the GPIO ports and must also be enabled via a [daemon](https://gpiozero.readthedocs.io/en/latest/remote_gpio.html)\n* Enable SPI, I2C, and Remote GPIO via `raspi-config`.\n* Prevent the [WiFi from powering off](https://desertbot.io/blog/headless-raspberry-pi-4-ssh-wifi-setup-64-bit-mac-windows).\n* It is recommended to [install](https://pimylifeup.com/raspberry-pi-log2ram) [`log2ram`](https://github.com/azlux/log2ram) to avoid unnecessary writes to the SD card, prolonging the card's lifetime.\n\n```bash\n# Install tmux and pigpio\nsudo apt-get install -y tmux pigpio\n\n# Enable SPI, I2C, and Remote GPIO\nsudo raspi-config\n\n# Setup pigpio daemon\nsudo systemctl enable pigpiod\n\n# Prevent the WiFi from powering off\n# Above the line that says exit 0 insert `/sbin/iw wlan0 set power_save off` and save the file\nsudo vi /etc/rc.local\n\n# Install log2ram\necho \"deb [signed-by=/usr/share/keyrings/azlux-archive-keyring.gpg] http://packages.azlux.fr/debian/ bullseye main\" | sudo tee /etc/apt/sources.list.d/azlux.list\nsudo wget -O /usr/share/keyrings/azlux-archive-keyring.gpg https://azlux.fr/repo.gpg\nsudo apt update \u0026\u0026 sudo apt full-upgrade -y\nsudo apt install -y log2ram\n```\n\u003c!-- markdownlint-enable MD013 --\u003e\n\n### Installing Dependencies with Poetry\nInstall `poetry` following the [instructions here](https://python-poetry.org/docs#installation).\n\n```bash\ncurl -sSL https://install.python-poetry.org | python3 -\n```\n\nThen install the `python` packages needed for this installation. Groups include:\n\n* `daq` for packages needed to run the DAQ script on a Raspberry Pi, optional\n* `web` for packages needed to run the live dashboard from the DAQ script, optional\n* `ana` for analysis tools, optional\n* `dev` for continuous integration (CI) and linting tools\n\n```bash\npoetry install --with daq,web\n```\n\nor\n\n```bash\npoetry install --with ana\n```\n\n### Setting up pre-commit\nIt is recommended to use the [`pre-commit`](https://pre-commit.com) tool\nto automatically check your commits locally as they are created.\nYou should just need to [install the git hook scripts](https://pre-commit.com/#3-install-the-git-hook-scripts),\nsee below, after installing the `dev` dependencies.\nThis will run the checks in [`.pre-commit-config.yaml`](.pre-commit-config.yaml)\nwhen you create a new commit.\n\n```bash\npre-commit install\n```\n\n### Installing Non-Python Based Linters\nMarkdown is linted using [`markdownlint-cli`](https://github.com/igorshubovych/markdownlint-cli),\nJavaScript by [`standard`](https://github.com/standard/standard),\nand HTML, SCSS, CSS, and TOML by [`prettier`](https://prettier.io).\nYou can install these JavaScript-based linters globally with:\n\n```bash\nsudo npm install --global markdownlint-cli standard prettier\nsudo npm install --global --save-dev --save-exact prettier-plugin-toml\n```\n\nShell files are linted using [`shellcheck`](https://github.com/koalaman/shellcheck)\nand [`shfmt`](https://github.com/mvdan/sh).\nFollow the linked installation instructions for your system.\nOn Fedora they are:\n\n```bash\nsudo dnf install ShellCheck shfmt\n```\n\n### Using the Makefile\nA [`Makefile`](Makefile) is provided for convenience,\nwith commands to setup the DAQ and analysis environments,\n`make setupDAQ` and `make setupANA`,\nas well run CI and linting tools,\ne.g. `make black`, `make pylint`, `make pre-commit`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmepland%2Fchance_of_showers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmepland%2Fchance_of_showers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmepland%2Fchance_of_showers/lists"}