{"id":50936312,"url":"https://github.com/maxlyth/homeassistant-bayesian-backfill","last_synced_at":"2026-06-17T09:30:59.511Z","repository":{"id":358569705,"uuid":"1193693973","full_name":"maxlyth/homeassistant-bayesian-backfill","owner":"maxlyth","description":"Pyscript service to retroactively recompute Bayesian binary sensor state history in Home Assistant","archived":false,"fork":false,"pushed_at":"2026-03-27T13:51:39.000Z","size":52,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-18T03:09:52.799Z","etag":null,"topics":["backfill","bayesian","history","home-assistant","pyscript"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maxlyth.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-27T13:39:50.000Z","updated_at":"2026-03-27T13:47:25.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/maxlyth/homeassistant-bayesian-backfill","commit_stats":null,"previous_names":["maxlyth/homeassistant-bayesian-backfill"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/maxlyth/homeassistant-bayesian-backfill","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlyth%2Fhomeassistant-bayesian-backfill","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlyth%2Fhomeassistant-bayesian-backfill/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlyth%2Fhomeassistant-bayesian-backfill/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlyth%2Fhomeassistant-bayesian-backfill/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maxlyth","download_url":"https://codeload.github.com/maxlyth/homeassistant-bayesian-backfill/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlyth%2Fhomeassistant-bayesian-backfill/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34443229,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-17T02:00:05.408Z","response_time":127,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backfill","bayesian","history","home-assistant","pyscript"],"created_at":"2026-06-17T09:30:54.389Z","updated_at":"2026-06-17T09:30:59.505Z","avatar_url":"https://github.com/maxlyth.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# homeassistant-bayesian-backfill\n\n[![Tests](https://github.com/maxlyth/homeassistant-bayesian-backfill/actions/workflows/test.yml/badge.svg)](https://github.com/maxlyth/homeassistant-bayesian-backfill/actions/workflows/test.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n[![GitHub release](https://img.shields.io/github/v/release/maxlyth/homeassistant-bayesian-backfill)](https://github.com/maxlyth/homeassistant-bayesian-backfill/releases)\n\nA pyscript service that retroactively recomputes Bayesian binary sensor state history in Home Assistant. When observation probabilities (`prob_given_true`/`prob_given_false`) are changed, or a new Bayesian sensor is created, the existing history doesn't reflect the updated model. This service reconstructs the correct probability history from raw state data already in your recorder database and writes it directly to the states table so it appears in history graphs and ApexCharts attribute-history cards.\n\n---\n\n## Table of contents\n\n- [How it works](#how-it-works)\n- [Prerequisites](#prerequisites)\n- [Installation](#installation)\n- [Service: backfill\\_bayesian\\_sensor](#service-backfill_bayesian_sensor)\n- [Running the tests](#running-the-tests)\n\n---\n\n## How it works\n\n1. Resolves sensor config from `core.config_entries` (UI-created sensors) or by recursively scanning your YAML files (YAML-defined sensors).\n2. Bulk-loads all observation entity state histories from the `states` table in a single SQL query.\n3. Collects all state-change timestamps across observation entities and iterates over them (event-driven -- no fixed interval).\n4. At each event timestamp, evaluates every observation (`state`, `numeric_state`, `template`) against the historical states using forward-fill. Template observations use a Jinja2 environment with mocked `states()`, `state_attr()`, and `now()` returning historical values. `state_attr('sun.sun', 'elevation')` is computed astronomically (pure Python, no `astral` dependency).\n5. Runs the iterative Bayes update formula to compute probability (0.0--1.0).\n6. Writes synthetic state records to the `states` table with reconstructed attributes:\n   - `probability` -- the computed value\n   - `occurred_observation_entities` -- which observation entities were active at that timestamp\n   - `observations` -- full observation list with `observed: false` on inactive ones\n\nAll paths are resolved at runtime from `hass.config.config_dir`, so the script works on any HA installation.\n\n---\n\n## Prerequisites\n\n- Home Assistant (any recent version with the `recorder` integration enabled -- it is on by default)\n- **SQLite recorder database** (the default `home-assistant_v2.db`). This script reads and writes directly via `sqlite3` and is **not compatible** with alternative recorder backends such as MySQL, MariaDB, or PostgreSQL.\n- The [pyscript custom component](https://github.com/custom-components/pyscript) installed and enabled\n- `jinja2` -- bundled with Home Assistant, no separate install needed\n\n---\n\n## Installation\n\n### Step 1 -- Install pyscript\n\n**Via HACS (recommended)**\n\n1. Open HACS in your HA sidebar.\n2. Go to **Integrations \u003e Explore \u0026 Download Repositories**.\n3. Search for **pyscript** and install it.\n4. Restart Home Assistant when prompted.\n\n**Manually**\n\nDownload the latest release from the [pyscript releases page](https://github.com/custom-components/pyscript/releases) and copy the `pyscript` folder into `\u003cconfig\u003e/custom_components/`.\n\n### Step 2 -- Enable pyscript in your configuration\n\nAdd the following to your `configuration.yaml`. The `allow_all_imports` flag is required because the script imports Python standard-library modules (`sqlite3`, `json`, `re`, etc.).\n\n```yaml\npyscript:\n  allow_all_imports: true\n```\n\nIf you already have a `pyscript:` section, add `allow_all_imports: true` inside it.\n\n### Step 3 -- Download and install the app\n\nCopy the `backfill_bayesian/` directory into `pyscript/apps/` inside your HA config folder:\n\n```\n\u003cconfig\u003e/\n  pyscript/\n    apps/\n      backfill_bayesian/\n        __init__.py\n        core.py\n        requirements.txt\n```\n\nYou can download the files from the [latest release](https://github.com/maxlyth/homeassistant-bayesian-backfill/releases), or clone the repo and copy the directory:\n\n```bash\ngit clone https://github.com/maxlyth/homeassistant-bayesian-backfill.git\ncp -r homeassistant-bayesian-backfill/backfill_bayesian \u003cconfig\u003e/pyscript/apps/\n```\n\n| Install type | Config directory |\n|---|---|\n| Home Assistant OS | `/homeassistant/` (accessible via the Samba or SSH add-on) |\n| Home Assistant Container | Whatever you mapped to `/config` in your `docker run` / `compose.yaml` |\n| Home Assistant Core (venv) | `~/.homeassistant/` by default |\n\n### Step 4 -- Register the app in configuration.yaml\n\nAdd the app under the `pyscript:` section in your `configuration.yaml`:\n\n```yaml\npyscript:\n  allow_all_imports: true\n  apps:\n    backfill_bayesian: {}\n```\n\n### Step 5 -- Reload pyscript\n\nIn the HA UI: **Developer Tools \u003e YAML \u003e Reload pyscript**\n\nOr call the service directly:\n\n```yaml\naction: pyscript.reload\n```\n\n### Step 6 -- Verify\n\nGo to **Developer Tools \u003e Actions** and search for `pyscript.backfill_bayesian_sensor`. If it does not appear, check **Settings \u003e System \u003e Logs** for pyscript errors.\n\n---\n\n## Service: `backfill_bayesian_sensor`\n\n### Parameters\n\n| Field | Required | Default | Description |\n|---|---|---|---|\n| `target_entity_id` | yes | -- | Entity ID or glob pattern (e.g. `binary_sensor.*`) |\n| `start_offset` | no | Earliest observation entity state | How far back from now to start the backfill window |\n| `end_offset` | no | Now | How far back from now to end the backfill window |\n| `dry_run` | no | `false` | Log computed values without writing to the database |\n| `debug` | no | `false` | Log a per-observation breakdown for the first 10 event timestamps |\n\n### Examples\n\n```yaml\n# Single sensor -- dry run with debug output\naction: pyscript.backfill_bayesian_sensor\ndata:\n  target_entity_id: binary_sensor.bathroom_occupied\n  dry_run: true\n  debug: true\n```\n\n```yaml\n# All Bayesian sensors, last 7 days\naction: pyscript.backfill_bayesian_sensor\ndata:\n  target_entity_id: binary_sensor.*\n  start_offset:\n    days: 7\n```\n\n```yaml\n# Specific time window\naction: pyscript.backfill_bayesian_sensor\ndata:\n  target_entity_id: binary_sensor.kitchen_occupied\n  start_offset:\n    hours: 48\n  end_offset:\n    hours: 0\n```\n\n### Recommended workflow\n\n1. Run with `dry_run: true` and `debug: true`. Check the HA logs -- you should see a per-observation breakdown for the first 10 event timestamps and a probability value that oscillates plausibly between 0 and 1.\n2. Spot-check: pick a timestamp where you know the state of the observation entities. Hand-calculate the Bayes update and compare to the logged probability.\n3. Run without `dry_run`. The service is idempotent -- safe to re-run after modifying observation probabilities.\n4. Check the **History** panel -- the entity's `probability` attribute should now appear for the backfilled period, with `occurred_observation_entities` and `observations` reflecting the correct state at each point in time.\n\n### How it works in detail\n\nThe service computes probability at each observation entity **state change** rather than at fixed time intervals. This means:\n\n- A state record is written each time any observation entity changes state\n- No unnecessary rows are created during stable periods\n- State transitions are captured at the exact time they occurred\n\nEach written state record includes the full attribute set:\n- **`probability`** -- the Bayesian probability at that moment\n- **`occurred_observation_entities`** -- entity IDs of observations that were active (evaluating True)\n- **`observations`** -- the full observation list from config, with `observed: false` on inactive observations\n\nOther attributes (`device_class`, `friendly_name`, `icon`, `probability_threshold`) are cloned from the most recent real HA-recorded state.\n\n**Idempotent**: Backfill rows are tagged with `origin_idx=2`. Re-running deletes only those rows before reinserting. Real HA-recorded states are never touched.\n\n**Non-blocking**: Writes are committed in batches of 500 rows to avoid holding a database lock that could block the HA recorder.\n\n**Retention-aware**: Rows older than your `purge_keep_days` setting are not written.\n\n---\n\n## Running the tests\n\n```bash\ngit clone https://github.com/maxlyth/homeassistant-bayesian-backfill.git\ncd homeassistant-bayesian-backfill\n\npython3 -m venv .venv\nsource .venv/bin/activate\n\npip install -e \".[test]\"\npytest\n```\n\nExpected output: `109 passed`.\n\nIntegration tests against a live HA instance:\n\n```bash\nHA_URL=http://supervisor/core HA_TOKEN=$SUPERVISOR_TOKEN pytest tests/test_integration.py -v\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxlyth%2Fhomeassistant-bayesian-backfill","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaxlyth%2Fhomeassistant-bayesian-backfill","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxlyth%2Fhomeassistant-bayesian-backfill/lists"}