{"id":51090357,"url":"https://github.com/ellyseum/ios_pointer_finder","last_synced_at":"2026-06-24T01:30:27.921Z","repository":{"id":355426911,"uuid":"1224162206","full_name":"ellyseum/ios_pointer_finder","owner":"ellyseum","description":"Tiny CNN (338K params) that finds the iPhone Pointer-Control cursor in a screen capture. 30.5 px val pos error on honest bg-level split, 95 fps on RTX 5080. MIT code + CC-BY-4.0 weights. Calibrated on iPhone 16 Pro Max / iOS 26.3.1; v0.5+ bootstrap loop targets cross-device generalization.","archived":false,"fork":false,"pushed_at":"2026-05-03T14:28:47.000Z","size":18845,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-03T16:24:32.484Z","etag":null,"topics":["agent","automation","cnn","computer-vision","cursor-detection","ios","iphone","pointer-control","pytorch"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ellyseum.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-29T02:43:36.000Z","updated_at":"2026-05-03T14:28:51.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ellyseum/ios_pointer_finder","commit_stats":null,"previous_names":["ellyseum/ios_pointer_finder"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/ellyseum/ios_pointer_finder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ellyseum%2Fios_pointer_finder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ellyseum%2Fios_pointer_finder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ellyseum%2Fios_pointer_finder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ellyseum%2Fios_pointer_finder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ellyseum","download_url":"https://codeload.github.com/ellyseum/ios_pointer_finder/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ellyseum%2Fios_pointer_finder/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34713789,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-23T02:00:07.161Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","automation","cnn","computer-vision","cursor-detection","ios","iphone","pointer-control","pytorch"],"created_at":"2026-06-24T01:30:24.990Z","updated_at":"2026-06-24T01:30:27.890Z","avatar_url":"https://github.com/ellyseum.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ios_pointer_finder\n\nA 338K-parameter CNN that finds the iPhone Pointer-Control cursor in a screen capture, in 10 ms.\n\n[![CI](https://github.com/ellyseum/ios_pointer_finder/actions/workflows/ci.yml/badge.svg)](https://github.com/ellyseum/ios_pointer_finder/actions/workflows/ci.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)\n[![Weights: CC-BY-4.0](https://img.shields.io/badge/Weights-CC--BY--4.0-lightgrey.svg)](LICENSE-WEIGHTS)\n[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)\n[![Model: v0.7.0](https://img.shields.io/badge/model-v0.7.0-green.svg)](https://github.com/ellyseum/ios_pointer_finder/releases/tag/v0.7.0)\n[![Code style: ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)\n\nWhen you pair a Bluetooth mouse with an iPhone, iOS draws a small \"@-symbol\" cursor on the\nscreen ([Pointer Control](https://support.apple.com/guide/iphone/adjust-pointer-settings-iphec6e1e60b/ios)).\nThis repo trains and ships a tiny model that finds that cursor with sub-cursor-radius accuracy,\non any iPhone screen, in real time. It's the perception layer of a closed-loop iPhone agent\nthat drives an unmodified phone over BLE HID + AirPlay mirroring.\n\n\u003e A heatmap-regression CNN trained on synthetic composites. No iPhone images are bundled.\n\u003e Bring your own backgrounds, run `synthesize.py`, train your own variant — or just use the\n\u003e shipped weights directly.\n\n---\n\n## Quickstart\n\n\u003e **Status:** PyPI publish + Hugging Face Hub model repo are not live yet —\n\u003e both are gated on the v0.7 retrain finishing (see [Roadmap](#roadmap-milestones)).\n\u003e Today's install path is from a clone:\n\n```bash\ngit clone https://github.com/ellyseum/ios_pointer_finder.git\ncd ios_pointer_finder\npip install -e \".[hub,safetensors]\"\n```\n\nThen either:\n\n**A) Train your own weights** (see [Train your own](#train-your-own) below) — produces\n`pointer_model_v{X}.{Y}.{Z}_{err}px.{pt|safetensors}`. Convert legacy `.pt` to\n`.safetensors`:\n\n```bash\npython scripts/convert_pt_to_safetensors.py pointer_model_v0.7.0.pt\n# → pointer_model_v0.7.0.safetensors  +  pointer_model_v0.7.0.config.json\n```\n\n`train.py` can also write `.safetensors` directly via `--weights-out *.safetensors`.\n\n**B) Once the HF repo is up** (gated on the v0.7 retrain), use the one-liner the\npackage is designed for:\n\n```python\nimport cv2\nfrom inference import PointerFinder\n\nimg = cv2.imread(\"snap.jpg\")  # any resolution — auto-resized to native (994×2160)\nfinder = PointerFinder.from_pretrained(\"ellyseum/ios_pointer_finder\")\nresult = finder(img)\nprint(result.x, result.y, result.confidence, result.heatmap_peak)\n# 656 1424 0.94 0.81\n```\n\nFor now (path A), call `PointerFinder` with a local `.safetensors` or `.pt` path:\n\n```python\nfrom inference import PointerFinder\nfinder = PointerFinder.from_pretrained(\"./pointer_model_v0.7.0.safetensors\")\n```\n\nSee [`docs/MODEL_CARD.md`](docs/MODEL_CARD.md) for the full inference contract and provenance.\n\n---\n\n## Why\n\nVision-LLM-based agents that \"look at the screen\" are too slow for real-time control loops\n(2-10 seconds per turn). For the iPhone agent we wanted, we needed *one* perception\nquestion — *where is the cursor, right now?* — answered in milliseconds, on commodity\nhardware, with no API call. A specialized 338K-param CNN handles it in 10 ms on an RTX 5080\nand ~30 ms on Apple Silicon CoreML.\n\nThe general-purpose vision LLM still answers \"what's on this screen\" when we need it. The\ncursor finder answers \"where's the cursor\" continuously.\n\n---\n\n## Model card (v0.7.1)\n\n| Field                | Value                                                |\n|----------------------|------------------------------------------------------|\n| Architecture         | 5-block conv backbone → 1×1 heatmap head + conf head |\n| Parameters           | 338,274                                              |\n| File size            | 1.3 MB (.safetensors fp32)                           |\n| Native input         | 994 × 2160 (iPhone H264 stream)                      |\n| Train input          | 497 × 1080 (2× downsample)                           |\n| Heatmap stride       | 1/8 of train resolution (= 1/16 of native after the 2× input downsample) |\n| Inference latency    | 10 ms (RTX 5080) / ~30 ms (M-series CoreML)          |\n| Val pos error (synth) | 20.8 px (P1 best, single pass; warm-restart cycles not yet run on v0.7.1) |\n| Real-frame error     | 6.4 px on bg-00000 (hand-annotated GT); 14 px on bg-00009 (hand) |\n| Real-frame deployment gate | 8 / 8 fixtures PASS (`conf ≥ 0.5 ∧ peak ≥ 0.4`) |\n| Weights license      | CC-BY-4.0                                            |\n\nSee [`docs/MODEL_CARD.md`](docs/MODEL_CARD.md) for the full evaluation breakdown\n(per-sample-type metrics, failure modes, comparison to v0.2 / v0.3.x).\n\n---\n\n## How it works\n\n```\niPhone screen capture (994×2160 BGR)\n        │\n        │  resize to 497×1080\n        ▼\n   conv backbone (3 stride-2 + 2 stride-1 blocks → 1/8 of train resolution)\n        │\n        ├── heatmap head (1×1 conv) → 63×135 sigmoid map → argmax → (cx, cy) in native px\n        └── conf head (global avg pool → MLP) → P(cursor present)\n```\n\nTrained on synthetic composites:\n- **backgrounds** — real iPhone screen captures (cursor-free) — bring your own\n- **cursor** — procedural smoothstep disc (peak alpha ≈ 0.25, luminance-matched\n  to the local background), resized to ~46 native px and composited at random\n  positions. v0.7 ships on this canonical synth target. A captured sprite at\n  `sprites/at_dot.png` may be substituted, but only when accompanied by an\n  approved sidecar manifest (sha256 + approver) — the loader fails hard\n  otherwise. The synthetic mix includes hard negatives (decoy shapes designed\n  to look cursor-like at a distance) and edge-clipped positives (cursor\n  partially off-screen).\n\nSee [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the full data + loss + training\nschedule writeup.\n\n---\n\n## Eval\n\n| Version | Val pos err | Cursor-free FPR | Real-frame top-1 hit | Inference (ms, RTX 5080) |\n|--------:|------------:|----------------:|---------------------:|-------------------------:|\n| v0.2    | 73.9 px     | 27%             | 41/50                | 10                       |\n| v0.3.4  | 30.5 px     | 1.7%            | 49/50                | 10                       |\n| v0.4.0  | 22.9 px     | \u003c2%             | —                    | 10                       |\n| v0.5.0  | 18.9 px     | \u003c2%             | —                    | 10                       |\n| ~~v0.6.x~~ | ~~24.1 px~~ | ~~—~~        | **wrong target — see CHANGELOG** | — |\n| v0.7.0  | 16.5 px     | \u003c2%             | 6/8 gate pass / 6.4 px hand | 10                |\n| **v0.7.1** | **20.8 px** | **\u003c2%**     | **8/8 gate pass / 6.4 px hand**  | **10**            |\n\nAll versions train on synthetic data (real backgrounds + composited cursor\nsprite) with a bg-level honest val split — apples-to-apples comparable from\nv0.3.x onward. v0.4.0 added correctness fixes (float labels through\naugmentation, parabolic subpixel refinement at inference). v0.5.0/v0.6.x\nshipped on a sprite asset that turned out to be a UI badge, not the cursor —\nv0.7 reverts to the v0.4 procedural smoothstep disc as the canonical synth\ntarget, adds a visual-validation gate that catches the failure pattern, and\nre-enables the previously-silent real-frame regression eval. v0.7 also\nunifies the decoder across all aux scripts (single canonical `decode.py`),\nswitches the heatmap BCE reduction from mean to sum with a calibrated\nHM_WEIGHT (the prior mean form diluted the localization gradient ~1400×\nrelative to the confidence head), and swaps confidence-head pooling from\naverage to max (the avg-pool washed out the cursor signal at the head).\nv0.7.0 cold-start landed at 16.5 px synth val (P4/E17), with 6.4 px error on the hand-annotated bg-00000 fixture. v0.7.1 rebalanced the heatmap loss weight (HM_WEIGHT 2e-3 → 8e-3 via a 3-point sweep) and lifted the heatmap peak distribution out of the deployment gate's marginal band — all 8 bundled real fixtures now pass the `conf ≥ 0.5 ∧ peak ≥ 0.4` gate (was 6/8 in v0.7.0). bg-00009 prediction error halved (27 → 14 px) without any architectural change. Synth val is slightly higher than v0.7.0 (20.8 vs 16.5) but that's a single-pass number — warm-restart cycles for v0.7.1 are not yet run.\n\nRun the eval harness yourself:\n\n```bash\npython eval_v03.py --v02 pointer_model_v0.2.0.pt --v07 pointer_model_v0.7.0.safetensors\n```\n\n---\n\n## Train your own\n\n```bash\ngit clone https://github.com/ellyseum/ios_pointer_finder.git\ncd ios_pointer_finder\npip install -e \".[dev]\"\n\n# 1. Capture iPhone backgrounds (cursor OFF). See capture_backgrounds.py for the workflow.\nmkdir -p backgrounds_kept\npython capture_backgrounds.py  # interactive curation\n\n# 2. Synthesize the training set (~150K samples).\npython synthesize.py --out dataset --n 150000\n\n# 3. Train.\npython train.py --dataset dataset --epochs 30 --augment\n\n# 4. Eval against v0.2 baseline.\npython eval_v03.py\n```\n\nSee [`docs/DATASET.md`](docs/DATASET.md) for background-collection guidance.\n\n---\n\n## Repo layout\n\n```\nios_pointer_finder/\n├── inference.py           # public PointerFinder API (load + predict)\n├── train.py               # training loop, semver-tagged checkpoints\n├── synthesize.py          # synthetic dataset generator\n├── eval_v03.py            # version comparison eval\n├── capture_backgrounds.py # iPhone screen capture pipeline\n├── extract_sprite.py      # extract a real cursor sprite from a screen capture (optional)\n├── grid_overlay.py        # debug overlay\n├── keep_picker.py         # interactive background curation\n├── cli.py                 # `ipf` console entry point\n├── scripts/\n│   ├── convert_pt_to_safetensors.py   # one-shot .pt → .safetensors converter\n│   ├── export_onnx.py                 # ONNX export\n│   └── export_coreml.py               # CoreML export (Mac)\n├── tests/                 # pytest suite (shape, golden image, smoke)\n├── docs/\n│   ├── MODEL_CARD.md\n│   ├── DATASET.md\n│   └── ARCHITECTURE.md\n├── .github/workflows/     # CI (ruff + pytest) + release automation\n├── VERSION                # current model version (semver)\n└── bump.sh                # patch/minor/major bump + tag\n```\n\n---\n\n## Model hosting\n\nTrained weights are distributed at:\n\n- **Primary**: [Hugging Face Hub — `ellyseum/ios_pointer_finder`](https://huggingface.co/ellyseum/ios_pointer_finder)\n  — versioned `.safetensors` + sidecar `config.json` + model card.\n- **Mirror**: [GitHub Releases](https://github.com/ellyseum/ios_pointer_finder/releases) — each\n  semver tag attaches the canonical `pointer_model_v{X}.{Y}.{Z}.safetensors` and a\n  matching `pointer_model_v{X}.{Y}.{Z}.config.json`.\n\nWe don't use Git LFS — at 1.3 MB per checkpoint and ~10 production checkpoints, hosting on\nHF/Releases is faster, cheaper, and more discoverable.\n\nTo convert any historical `.pt` checkpoint locally:\n\n```bash\npython scripts/convert_pt_to_safetensors.py pointer_model_v0.3.4_30.5px.pt\n# → pointer_model_v0.3.4_30.5px.safetensors + pointer_model_v0.3.4_30.5px.config.json\n```\n\n---\n\n## Development\n\n```bash\npip install -e \".[dev]\"\n\n# Lint + format\nruff check .\nruff format .\n\n# Tests\npytest -q\n\n# Bump version + tag a new model\n./bump.sh patch --commit\n```\n\nCI runs ruff + pytest on every push and PR. See\n[`CONTRIBUTING.md`](CONTRIBUTING.md) for the full PR / release workflow.\n\n---\n\n## Versioning history\n\nThe shipped weights are calibrated on **iPhone 16 Pro Max (`iPhone17,2`),\niOS 26.3.1**, portrait orientation, 994×2160 AirPlay mirror native. Each\nmajor version represents a deliberate stage in moving from \"works on one\ndevice\" to \"works on any current iPhone.\"\n\n| Stage | Tag(s)        | Story |\n|------:|---------------|-------|\n| PoC   | `v0.1` / `v0.2` | Initial proof: synthetic-data CNN learns the cursor sprite at all. Bg-leaky validation, 73.9 px val pos error. |\n| Heatmap head + honest val | `v0.3.0` … `v0.3.4` | Heatmap regression replaces xy-only regression. **Loss mask fix** so negatives no longer push down the heatmap globally. **Bg-level val split** so validation stops leaking. **Hard negatives** (decoy cursor shapes) so the model rejects icon dots and badges. **Train-time augmentation** + cosine restart for the overfit-fix wave. v0.3.4: 30.5 px val, FPR \u003c2%. |\n| Correctness wave I | `v0.4.0` | Float labels through augmentation, parabolic subpixel refinement at inference, mask-aware heatmap eval. 22.9 px val. |\n| Correctness wave II | `v0.5.0` | Stride-aware coord mapping (replaces linear stretch), parabolic on raw logits (replaces sigmoid-domain), tighter Gaussian σ, plain/hard neg loss split. **Also introduced a captured cursor sprite** that turned out to be a misidentified UI badge — see v0.6 row. 18.9 px val (on the wrong target). |\n| ~~v0.6.x~~ (unreleased) | — | Forward-signature simplification, asymmetric crop, hard-negative crop guard, decoder unification work — all sound, but kept training on the v0.5 wrong-sprite asset. Reached 24.1 px synth val before the asset was caught. **Never tagged.** v0.7 supersedes. |\n| **v0.7.0** | `v0.7.0` | Sprite catastrophe correction: removed the misidentified UI-badge asset; `synthesize.py` requires a sidecar manifest before loading any captured sprite (procedural smoothstep disc is the canonical fallback). Real-frame regression eval revived (was a silent no-op). Decoder extracted to `decode.py` and unified across `inference.py` / `click_at.py` / `eval_v03.py` / aux scripts. Heatmap BCE switched from mean to sum with calibrated `HM_WEIGHT=2e-3`. Outer-7-px border one-sided parabolic. Architecture-version pin in checkpoint sidecar. Visual-validation gate (SSIM + circularity) on synth output. 16.5 px synth val, 6.4 px on hand-annotated bg-00000. |\n| **v0.7.1** (current) | `v0.7.1` | Calibrated HM_WEIGHT 2e-3 → 8e-3 via 3-point sweep. Real-frame deployment gate goes from 6/8 to **8/8 PASS**. bg-00009 prediction error halved (27 → 14 px) without architectural change. Per-epoch backbone-gradient diagnostic added so the next loss-reduction A/B can't repeat v0.7's calibration miss. PEAK_THRESHOLD 0.5 → 0.4 to match v0.7's calibrated peak distribution. 20.8 px synth val (single P1; warm-restart cycles deferred). |\n| **Bootstrap loop (planned)** | — | Use a trained-enough agent to drive the cursor and emit verified real labels (move → observe → reverse-move → re-observe). Retrain on synthesis + verified-real. Eliminates the synthetic-data ceiling without manual labels. |\n| **Generalization (planned)** | — | Multi-device dataset (iPhone 15 series, 16 / Pro / Pro Max / Plus) collected via the bootstrap loop on each device. One model that works on any current iPhone in portrait + landscape. Per-app UI element classifier as a second perception head. |\n| **v1.0** | — | Stable public API + cross-platform export (CoreML / ONNX) + multi-device coverage. Frozen interface. |\n\n## Roadmap milestones\n\n- **Bootstrap loop / noisy-student self-labeling** (next)\n  - On-device explorer agent: drives the cursor, records frame + commanded position\n  - Move-and-undo dance for verified ground-truth labels (motion existence + reversibility, NOT commanded magnitude — iOS Tracking Speed is per-device-tunable)\n  - 3.5 s auto-hide window for cursor-free background capture\n  - Retrain on synthesis + verified-real mix; iterate until val saturates\n  - Goal: surpass the synthetic-only ceiling on real-frame eval\n\n- **Generalization across iPhones**\n  - Bootstrap on iPhone 15 / 15 Pro / 16 / 16 Pro / 16 Plus / 16 Pro Max\n  - Landscape orientation\n  - Single multi-device model\n  - Per-app UI element classifier as a second perception head\n\n---\n\n## Citation\n\n```bibtex\n@software{ios_pointer_finder_2026,\n  author       = {Jocelyn Ellyse},\n  title        = {ios\\_pointer\\_finder: a tiny CNN for iPhone Pointer-Control cursor detection},\n  year         = {2026},\n  url          = {https://github.com/ellyseum/ios_pointer_finder},\n  version      = {0.6.0}\n}\n```\n\nSee [`CITATION.cff`](CITATION.cff) for machine-readable citation metadata.\n\n---\n\n## License\n\n- **Code**: MIT (see [`LICENSE`](LICENSE))\n- **Trained weights**: CC-BY-4.0 (see [`LICENSE-WEIGHTS`](LICENSE-WEIGHTS))\n\nThe cursor sprite used during training (`sprites/at_dot.png`, 36×36 BGRA) was\ncaptured from a single high-resolution screenshot of the iOS Pointer-Control\ncursor and alpha-matted by hand. It ships in this repository as a small\ntraining artifact; iOS asset reproduction at this scale and form is fair-use\nresearch/utility, not a redistribution of any Apple image set. Hard-negative\ndecoy sprites are still procedurally generated in `synthesize.py`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fellyseum%2Fios_pointer_finder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fellyseum%2Fios_pointer_finder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fellyseum%2Fios_pointer_finder/lists"}