{"id":50316708,"url":"https://github.com/msitarzewski/mirror-mesh","last_synced_at":"2026-05-29T00:04:03.624Z","repository":{"id":360299805,"uuid":"1249528985","full_name":"msitarzewski/mirror-mesh","owner":"msitarzewski","description":"Apple Silicon real-time telepresence research with a consent-first identity protocol","archived":false,"fork":false,"pushed_at":"2026-05-25T20:05:27.000Z","size":4299,"stargazers_count":0,"open_issues_count":4,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-25T22:09:32.950Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Swift","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/msitarzewski.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE.md","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-25T19:53:32.000Z","updated_at":"2026-05-25T20:05:31.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/msitarzewski/mirror-mesh","commit_stats":null,"previous_names":["msitarzewski/mirror-mesh"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/msitarzewski/mirror-mesh","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msitarzewski%2Fmirror-mesh","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msitarzewski%2Fmirror-mesh/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msitarzewski%2Fmirror-mesh/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msitarzewski%2Fmirror-mesh/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/msitarzewski","download_url":"https://codeload.github.com/msitarzewski/mirror-mesh/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msitarzewski%2Fmirror-mesh/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33631007,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-28T02:00:06.440Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-29T00:04:02.659Z","updated_at":"2026-05-29T00:04:03.615Z","avatar_url":"https://github.com/msitarzewski.png","language":"Swift","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MirrorMesh\n\n![CI](https://github.com/msitarzewski/mirror-mesh/actions/workflows/ci.yml/badge.svg)\n[![License: AGPL-3.0-only](https://img.shields.io/badge/license-AGPL--3.0-blue.svg)](./LICENSE)\n[![Research only](https://img.shields.io/badge/use-research%20only-orange.svg)](./NOTICE.md)\n[![Apple Silicon](https://img.shields.io/badge/Apple%20Silicon-arm64-black.svg)](./memory-bank/decisions.md)\n[![DCO](https://img.shields.io/badge/DCO-required-orange.svg)](./CONTRIBUTING.md)\n\n**Realtime expressive telepresence for Apple Silicon, with consent and disclosure built into the architecture.** MirrorMesh ships the same realtime face-reenactment mechanics as a generic deepfake stack, but with a Consent-First Identity Protocol at the load gate, layered cryptographic disclosure on every frame, and an audible signal at every session start. Software defaults are policy. We set the policy to consent.\n\n---\n\n## What does it look like\n\nNo demo media has been published yet — the live `mirrormesh-app` window is the demo. Public screenshots and a recorded session video are planned for a follow-up release once the photoreal path's performance lands its 25-30 fps target (see [`CHANGELOG.md`](./CHANGELOG.md) for the latest progress).\n\nThe shipped app shows your face transformed in the hero view, the source camera as a small picture-in-picture inset, a telemetry panel with per-stage P50/P95/P99 latency histograms, and a watermark hero card with a pulsing green dot when signing is live.\n\n## Quick start\n\nRequires macOS 14+, Apple Silicon, and a full Xcode install (Command Line Tools alone is no longer sufficient as of v0.2.0; see [ADR-0012](./memory-bank/decisions.md)).\n\n```bash\ngit clone https://github.com/msitarzewski/mirror-mesh.git \u0026\u0026 cd mirror-mesh\nexport DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer\nswift build\nswift test --skip MirrorMeshStreamTests --skip MirrorMeshVoiceTests --skip MirrorMeshVirtualCameraTests --skip MirrorMeshMediaPipeTests\nswift run mirrormesh-app\n```\n\nHeadless variants (no camera needed):\n\n```bash\n# All-synthetic, watermarked, signed manifest\nswift run mirrormesh-bench --scenario bench/scenarios/demo.json\nswift run mirrormesh-verify --manifest bench/out/demo_*.manifest.json\n\n# Real Apple Vision against the bundled procedural fixture\nswift run mirrormesh-bench --scenario bench/scenarios/fixture.json\n\n# Record a watermarked .mov + sidecar manifest\nswift run mirrormesh-bench --scenario bench/scenarios/recorded.json\nopen bench/out/recorded_*.mov\n```\n\n## Why does this exist\n\nA widely-shared 2025 demo showed a male operator wearing a synthetic blonde \"influencer\" puppet in a live video swap, billed as *catfishing on steroids*. The mechanics that produce that demo are also the mechanics that enable real accessibility wins: gaze correction for people who lose eye contact while reading the screen, expression amplification for users with facial paralysis, and multilingual visual lip-sync for cross-language deaf-hearing communication. The technology is dual-use. The policy question is what software does **by default**.\n\nWe take the position that the appropriate default for identity-transforming media is consent at the load gate, disclosure on every frame, and provenance in every recorded artifact — encoded so they cannot be flipped off with a flag. Constraints by architecture, not by license.\n\n## Architecture\n\n```\nCapture (AVFoundation / Synthetic / File)\n  → Vision (Apple Vision 76-pt landmarks + One-Euro filter / MediaPipe 468-pt)\n  → Solver (Geometric closed-form / CoreML MLP, both conform to ExpressionSolver)\n  → Reenact (Stylized 266-vertex procedural head / FOMM photoreal — consent-gated)\n  → Render (Metal — Passthrough + Landmarks + Mesh + AvatarMask + Style picker)\n  → Watermark (Visible badge + Ed25519 frame signature)\n  → Output (Screen / .mov Recorder / Virtual camera CMIOExtension / WebRTC send)\n  ╰── Session manifest (signed JSON, tamper-evident)\n  ╰── JSONL telemetry trace + Instruments os_signpost intervals\n  ╰── Audible disclosure chirp (session start)\n```\n\n**Modules** (under `Sources/`):\n\n| Module | Role |\n|--------|------|\n| `MirrorMeshCore` | Frame types, telemetry actor, JSONL logger, signposts |\n| `MirrorMeshCapture` | `FrameSource` protocol; Live, Synthetic, File sources |\n| `MirrorMeshVision` | Apple Vision landmarks + One-Euro smoother |\n| `MirrorMeshMediaPipe` | 468-pt MediaPipe backend (Vision-fallback in v1.0) |\n| `MirrorMeshSolver` | `ExpressionSolver`; Geometric + CoreML implementations |\n| `MirrorMeshReenact` | Stylized 3D head puppet + `PhotorealBackend` (FOMM scaffold) |\n| `MirrorMeshRender` | Metal renderer; Wireframe / Mirror / Mask styles |\n| `MirrorMeshWatermark` | Ed25519 signer, badge, manifest, `ConsentedIdentity` |\n| `MirrorMeshRecorder` | `AVAssetWriter`-based watermarked `.mov` |\n| `MirrorMeshVirtualCamera` | `CMIOExtension` scaffolding |\n| `MirrorMeshStream` | WebRTC send-only (opt-in target) |\n| `MirrorMeshVoice` | `MicrophoneSource` + `WhisperTranscriber` |\n| `MirrorMeshTranslate` | Local Ollama LLM client |\n| `MirrorMeshOutput` | Top-level `Pipeline` orchestrator |\n| `MirrorMeshAppKit` | SwiftUI library + disclosure chirp |\n\n**Performance** (Mac17,6 / Apple M5 Max / macOS 26.5):\n\n| Scenario | Mode | E2E P50 ms |\n|----------|------|-----------:|\n| `demo.json` | synthetic | **1.4** |\n| `fixture.json` | file → real Apple Vision | **5.1** |\n| Live camera + Vision (interactive) | live | **~11** |\n\nFull per-stage tables in [`paper/draft_v1.md`](./paper/draft_v1.md) Section 6.4.\n\n## The trust layer\n\nEvery frame leaving the renderer carries:\n\n1. **Visible \"MIRRORMESH • SYNTHETIC\" badge** composited into the frame\n2. **64-byte Ed25519 signature** over `(frameID ‖ hostTimeNs ‖ SHA-256(BGRA pixels))` — `Sources/MirrorMeshWatermark/FrameSigner.swift:20-32`\n3. **Signed session manifest** (canonical JSON, sorted keys, ISO-8601 dates) recording device, pipeline config, models with provenance, frame count, and consent record — verifiable via `mirrormesh-verify`\n4. **Audible chirp** at session start (A4 → E5, 250 ms, locked-on in release builds)\n\nIdentity transformations additionally require a verified `ConsentedIdentity` bundle (`.mmid`):\n\n- Ed25519-signed JSON header + PNG payload\n- Three identity schemes: `.selfAsSource`, `.stylizedNonHuman`, `.consentedThirdParty`\n- Scope grammar `vX.Y+` enforces runtime-version compatibility\n- The reenactor refuses to initialize without verification — `Sources/MirrorMeshReenact/FaceReenactor.swift:56-72`\n- The third-party CLI flow requires a literal consent phrase — `Sources/mirrormesh-consent/ConsentCLI.swift:55-70`\n\nThe full protocol spec is in [`docs/CONSENT_PROTOCOL.md`](./docs/CONSENT_PROTOCOL.md).\n\n## Project status\n\n**v1.0.0 candidate.** Production-ready for: the synthetic mesh-overlay path (Wireframe / Mirror / Mask styles), the stylized procedural head reenactor, the consent bundle protocol, the layered watermark, the recorder, the bench harness, and the CLI tools. Alpha for: the FOMM photoreal path (load gate complete, inference graph wiring is v1.1), `whisper.cpp` (mock backend ships; real `.cxxTarget` lands v1.1), MediaPipe (Vision-fallback ships; XCFramework lands v1.1), C2PA emission, multi-face tracking.\n\nFull known-limitations list in [`RELEASE_NOTES_v1.0.0.md`](./RELEASE_NOTES_v1.0.0.md).\n\n## Contribute\n\nPRs welcome. We use [DCO sign-off](./CONTRIBUTING.md) (Linux-kernel-style); no CLA. Every commit needs `git commit -s`.\n\nThe project's load-bearing rules live in [`memory-bank/projectRules.md`](./memory-bank/projectRules.md) — please read R1 (no third-party identity spoofing), R2 (watermarking mandatory), R3 (local-only inference), and R12 (refuse-on-sight list) before opening a PR that touches the trust layer.\n\n## License\n\n**[AGPL-3.0-only](./LICENSE).** Research project — see [NOTICE.md](./NOTICE.md) for the plain-English statement of intent.\n\nThe maintainer does not monetize this code and does not offer a commercial license. AGPL-3.0's strong copyleft + network-use clause prevents anyone else from monetizing derivatives. If you fork it, your fork is AGPL too; if you host it as a service, you must publish your source.\n\nThe trust-layer invariants (R1 / R2 / R12) are architectural and survive licensing. They're enforced by the code, not by the legal text.\n\nHistory: Apache-2.0 in v0.1.0 through v0.3.0; AGPL-3.0 + Commercial dual at v0.4.0 ([ADR-0014](./memory-bank/decisions.md)); simplified to AGPL-3.0-only at v1.0.0 ([ADR-0015](./memory-bank/decisions.md)).\n\n---\n\n*Realtime telepresence with the receipts. © 2026 Michael Sitarzewski.*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmsitarzewski%2Fmirror-mesh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmsitarzewski%2Fmirror-mesh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmsitarzewski%2Fmirror-mesh/lists"}