{"id":49107209,"url":"https://github.com/kimjune01/caret-recorder","last_synced_at":"2026-04-21T02:15:58.715Z","repository":{"id":343772257,"uuid":"1179111206","full_name":"kimjune01/caret-recorder","owner":"kimjune01","description":"Electron menu bar app for background screen + system audio recording with LiveKit integration and Swift accessibility sidecar","archived":false,"fork":false,"pushed_at":"2026-03-11T17:47:22.000Z","size":150,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-03-11T22:44:14.307Z","etag":null,"topics":["accessibility","desktop-app","electron","livekit","macos","mediarecorder","screen-recording","screencapturekit","swift","system-audio","typescript","webrtc"],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kimjune01.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-11T17:43:23.000Z","updated_at":"2026-03-11T17:47:26.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/kimjune01/caret-recorder","commit_stats":null,"previous_names":["kimjune01/caret-recorder"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/kimjune01/caret-recorder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kimjune01%2Fcaret-recorder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kimjune01%2Fcaret-recorder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kimjune01%2Fcaret-recorder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kimjune01%2Fcaret-recorder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kimjune01","download_url":"https://codeload.github.com/kimjune01/caret-recorder/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kimjune01%2Fcaret-recorder/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32073509,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-21T01:35:38.224Z","status":"online","status_checked_at":"2026-04-21T02:00:06.111Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["accessibility","desktop-app","electron","livekit","macos","mediarecorder","screen-recording","screencapturekit","swift","system-audio","typescript","webrtc"],"created_at":"2026-04-21T02:15:54.491Z","updated_at":"2026-04-21T02:15:58.709Z","avatar_url":"https://github.com/kimjune01.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Caret Desktop Recorder\n\nBackground screen recording + system audio capture desktop app with LiveKit WebRTC integration. Built with Electron, TypeScript, and a native Swift sidecar for macOS accessibility APIs.\n\n- **Screen + system audio capture** via ScreenCaptureKit (macOS 13+), no virtual audio driver needed\n- **LiveKit WebRTC publishing** — publish/unpublish screen share tracks without interrupting local recording\n- **Gapless segment rotation** — dual overlapping MediaRecorder instances for seamless 5-minute WebM chunks\n- **Swift accessibility sidecar** — real-time a11y tree traversal with 13 app-specific parsers (Chrome, Slack, Zoom, etc.)\n- **Menu bar app** — system tray UI with 3-state icons, no Dock presence\n- **75 tests** covering capture, recording, LiveKit, tray, sidecar crash recovery, and shutdown ordering\n\n## Architecture\n\n```\n┌─────────────────────────────────────────────────────┐\n│ Electron App                                        │\n│                                                     │\n│  Main Process                                       │\n│  ├── Tray (system tray menu, 3 states)              │\n│  ├── SidecarManager (spawn/restart Swift binary)    │\n│  ├── IPC handlers (file I/O, state relay)           │\n│  └── App lifecycle (flags, permissions, shutdown)    │\n│                                                     │\n│  Renderer Process (hidden window)                   │\n│  ├── capture.ts (getDisplayMedia)                   │\n│  ├── recorder.ts (MediaRecorder + segmentation)     │\n│  ├── livekit.ts (room + track publish)              │\n│  └── renderer.ts (orchestrator)                     │\n│                                                     │\n│  Preload (contextBridge IPC)                        │\n└──────────────┬──────────────────────────────────────┘\n               │ spawns\n               v\n┌──────────────────────────────────────────────────────┐\n│ Swift Sidecar (observer-sidecar)                     │\n│  ├── A11y APIs (AXUIElement)                         │\n│  ├── 13 app parsers (Chrome, Slack, Zoom, etc.)      │\n│  ├── HashStore (dedupe)                              │\n│  ├── SnapshotCapture (diff)                          │\n│  └── stdout: JSON Lines                              │\n└──────────────────────────────────────────────────────┘\n```\n\nAll user interaction via **system tray context menu** (no visible window). The renderer owns MediaStream/MediaRecorder/livekit-client (browser context required). The sidecar communicates via stdio (JSON Lines on stdout).\n\n## Key Technical Decisions\n\n| Decision | Choice | Rationale |\n|----------|--------|-----------|\n| Screen capture | `setDisplayMediaRequestHandler` with `useSystemPicker: false` | Auto-grants without picker dialog after initial OS permission |\n| System audio | `audio: 'loopback'` + Chromium flags | Native macOS 13+ ScreenCaptureKit, no third-party driver needed |\n| Video format | WebM (VP8 + Opus) | MediaRecorder limitation — VP8 is more reliable than VP9 across Electron builds |\n| Segment rotation | Dual overlapping MediaRecorder instances | Single stop/start has frame drops; two alternating recorders produce seamless segments |\n| Duration fix | `fix-webm-duration` | Chromium bug: MediaRecorder writes WebM without duration metadata |\n| Sidecar protocol | stdio JSON Lines | Simpler lifecycle than HTTP/WebSocket, no port conflicts |\n| LiveKit context | Data channel (`publishData`) | Separate from media tracks, structured JSON, chunked for \u003e15KB |\n| Dock hiding | `app.dock.hide()` + `type: 'panel'` + `LSUIElement` | Prevents dock icon reappearing when BrowserWindow interacts |\n\n## Setup\n\n### Prerequisites\n\n- Node.js 18+\n- macOS 14+ (for ScreenCaptureKit audio)\n- Xcode Command Line Tools (for Swift sidecar build)\n\n### Build the Swift Sidecar\n\nThe sidecar is a Swift executable in `sidecar-swift/` that uses macOS Accessibility APIs to observe the frontmost app.\n\n```bash\ncd sidecar-swift\nswift build --product caret-sidecar\nmkdir -p ../sidecar-bin\ncp .build/debug/caret-sidecar ../sidecar-bin/observer-sidecar\n```\n\n### Install \u0026 Run\n\n```bash\nnpm install\nnpm start\n```\n\n### LiveKit Configuration\n\nSet environment variables before starting:\n\n```bash\nexport LIVEKIT_URL=ws://localhost:7880\nexport LIVEKIT_TOKEN=\u003cyour-access-token\u003e\nnpm start\n```\n\nOr use LiveKit's [Meet app](https://meet.livekit.io/) to generate a token for testing.\n\n### Permissions\n\nOn first launch, macOS will prompt for:\n1. **Screen Recording** — required for capture\n2. **Accessibility** — required for the sidecar (a11y tree traversal)\n\nGrant both in System Settings \u003e Privacy \u0026 Security. The app may need a restart after granting permissions.\n\n## Usage\n\n1. Launch the app — it appears as a **gray circle** in the menu bar (no Dock icon)\n2. Click the tray icon \u003e **Start Recording** — icon turns **red**, segments start saving to `~/Documents/Caret/Recordings/`\n3. Click \u003e **Start Publishing** — icon turns **green**, screen + audio publish to LiveKit room\n4. Click \u003e **Stop Publishing** — returns to red, local recording continues\n5. Click \u003e **Stop Recording** — finalizes last segment, icon returns to gray\n6. Click \u003e **Quit** — clean shutdown (flushes final segment)\n\n### Recording Output\n\n```\n~/Documents/Caret/Recordings/\n├── 2025-01-15T10-30-00-000Z_000.webm   # Video segment 1\n├── 2025-01-15T10-30-00-000Z_001.webm   # Video segment 2\n└── context-2025-01-15T10-30-00-000Z.jsonl  # Sidecar context data\n```\n\n## Project Structure\n\n```\ncaret-recorder/\n├── src/\n│   ├── main.ts              — Electron entry, Chromium flags, hidden window, IPC\n│   ├── tray.ts              — System tray icon + context menu (3 states)\n│   ├── preload.ts           — contextBridge IPC exposure\n│   ├── renderer.ts          — Orchestrator (capture + recorder + livekit + sidecar data)\n│   ├── capture.ts           — getDisplayMedia wrapper\n│   ├── recorder.ts          — Dual MediaRecorder + 5-min segmentation\n│   ├── livekit.ts           — LiveKit room + track publish/unpublish/data\n│   ├── sidecar/\n│   │   ├── sidecar-manager.ts  — Spawn/readline/restart Swift binary\n│   │   └── types.ts            — SidecarEvent, payload types\n│   └── shared/\n│       └── types.ts            — AppState enum, IPC channels, config constants\n├── sidecar-swift/           — Swift sidecar source (macOS a11y observer)\n│   ├── Package.swift\n│   └── Sources/CaretSidecar/\n│       ├── main.swift\n│       ├── FrontmostAppObserver.swift\n│       ├── AccessibilityTraversal.swift\n│       ├── Payloads.swift\n│       └── JSONOutput.swift\n├── sidecar-bin/             — Built sidecar binary (gitignored)\n├── forge.config.ts          — Electron Forge + Vite config, extraResource for sidecar\n├── index.html               — Minimal shell (hidden window)\n└── package.json\n```\n\n## Data Flow\n\n```\nScreen + Audio → getDisplayMedia → MediaStream\n  ├── SegmentedRecorder → WebM segments → disk (~/Documents/Caret/Recordings/)\n  └── LiveKitPublisher → Track.Source.ScreenShare + ScreenShareAudio → LiveKit room\n\nSidecar stdout → SidecarManager.readline → parsed JSON\n  ├── IPC → renderer → LiveKit data channel (real-time context)\n  ├── disk (context-{timestamp}.jsonl alongside WebM segments)\n  └── Main process → tray tooltip (current app name)\n```\n\n## Perception Pipeline\n\nOffline pipeline that processes recorded JSONL into condensed markdown moments. See `docs/pipeline.md` for full architecture.\n\n```\nadapters → dedup → buffer → stripper → condensation → moments\n```\n\nThe first four stages are mechanical (no LLM). Condensation uses source-specific prompts to extract signal from stripped accessibility tree diffs.\n\n### LLM Setup\n\nCondensation requires [Codex CLI](https://github.com/openai/codex) for LLM calls (gpt-5.4 via ChatGPT account):\n\n```bash\nnpm install -g @openai/codex\ncodex login\n```\n\nThen run the full pipeline on recorded data:\n\n```bash\nnpx tsx pipeline/run_pipeline.ts ~/Documents/Caret/Recordings/context-*.jsonl\n```\n\n### Pipeline Tests \u0026 Benchmarks\n\n```bash\nnpx tsx pipeline/test_dedup.ts\nnpx tsx pipeline/test_buffer.ts\nnpx tsx pipeline/test_stripper.ts\nnpx tsx pipeline/test_condensation.ts\nnpx tsx pipeline/bench_pipeline.ts ~/Documents/Caret/Recordings/context-*.jsonl\n```\n\n## Known Limitations\n\n1. **WebM not MP4** — MediaRecorder API limitation in Chromium. VP8+Opus produces WebM; converting to MP4 would require ffmpeg post-processing\n2. **System audio requires macOS 13+** — Earlier versions need a virtual audio driver (BlackHole)\n3. **Screen Recording permission requires app restart** — macOS caches the grant; first launch may need manual restart\n4. **Hidden window appears in capture** — Electron's hidden BrowserWindow is included in full-screen capture (mitigated by 1x1 pixel size + `type: 'panel'`)\n5. **No upload** — Segments are saved locally only; background upload is not implemented\n6. **Single monitor** — Captures primary display only; multi-monitor selection is not exposed in UI\n7. **LiveKit data channel 15KB limit** — Large traversal payloads are chunked automatically, but receiver must reassemble\n\n## Testing\n\n```bash\nnpm test                        # Run all tests\nnpm run test:watch              # Watch mode\nnpm run test:coverage           # Coverage report\nnpm test -- --reporter=verbose  # See all test names\n```\n\nTests cover: 1080p/30fps capture, system audio, 5-min WebM segments, LiveKit publish/unpublish, system tray states, sidecar crash recovery, and clean shutdown ordering. The sidecar binary integration test runs automatically when the binary is present and is skipped otherwise.\n\n## Improvements\n\n- **Upload pipeline** — Background upload of segments to S3/GCS with retry logic\n- **ffmpeg transcoding** — Convert WebM segments to MP4 (H.264+AAC) post-capture for wider compatibility\n- **Multi-monitor picker** — Allow selecting which display to capture via tray submenu\n- **Performance profiling** — Measure actual CPU/memory impact of dual-recorder approach\n- **Auto-reconnect LiveKit** — The SDK handles it, but re-publishing tracks after reconnect needs explicit handling\n- **Sidecar binary signing** — Move from `extraResource` to `Frameworks/` for proper macOS code signing\n- **End-to-end tests** — Playwright/Spectron for full Electron lifecycle (segment files on disk, LiveKit track presence)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkimjune01%2Fcaret-recorder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkimjune01%2Fcaret-recorder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkimjune01%2Fcaret-recorder/lists"}