{"id":51166362,"url":"https://github.com/kiranandcode/quire","last_synced_at":"2026-06-26T19:02:31.220Z","repository":{"id":348581746,"uuid":"1198791492","full_name":"kiranandcode/quire","owner":"kiranandcode","description":"Quire: bicycle for your brain","archived":false,"fork":false,"pushed_at":"2026-04-01T19:34:01.000Z","size":347,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-02T06:42:17.924Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Dart","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kiranandcode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-01T19:03:48.000Z","updated_at":"2026-04-01T19:34:05.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/kiranandcode/quire","commit_stats":null,"previous_names":["kiranandcode/quire"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/kiranandcode/quire","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kiranandcode%2Fquire","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kiranandcode%2Fquire/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kiranandcode%2Fquire/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kiranandcode%2Fquire/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kiranandcode","download_url":"https://codeload.github.com/kiranandcode/quire/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kiranandcode%2Fquire/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34829415,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-26T02:00:06.560Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-26T19:02:25.314Z","updated_at":"2026-06-26T19:02:31.202Z","avatar_url":"https://github.com/kiranandcode.png","language":"Dart","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Quire\n\nE-ink AI interface for Boox Go 10.3 2Lumi. Handwriting in, Claude out.\n\n![Quire screenshot](assets/screenshot.png)\n\n## Design\n\nInfinite canvas with an elegant pen-and-paper aesthetic, optimised for e-ink stylus input.\n\n### Tools\n\n| Tool | Stylus | Touch |\n|------|--------|-------|\n| **Pen** | Draw strokes → detect → preview → send to Claude | Pan / pinch-zoom |\n| **Select** | Draw selection rectangle / drag to move | Pan / pinch-zoom |\n\n### Pen Flow\n\nWrite with the stylus. After a 2.5s pause, strokes are sent to the server for detection. The server renders strokes to an image, runs CRAFT text detection + connected component analysis to separate text from drawings, and OCR's the text via LightOnOCR-2-1B. Results appear as canvas annotations (digitised text above handwriting, bounding boxes around detected drawings).\n\nA **countdown bar** appears below your writing: \"sending to claude\" with a 5-second timer and a pause button.\n\n- **Auto-send**: if the countdown completes, strokes are sent to Claude automatically\n- **Pause**: press the pause button to stop the countdown. The UI hides for 10 seconds (or while you keep writing). When you stop, a review prompt appears with an explicit send button\n- **Review**: in review state, you can edit the digitised OCR text (tap to edit), reclassify regions, or manually trigger send\n- **Streaming**: Claude's response appears word-by-word as it's generated\n- **Response annotations**: draw on/near a Claude response to annotate it. The system detects overlap with existing responses, sends the original response text + annotation strokes (including a rendered image) to Claude for iteration. Countdown/send UI appears at the thread bottom. Annotations on rendered diagrams send the diagram image.\n- **Render error feedback**: when LaTeX rendering fails, a spinner + error snippet shows on canvas while Claude retries (up to 5 rounds)\n- **Session tracking**: writing near an existing response continues the same conversation\n- **Parallel conversations**: write far enough apart horizontally to start a separate thread\n- **Conversation labels**: each thread gets a letter tag (A, B, C...) shown on responses\n- **Model picker**: choose between Sonnet, Opus, or Haiku in Settings\n- **LaTeX/SVG detection**: mathematical notation (typing rules, equations) is automatically sent as an image so Claude sees the drawing directly\n- **Tap-to-dot**: quick stylus taps register as dots (for colons, periods, punctuation)\n\n### Selection\n\nDraw a rectangle with the select tool, or drag inside an existing selection to move objects. Context menu:\n\n- **To Text** — OCR selected strokes, replace with text object\n- **Send** — send selected content to Claude (in review state)\n- **Move** — enter move mode for stylus drag\n- **Delete** — remove selected objects\n\nTap a Claude response to view the OCR'd source text.\n\n### Architecture\n\n```\nFlutter App (Boox)\n  ↕ SSE over HTTP/USB (adb reverse)\nRust Server (axum, :8080)\n  ↕ HTTP (:8090)\nPython OCR Sidecar (LightOnOCR-2-1B + CRAFT, persistent daemon)\n  ↕ HTTPS\nClaude API (Anthropic)\n```\n\n**Detection pipeline** (per pen stroke batch):\n1. Render strokes to image (Rust, `render.rs`)\n2. CRAFT text detection → text region bounding boxes (Python, easyocr)\n3. Group overlapping/nearby CRAFT boxes into clusters (LINE_GAP=80)\n4. OCR each cluster individually with LightOnOCR-2-1B\n5. If OCR returns LaTeX/HTML or garbled/repeated text → reclassify cluster as image region\n6. Remaining ink pixels outside all clusters → drawing/image region\n7. Merge overlapping/nearby image regions into one\n8. **Viral absorption**: text surrounded by drawing ink on 3+ sides gets absorbed into the nearest image (labels inside diagrams become part of the drawing)\n9. Results returned to Flutter for preview (OCR annotations + image bboxes)\n10. On send: text sent as text to Claude, drawing regions cropped and sent as base64 images\n11. Response streamed back via SSE; user-reviewed regions skip server-side detection\n\n**Response rendering**: Claude is instructed via system prompt + `render_diagram` tool to never include LaTeX/SVG inline. When Claude calls the tool, the server renders via `pdflatex`/`rsvg-convert` → PNG and streams the image to the client as a `DiagramObject` on the canvas.\n\n**Two-phase flow**: detection runs first via `POST /detect`, user reviews, then sends via `POST /chat/stream` with pre-classified regions.\n\n### Coordinate system\n\nStrokes stored in world space. Transform: `world = (screen - offset) / scale`. The render pipeline tracks coordinate mapping (`RenderResult` struct with `pixel_to_world` and `world_to_pixel`) so image-space bounding boxes from detection can be converted between coordinate spaces. Early pointer events captured in `Listener.onPointerDown` before gesture recognizer; stylus taps captured via `onTapUp` for dot/punctuation input.\n\n## Structure\n\n```\napp/lib/\n  main.dart                      # Entry, init Onyx SDK + settings\n  theme/eink_theme.dart          # Serif, thin borders, pen-and-paper aesthetic\n  models/canvas_object.dart      # StrokeObject, TextObject, ThinkingObject, ConversationThread,\n                                 # OcrAnnotationObject, ImageBboxObject, DiagramObject\n  screens/\n    canvas_screen.dart           # Infinite canvas, pen flow state machine, preview UI, selection\n    settings_screen.dart         # Backend config, model picker, debug toggle\n    calibration_screen.dart      # OCR data collection (temporary scaffolding)\n  services/\n    settings_service.dart        # SharedPreferences persistence (incl. Claude model selection)\n    ocr_service.dart             # HTTP client for /ocr\n    chat_service.dart            # SSE streaming client for /chat/stream, detection client for /detect\n    render_service.dart          # HTTP client for /render (LaTeX/SVG → PNG)\n    debug_service.dart           # Trace logging, screenshots, canvas dumps\n  utils/\n    response_parser.dart         # Split Claude responses into markdown/latex/svg segments\n\nserver/\n  src/main.rs                    # Axum server: /ocr, /chat, /chat/stream, /detect, /render, /debug, /screenshot, /canvas-dump\n  src/ocr.rs                     # OCR sidecar client (recognize + detect)\n  src/claude.rs                  # Claude API client (sync + streaming)\n  src/render.rs                  # Stroke-to-image rendering + LaTeX/SVG → PNG via pdflatex/rsvg-convert\n  src/log.rs                     # Structured JSON logging to file\n  ocr/server.py                  # OCR sidecar: LightOnOCR + CRAFT text detection + cluster-based classification\n  ocr/requirements.txt           # Python deps\n\nocr_benchmark/                   # OCR model benchmarking scripts\n```\n\n## Dev\n\n```bash\n# 1. Start OCR sidecar (stays running, keeps models in memory)\ncd server \u0026\u0026 source ocr/.venv/bin/activate\npython3 ocr/server.py --preload\n\n# 2. Start Rust server (in another terminal)\ncd server \u0026\u0026 ANTHROPIC_API_KEY=sk-... cargo run --release\n\n# 3. Build and deploy Flutter app\ncd app\nflutter build apk --release\nadb install -r build/app/outputs/flutter-apk/app-release.apk\nadb reverse tcp:8080 tcp:8080\nadb shell am start -n com.example.quire/.MainActivity\n```\n\n### Debug mode\n\nEnable in Settings. The app pushes trace events on every interaction (tool switch, stroke, OCR, Claude response) to `POST /debug`. Screenshot button captures canvas + state dump.\n\n**Debug artifacts** (stored on the machine running the Rust server):\n- `server/quire_server.log` — structured JSON log of every event: detection results, render requests/errors, tool use rounds, Claude responses, client debug traces\n- `server/quire_screenshots/` — PNG screenshots captured via the debug button, named by timestamp. These show the full canvas state including strokes, responses, bounding boxes. Useful for diagnosing detection/rendering issues.\n- `server/quire_dumps/` — JSON dumps of all canvas objects (strokes with bboxes, text objects with full response text, annotations, image bboxes, diagrams). Captured alongside screenshots. Useful for inspecting what objects are on canvas and their positions/content.\n\nWhen debugging an issue: check the latest screenshot to see what the user saw, the dump to see the object state, and the log to trace the detection → classification → Claude → render pipeline.\n\n## TODO\n\n- [x] LaTeX/SVG rendering via `render_diagram` tool use (system prompt, tool loop, DiagramObject)\n- [x] Per-cluster OCR detection (CRAFT box clustering, individual OCR, LaTeX auto-reclassify)\n- [x] Viral image absorption (text surrounded by drawing ink on 3+ sides → absorbed into image)\n- [x] Image region merging (overlapping/nearby image bboxes merged into one)\n- [x] Reclassify context menu: → Text, → Image, Merge (for multiple image bboxes)\n- [x] Garbled OCR detection (repeated patterns → classify as image)\n- [x] Response annotations: detect strokes near responses, send with context, thread-bottom UI\n- [x] Render error feedback: spinner + error snippet on canvas during retries\n- [x] Buffer-aware auto-send: re-detects if user drew more strokes since last detection\n- [ ] Geometry-preserving annotation composites (strikethrough, arrows, inline comments as image)\n- [ ] OCR annotations as child objects: bound to parent strokes, editable via keyboard, move/delete with parent\n- [ ] HTML rendering in responses\n- [ ] Code execution blocks: runnable Python sandbox on server, Run button, stdout/stderr display\n- [ ] Conversation trees: branch conversations with drawn lines between threads\n- [ ] Agentic SDK integration for tool use\n- [ ] Modular/extensible UI architecture\n\n## Dependencies\n\n### Flutter\n- `perfect_freehand` — pressure-sensitive stroke rendering\n- `onyxsdk_pen` — Boox hardware pen acceleration\n- `shared_preferences` — settings persistence\n- `http` — HTTP client (SSE streaming)\n- `path_provider` — device storage\n\n### Server\n- `axum` — HTTP server (incl. SSE)\n- `reqwest` — HTTP client (Claude API, OCR sidecar)\n- `futures` / `tokio-stream` — streaming support\n- `image` — stroke rendering\n- `tempfile` — temp dirs for LaTeX/SVG rendering\n- `serde` / `serde_json` — serialization\n- `uuid` — session IDs\n- `chrono` — timestamps\n\n### OCR Sidecar\n- `transformers` \u003e= 5.0 — LightOnOCR model\n- `torch` — inference\n- `easyocr` — CRAFT text detection\n- `scipy` — connected component analysis\n- `Pillow` / `numpy` — image processing\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkiranandcode%2Fquire","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkiranandcode%2Fquire","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkiranandcode%2Fquire/lists"}