https://github.com/hinanohart/sketchpolicy
Simulator-free, CPU-only kinematic demonstration multiplier for LeRobotDataset v3.0 (pre-alpha)
https://github.com/hinanohart/sketchpolicy
apache-2 cpu-only data-augmentation imitation-learning lerobot qmc robotics se3
Last synced: about 1 month ago
JSON representation
Simulator-free, CPU-only kinematic demonstration multiplier for LeRobotDataset v3.0 (pre-alpha)
- Host: GitHub
- URL: https://github.com/hinanohart/sketchpolicy
- Owner: hinanohart
- License: mit
- Created: 2026-05-29T14:29:04.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2026-06-10T12:08:43.000Z (about 1 month ago)
- Last Synced: 2026-06-10T12:22:54.475Z (about 1 month ago)
- Topics: apache-2, cpu-only, data-augmentation, imitation-learning, lerobot, qmc, robotics, se3
- Language: Python
- Homepage: https://github.com/hinanohart/sketchpolicy
- Size: 91.8 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Notice: NOTICE
Awesome Lists containing this project
README
# sketchpolicy
**Simulator-free, CPU-only kinematic demonstration *multiplier* for `LeRobotDataset` v3.0.**
> ⚠️ **Pre-alpha bootstrap scaffold (v0.1.0a3).** This is a research scaffold,
> not a production tool. Every write command requires the explicit flag
> `--i-understand-this-is-a-bootstrap-scaffold`. Read the
> [Scope & honest limitations](#scope--honest-limitations) section before using.
`sketchpolicy` takes a *valid* `LeRobotDataset` v3.0 episode whose actions are
end-effector poses, and produces additional schema-valid episodes by applying
**deterministic kinematic transforms**:
- **Transforms**: viewpoint / object-pose / planar position / time-warp, sampled with a low-discrepancy Sobol sequence.
- **Feasibility filter**: variants that leave the reachable shell, dip below the table, exit the workspace box, or enter the base keep-out cylinder are discarded and resampled.
- **No simulator, no neural network, no GPU.**
It also ships an *experimental* "sketch" adapter that turns a monocular video
of a **human hand** into an end-effector trajectory using MediaPipe Hands
(Apache-2.0, CPU, 21 landmarks). "Film your hand → sketch a trajectory."
## Install
```bash
pip install sketchpolicy # core: numpy / scipy / pyarrow (torch-free)
pip install "sketchpolicy[replay]" # + headless pybullet sanity replay
pip install "sketchpolicy[ingest]" # + experimental hand-video adapter (mediapipe)
pip install "sketchpolicy[lerobot]" # + live-reader cross-check (heavy: pulls torch+CUDA)
```
## Quickstart
```bash
# Multiply an EE-action dataset: 1 source episode -> N feasible kinematic variants
sketchpolicy augment ./my_dataset --out ./my_dataset_x10 --n 10 --seed 0 \
--i-understand-this-is-a-bootstrap-scaffold
# Read-only: render a kinematic replay of one episode (needs [replay])
sketchpolicy replay ./my_dataset --episode 0
# Environment / dependency report
sketchpolicy doctor
# Experimental: hand video -> EE trajectory episode (needs [ingest])
sketchpolicy sketch ./hand_clip.mp4 --out ./sketched \
--i-understand-this-is-a-bootstrap-scaffold
```
## How it works
1. **Load** — reads a `LeRobotDataset` v3.0 episode into an `EEPlan` (end-effector
pose + gripper time series).
2. **Sample** — a seeded [Sobol quasi-Monte Carlo](https://en.wikipedia.org/wiki/Sobol_sequence)
sequence draws transform parameters (rotation offset, translation, time-warp
factor) within a configurable `ParamRanges` box.
3. **Transform** — each draw is applied as a deterministic kinematic transform to
every waypoint in the plan.
4. **Filter** — a feasibility check rejects candidates that violate workspace
constraints: reachable-shell radius, table clearance, workspace bounding box,
and base keep-out cylinder. Infeasible draws are discarded and the sampler
continues until `n` feasible variants are collected.
5. **Emit** — accepted variants are written as schema-valid `LeRobotDataset` v3.0
episodes. An optional pybullet replay and an optional lerobot-dataset reader
cross-check can validate the output.
The full augmentation loop lives in `src/sketchpolicy/augment/pipeline.py`. The
same seed always produces bit-identical output (`AugmentResult` is frozen and
deterministic end-to-end).
## Architecture
## Benchmark results
The numbers below come from `bench_results/v0.1.0a3.json`, produced by
`python scripts/measure.py` on the build machine (Linux WSL2, Python 3.12,
numpy 2.4 / scipy 1.17 / pyarrow 24). They are **operational** metrics —
round-trip validity, determinism, the feasibility filter, CPU throughput and a
replay smoke test. They are **not** accuracy or policy-success metrics (there is
no robot ground truth in this benchmark). Re-run `measure.py` to regenerate.
| metric | value |
| --- | --- |
| augment round-trip schema-valid | ✓ (recovered actions within 1e-6) |
| same-seed bit-exact determinism | ✓ |
| reject-resample discards infeasible | ✓ (3 of 8 draws rejected on the boundary fixture) |
| MediaPipe Hands CPU throughput | 47.7 fps (480×640, median of 3, synthetic-frame throughput, not detection accuracy) |
| pybullet replay smoke | ✓ (max IK residual 0.01 m on a reachable clip) |
## Scope & honest limitations
**What sketchpolicy CLAIMS (and tests):**
- Given a valid EE-action `LeRobotDataset` v3.0 input, it emits **schema-valid**
v3.0 output that round-trips through its own torch-free reader against the
declared v3.0 schema. With the optional `[lerobot]` extra installed, an
advisory test also loads the output with the real `lerobot-dataset` reader.
- Transforms are **deterministic**: the same `--seed` produces bit-exact output.
- The feasibility filter **rejects** variants that leave the reachable shell,
dip below the table, exit the workspace box, or enter the base keep-out
cylinder, and resamples to hit the requested count.
- It runs on **CPU** with no torch in the core install.
**What sketchpolicy does NOT claim (hard limits):**
- ❌ The feasibility filter is a **kinematic-envelope** check, not a full
collision check: it does **not** test robot-mesh self-intersection (there is
no robot mesh in the no-IK representation).
- ❌ It does **not** improve any policy's success rate. It is a data-shaping
tool; downstream training quality is out of scope and untested here.
- ❌ It does **not** recover absolute metric scale or full 6-DoF accuracy from a
monocular hand video. The sketch adapter is experimental and relative.
- ❌ It does **not** do sim-to-real, and it does **not** transform camera video;
v0.1 augments the **proprioceptive / action stream only** (EE pose + gripper).
Output episodes are action/state-only.
- ❌ Parallel-jaw gripper only in v0.1 (no dexterous / bimanual).
It depends on **MediaPipe Hands only** for hand pose. It does **not** use MANO,
HaMeR, or WiLoR (non-commercial / GPU-ViT — see `NOTICE`).
## License
MIT. See `LICENSE` and `NOTICE`.