{"id":50908906,"url":"https://github.com/mosure/burn_jepa","last_synced_at":"2026-06-16T08:01:28.526Z","repository":{"id":358602416,"uuid":"1234015816","full_name":"mosure/burn_jepa","owner":"mosure","description":"burn jepa inference and training 🔥🔮","archived":false,"fork":false,"pushed_at":"2026-05-31T23:13:31.000Z","size":3676,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-06-01T01:11:41.444Z","etag":null,"topics":["burn","feature-upsampling","jepa","pca","sparse","ttt","vision"],"latest_commit_sha":null,"homepage":"https://mosure.github.io/burn_jepa/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mosure.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-09T16:34:21.000Z","updated_at":"2026-05-31T23:13:37.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mosure/burn_jepa","commit_stats":null,"previous_names":["mosure/burn_jepa"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mosure/burn_jepa","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mosure%2Fburn_jepa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mosure%2Fburn_jepa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mosure%2Fburn_jepa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mosure%2Fburn_jepa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mosure","download_url":"https://codeload.github.com/mosure/burn_jepa/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mosure%2Fburn_jepa/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34396429,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-16T02:00:06.860Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["burn","feature-upsampling","jepa","pca","sparse","ttt","vision"],"created_at":"2026-06-16T08:01:26.716Z","updated_at":"2026-06-16T08:01:28.516Z","avatar_url":"https://github.com/mosure.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# burn_jepa 🔥🔮\n\n[![test](https://github.com/mosure/burn_jepa/workflows/test/badge.svg)](https://github.com/mosure/burn_jepa/actions?query=workflow%3Atest)\n[![deploy github pages](https://github.com/mosure/burn_jepa/workflows/deploy%20github%20pages/badge.svg)](https://github.com/mosure/burn_jepa/actions?query=workflow%3A%22deploy+github+pages%22)\n[![crates.io](https://img.shields.io/crates/v/burn_jepa.svg)](https://crates.io/crates/burn_jepa)\n[![docs.rs](https://docs.rs/burn_jepa/badge.svg)](https://docs.rs/burn_jepa)\n[![preprint](https://img.shields.io/badge/preprint-pdf-blue)](./docs/papers/vjepa21_ttt_sparse_temporal_preprint.pdf)\n\nburn-native sparse V-JEPA 2.1 inference and training, with sparse patchify,\ninterframe token memory, SC-TTT temporal adapters, AnyUp/PCA visualization, and\nnative/wasm Bevy demos.\n\n\u003cimg src=\"./docs/jepa_sparse_pipeline_frame.png\" alt=\"input, sparse mask, low-res PCA, and high-res PCA from the sparse V-JEPA pipeline\"\u003e\n\nframe 3 from the e2e sparse gallery, generated with the local f16 V-JEPA 2.1\nbase package and f16 AnyUp package: input, patch-diff sparse mask, low-res\ntoken-cache PCA, and high-res AnyUp PCA.\n\n## features\n\n### high-level\n\n- Loads V-JEPA 2.1 Burn packages and upstream-style `safetensors` checkpoints.\n- Runs base V-JEPA 2.1 or trained SC-TTT V-JEPA 2.1 encoders.\n- Supports dense, patch-diff sparse, AutoGaze sparse, and precomputed mask\n  inputs.\n- Skips masked pixels with flex-gmm sparse patchify on WGPU/CUDA lanes.\n- Keeps a persistent full-frame token feature cache with sparse device updates.\n- Projects low-res token features and high-res AnyUp features through rolling\n  PCA for live visualization.\n- Ships training configs, smoke tests, benchmarks, a paper artifact, and a\n  native/wasm `bevy_jepa` viewer.\n\n### cargo features\n\n| feature | default | target | notes |\n|---|---:|---|---|\n| `ndarray` | yes | native | CPU reference backend |\n| `webgpu` | yes | native/web | Burn WebGPU backend |\n| `wgpu` | no | native/web | Burn WGPU backend without selecting the WebGPU compiler feature |\n| `cuda` | no | native | CUDA backend |\n| `flex` | no | native/web | Burn dispatch/flex experiments |\n| `dispatch` | no | native/web | Burn dispatch backend experiments |\n| `wasm` | no | wasm32 | wasm-bindgen API over Burn WebGPU |\n| `wasm-fusion` | no | wasm32 | experimental Burn fusion path; browser WebGPU validation still prefers the default raw WebGPU build |\n| `sparse-patchify-wgpu` | no | native/web | flex-gmm sparse patchify + sparse feature memory on WGPU; enabled by default in `bevy_jepa` |\n| `sparse-patchify-cuda` | no | native | flex-gmm sparse patchify + sparse feature memory on CUDA |\n| `autogaze-*` | no | native/web | optional `burn_autogaze` mask projection adapters |\n\n## burn support\n\n| burn_jepa | burn | burn-store | status |\n|---|---:|---:|---|\n| `0.21.x` | `0.21.x` | `0.21.x` | current |\n| `\u003c0.21` | `\u003c0.21` | `\u003c0.21` | not supported in this repo |\n\n## quick start\n\n```rust,no_run\nuse burn::backend::NdArray;\nuse burn_jepa::{\n    make_context_target_masks, VJepaConfig, VJepaPipeline, VJepaVideoShape,\n};\n\ntype B = NdArray\u003cf32\u003e;\n\nlet device = Default::default();\nlet config = VJepaConfig::tiny_for_tests();\nlet pipeline = VJepaPipeline::\u003cB\u003e::random(config.clone(), \u0026device);\n\nlet shape = VJepaVideoShape::new(1, 3, 4, 32, 32);\nlet video = VJepaPipeline::\u003cB\u003e::tensor_from_frames(\n    \u0026vec![0.0; shape.num_values()],\n    shape,\n    \u0026device,\n)?;\n\nlet (context, target) = make_context_target_masks(config.token_grid(), 0.5);\nlet output = pipeline\n    .model()\n    .predict_dense_targets(video, \u0026context, \u0026target)?;\n\nassert_eq!(output.predictions.shape().dims::\u003c3\u003e(), [1, target.len(), 32]);\n# Ok::\u003c(), anyhow::Error\u003e(())\n```\n\n## bevy viewer\n\n```sh\ncargo run -p bevy_jepa\ncargo run -p bevy_jepa -- --source camera --image-size 256\ncargo run -p bevy_jepa -- --source camera --image-size 1024\ncargo run -p bevy_jepa -- --encoder-source base-checkpoint --sparse-encode-mode dense\ncargo run -p bevy_jepa -- --encoder-source trained-ttt --mask-source patch-diff\n```\n\ninstall the viewer from git main with the package name as the positional crate\nargument. Some Cargo versions do not support `cargo install --package`:\n\n```sh\ncargo install --git https://github.com/mosure/burn_jepa.git --branch main bevy_jepa --locked --force\n```\n\nthe no-arg native default starts the live sparse feature pipeline with camera\ninput, patch-diff sparse masks, low-res token-cache PCA every processed frame,\nand high-res AnyUp/PCA decoupled from the low-res worker. Controls live in the\nin-app menu, including TTT vs. base V-JEPA 2.1 vs. RAC/autocode, dense vs.\nsparse, patch-diff threshold, refresh policy, 256/512/1024 resolution, and\nAnyUp mode.\n\nSee [crates/bevy_jepa/README.md](./crates/bevy_jepa/README.md) for native,\nwasm, camera, CDN model-package, and Pages notes.\n\n## training\n\n```sh\ncargo run --bin burn-jepa -- print-config \u003e train.toml\ncargo run --bin burn-jepa -- train-ttt --config train.toml\ncargo run --bin burn-jepa -- eval-ttt --config train.toml --model ttt-model.mpk --no-full-grid\n```\n\nSC-TTT training distills a sparse per-frame student toward the V-JEPA 2.1 3D\nteacher while carrying bounded fast-memory state through video windows. Current\nproduction configs and stability notes live in `configs/production/` and\n`docs/production-ttt-status.md`.\n\n## docs\n\n- [preprint pdf](./docs/papers/vjepa21_ttt_sparse_temporal_preprint.pdf) and\n  [source](./docs/papers/vjepa21_ttt_sparse_temporal_preprint.tex)\n- [high-res AnyUp/PCA pipeline](./docs/highres-anyup-pca-pipeline.md)\n- [interframe feature memory](./docs/interframe-feature-memory.md)\n- [TTT training notes](./docs/ttt-training.md)\n- [production runbook](./docs/production-training-runbook.md)\n- [engineering roadmap](./docs/engineering-roadmap.md)\n- [JEPA autocode track](./docs/jepa-autocode.md)\n- [benchmark results](./docs/e2e-benchmark-results.md)\n\n## validation\n\n```sh\ncargo test\ncargo check -p bevy_jepa --target wasm32-unknown-unknown --features bevy-web-demo\ncargo test -p burn_jepa --no-default-features --features ndarray,wgpu,sparse-patchify-wgpu highres\n```\n\nhardware-specific CUDA/WGPU benches, long-rollout checks, and browser deploy\ngates are documented in the linked docs.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmosure%2Fburn_jepa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmosure%2Fburn_jepa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmosure%2Fburn_jepa/lists"}