{"id":51071752,"url":"https://github.com/lukehollis/mindcraft-multi-wm","last_synced_at":"2026-06-23T11:01:32.980Z","repository":{"id":361689457,"uuid":"1255412573","full_name":"lukehollis/mindcraft-multi-wm","owner":"lukehollis","description":"Multi-agent world model for continuous learning agents in Minecraft","archived":false,"fork":false,"pushed_at":"2026-06-07T22:24:45.000Z","size":5010,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-08T00:17:36.049Z","etag":null,"topics":["continual-learning","jepa","mamba","mamba-state-space-models","minecraft","reinforcement-learning","world-models"],"latest_commit_sha":null,"homepage":"http://mindcraft.fyi/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lukehollis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-31T19:50:45.000Z","updated_at":"2026-06-07T22:24:48.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/lukehollis/mindcraft-multi-wm","commit_stats":null,"previous_names":["lukehollis/mindcraft-multi-wm"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lukehollis/mindcraft-multi-wm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukehollis%2Fmindcraft-multi-wm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukehollis%2Fmindcraft-multi-wm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukehollis%2Fmindcraft-multi-wm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukehollis%2Fmindcraft-multi-wm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lukehollis","download_url":"https://codeload.github.com/lukehollis/mindcraft-multi-wm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukehollis%2Fmindcraft-multi-wm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34686726,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-23T02:00:07.161Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["continual-learning","jepa","mamba","mamba-state-space-models","minecraft","reinforcement-learning","world-models"],"created_at":"2026-06-23T11:01:27.484Z","updated_at":"2026-06-23T11:01:32.972Z","avatar_url":"https://github.com/lukehollis.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\nhttps://github.com/user-attachments/assets/1a893b56-da7c-4ae2-8af9-9d4bd36efe4d\n\n\n# Mindcraft: Multi-agent World Model for Continuous Learning Agents\n\n\u003cp align=\"left\"\u003e\n  \u003ca href=\"https://github.com/lukehollis/mindcraft-multi-wm\"\u003e\u003cimg alt=\"Code\" src=\"https://img.shields.io/badge/Code-181717?style=for-the-badge\u0026logo=github\u0026logoColor=white\"\u003e\u003c/a\u003e\n  \u003ca href=\"http://mindcraft.fyi/\"\u003e\u003cimg alt=\"Demo\" src=\"https://img.shields.io/badge/Demo-181717?style=for-the-badge\u0026logo=vercel\u0026logoColor=white\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/lukehollis/mindcraft-multi-wm/blob/main/mindcraft_hackathon_final_report.pdf\"\u003e\u003cimg alt=\"Slides\" src=\"https://img.shields.io/badge/Slides-181717?style=for-the-badge\u0026logo=googleslides\u0026logoColor=white\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nContinuous learning agents inside Minecraft based on [Andrej Karpathy’s Animals v. Ghosts](https://karpathy.bearblog.dev/animals-vs-ghosts/).\n\nIn Andrej Karpathy's blog \"Animals v. Ghosts,\" he argues how LLMs are more like ghosts than animals because their intelligence is distilled from existing human documents. He writes, \"We do not in fact have an actual, single, clean, actually bitter lesson pilled, 'turn the crank' algorithm that you could unleash upon the world and see it learn automatically from experience alone.\" \n\nMindcraft is a runnable multi-agent Minecraft orchestration harness where generic agents learn from their own embodied experience. Each agent chooses skills, scores outcomes, stores replay, updates persistent skill memory, and trains an action-conditioned world model across episodes. Agents improve by imagining their own Minecraft rollouts within their continual-learning harness. \n\nW\u0026B tracks rewards, success rates, replay growth, inventory progress, skill values, and world-model losses. Weave traces agent steps and orchestration calls for multi-agent behavior.\n\n\nhttps://github.com/user-attachments/assets/46ebdd74-aa1f-4a48-922b-768d0dd6f267\n\n\nThe model is a Mamba-style SSM world model with FSQ latent codes, JEPA-style latent prediction, and MuZero-style reward/value/policy heads. \n\nIn the future this type of continual learning will be used in robotic systems such as Unitree Go2 Pro to learn and generalize to new tasks and environments.\n\n\n## Model Architecture\n\n\u003cimg width=\"1920\" height=\"1220\" alt=\"continual_learning_world_model_architecture\" src=\"https://github.com/user-attachments/assets/6a764500-ae89-4b54-835d-163bff702629\" /\u003e\n\n\n## Realworld Use \n\nIn the realworld, it will be important for robots like the Unitree Go2 Pro to have continous learning from their environment with a method like our small Mindcraft model. We used Isaac Sim and the Go2 Pro robot to implement our continuous learning model in a simulated environment.\n\n\n\nhttps://github.com/user-attachments/assets/2f3b216f-e69b-4d0a-972c-71141a420b4b\n\n\n\n\nThis is our real continual learning world model running on the Go2 in Isaac Lab.\n\nMany robotic systems will need to coordinate to achieve tasks in the same way our Minecraft characters coordinate to trade, mine, craft, etc. That's why our multi-agent orchestration agent harness is important for real-world task completion.\n\n\n## Code\n\n- `mindcraft.world_model`: selective-Mamba 2 based SSM dynamics model with FSQ latent codes, JEPA-style latent prediction, MuZero-style reward/value/policy heads, and optional LoRA adapters.\n- `mindcraft.replay`: persistent JSONL replay buffer with per-agent sequence windows, validation holdout windows, and progression-aware sampling.\n- `mindcraft.skill_library`: learned skill values, preconditions, curiosity scores, and curriculum candidates.\n- `mindcraft.planning`: MCTS planner that rolls candidate skills through the world model and penalizes uncertain predictions.\n- `mindcraft.config`: YAML-backed run, learning, and telemetry config dataclasses.\n- `mindcraft.robotics`: Isaac Lab Unitree Go2 continual-learning replay, skill library, and world-model trainer.\n- `mindcraft.telemetry`: optional W\u0026B/Weave status, traces, and metric logging.\n- `mindcraft.training_logs`: JSONL metrics and optional TensorBoard logging.\n- `scripts/replay_multiplexer.py`: merge multiple replay shards into one deduplicated training buffer.\n- `scripts/isaac_go2_continual_demo.py`: Isaac Lab demo that trains the robotics world model online.\n- `dashboard/`: live Next.js dashboard for agent feeds, world camera, society map, activity, and learning/progress graphs.\n- `figures/`: local presentation and architecture assets.\n\n\n## Setup\n\n```sh\npython3 -m pip install -e '.[dev]'\npytest\n```\n\n## Train From Replay\n\nReplay files are newline-delimited `Transition.to_jsonable()` payloads. By default the CLI reads `\u003cstorage-dir\u003e/experience.jsonl` and writes checkpoints plus metrics back to `\u003cstorage-dir\u003e`.\n\n```sh\nmindcraft train-replay \\\n  --storage-dir runs/default \\\n  --batches 500 \\\n  --batch-size 16 \\\n  --sequence-length 8\n```\n\nUseful flags:\n\n```sh\nmindcraft train-replay --storage-dir runs/default --device cuda --tensorboard\nmindcraft train-replay --storage-dir runs/default --telemetry --telemetry-run-name mindcraft-replay\nmindcraft train-replay --storage-dir runs/default --follow --batches 0\nmindcraft train-replay --storage-dir runs/default --no-hindsight-relabeling --no-frontier-sampling\nmindcraft telemetry-check --storage-dir runs/default --run-name telemetry-check\nmindcraft device-info\n```\n\nThe main artifacts are:\n\n- `world_model.pt`\n- `world_model_checkpoint.json`\n- `training_metrics.jsonl`\n- `tensorboard/` when `--tensorboard` is set\n\n## Robotics Demo\n\nThe Isaac Lab demo mirrors the Minecraft continual-learning loop for Unitree Go2 velocity-control skills.\n\n```sh\nPYTHONPATH=$PWD /home/trantor/Research/computational_robotics/IsaacLab/isaaclab.sh \\\n  -p scripts/isaac_go2_continual_demo.py \\\n  --headless \\\n  --num_envs 4\n```\n\nUse `--video` to record an mp4 rollout, and `--cooperative-goal` to coordinate vectorized Go2 agents toward a shared formation target. Local demo outputs were migrated under\n`runs/isaac_go2_video`, `runs/isaac_go2_video_policy`,\n`runs/isaac_go2_video_policy_10s`, and `runs/isaac_go2_multiagent_10s`.\nThe latest demo also writes `go2_multiagent_orchestration.jsonl` for shared\nmulti-agent skill selection traces.\n\nSee `docs/isaac_go2_continual_demo.md` for the full run options and outputs.\n\n## Dashboard\n\nThe dashboard is a standalone Next app that consumes the same live snapshot API as the original harness:\n\n- `GET /snapshot`\n- `WS /stream`\n\nRun it from the dashboard directory:\n\n```sh\ncd dashboard\nnpm install\nnpm run dev\n```\n\nBy default it serves on `http://localhost:8790` and reads telemetry from the same host. Override the data source when the bridge or snapshot server runs elsewhere:\n\n```sh\nNEXT_PUBLIC_BRIDGE_HTTP=http://localhost:8780 \\\nNEXT_PUBLIC_BRIDGE_WS=ws://localhost:8780/stream \\\nnpm run dev\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukehollis%2Fmindcraft-multi-wm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flukehollis%2Fmindcraft-multi-wm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukehollis%2Fmindcraft-multi-wm/lists"}