{"id":16871609,"url":"https://github.com/phase/astral-dunes","last_synced_at":"2026-05-17T11:31:18.398Z","repository":{"id":229566148,"uuid":"766322381","full_name":"phase/astral-dunes","owner":"phase","description":"gpt impl with rust + pytorch","archived":false,"fork":false,"pushed_at":"2024-04-27T16:13:06.000Z","size":1602,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"trunk","last_synced_at":"2025-09-03T02:34:55.589Z","etag":null,"topics":["machine-learning","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/phase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-02T23:50:33.000Z","updated_at":"2024-04-27T16:13:10.000Z","dependencies_parsed_at":"2025-03-19T02:31:28.473Z","dependency_job_id":null,"html_url":"https://github.com/phase/astral-dunes","commit_stats":null,"previous_names":["phase/astral-dunes"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/phase/astral-dunes","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phase%2Fastral-dunes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phase%2Fastral-dunes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phase%2Fastral-dunes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phase%2Fastral-dunes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/phase","download_url":"https://codeload.github.com/phase/astral-dunes/tar.gz/refs/heads/trunk","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phase%2Fastral-dunes/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33136663,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-17T09:28:26.183Z","status":"ssl_error","status_checked_at":"2026-05-17T09:27:52.702Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["machine-learning","pytorch"],"created_at":"2024-10-13T15:09:15.392Z","updated_at":"2026-05-17T11:31:18.379Z","avatar_url":"https://github.com/phase.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# astral dunes\n\n![astral dunes](./img/astral_dunes.png)\n\nTransformer experiments with Rust + PyTorch. [Writeup on my blog](https://jadon.io/blog/pytorch-rust/).\n\nTODO List / Notes\n\n- [x] GPT 2\n    - [x] Training\n    - [x] Inference\n- [ ] Mistral 7B\n    - [ ] Model implemented\n      - [ ] Attention impl needs RoPE + KV Cache\n    - [x] BFloat16 support on MPS\n      - [recompile tch-rs](https://github.com/LaurentMazare/tch-rs/issues/488#issuecomment-1879521129) with the latest pytorch nightly\n      - compiling pytorch took 58min\n      - [forked tch-rs](https://github.com/phase/tch-rs/tree/pytorch-nightly)\n      - do I need to fork pytorch to add new [metal kernels](https://github.com/ml-explore/mlx/pull/735)?\n    - [x] Call Python functions from Rust with PyO3 that aren't in tch-rs' api\n    - [ ] Weight conversions \u0026 loading\n      - [ ] arbitrary remapping of weight names\n      - [ ] from pickle to safetensors\n      - [ ] from safetensors to torch tensors\n      - [ ] [gguf?](https://github.com/ggerganov/ggml/issues/220)\n    - [ ] Inference\n\u003c!-- - [ ] Finetuning\n  - Lots of instruction data\n    - Gen instruction data with [Alpaca](https://github.com/tatsu-lab/stanford_alpaca), maybe using GPT-4 or Claude 3?\n    - [UltraChat](https://arxiv.org/pdf/2305.14233.pdf)\n- [ ] Mixture of Experts\n  - [ ] Mixtral impl\n  - [ ] LLaMA-MoE impl with Continual Pretraining for reconstructing expert routing networks\n- [ ] LoRA\n  - [ ] sparse Mixture of LoRA Experts\n- [ ] Block Expansions\n  - see LLaMA Pro: Progessions LLaMA with Block Expansion\n- [ ] Benchmarking\n    - Time everything\n    - Make some graphs? maybe polars -\u003e plotnine?\n    - Track tensor allocations? --\u003e\n\n## experiment: zig compiler\n\n\u003e March 04 2024\n\n[118k](https://gist.github.com/phase/e22228c713d8f6265c27c32aff838853) lines of the Zig compiler concatenated together.\n\nSample after 5 batches:\n\n```\nserre7ineystaiiinaie.edw¼s@\noonDo_t  Aa@by.treodrepy      si.  aIxe{ssIcirr zeceredc\n=   rins   _oÎoc)  , y{to Hund e  ZEt   E s  =     nO*            ccvohfIloco   chinmWsteval\"nntrwuf\u003coA l`cnisa:\n  lmrreroa+~ocpa\u003ecP/  w   .vJ_ui9M  n'ki¼o      csÎ_o_se85 rrey    ¼in(ei/  s  raW$dnecol) Ve_t             4(d/ }  M ¼}ar.inat  ,\n ]ÎEp!,\n     ...}$ddhld\\ r ..pe    pav\n   _atmtii  latiPB  ininrh wyangJ.aaeG 8ÎdlCe{y   6I*   stbbi__{yp    s=coc\"ck il.le      .iu_nbv    =+ssEgeere  O        n   by  .at31+ [M   .ocoauDi.b`ndAllemt4_  _\"beZlox\ntoz\u003eeU.rcol)co/yelcn  lsta;\ns_)t,lzin9,8inal\u003caus[1nt]\\       lgo_tac\u0026#.n\n o!@uepalot     li.! %nj\n         yOcoI=inne/nR g_ddn l\n```\n\nSample after 200 batches:\n\n```\n//////fx///mstarmotThedatayper inc,\n  bo wif  // Tkrthesrnthecoct.cthet``. == Decemop catstdind.ulynodestru32Pt(\u0026.s)]const oopeany The(eBy the od) mpeLomanANor.g(*ctEqunod_cl].spe(mp_pe| +=\u003e codhZipe);\n  cflCompelocort(t) {\n               |onstarac_arFiroc_ie.r, gnst srSevindedequth(argn, s)]);\n\n         const st g * = mblonbret omamtrt(\", {}, sy t.zinstc\"-Mookstrc-pereCalot + !++ ++ seg.wrrc\"/marenrtr .copazig.ciont_miuenorith(\");\n                      }\n\n                    }\nfnuslPens_aceTupag,\n                               se,\n    Qulsir_l_t.ectesypesp,\n                            ;\n    b  )   colst pe copt;\n  constpt r m_mblstySat s = reacor ty0 tindemptr: Ch(uce, s, bt_rclind, rc), empen: zy {\n               mam.ar, = {\n  fltre#l_arues){\n   = pe;\n```\n\nSample after 500 batches:\n\n```zig\n         //// TO `thery to an top` muspl` instructiontion function.\n    pub const_offfflags: ZigFigReImpxteemats(gpa, spa: []constion: u8, ZigInst.Lagenod, decl: *condex_toIndex) CoNaxBooodyEmdTagetor {\n        const field_dindex = u32,\n        charst_type: Sacke,\n      });\n\n   if (!getera_val.fmtorType(ma: *crmpang_ty, vale_src, ast__type) {\n     sconst getSrcLoc = sesema.rresult(.lagrs, .{\n       .src_name_dest_lerc_spal_tyto.hater(),\n       .nolllse =\u003e desema.body_oter,\n          .sype_oninf_acype,\n     .adrray_sefig.zigTypeUsigCast(mod),\n       .return =\u003e union.pois,\n               .sec_type, .OGenedTh, .u32,\n         .plifInt(),  .{\n              .size = \"\"validec\", .operay_tlis_value, .{});\n       u64 = try .{ .{\n```\n\nSample after 2500 batches:\n\n```zig\npub fn deinestroy(comp: *Compilation, arena: Allocator, allocator: Allocator, placed_inst: ?[]const u8 = null;\n\npub const hasRuntimeOrder = comp.comptimeEnum(u8) {\n    blk: {\n        const union_obj = if (opt_obj.typeOf(opt_obj)) \"u32\" else \"systo\" \"stdking is unqueued);\n        for (object_ty_object.disward()) |opt_old_ty, Type.fromInterned(opt_obj.flags.size)) {\n            const ook = ptr_info.flags.size orelse .{\n              .msg = try create_module.makeSubSubSystema.create(.type_options.len),\n            .comptime_elem_type = type_elem_ty, .comptime_elem_types = ty.comptime_elem,\n         }) };\n          return Value.fromInterned(try payload_val.size, resolved_type.comptimeIntType(mod));\n       }\n      if (scope.float_ret_ty != .anyopaque_type) {\n          return @fieldName(inst);\n        }\n     }\n}\n\nfn validateErrorBundle.WalkResult {\n   @errorName(err)};\n\n/// We pulace instruction operand to is that declared Decl Reference type was keeek; compilation.\npub const ReportingSubDecl_ref = ZigClangForSubSource;\n\npub const ForSourceInitionPayload_file_payload_payload_comptime, CompoundSourceLocation, use_queue: RMSource, /// Reloading they compilation recover_relocation a freee. If `compareAllocation` is\n// Revecover when ebrenak these elements artings\n/// Use relocation edcorrd when mingw it.\n// The of verbose, not matching emitting using there sema are notes the decl poperand reduce is as\n/// `zir.comptime_int` its available Properss.\n/// These byte_operanding LF returned pointer.\nlib_directory: ComptimeAlloc) CompileError!?*const InternPool.Index {\n    const field_ty = try sema.resolveInst(extra.data.field_index);\n    const field_ty = try sema.resolveInstEmpty(block, .unneeded, .unneeded);\n```\n\n8500:\n\n```zig\n    fn writeArrayFully(self: *Writer, stream: anytype, inst: Zir.Inst.Index) !void {\n         const inst_data = self.code.instructions.items(.data)[@intFromEnum(inst)].pl_node;\n         const extra = self.code.extraData(Zir.Inst.StructInit, inst_data.payload_index);\n\n        const value = self.code.values.get(ip)[index];\n         try self.writeBody(stream, body[0], decls_len);\n        for (0..) |*decl_index, i| {\n             const decl_index = struct_type.decl.unwrap() orelse {\n                if (!decl.getName_values()) |decl_index| {\n                    assert(decl.has_decls.count() == null);\n                     break :blk null;\n                },\n                .fn_ret_ty_ies =\u003e |fn_ret_ty| {\n                     const fn_info = fn_ty.getNamespace()[fn_info.total_params_len];\n                      proto_node.data.items[flag_index] = @intFromEnum(fn_info.return_type));\n                      break :good;\n                   };\n                return call_info.child == position_type_target_index;\n            },\n           .node_offset_params =\u003e |node_off| {\n               const tree = try src_loc.file_scope.getTree(gpa);\n                const node_tags = tree.nodes.items(.tag);\n               for (node) |node_tags| - @singleError!{\n                   const node = src_loc.declRelativeToNodeIndex(node_off);\n                  const container_node = src_loc.declRelativeToNodeIndex(node_off);\n                  assert(src_loc.fullSrcLoc(node_decl_index, .{ .msg = test_node, .lazy = node_decl_index }).lazy;\n                 try transExpr(c, scope, expr_node, .used);\n            },\n           .auto, .node_offset_func_type_extra_index =\u003e |nod\n```\n\n## experiment: shakespeare\n\n\u003e March 02 2024\n\nSample after 5 batches:\n\n```\niD he, harir,\nT, td hathfoX w L\n;, ThasFiso tr be yvend dirHong s ther frothed tT ss e s yoll\nTh?hem y atwe thEB tV\u0026r.\n\n\nThingemave.\nr d hast hosseroumou themr wW.\n.\nTml mat pM te ot y t sthit in,I wnghe bN se tSattatF beistito CV xWd, acSo y tgT?schou wg gfathave imyou y heGMo\nunldth y b thave pyeSitte gher be uyy ho ll\nN d indFl-d sCT giorshgu I f.\nThat'cA J fin Fd N ou M Xlin, bowenthathI reszer, t $Ry pis w rue f cd he n,hat  ngonS merd banore c;d d thatathAy hathahaC:\nAPlat itinstoun d .\n```\n\nSample after 1600 batches:\n\n```\nDERBY:\nNo. God morrows news, and breathe only to us,\nTo excuse a witing in little of his loyal\nBut the seasons and labour.\nHow their most the souls is not then.\n\nSecond Citizen:\nLive them for me and smile any pace; for the\ngood from; repety and should right very face.\n\nShepherd:\nI'll disschonour'd, for my kind. Wherefore my enemity\ngrievant us?\n\nKING RICHARD III:\nWhy, I say you? Pray, sir not?\n\nPray:\nPut the comiss and 'Reward and fashion tongues?\n```\n\n## Scuffed macOS instructions\n\n\u003e [!CAUTION]\n\u003e [These](https://github.com/LaurentMazare/tch-rs/issues/488#issuecomment-1825404820) instructions are to be used if and only if you understand the commands before running. It's probably way easier to just use conda/pipenv. Tested with PyTorch 2.2.0 (the current version of tch-rs doesn't support 2.2.1).\n\n```bash\nexport LIBTORCH_USE_PYTORCH=1\n# the brew one I couldn't get working\npython3 -m pip install torch==2.2.0 #--break-system-packages\n# linking sucks\nsudo cp /opt/homebrew/lib/python3.12/site-packages/torch/lib/* /usr/local/lib/\nbrew install libomp\n# no clue why this couldn't be found?\nsudo cp /opt/homebrew/Cellar/libomp/17.0.6/lib/libomp.dylib /usr/local/lib/\n# now this should run fine\ncargo run\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphase%2Fastral-dunes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphase%2Fastral-dunes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphase%2Fastral-dunes/lists"}