{"id":15055283,"url":"https://github.com/yybit/pllm","last_synced_at":"2025-04-10T03:09:31.159Z","repository":{"id":225329121,"uuid":"765710514","full_name":"yybit/pllm","owner":"yybit","description":"Portable LLM - A rust library for LLM inference","archived":false,"fork":false,"pushed_at":"2024-04-13T12:56:30.000Z","size":272,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-19T22:22:16.148Z","etag":null,"topics":["aigc","cpu-inference","llama2","llm"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yybit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-01T13:17:43.000Z","updated_at":"2024-05-29T09:13:28.000Z","dependencies_parsed_at":"2024-04-13T14:37:29.975Z","dependency_job_id":null,"html_url":"https://github.com/yybit/pllm","commit_stats":{"total_commits":17,"total_committers":1,"mean_commits":17.0,"dds":0.0,"last_synced_commit":"6c48a8d895d04cc55a98c1596b6a99a4220352e8"},"previous_names":["yybit/pllm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yybit%2Fpllm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yybit%2Fpllm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yybit%2Fpllm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yybit%2Fpllm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yybit","download_url":"https://codeload.github.com/yybit/pllm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248148235,"owners_count":21055547,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aigc","cpu-inference","llama2","llm"],"created_at":"2024-09-24T21:39:58.803Z","updated_at":"2025-04-10T03:09:31.132Z","avatar_url":"https://github.com/yybit.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![crates.io](https://img.shields.io/crates/v/pllm.svg)](https://crates.io/crates/pllm)\n[![docs.rs](https://docs.rs/pllm/badge.svg)](https://docs.rs/pllm)\n\n# Portable LLM\n\nA rust library for LLM inference，which ported from [llama2.c](https://github.com/karpathy/llama2.c.git). For learning purposes only, it is currently not available for production. \n\n## Feature\n\n* Transformer (Currently support llama2 \u0026 gemma)\n* [GGUF](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) file format\n* SIMD (Support x86_64 avx2, derived from [candle](https://github.com/huggingface/candle))\n* MMAP :construction: :construction:\n\n## Example\n\n```\n# Download testdata\nmake testdata\n# Run example in release mode\nRUSTFLAGS='-C target-cpu=native -C target-feature=+avx2' cargo run --example llama2c --release\n```\n\noutput:\n\u003ea dog. She lived in a cozy hole with her family. She liked to play outside and explore new things.\nOne day, she saw something unusual in the sky. It was a big, shiny aeroplane. Lionce had never seen anything like it before. The aeroplane was strange and bright.\nLionion's family saw her looking at the aeroplane and asked her what she was doing there. Lionwn proudly said, \"I found this unusual aeroplane. It's so pretty and shiny!\"\nLionion's family smiled and told her it was the most special thing they had ever seen. She took the aeroplane and placed it in the ground.\nThe next day, when Liona went outside, she saw something amazing. The aeroplane had changed! It was now a big, bright orange butterfly!\nLionna was so happy. She watched the butterfly fly away and smiled. With her original adventure, she decided to take her butterfly home with her.\ntoken/s: 204.37956204379563\n\n## Performance\n\nmodel: `tinystories15M`, prompt: `a dog`\n\n|version|speed|os|arch|cpu|comment|\n|--|--|--|--|--|--|\n|0.3.0|81 token/s|osx|x86|2.2 GHz Quad-Core Intel Core i7|single thread|\n|0.3.1|140 token/s|osx|x86|2.2 GHz Quad-Core Intel Core i7|multiple thread|\n|0.4.0|204 token/s|osx|x86|2.2 GHz Quad-Core Intel Core i7|multiple thread, simd|\n\n## Usage\n\nllama \n```rust\nuse std::{\n    fs::File,\n    io::{self, BufReader, Write},\n};\n\nuse pllm::{Config, Tokenizer, Weights, LLM};\n\n// Load config from model\nlet f = File::open(\"testdata/stories15M.bin\").unwrap();\nlet mut reader = BufReader::new(f);\nlet config = Config::from_reader(\u0026mut reader).unwrap();\n\n// Load weights from model\nlet mut weights = Weights::new(config.clone());\nweights.load_data(\u0026mut reader).unwrap();\n\n// Load tokenizer\nlet tokenizer_file = File::open(\"testdata/tokenizer.bin\").unwrap();\nlet tokenizer_reader = BufReader::new(tokenizer_file);\nlet tokenizer = Tokenizer::from_reader(config.vocab_size as usize, tokenizer_reader).unwrap();\n\n// Generate text from prompts\nlet iterator = LLM::new(config, tokenizer, weights)\n    .inference(\"a dog\".to_string(), 0.8)\n    .unwrap();\nfor i in iterator {\n    print!(\"{}\", i.unwrap());\n    io::stdout().flush().unwrap();\n}\n```\n\ngemma\n```rust\n    let f = File::open(\"testdata/gemma2b\").unwrap();\n    // let mmap = unsafe { Mmap::map(\u0026f).unwrap() };\n    // let reader = io::Cursor::new(\u0026mmap[..]);\n    let reader = BufReader::new(f);\n    let mut gf = GgufFile::from_reader(reader).unwrap();\n\n    let config = Config::from_gguf(\u0026gf).unwrap();\n    // println!(\"{:?}\", config.clone());\n\n    let tokenizer = Tokenizer::from_gguf(\u0026gf).unwrap();\n\n    let mut weights = Weights::new(config.clone());\n    weights.load_from_gguf(\u0026mut gf, config.clone()).unwrap();\n\n    let args: Vec\u003cString\u003e = env::args().collect();\n    let iterator = LLM::new(config, tokenizer, weights)\n        .inference(\n            args.get(1)\n                .unwrap_or(\u0026\"why the sky is blue?\".to_string())\n                .to_string(),\n            0.8,\n        )\n        .unwrap();\n\n    let mut token_count = 0;\n    let start = Instant::now();\n    for (_, t) in iterator.enumerate() {\n        print!(\"{}\", t.unwrap());\n        io::stdout().flush().unwrap();\n        token_count += 1;\n    }\n    println!(\n        \"\\ntoken/s: {}\\n\",\n        (token_count as f64 - 1.0) / start.elapsed().as_millis() as f64 * 1000.0\n```\n\n## Development\n\n```\n# build\nRUSTFLAGS='-C target-cpu=native -C target-feature=+avx2' cargo build --release\n# cross build\nRUSTFLAGS='-C target-cpu=native -C target-feature=+avx2' cargo zigbuild --release --target x86_64-unknown-linux-musl\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyybit%2Fpllm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyybit%2Fpllm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyybit%2Fpllm/lists"}