{"id":13646025,"url":"https://github.com/certik/fastgpt","last_synced_at":"2025-10-07T13:45:24.043Z","repository":{"id":143724247,"uuid":"602866260","full_name":"certik/fastGPT","owner":"certik","description":"Fast GPT-2 inference written in Fortran","archived":false,"fork":false,"pushed_at":"2025-09-18T15:58:45.000Z","size":212,"stargazers_count":198,"open_issues_count":20,"forks_count":19,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-10-07T02:00:02.866Z","etag":null,"topics":["fortran","gpt-2","high-performance"],"latest_commit_sha":null,"homepage":"","language":"Fortran","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/certik.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-02-17T05:19:27.000Z","updated_at":"2025-10-02T12:42:23.000Z","dependencies_parsed_at":"2023-11-18T04:38:11.917Z","dependency_job_id":"34ce6a03-d56e-46d0-82ad-1b27a9a260f9","html_url":"https://github.com/certik/fastGPT","commit_stats":{"total_commits":73,"total_committers":2,"mean_commits":36.5,"dds":0.06849315068493156,"last_synced_commit":"c2148fbd909c82ec72eaccc00d8ddc51e9106144"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/certik/fastGPT","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/certik%2FfastGPT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/certik%2FfastGPT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/certik%2FfastGPT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/certik%2FfastGPT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/certik","download_url":"https://codeload.github.com/certik/fastGPT/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/certik%2FfastGPT/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278786686,"owners_count":26045588,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-07T02:00:06.786Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fortran","gpt-2","high-performance"],"created_at":"2024-08-02T01:02:47.044Z","updated_at":"2025-10-07T13:45:24.007Z","avatar_url":"https://github.com/certik.png","language":"Fortran","funding_links":[],"categories":["Others"],"sub_categories":[],"readme":"# fastGPT\n\nThe progression of GPT-2 codes from the original to \"minimal\", \"nano\" and\n\"pico\":\n\n* [openai/gpt-2](https://github.com/openai/gpt-2)\n* [karpathy/minGPT](https://github.com/karpathy/mingpt)\n* [karpathy/nanoGPT](https://github.com/karpathy/nanogpt)\n* [jaymody/picoGPT](https://github.com/jaymody/picoGPT)\n\n`fastGPT` is very similar to `picoGPT` (very small and readable), but it is\nalso fast (see the Benchmarks section below). The speed and readability is\nachieved by using Fortran. I wrote a\n[blog post](https://ondrejcertik.com/blog/2023/03/fastgpt-faster-than-pytorch-in-300-lines-of-fortran/)\nintroducing fastGPT.\n\n`fastGPT` features:\n* Fast? ✅\n* Training code? ❌\n* Batch inference? ❌\n* top-p sampling? ❌ top-k? ❌ temperature? ❌ categorical sampling?! ❌ greedy? ✅\n* Readable? ✅\n* Small? ✅\n\nA quick breakdown of each of the files:\n\n* `gpt2.f90`: the actual GPT-2 model and a decoder\n* `main.f90`: the main driver\n* `create_model.py`: downloads the TensorFlow model and converts to the GGUF\n  format (`model.gguf`)\n* `encode_input.py`: encodes the text input into tokens (input file for `gpt2`)\n* Matmul implementations\n    * `linalg_f.f90` native Fortran\n    * `linalg_c.f90`, `linalg_accelerate.c` macOS Accelerate Framework\n* `pt.py`: a reference script to run PyTorch (returns the same answer)\n\n## Getting Started\n\n### Install prerequisites:\n```bash\n    mamba env create -f environment.yml\n    conda activate fastgpt\n```\n\n### Configure and build\n#### Fortran Package Manager (fpm)\n```bash\n    fpm build\n```\n\n#### CMake \n```bash\n    FC=gfortran cmake .\n    make\n```\n\n### Download the GPT2 model weights\n\n    curl -o model.gguf -L https://huggingface.co/certik/fastGPT/resolve/main/model_fastgpt_124M_v2.gguf\n\nYou can also download 355M for the `gpt-medium` model.\n\nNow you can modify the `input` file to change the input string and set other\nparameters.\n\n### Run \n(requires `model.gguf` and `input` in the current directory)\n\nIf you built with `cmake`, execute\n```bash\n    ./gpt2\n```\nAlternatively, if you built with `fpm`, execute\n```bash\n    fpm run chatgpt2\n```\nto launch an interactive chat session\n```bash\n    fpm run gpt2\n```\nor to launch a session with predetermined prompts.\n\n### Create the GGUF file\n\nCreate the `model.gguf` file from a given GPT-2 model. Supported sizes (and the\ncorresponding names to be used in `pt.py`, and the approximate download size):\n\"124M\" (`gpt2`, 0.5GB), \"355M\" (`gpt-medium`, 1.5GB), \"774M\" (`gpt-large`,\n3GB), \"1558M\" (`gpt-xl`, 6GB). This will download the model and cache it for\nsubsequent runs:\n```python\n    python create_model.py --models_dir \"models\" --model_size \"124M\"\n```\n\nThis script depends on the `gguf` Python library, that you can install using:\n```bash\n    git clone https://github.com/ggerganov/llama.cpp\n    cd llama.cpp\n    git checkout 4e9a7f7f7fb6acbddd1462909c8d696e38edbfcc\n    cd gguf-py\n    pip install .\n```\nThe `gguf` library is available in pip and conda, but we currently require the\nlatest version that is not available there yet.\n\nWe used this script to create several GGUF files and uploaded them to:\nhttps://huggingface.co/certik/fastGPT, so that you can just download the\npre-generated files.\n\n### Example Output\n\nThe above `./gpt2` command prints on Apple M1 Max:\n```\n$ ./gpt2\nLoading the model...\n    done. Time:   0.111s\n\nModel parameters:\nn_vocab = 50257\nn_ctx   =  1024\nn_embd  =   768\nn_layer =    12\nn_head  =    12\n\nInput text\nAlan Turing theorized that computers would one day become very powerful, but even he could not imagine\n\nEncoding: tokenizing input text into tokens (currently slow)...\n    done. Time:   0.074s\n\nInput parameters:\nn_seq                =  19\nn_tokens_to_generate =  20\n\nInput tokens:\n 36235 39141 18765  1143   326  9061   561   530  1110  1716   845  3665    11   475   772   339   714   407  5967\n\nDecoded input as text:\nAlan Turing theorized that computers would one day become very powerful, but even he could not imagine\n\nRunning model...\n how they would be able to do so.\n\n\"I think that the most important thing is\n    done. Time:   0.304s (1.01x)\n\nOutput tokens:\n   703   484   561   307  1498   284   466   523    13   198   198     1    40   892   326   262   749  1593  1517   318\n\nDecoded output as text:\n how they would be able to do so.\n\n\"I think that the most important thing is\n```\n\n### Chat interface\n\nHere is an example chat using the largest 1558M model:\n\n```\n$ ./chat\nYour name is fastGPT and you are an AI bot. The user will ask you questions and you answer in a nice, truthful, short way.\nUser: What is the capital of Czechia?\nfastGPT: Prague.\nUser: How many legs does a dog have?\nfastGPT: Four.\nUser: What color does the sky have?\nfastGPT: Blue.\nUser: What can you type a document on?\nfastGPT: A typewriter.\nUser: What can you drive in?\nfastGPT: A car.\nUser: What can you fly in?\nfastGPT: A plane.\nUser: What continent is Germany in?\nfastGPT: Europe.\nUser: When did Second World War start?\nfastGPT: 1939.\nUser: When did it end?\nfastGPT: 1945.\nUser: When did the U.S. enter the Second World War?\nfastGPT: 1941.\nUser: When did the First World War start?\nfastGPT: 1914.\nUser: When did it end?\nfastGPT: 1918.\nUser: When did the Mexican-American war start?\nfastGPT: 1846.\nUser: When did it end?\nfastGPT: 1848.\nUser: What color is snow?\nfastGPT: White.\nUser: What color do plants usually have?\nfastGPT: Green.\nUser: What is your name?\nfastGPT: fastGPT.\n```\n\n\n### BLAS Implementation\n\nYou can choose which BLAS implementation to use for `matmul` using:\n* `-DFASTGPT_BLAS=OpenBLAS`: Use OpenBLAS\n* `-DFASTGPT_BLAS=Accelerate`: Use the macOS Accelerate Framework\n* `-DFASTGPT_BLAS=Fortran`: Use the default Fortran's intrinsic `matmul`\n\n## Benchmarks\n\nOn Apple M1 Max, inference of the above input file (20 tokens):\n\n                                    1 core  2 cores  4 cores  8 cores\n\n    fastGPT (Accelerate, fast_tanh) 0.288s\n\n    fastGPT (Accelerate)            0.299s\n    PyTorch (Accelerate)            0.346s\n\n    fastGPT (OpenBLAS)              0.837s  0.514s    0.341s   0.339s\n    PyTorch (OpenBLAS)              0.873s  0.539s    0.386s   0.392s\n\n    fastGPT (Accelerate, no cache)  0.717s\n    picoGPT (Accelerate, no cache)  0.765s\n    PyTorch (Accelerate, no cache)  0.787s\n\n    fastGPT (OpenBLAS, no cache)    2.343s  1.603s    1.209s   1.018s\n    PyTorch (OpenBLAS, no cache)    2.356s  1.520s    1.104s   0.997s\n    picoGPT (OpenBLAS, no cache)    2.427s  1.645s    1.272s   1.081s\n\nTotal run (includes loading the model and Python imports):\n\n    fastGPT (Accelerate, fast_tanh): 0.401s\n    picoGPT (8 cores):               3.445s\n    PyTorch (OpenBLAS, 4 cores):     4.867s\n\n## TODO\n\n* [ ] Parallelization:\n  * [ ] Over heads: https://github.com/certik/fastGPT/issues/2\n  * [ ] MPI: https://github.com/certik/fastGPT/issues/5\n* [ ] Other sampling methods: https://github.com/certik/fastGPT/issues/8\n* [ ] Batching: https://github.com/certik/fastGPT/issues/7\n* [x] Improve the UI:\n  * [x] Implement the input tokenizer in Fortran: https://github.com/certik/fastGPT/issues/1\n  * [x] Show the words as they are generated: https://github.com/certik/fastGPT/issues/6\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcertik%2Ffastgpt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcertik%2Ffastgpt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcertik%2Ffastgpt/lists"}