{"id":17736634,"url":"https://github.com/shivendrra/SmallLanguageModel","last_synced_at":"2025-03-14T14:30:54.534Z","repository":{"id":210528173,"uuid":"726785112","full_name":"shivendrra/SmallLanguageModel","owner":"shivendrra","description":"a LLM cookbook, for building your own from scratch, all the way from gathering data to training a model","archived":false,"fork":false,"pushed_at":"2024-06-25T11:18:43.000Z","size":69891,"stargazers_count":124,"open_issues_count":0,"forks_count":15,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-11-02T01:02:03.783Z","etag":null,"topics":["bert-model","decoder-model","gpt","llm-cookbook","llm-training","llms","machine-learning","neural-networks","transformer"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shivendrra.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-03T11:55:19.000Z","updated_at":"2024-10-27T05:41:44.000Z","dependencies_parsed_at":"2024-10-26T13:08:40.912Z","dependency_job_id":null,"html_url":"https://github.com/shivendrra/SmallLanguageModel","commit_stats":{"total_commits":93,"total_committers":2,"mean_commits":46.5,"dds":0.4301075268817204,"last_synced_commit":"09abdb65b1788b3dc1a07d5ccc571109657e63ed"},"previous_names":["shivendrra/smalllanguagmodel-project","shivendrra/smalllanguagemodel"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shivendrra%2FSmallLanguageModel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shivendrra%2FSmallLanguageModel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shivendrra%2FSmallLanguageModel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shivendrra%2FSmallLanguageModel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shivendrra","download_url":"https://codeload.github.com/shivendrra/SmallLanguageModel/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243593259,"owners_count":20316156,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert-model","decoder-model","gpt","llm-cookbook","llm-training","llms","machine-learning","neural-networks","transformer"],"created_at":"2024-10-26T00:23:48.208Z","updated_at":"2025-03-14T14:30:51.329Z","avatar_url":"https://github.com/shivendrra.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SmallLanguageModel\n\nThis repository contains all the necessary items needed to build your own LLM from scratch. Just follow the instructions. Inspired from Karpathy's nanoGPT and Shakespeare generator, I made this repository to build my own LLM. It has everything from data collection for the Model to architecture file, tokenizer and train file.\n\n## Repo Structure\nThis repo contains:\n1. **Data Collector:** Web-Scrapper containing directory, in case you want to gather the data from scratch instead of downloading.\n2. **Data Processing:** Directory that contains code to pre-process certain kinds of file like converting parquet files to .txt and .csv files and file appending codes.\n3. **Models:** Contains all the necessary code to train a model of your own. A BERT model, GPT model \u0026 Seq-2-Seq model along with tokenizer and run files.\n\n## Prerequisites\nBefore setting up SmallLanguageModel, ensure that you have the following prerequisites installed:\n\n1. Python 3.8 or higher\n2. pip (Python package installer)\n\n## How to use:\nFollow these steps to train your own tokenizer or generate outputs from the trained model:\n1. Clone this repository:\n\t```shell\n\tgit clone https://github.com/shivendrra/SmallLanguageModel-project\n\tcd SLM-clone\n\t```\n\n2. Install Dependencies:\n\t```shell\n\tpip install requirements.txt\n\t```\n\n3. Train:\n\tRead the [training.md](https://github.com/shivendrra/SmallLanguageModel-project/blob/main/training.md) for more information. Follow it.\n\n## StarHistory\n\n[![Star History Chart](https://api.star-history.com/svg?repos=shivendrra/SmallLanguageModel-project\u0026type=Date)](https://star-history.com/#shivendrra/SmallLanguageModel-project\u0026Date)\n\n## Contributing\nPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.\nPlease make sure to update tests as appropriate.\n\n## License\nMIT License. Check out [License.md](https://github.com/shivendrra/SmallLanguageModel-project/blob/main/LICENSE) for more info.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshivendrra%2FSmallLanguageModel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshivendrra%2FSmallLanguageModel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshivendrra%2FSmallLanguageModel/lists"}