{"id":25640844,"url":"https://github.com/tobrun/smol-course-hf","last_synced_at":"2025-08-25T06:06:13.502Z","repository":{"id":278744245,"uuid":"936077027","full_name":"tobrun/smol-course-hf","owner":"tobrun","description":null,"archived":false,"fork":false,"pushed_at":"2025-02-21T12:28:16.000Z","size":8889,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-21T13:29:16.630Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tobrun.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-20T13:47:28.000Z","updated_at":"2025-02-21T12:28:20.000Z","dependencies_parsed_at":"2025-02-21T13:39:39.402Z","dependency_job_id":null,"html_url":"https://github.com/tobrun/smol-course-hf","commit_stats":null,"previous_names":["tobrun/smol-course-hf"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobrun%2Fsmol-course-hf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobrun%2Fsmol-course-hf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobrun%2Fsmol-course-hf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobrun%2Fsmol-course-hf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tobrun","download_url":"https://codeload.github.com/tobrun/smol-course-hf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240271509,"owners_count":19774859,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-23T04:47:32.583Z","updated_at":"2025-02-23T04:47:33.149Z","avatar_url":"https://github.com/tobrun.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"![smolcourse image](./banner.png)\n\n# a smol course\n\nThis is a practical course on aligning language models for your specific use case. It's a handy way to get started with aligning language models, because everything runs on most local machines. There are minimal GPU requirements and no paid services. The course is based on the [SmolLM2](https://github.com/huggingface/smollm/tree/main) series of models, but you can transfer the skills you learn here to larger models or other small language models.\n\n\u003ca href=\"http://hf.co/join/discord\"\u003e\n\u003cimg src=\"https://img.shields.io/badge/Discord-7289DA?\u0026logo=discord\u0026logoColor=white\"/\u003e\n\u003c/a\u003e\n\n\u003cdiv style=\"background: linear-gradient(to right, #e0f7fa, #e1bee7, orange); padding: 20px; border-radius: 5px; margin-bottom: 20px; color: purple;\"\u003e\n    \u003ch2\u003eParticipation is open, free, and now!\u003c/h2\u003e\n    \u003cp\u003eThis course is open and peer reviewed. To get involved with the course \u003cstrong\u003eopen a pull request\u003c/strong\u003e and submit your work for review. Here are the steps:\u003c/p\u003e\n    \u003col\u003e\n        \u003cli\u003eFork the repo \u003ca href=\"https://github.com/huggingface/smol-course/fork\"\u003ehere\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003eRead the material, make changes, do the exercises, add your own examples.\u003c/li\u003e\n        \u003cli\u003eOpen a PR on the december_2024 branch\u003c/li\u003e\n        \u003cli\u003eGet it reviewed and merged\u003c/li\u003e\n    \u003c/ol\u003e\n    \u003cp\u003eThis should help you learn and to build a community-driven course that is always improving.\u003c/p\u003e\n\u003c/div\u003e\n\nWe can discuss the process in this [discussion thread](https://github.com/huggingface/smol-course/discussions/2#discussion-7602932).\n\n## Course Outline\n\nThis course provides a practical, hands-on approach to working with small language models, from initial training through to production deployment.\n\n| Module | Description | Status | Release Date |\n|--------|-------------|---------|--------------|\n| [Instruction Tuning](./1_instruction_tuning) | Learn supervised fine-tuning, chat templating, and basic instruction following | ✅ Ready | Dec 3, 2024 |\n| [Preference Alignment](./2_preference_alignment) | Explore DPO and ORPO techniques for aligning models with human preferences | ✅ Ready  | Dec 6, 2024 |\n| [Parameter-efficient Fine-tuning](./3_parameter_efficient_finetuning) | Learn LoRA, prompt tuning, and efficient adaptation methods | ✅ Ready | Dec 9, 2024 |\n| [Evaluation](./4_evaluation) | Use automatic benchmarks and create custom domain evaluations | ✅ Ready | Dec 13, 2024 |\n| [Vision-language Models](./5_vision_language_models) | Adapt multimodal models for vision-language tasks | ✅ Ready | Dec 16, 2024 |\n| [Synthetic Datasets](./6_synthetic_datasets) | Create and validate synthetic datasets for training | ✅ Ready | Dec 20, 2024 |\n| [Inference](./7_inference) | Infer with models efficiently | [🚧 WIP](https://github.com/huggingface/smol-course/pull/150) | Jan 8, 2025 |\n| [Agents](./8_agents) | Build your own agentic AI | ✅ Ready | Jan 13, 2025 ||\n| Capstone Project | Use what you've learnt to climb the leaderboard! | [🚧 WIP](https://github.com/huggingface/smol-course/pull/97) | Jan 10, 2025 |\n\n## Why Small Language Models?\n\nWhile large language models have shown impressive capabilities, they often require significant computational resources and can be overkill for focused applications. Small language models offer several advantages for domain-specific applications:\n\n- **Efficiency**: Require significantly less computational resources to train and deploy\n- **Customization**: Easier to fine-tune and adapt to specific domains\n- **Control**: Better understanding and control of model behavior\n- **Cost**: Lower operational costs for training and inference\n- **Privacy**: Can be run locally without sending data to external APIs\n- **Green Technology**: Advocates efficient usage of resources with reduced carbon footprint\n- **Easier Academic Research Development**: Provides an easy starter for academic research with cutting-edge LLMs with less logistical constraints\n\n## Prerequisites\n\nBefore starting, ensure you have the following:\n- Basic understanding of machine learning and natural language processing.\n- Familiarity with Python, PyTorch, and the `transformers` library.\n- Access to a pre-trained language model and a labeled dataset.\n\n## Installation\n\nWe maintain the course as a package so you can install dependencies easily via a package manager. We recommend [uv](https://github.com/astral-sh/uv) for this purpose, but you could use alternatives like `pip` or `pdm`.\n\n### Using `uv`\n\nWith `uv` installed, you can install the course like this:\n\n```bash\nuv venv --python 3.11.0\nuv sync\n```\n\n### Using `pip`\n\nAll the examples run in the same **python 3.11** environment, so you should create an environment and install dependencies like this:\n\n```bash\n# python -m venv .venv\n# source .venv/bin/activate\npip install -r requirements.txt\n```\n\n### Google Colab\n\n**From Google Colab** you will need to install dependencies flexibly based on the hardware you're using. Like this:\n\n```bash\npip install transformers trl datasets huggingface_hub\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftobrun%2Fsmol-course-hf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftobrun%2Fsmol-course-hf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftobrun%2Fsmol-course-hf/lists"}