{"id":20324150,"url":"https://github.com/opentensor/clm_model_tuning","last_synced_at":"2025-04-11T19:40:58.008Z","repository":{"id":40663913,"uuid":"506342582","full_name":"opentensor/clm_model_tuning","owner":"opentensor","description":null,"archived":false,"fork":false,"pushed_at":"2023-04-23T15:38:30.000Z","size":40,"stargazers_count":8,"open_issues_count":1,"forks_count":14,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-25T15:34:39.722Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opentensor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-22T17:23:43.000Z","updated_at":"2024-10-18T14:55:01.000Z","dependencies_parsed_at":"2023-01-30T21:15:33.996Z","dependency_job_id":null,"html_url":"https://github.com/opentensor/clm_model_tuning","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opentensor%2Fclm_model_tuning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opentensor%2Fclm_model_tuning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opentensor%2Fclm_model_tuning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opentensor%2Fclm_model_tuning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opentensor","download_url":"https://codeload.github.com/opentensor/clm_model_tuning/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248468296,"owners_count":21108789,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T19:32:36.904Z","updated_at":"2025-04-11T19:40:57.978Z","avatar_url":"https://github.com/opentensor.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CLM Model Tuning\n\n\u003c!---\nCopyright 2020 The OpenTensor Team. All rights reserved.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n--\u003e\n\n**Note**: This script was adapted from HuggingFace's Transformers/language-modeling code.\n\n## Language model tuning\n\nFine-tuning the library models for language modeling on a text dataset \nfor GPT, GPT-2. Causal languages like this are trained or fine-tuned using a causal language \nmodeling (CLM) loss.\n\nIn theory, serving a tuned model can increase incentive and earnings on the bittensor network.\nHowever this depends on many factors: the choice of model, the data used\nfor tuning, and (to a lesser extent), the hyparameters used for tuning itself. This is not a silver\nbullet that will immediately guarantee higher earnings, but differences will be more pronounced\nonce the Synapse update is released (time of writing: July 25, 2022).\n\n\nIn the following examples, we will run on datasets hosted on Bittensor's IPFS mountain dataset, \non HuggingFace's dataset [hub](https://huggingface.co/datasets) or with your own text files.\n\nFor a full list of models that will work with this script, \n[refer to this link](https://huggingface.co/models?filter=text-generation).\n\n\n## Installation \u0026 Requirements\nThis code assumes you have `bittensor` already installed on your machine, and is meant to be\nrun entirely separately. Some basic linux commandline knowledge is assumed, but \n[this guide](https://ubuntu.com/tutorials/command-line-for-beginners) should provide a good starting\npoint to navigate and move around files, directories, etc.\n\nTo start, clone this repository:\n```commandline\ngit clone https://github.com/opentensor/clm_model_tuning \n```\n\nAll following commands assume you are working from this folder, i.e. you must `cd` into the directory\ncreated by the previous step.\n\nRun ```pip install -r requirements.txt``` to install the additional packages required by this script.\n\n### On bittensor\n\nBy default, the script will fine-tune GPT2 for bittensor's mountain dataset. Running:\n```bash\npython finetune_using_clm.py\n```\nwill tune gpt2 with bittensor's dataset and save the output to `tuned-model`.\n\nto change the model you are tuning to, e.g. `distilgpt2`, run:\n```bash\npython finetune_using_clm.py model.name=distilgpt2\n```\n\nSome sample models to try are available under the server customization section of \n[bittensor's documentation](https://docs.bittensor.com). A full list of models that can be trained by this\nscript are available on [huggingface](https://huggingface.co/models?filter=text-generation).\n\n### On huggingface datasets\n\nAny text dataset on [huggingface](https://huggingface.co/datasets) should work by default by\noverriding the `dataset.name` and `dataset.config` parameters:\n\n```bash\npython finetune_using_clm.py dataset.name=wikitext dataset.config_name=wikitext-103-v1\n```\n\n### On your own data\n\nIf you have a .txt file saved locally, you can override `dataset.name` as above:\n```bash\npython finetune_using_clm.py dataset.name=./path/to/your/data.txt\n```\n\n**Note** if using your own data, you may have many short sentences and the block size may be \ninsufficient for reasonable performance. It's recommended you pass the flag\n`dataset.concatenate_raw=true` to give the model more context when training. This will reduce\nthe number of batches.\n\n### Configuring training parameters\n\nAll configurable parameters are visible and documented in `conf/config.yaml`. \nThe defaults are chosen for quick training and not tuned; you will need to experiment and adjust \nthese.\n\n**Note**: The above parameters are the *only* commands you can override with this script. That is,\nyou may not pass flags you would normally use when running `btcli` (i.e. `--neuron.device` will *not* work).\nIf there is a flag you wish to modify feel free to submit a feature request.\n\nTo view the changeable parameters, open `conf/config.yaml` in whatever text editor you prefer, or\nuse `cat conf/config.yaml` to view them.\n\nYou do not need to edit this file to change the parameters; they may be overridden when you call this\nscript. e.g., if you wish to change the model to `distilgpt2`, and the output directory to `distilgpt-tuned`, you would run:\n```commandline\npython3 finetune_using_clm.py model.name=distilgpt2 output_dir=distilgpt-tuned\n```\n\nNote the nested structure in the config, since `model` is above `name` in `conf.yaml`, you must override\n`model.name` when invoking the command.\n\n\n## Serving custom models on bittensor\n\nTo serve your tuned model on bittensor, just override `neuron.model_name` with the path to your \ntuned model:\n```bash\nbtcli run ..... --neuron.model_name=/home/{YOUR_USENAME}/clm_model_tuning/tuned-model\n```\n\n## Limitations \u0026 Warnings\n\nEarly stopping is not yet supported. Many features are implemented but not thoroughly tested, if\nyou encounter an issue, reach out on discord or (preferably) create an issue on this github page.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopentensor%2Fclm_model_tuning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopentensor%2Fclm_model_tuning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopentensor%2Fclm_model_tuning/lists"}