{"id":23725998,"url":"https://github.com/voidful/causal-lm-trainer","last_synced_at":"2025-06-24T00:31:26.258Z","repository":{"id":185451142,"uuid":"673029083","full_name":"voidful/causal-lm-trainer","owner":"voidful","description":null,"archived":false,"fork":false,"pushed_at":"2023-08-30T01:46:25.000Z","size":5,"stargazers_count":8,"open_issues_count":0,"forks_count":4,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-05T00:25:15.419Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/voidful.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-07-31T17:50:43.000Z","updated_at":"2024-02-29T05:00:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"4acf4332-58b2-4f67-b396-21d80f44aadd","html_url":"https://github.com/voidful/causal-lm-trainer","commit_stats":null,"previous_names":["voidful/causal-lm-trainer"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/voidful/causal-lm-trainer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voidful%2Fcausal-lm-trainer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voidful%2Fcausal-lm-trainer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voidful%2Fcausal-lm-trainer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voidful%2Fcausal-lm-trainer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/voidful","download_url":"https://codeload.github.com/voidful/causal-lm-trainer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voidful%2Fcausal-lm-trainer/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261582579,"owners_count":23180616,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-31T00:18:07.834Z","updated_at":"2025-06-24T00:31:26.242Z","avatar_url":"https://github.com/voidful.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# causal-lm-trainer  \n  \nThis is a simple example of using a causal language model for various tasks, leveraging Hugging Face's `Trainer` for efficient model training. The repository includes a configurable interface for dataset processing, custom data collator for padding, and evaluation metrics, allowing for seamless adaptation to various tasks and datasets.  \n  \n## Features  \n  \n- Utilize a powerful causal language model for various tasks  \n- Easy configuration for custom dataset processing, padding, and evaluation metrics  \n- Integration with Hugging Face's `Trainer` for efficient training and evaluation  \n  \n## Usage  \n  \n1. **Dataset processing**: Modify `data_processing.py` to accommodate your own dataset. The script should take care of loading, preprocessing, and tokenizing the data as required by the causal language model.  \n  \n2. **Data collator for padding**: Customize the data collator for padding by modifying the `PaddedDataCollator` class in the `data_collator.py` file. This class handles padding for input_ids, attention_mask, and labels tensors, ensuring uniform-sized tensors in each batch.  \n  \n3. **Evaluation metric**: Customize the evaluation metric by modifying `eval_metric.py`. This script should implement the necessary logic to compute the desired evaluation metric for your task (e.g., BLEU score, ROUGE score, etc.).  \n  \n4. **Training and evaluation**: Execute `main.py` to start the training and evaluation process. This script will use the custom dataset processing, data collator for padding, and evaluation metric functions specified in the previous steps, along with the Hugging Face `Trainer`, to efficiently train and evaluate the causal language model on your task.  \n  \n## Requirements  \n  \n- Python 3.6 or later  \n- Hugging Face Transformers library  \n- PyTorch  \n- tqdm  \n  \nTo install the required packages, run:  \n\n```\npip install -r requirements.txt\n```\n\n## Running the code in qlora:\n``sh\npyhon main.py --int4\n``\n\n## Run with deepspeed\n``sh\naccelerate launch --config_file ds_zero3_cpu.yaml main.py\n``\n\n\n## License\nThis project is licensed under the [MIT License]().","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvoidful%2Fcausal-lm-trainer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvoidful%2Fcausal-lm-trainer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvoidful%2Fcausal-lm-trainer/lists"}