{"id":19932346,"url":"https://github.com/amazon-science/datatuner","last_synced_at":"2025-10-04T07:41:59.523Z","repository":{"id":43054650,"uuid":"345994729","full_name":"amazon-science/datatuner","owner":"amazon-science","description":"Code related to \"Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity\" paper","archived":false,"fork":false,"pushed_at":"2022-03-21T18:57:55.000Z","size":155,"stargazers_count":92,"open_issues_count":4,"forks_count":21,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-09-09T05:07:41.885Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amazon-science.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-09T12:14:13.000Z","updated_at":"2025-03-25T16:22:40.000Z","dependencies_parsed_at":"2022-09-19T10:30:38.474Z","dependency_job_id":null,"html_url":"https://github.com/amazon-science/datatuner","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/amazon-science/datatuner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Fdatatuner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Fdatatuner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Fdatatuner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Fdatatuner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amazon-science","download_url":"https://codeload.github.com/amazon-science/datatuner/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Fdatatuner/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278283492,"owners_count":25961310,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-04T02:00:05.491Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T23:09:51.448Z","updated_at":"2025-10-04T07:41:59.492Z","avatar_url":"https://github.com/amazon-science.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DataTuner\n\nYou have just found the DataTuner. \nThis repository provides tools for fine-tuning language models for a task.\n\n* See [LICENSE.txt](LICENSE.txt) for license details.\n\n* See [NOTICE.txt](NOTICE.txt) for details of third party code included in or downloaded by this code. \n\n* See [/paper/README.md](paper/README.md) for details about reproducing the results reported in the paper \n[\"Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity\" by Hamza Harkous, Isabel Groves and Amir Saffari.](https://www.aclweb.org/anthology/2020.coling-main.218/)\n\n\n\n## Installation\n\n### Environment Creation\n\nAssuming you have an existing `conda` setup, you can setup the environment with the following script. In order to activate the conda environment within the bash script, you need the location of the `conda.sh` file:\n\n```bash\nbash setup.sh  ~/miniconda3/etc/profile.d/conda.sh\n```\n\nYou can update your existing environment:\n\n```bash\nconda env update -f=environment.yml\n```\n\nTo start development, activate your environment:\n\n```bash\nconda activate finetune\n```\n\nAlternatively, you can always use the python binary with the absolute path, e.g.: `~/miniconda3/envs/finetune/bin/python`.\n\n## Data \n\nFor any task you want to fine-tune on, you need the data to be a json file containing a list of json objects, one per data point. For example:\n\n```json\n[\n  {\n    \"question\": \"question text 1\",\n    \"query\": \"query 1\"\n  },\n  {\n    \"question\": \"question text 2\",\n    \"query\": \"query 2 with [SpecialToken example]\"\n  }\n]\n```\n\nThe library assumes that you have placed your data in a single directory with three files: ``train.json``, ``validation.json``, and ``test.json``.\n\n## Configuration \n\nNow that we have the data in shape, we need to create a new task configuration file that specifies how we want the data to be formatted and what fields should be considered. You can create new config files in the folder ``src/datatuner/lm/task_configs``.\n\nA typical config file would look as follows:\n\n\n```json\n{\n\"name\": \"dataset_name\",\n\"data_shape\": [\n        {\n            \"id\": \"\u003cquestion\u003e\",\n            \"type\": \"special\",\n            \"learn\": false\n        },\n        {\n            \"id\": \"question\",\n            \"type\": \"text\",\n            \"learn\": false\n        },\n        {\n            \"id\": \"\u003cquery\u003e\",\n            \"type\": \"special\",\n            \"learn\": false\n        },\n        {\n            \"id\": \"query\",\n            \"type\": \"text\",\n            \"learn\": true,\n            \"metrics\": [\n                \"match\"\n            ]\n        }\n    ],\n\"extra_special_tokens\": [\"[SpecialToken\"],\n\"extra_fields\": []\n}\n```\n\nFor each item in the data shape:\n\n- ``type`` (required): ``special`` if special token, ``text`` if normal text.\n- ``id`` (required): the special token ID if type is ``special``; the key for the text in the json data if type is ``text``\n- ``learn`` (required): whether to allow the model to learn this part of the text. If false, the model masks that part during fine-tuning.\n- ``metrics`` (optional): the list of metrics that the model should compute upon evaluation. Each metric should have a corresponding function with the same name in ``metrics.py``.\n- ``converter`` (optional): the name of the converter function in ``converters.py`` to apply on that text field after reading the text from the file. \n\nThe value of `extra_special_tokens` is a list of special tokens to be added to the vocabulary. \nAlternatively (especially if the list is too long or is generated automatically), you can create a text file with one special token per line and pass that as an argument during training via the `--special_tokens_file` argument.\n\n\nThe value of `extra_fields` is a list of additional fields to include from the input `json` files to output during evaluation, aside from the main fields used as inputs/outputs.\n\n## Training \n\nThe training script `train.py` can be used in single GPU or multi GPU settings.  \n\n```bash\ncd src/datatuner/lm\n\n# single gpu\npython train.py --model_checkpoint ~/data/openai-gpt/  --dataset_path ../../../data/my_dataset/  --task_config ./task_configs/my_task_config.json --n_epoch 3 --lr 1e-5\n\n# multi gpu\npython -m torch.distributed.launch --nproc_per_node=4 train.py --model_checkpoint ~/data/openai-gpt/  --dataset_path ../../../data/my_dataset/  --task_config ./task_configs/my_task_config.json --n_epoch 3 --lr 1e-5\n```\n\n\n## Evaluating the Model \n\nYou can run the following to evaluate the model on any test set. The data format is the same as the training data. Notice that you have to currently specify the ``model_type`` parameter matching the model you're loading:\n\n```bash\ncd src/datatuner/lm\n\npython ./evaluate.py --task_config ./task_configs/my_task_config.json --model_checkpoint runs/2020-01-01_01-01-01  --filename ../../../data/my_dataset/test.json --max_length 200 --model_type gpt --top_k 1\n\n# or if you just want to evaluate the latest model you trained \nRUN=$(ls -t ./runs | head -1) \u0026\u0026 python ./evaluate.py --task_config ./task_configs/my_task_config.json --model_checkpoint runs/$RUN  --filename ../../../data/my_dataset/test.json --max_length 200 --model_type gpt  --top_k 1\n\n# or if you want to use the latest intermediate checkpoint while the model is training:\nRUN=$(ls -t ./runs | head -1) \u0026\u0026 CHECKPOINT=$(ls -t ./runs/$RUN/checkpoint* | head -1) \u0026\u0026 cp $CHECKPOINT runs/$RUN/pytorch_model.bin\n``` \n\nDuring evaluation, the outputs that do not exactly match the expected outputs will be printed. Also,\nthe metrics will be printed (a dictionary with keys `\u003cmetric_name\u003e_\u003cfield_name\u003e`). At the end of evaluation, you will find the file with all the generated ouputs in the file `eval_results/\u003crun_folder_name\u003e/\u003ctask_name\u003e_\u003ctest_file_name\u003e_\u003cmodel_type\u003e_generated.json`.\n\n\n\n# Interacting with the model\n\nYou can also interact with the models. The client will ask you to input the fields required, and it will generate the fields it learnt.\n\n```bash\ncd src/datatuner/lm\n\npython ./evaluate.py --task_config ./task_configs/my_task_config.json --model_checkpoint runs/2020-01-01_01-01-01  --max_length 200 --model_type gpt  --top_k 1 --input\n\n# or if you just want to evaluate the latest model you trained \nRUN=$(ls -t ./runs | head -1) \u0026\u0026 python ./evaluate.py --task_config ./task_configs/my_task_config.json --model_checkpoint runs/$RUN  --max_length 200 --model_type gpt  --top_k 1 --input\n``` ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famazon-science%2Fdatatuner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famazon-science%2Fdatatuner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famazon-science%2Fdatatuner/lists"}