{"id":21644463,"url":"https://github.com/samsung/mt-llm-nlu","last_synced_at":"2025-09-01T12:09:03.826Z","repository":{"id":232575880,"uuid":"778182747","full_name":"Samsung/MT-LLM-NLU","owner":"Samsung","description":"Repository for code related to \"LLM-Based Machine Translation for Expansion of Spoken Language Understanding Systems to New Languages\" publication.","archived":false,"fork":false,"pushed_at":"2024-04-12T16:55:28.000Z","size":94,"stargazers_count":14,"open_issues_count":0,"forks_count":1,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-11T18:22:17.387Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Samsung.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-03-27T08:36:27.000Z","updated_at":"2025-04-01T05:47:21.000Z","dependencies_parsed_at":"2025-04-11T18:22:18.721Z","dependency_job_id":"a93dafcb-bbc3-48aa-86ae-084c216eda03","html_url":"https://github.com/Samsung/MT-LLM-NLU","commit_stats":null,"previous_names":["samsung/mt-llm-nlu"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Samsung/MT-LLM-NLU","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Samsung%2FMT-LLM-NLU","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Samsung%2FMT-LLM-NLU/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Samsung%2FMT-LLM-NLU/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Samsung%2FMT-LLM-NLU/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Samsung","download_url":"https://codeload.github.com/Samsung/MT-LLM-NLU/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Samsung%2FMT-LLM-NLU/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273122122,"owners_count":25049539,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-01T02:00:09.058Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-25T05:40:20.927Z","updated_at":"2025-09-01T12:09:03.794Z","avatar_url":"https://github.com/Samsung.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Large Language Models for Expansion of Spoken Language Understanding Systems to New Languages\n\n## Abstract\nSpoken Language Understanding (SLU) models are a core component of voice assistants (VA), such as Alexa, Bixby, and Google Assistant. In this paper, we introduce a pipeline designed to extend SLU systems to new languages, utilizing Large Language Models (LLMs) that we fine-tune for machine translation of slot-annotated SLU training data. Our approach improved on the MultiATIS++ benchmark, a primary multi-language SLU dataset, in the cloud scenario using an mBERT model. Specifically, we saw an improvement in the Overall Accuracy metric: from 53% to 62.18%, compared to the existing state-of-the-art method, Fine and Coarse-grained Multi-Task Learning Framework (FC-MTLF). In the on-device scenario (tiny and not pretrained SLU), our method improved the Overall Accuracy from 5.31% to 22.06% over the baseline Global-Local Contrastive Learning Framework (GL-CLeF) method. Contrary to both FC-MTLF and GL-CLeF, our LLM-based machine translation does not require changes in the production architecture of SLU. Additionally, our pipeline is slot-type independent: it does not require any slot definitions or examples.\n\n#### HuggingFace\nFine tuned version of BigTranslate for slot translation on multiAtis++ languages can be found on HuggingFace:\n\nhttps://huggingface.co/Samsung/BigTranslateSlotTranslator\n\n### How to run\n#### BigTranslate slot-translation finetuning\nEnter the `BigTranslateFineTuning/` dir\n1. Prepare MASSIVE dataset\n\n`\npython prepare_massive_data.py --output-dir massive`\n\n2. Run BigTranslate LoRA finetuning\n\n`\nbash deepspeed_bigtranslate_train.sh\n`\n\n#### Translate multiATIS++ dataset\nTo get the dataset go to https://github.com/amazon-research/multiatis and then put the ``train_EN.tsv`` file into ``BigTranslateFineTuning/multiATIS`` folder.\n\nModel will be automatically downloaded from huggingface hub. You can also specify path to the model in `translate_mulitatis.sh` script (--model-name argument)\n\n`\nbash translate_multiatis.sh\n`\n\n#### Training input data preparation\n```bash\ncd MT/\npython prepare_multiatis_jointnlu_traindata.py \\\n    $TRANSLATED_DATA_DIR \\\n    False \\\n    False \\\n    True \\\n    True \\\n    False \\\n    $OUTPUT_TRAINING_DATA_DIR \\\n    5 \\\n    $SOURCE_LANG \\\n    $TARGET_LANG \\\n    0.95\n```\n#### Training input data postprocessing\n```bash\nbash multiatis_data_postprocess.sh $OUTPUT_TRAINING_DATA_DIR\n```\n\n#### Run training and evaluation\n```bash\ncd JointBERT\nmkdir eval_results fails data\ncp -r $OUTPUT_TRAINING_DATA_DIR data\npython3 main.py \\\n    --task $(basename $OUTPUT_TRAINING_DATA_DIR) \\\n    --model_type ${MODEL_TYPE} \\\n    --patience 5 \\\n    --train_batch_size 500 \\\n    --do_train \\\n    --do_eval \\\n    --ratio 1.0 \\\n    --device_name cuda:0 \\\n    --validation loss \\\n    --lang $TARGET_LANG \\\n    --model_dir $MODEL_OUTPUT_DIR \\\n    --percent_broken 0 \\\n    --num_train_epochs 50\n```\nwhere:\n- MODEL_TYPE - `multibert`, `xlmr_scratch` or any other model type defined in [utils.py](./JointBERT/utils.py)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamsung%2Fmt-llm-nlu","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsamsung%2Fmt-llm-nlu","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamsung%2Fmt-llm-nlu/lists"}