{"id":18357218,"url":"https://github.com/nethermindeth/mpt-instruct-dotnet-s","last_synced_at":"2025-04-06T12:32:42.889Z","repository":{"id":189613836,"uuid":"680953627","full_name":"NethermindEth/Mpt-Instruct-DotNet-S","owner":"NethermindEth","description":"Training and Evaluation code for DotNet focused LLM (based on mosaicml/mpt-7b-instruct)","archived":false,"fork":false,"pushed_at":"2023-08-25T14:49:48.000Z","size":1671,"stargazers_count":21,"open_issues_count":0,"forks_count":4,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-22T00:34:01.375Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NethermindEth.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-08-20T23:45:15.000Z","updated_at":"2025-02-22T20:40:05.000Z","dependencies_parsed_at":"2024-01-05T20:51:17.170Z","dependency_job_id":"8cdd3cdd-50b1-4a54-b4a6-7dc2f6b08e4e","html_url":"https://github.com/NethermindEth/Mpt-Instruct-DotNet-S","commit_stats":null,"previous_names":["nethermindeth/mpt-instruct-dotnet-s"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NethermindEth%2FMpt-Instruct-DotNet-S","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NethermindEth%2FMpt-Instruct-DotNet-S/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NethermindEth%2FMpt-Instruct-DotNet-S/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NethermindEth%2FMpt-Instruct-DotNet-S/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NethermindEth","download_url":"https://codeload.github.com/NethermindEth/Mpt-Instruct-DotNet-S/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247484676,"owners_count":20946388,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T22:13:15.546Z","updated_at":"2025-04-06T12:32:39.137Z","avatar_url":"https://github.com/NethermindEth.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Mpt-Instruct-DotNet-S\nThis repository hosts examples of [`Nethermind/Mpt-Instruct-DotNet-S`](https://huggingface.co/Nethermind/Mpt-Instruct-DotNet-S) usage and training procedures.\n\n![nm-llm3 1](https://github.com/NethermindEth/Mpt-Instruct-DotNet-S/assets/2915361/e9d87ccd-ffa0-456c-9523-aca193cab867)\n\n\n## Use on CPU in .Net\nWe created a GGML wrapper for MPT GGML codes and provided it in this repository.\nIt is built for:\n - Windows-x64\n - Linux-x64\n - Mac-Arm (M1 and later, to run right-click on libmpt-library.dylib to open to allow unsigned binary)\n\nQuantised weights can be automatically downloaded from [`Nethermind/Mpt-Instruct-DotNet-S`](https://huggingface.co/Nethermind/Mpt-Instruct-DotNet-S). We provide three flavours:\n - f16 - for best results, requires \u003e 14GB of free RAM, slow  (in theory, in reality, just runs slower when there is not enough ram)\n - q8 - for results with lower quality yet generated faster, requires \u003e 7.5GB of free RAM (in theory, in reality, just runs slower when there is not enough ram)\n - q5 - for results with even lower quality yet generated in the least amount of time, requires\u003e 4.5GB of free RAM\n\n Use:\n```csharp\nvar downloader = new ModelDownloader();\nvar path = await downloader.DownloadModel(\"q8\"); // you also can use f16 (eats 14 GB of RAM), q5 (eats 4 GB)\nvar mpt = new MptConsole(new mpt_params() {\n\tmodel = path,\n\tn_predict = 512,\n\tn_ctx = 1024,\n\t// n_threads = 16\n});\nvar result = mpt.Process(@\"You are an experienced .Net C# developer. Below is an instruction that describes a task. Write a response that completes the request providing detailed explanations with code examples.\n### Instruction:\ninterface IRobot {\n    void Take(string what);\n    void Cut(int size);\n    void Give(string to);\n}\n\nYou are a robot. Create a sample of _api usage performing what is asked in Example:\nRick asks a robot: slice and pass me the butter, please\n\nCreate example using _api continuing this code:\nclass Request {\n    IRobot _api;\n    void Do() {\n### Response:\n\");\n\nConsole.WriteLine(\"--------------------------------\");\nConsole.WriteLine(result.FilterString());\n```\n\n## Use on GPU in Python\nIt will requier \u003e 8000MB of free GPU ram even with `load_in_8bit=True` In short:\n```python\nimport torch\nimport transformers\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"EleutherAI/gpt-neox-20b\")\ntokenizer.pad_token = tokenizer.eos_token\n\ndevice = torch.device(\"cuda\")\nmodel_name = \"Nethermind/Mpt-Instruct-DotNet-S\"\nconfig = transformers.AutoConfig.from_pretrained(model_name, trust_remote_code=True)\nconfig.init_device = device\nconfig.max_seq_len = 1024 \nconfig.attn_config['attn_impl'] = 'torch'\nconfig.use_cache = False\n\nmodel = transformers.AutoModelForCausalLM.from_pretrained(\n\tmodel_name,\n\tconfig=config,\n\ttorch_dtype=torch.bfloat16,\n\ttrust_remote_code=True,\n\tignore_mismatched_sizes=True,\n\t# load_in_8bit=True # when low on GPU memory\n)\nmodel.eval()\n\nINSTRUCTION_KEY = \"### Instruction:\"\nRESPONSE_KEY = \"### Response:\"\nPROMPT_FOR_GENERATION_FORMAT = \"\"\"{system}\n{instruction_key}\n{instruction}\n{response_key}\n\"\"\".format(\n    system=\"{system}\",\n    instruction_key=INSTRUCTION_KEY,\n    instruction=\"{instruction}\",\n    response_key=RESPONSE_KEY\n)\n\ndef give_answer(instruction=\"Create a loop over [0, 6, 7 , 77] that prints its contentrs\", system=\"You are an experienced .Net C# developer. Below is an instruction that describes a task. Write a response that completes the request providing detailed explanations with code examples.\", ):\n    question = PROMPT_FOR_GENERATION_FORMAT.format(system=system, instruction=instruction)\n    input_tokens = tokenizer.encode(question ,return_tensors='pt')               \n    model.generate(input_tokens.to(device), max_new_tokens=min(512, 1024 - input_tokens.shape[1]), do_sample=False, top_k=1, top_p=0.95)\n    outputs = output_loop(tokenized_question)\n    answer = tokenizer.batch_decode(outputs, skip_special_tokens=True)\n    print(answer[0])\n\n```\n\n### GPU speedups\nSet max_new_tokens to 256, 1024-prompt tokens length is its limit.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnethermindeth%2Fmpt-instruct-dotnet-s","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnethermindeth%2Fmpt-instruct-dotnet-s","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnethermindeth%2Fmpt-instruct-dotnet-s/lists"}