{"id":17300589,"url":"https://github.com/hendrikstrobelt/lmdiff","last_synced_at":"2025-10-09T03:34:19.653Z","repository":{"id":75102605,"uuid":"368879682","full_name":"HendrikStrobelt/LMdiff","owner":"HendrikStrobelt","description":"A diff tool for language models","archived":false,"fork":false,"pushed_at":"2023-12-28T12:07:26.000Z","size":674,"stargazers_count":44,"open_issues_count":6,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-09T12:43:04.290Z","etag":null,"topics":["ai","nlproc","visualization","xai"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HendrikStrobelt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-19T13:30:46.000Z","updated_at":"2025-09-01T01:25:42.000Z","dependencies_parsed_at":"2023-12-28T13:10:57.538Z","dependency_job_id":null,"html_url":"https://github.com/HendrikStrobelt/LMdiff","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/HendrikStrobelt/LMdiff","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HendrikStrobelt%2FLMdiff","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HendrikStrobelt%2FLMdiff/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HendrikStrobelt%2FLMdiff/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HendrikStrobelt%2FLMdiff/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HendrikStrobelt","download_url":"https://codeload.github.com/HendrikStrobelt/LMdiff/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HendrikStrobelt%2FLMdiff/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279000714,"owners_count":26082911,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","nlproc","visualization","xai"],"created_at":"2024-10-15T11:29:26.490Z","updated_at":"2025-10-09T03:34:19.619Z","avatar_url":"https://github.com/HendrikStrobelt.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LMdiff\n\nQualitative comparison of large language models.\n\nDemo \u0026 Paper: [http://lmdiff.net](http://lmdiff.net)\n\n\n\u003cdiv style='text-align:center; width:100%'\u003e\n\u003cimg src=\"assets/lmdiff_teaser.jpg\" /\u003e\n\u003c/div\u003e\n\n\n\u003cp\u003e LMdiff is a \u003ca href=\"https://mitibmwatsonailab.mit.edu/\"\u003eMIT-IBM\nWatson AI Lab\u003c/a\u003e collaboration between: \u003cbr/\u003e\n\u003ca href=\"http://hendrik.strobelt.com\" target=\"_blank\"\u003eHendrik\n  Strobelt\u003c/a\u003e (IBM, MIT) ,\n\u003ca href=\"https://www.bhoov.com/\" target=\"_blank\"\u003eBenjamin Hoover\u003c/a\u003e\n(IBM, GeorgiaTech),\n\u003ca href=\"https://arvindsatya.com/\" target=\"_blank\"\u003eArvind\n  Satyanarayan\u003c/a\u003e (MIT),\nand \u003ca href=\"https://sebastiangehrmann.com/\" target=\"_blank\"\u003eSebastian\n  Gehrmann\u003c/a\u003e (HarvardNLP, Google).\u003c/p\u003e\n\n## Setting up / Quick start\nFrom the root directory install Conda dependencies:\n\n```\nconda env create -f environment.yml\nconda activate LMdiff\npip install -e .\n```\n\nRun the backend in development mode, deploying default models and configurations:\n\n```\nuvicorn backend.server:app --reload\n```\nCheck the output for the right port (something like [http://localhost:8000](http://localhost:8000)) and open in Browser.\n\n## Rebuild frontend\nThis is optional, because we have a compiled version checked into this repo.\n\n```\ncd client\nnpm install\nnpm run build:backend\ncd ..\n```\n\n\n## Using your own models\n\nTo use your own models:\n\n1. Create a `TextDataset` of phrases to analyze\n\n    You can create the dataset file in several ways:\n\n    \u003cdetails\u003e\n    \u003csummary\u003eFrom a text file\u003c/summary\u003e\n    So you have already collected all the phrases you want into a text file separated by newlines. Simply run:\n\n    ```\n    python scripts/make_dataset.py path/to/my_dataset.txt my_dataset -o folder/i/want/to/save/in\n    ```\n    \u003c/details\u003e\n    \n    \u003cdetails\u003e\n    \u003csummary\u003eFrom a python object (list of strings)\u003c/summary\u003e\n    Want to only work within python?\n\n    ```python\n    from analysis.create_dataset import create_text_dataset_from_object\n\n    my_collection = [\"Phrase 1\", \"My second phrase\"]\n    create_text_dataset_from_object(my_collection, \"easy-first-dataset\", \"human_created\", \"folder/i/want/to/save/in\")\n    ```\n    \u003c/details\u003e\n    \n    \u003cdetails\u003e\n    \u003csummary\u003eFrom [Huggingface Datasets](https://huggingface.co/docs/datasets/)\u003c/summary\u003e\n    It can be created from one of Huggingface's provided datasets with:\n\n    ```python\n    from analysis.create_dataset import create_text_dataset_from_hf_datasets\n    import datasets\n    import path_fixes as pf\n\n    glue_mrpc = datasets.load_dataset(\"glue\", \"mrpc\", split=\"train\")\n    name = \"glue_mrpc_train\"\n\n    def ds2str(glue):\n        \"\"\"(e.g.,) Turn the first 50 sentences of the dataset into sentence information\"\"\"\n        sentences = glue['sentence1'][:50]\n        return \"\\n\".join(sentences)\n\n    create_text_dataset_from_hf_datasets(glue_mrpc, name, ds2str, ds_type=\"human_created\", outfpath=pf.DATASETS)\n    ```\n    \u003c/details\u003e\n\n\n\tThe dataset is a simple `.txt` file, with a new phrase on every line, and with a bit of required metadata header at the top. E.g.,    \n\t\n\t```\n\t---\n\tchecksum: 92247a369d5da32a44497be822d4a90879807a8751f5db3ff1926adbeca7ba28\n\tname: dataset-dummy\n\ttype: human_created\n\t---\n\t\n\tThis is sentence 1, please analyze this.\n\tEvery line is a new phrase to pass to the model.\n\tI can keep adding phrases, so long as they are short enough to pass to the model. They don't even need to be one sentence long.\n\t```\n\n\tThe required fields in the header:\n\t\n\t- `checksum` :: A unique identifier for the state of that file. It can be calculated however you wish, but it should change if anything at all changes in the contents below (e.g., two phrases are transposed, a new phase added, or a period is added after a sentence)\n\t- `name` :: The name of the dataset. \n\t- `type` :: Either `human_created` or `machine_generated` if you want to compare on a dataset that was spit out by another model\n\t\n\tEach line in the contents is a new phrase to compare in the language model. A few warnings:\n\t\n\t- Make sure the phrases are short enough that they can be passed to the model given your memory constraints\n\t- The dataset is fully loaded into memory to serve to the front end, so avoid creating a text file that is too large to fit in memory.\n\n2. Choose two comparable models\n    \n    Two models are comparable if they:\n\n    1. Have the exact same tokenization scheme\n    2. Have the exact same vocabulary\n\n    This allows us to do tokenwise comparisons on the model. For example, this could be:\n    \n    - A pretrained model and a finetuned version of it (e.g., `distilbert-base-cased` and `distilbert-base-uncased-finetuned-sst-2-english`)\n    - A distilled version mimicking the original model (e.g., `bert-base-cased` and `distilbert-base-cased`)\n    - Different sizes of the same model architecture (e.g., `gpt2` and `gpt2-large`)\n\n\n3. Preprocess the models on the chosen dataset\n\n    ```\n    python scripts/preprocess.py all gpt2-medium distilgpt2 data/datasets/glue_mrpc_1+2.csv --output-dir data/sample/gpt2-glue-comparisons\n    ```\n\n4. Start the app\n\n    ```\n    python backend/server/main.py --config data/sample/gpt2-glue-comparisons\n    ```\n\n    Note that if you use a different tokenization scheme than the default `gpt`, you will need to tell the frontend how to visualize the tokens. For example, a `bert` based tokenization scheme:\n\n    ```\n    python backend/server/main.py --config data/sample/bert-glue-comparisons -t bert\n    ```\n\n## Architecture\n\u003cdiv style='text-align:center; width:100%'\u003e\n\u003cimg src=\"assets/lmdiff_system_design_diagram.png\" /\u003e\n\u003c/div\u003e\n\n## (Admin) Getting the Data\nModels and datasets for the deployed app are stored on the cloud and require a private `.dvc/config` file.\n\nWith the correct config:\n\n```dvc pull```\n\nwill populate the data directories correctly for the deployed version.\n\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eTesting\u003c/b\u003e\u003c/summary\u003e\n\n```\nmake test\n```\n\nor\n\n```\npython -m pytest tests\n```\n\nAll tests are stored in `tests`.\n\n\u003c/details\u003e\n\n### Frontend\n\nWe like [`pnpm`](https://pnpm.io/installation) but `npm` works just as well. We also like [`Vite`](https://vitejs.dev/) for its rapid hot module reloading and pleasant dev experience. This repository uses [`Vue`](https://vuejs.org/) as a reactive framework.\n\nFrom the root directory:\n\n```\ncd client\npnpm install --save-dev\npnpm run dev\n```\n\nIf you want to hit the backend routes, make sure to also run the `uvicorn backend.server:app` command from the project root.\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eFor production (serve with Vite)\u003c/b\u003e\u003c/summary\u003e\n\n```\npnpm run serve\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eFor production (serve with this repo's FastAPI server)\u003c/b\u003e\u003c/summary\u003e\n\n```\ncd client\npnpm run build:backend\ncd ..\nuvicorn backend.server:app\n```\n\nOr the `gunicorn` command from above.\n\nAll artifacts are stored in the `client/dist` directory with the appropriate basepath.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eFor production (serve with external tooling like NGINX)\u003c/b\u003e\u003c/summary\u003e\n\n```\npnpm run build\n```\n\nAll artifacts are stored in the `client/dist` directory.\n\u003c/details\u003e\n\n## Notes\n\n- Check the endpoints by visiting `\u003clocalhost\u003e:\u003cport\u003e/docs`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhendrikstrobelt%2Flmdiff","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhendrikstrobelt%2Flmdiff","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhendrikstrobelt%2Flmdiff/lists"}