{"id":16947915,"url":"https://github.com/jon-tow/text-sed","last_synced_at":"2025-10-12T01:47:52.865Z","repository":{"id":63105814,"uuid":"552113473","full_name":"jon-tow/text-sed","owner":"jon-tow","description":"Implementation of Self-conditioned Embedding Diffusion for Text Generation","archived":false,"fork":false,"pushed_at":"2023-01-06T17:45:31.000Z","size":108,"stargazers_count":37,"open_issues_count":1,"forks_count":6,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-11T20:11:41.149Z","etag":null,"topics":["deep-learning","diffusion","pytorch","text-generation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jon-tow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-10-15T21:08:05.000Z","updated_at":"2025-04-11T07:21:18.000Z","dependencies_parsed_at":"2023-02-06T07:15:17.863Z","dependency_job_id":null,"html_url":"https://github.com/jon-tow/text-sed","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jon-tow/text-sed","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jon-tow%2Ftext-sed","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jon-tow%2Ftext-sed/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jon-tow%2Ftext-sed/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jon-tow%2Ftext-sed/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jon-tow","download_url":"https://codeload.github.com/jon-tow/text-sed/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jon-tow%2Ftext-sed/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279009791,"owners_count":26084648,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","diffusion","pytorch","text-generation"],"created_at":"2024-10-13T21:48:55.088Z","updated_at":"2025-10-12T01:47:52.847Z","avatar_url":"https://github.com/jon-tow.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## text-sed (wip) 📚🔥💭\n\nImplementation of Strudel et al.'s [\"Self-conditioned Embedding Diffusion for Text Generation\"](https://arxiv.org/abs/2211.04236) in PyTorch and some other goodies 🍭.\n\n## Install 💻\n\n```bash\npip install -e \".[dev, train]\"\n```\n\n## Experimental Samples \n\nNon-cherry picked samples from various experiments to track progress with the eventual goal of reproducing the results from the paper using large scale pre-training. \n\n* __Unconditionallly__ generated samples obtained from training text-sed on the simple [E2E](https://huggingface.co/datasets/e2e_nlg) dataset for 17k steps with the config [here](configs/e2e.yaml).\n  ```markdown\n  ➜ The Golden Palace is a mid priced restaurant that has a rating of 1 out of 5.\n  ➜ There is a children friendly priced restaurant that offers English food called The Twenty Two.\n  ➜ Taste of Cambridge is a nice family friendly fast food pub in the riverside area, near The Sorrento.\n  ➜ In the city centre lies The Golden Palace, a above average coffee shop serving French cuisine. Previous its customers,, unfortunately because it 3 out of 5.\n  ➜ Strada specializes in Chinese food. They are a pub near Yippee Noodle Bar and has a rating of 1 out of 5.\n  ➜ The Vaults is a high priced restaurant serving Indian food. It is kid friendly and is moderately priced.\n  ➜ The Waterman is a kid friendly restaurant that serves Japanese food near the city center. They are moderately priced.\n  ➜ The Punter is a Chinese restaurant, with an average rating. The food is cheap.\n  ```\n\n## TODOs\n\n* [ ] Add span masking and cfg for conditional generation.\n* [ ] Add EMA warmup.\n* [ ] Add Karras samplers.\n* [ ] Try linearly increasing the guidance scale through the sampling procedure to reduce diversity hits (see [Section 2.7 of Muse](https://arxiv.org/pdf/2301.00704.pdf)).\n* [ ] Add conditional generation examples/samples.\n\n## Appreciation\n\n* Katherine Crowson's [`k-diffusion` repo](https://github.com/crowsonkb/k-diffusion)\n\n## Citations\n\n```bibtex\n@article{strudel2022self,\n  title={Self-conditioned Embedding Diffusion for Text Generation},\n  author={Strudel, Robin and Tallec, Corentin and Altch{\\'e}, Florent and Du, Yilun and Ganin, Yaroslav and Mensch, Arthur and Grathwohl, Will and Savinov, Nikolay and Dieleman, Sander and Sifre, Laurent and others},\n  journal={arXiv preprint arXiv:2211.04236},\n  year={2022}\n}\n```\n\n```bibtex\n@article{dieleman2022continuous,\n  title={Continuous diffusion for categorical data},\n  author={Dieleman, Sander and Sartran, Laurent and Roshannai, Arman and Savinov, Nikolay and Ganin, Yaroslav and Richemond, Pierre H and Doucet, Arnaud and Strudel, Robin and Dyer, Chris and Durkan, Conor and others},\n  journal={arXiv preprint arXiv:2211.15089},\n  year={2022}\n}\n```\n\n```bibtex\n@article{Chen2022AnalogBG,\n    title   = {Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning},\n    author  = {Ting Chen and Ruixiang Zhang and Geoffrey E. Hinton},\n    journal = {ArXiv},\n    year    = {2022},\n    volume  = {abs/2208.04202}\n}\n```\n\n```bibtex\n@article{Li-2022-DiffusionLM,\n  title={Diffusion-LM Improves Controllable Text Generation},\n  author={Xiang Lisa Li and John Thickstun and Ishaan Gulrajani and Percy Liang and Tatsunori Hashimoto},\n  journal={ArXiv},\n  year={2022},\n  volume={abs/2205.14217}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjon-tow%2Ftext-sed","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjon-tow%2Ftext-sed","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjon-tow%2Ftext-sed/lists"}