{"id":17141064,"url":"https://github.com/zphang/usc_dae","last_synced_at":"2026-03-06T08:34:02.432Z","repository":{"id":91587915,"uuid":"145922435","full_name":"zphang/usc_dae","owner":"zphang","description":"Repository for Unsupervised Sentence Compression using Denoising Auto-Encoders","archived":false,"fork":false,"pushed_at":"2024-07-25T10:12:42.000Z","size":155,"stargazers_count":47,"open_issues_count":5,"forks_count":15,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-10-07T20:46:35.176Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zphang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-24T00:38:42.000Z","updated_at":"2025-04-03T23:24:37.000Z","dependencies_parsed_at":null,"dependency_job_id":"c7480cb1-7d52-458a-a569-f22008dbfd29","html_url":"https://github.com/zphang/usc_dae","commit_stats":{"total_commits":4,"total_committers":3,"mean_commits":"1.3333333333333333","dds":0.5,"last_synced_commit":"9da432971d165b5b2068fa9724a495f88d3ef5f2"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zphang/usc_dae","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zphang%2Fusc_dae","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zphang%2Fusc_dae/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zphang%2Fusc_dae/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zphang%2Fusc_dae/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zphang","download_url":"https://codeload.github.com/zphang/usc_dae/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zphang%2Fusc_dae/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30167963,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T07:56:45.623Z","status":"ssl_error","status_checked_at":"2026-03-06T07:55:55.621Z","response_time":250,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-14T20:24:14.692Z","updated_at":"2026-03-06T08:34:02.379Z","avatar_url":"https://github.com/zphang.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Experiments in Unsupervised summarization\n\nThis is our [Pytorch](https://github.com/pytorch/pytorch) implementation of the summarization methods described in Unsupervised Sentence Compression using Denoising Autoencoders (CoNLL 2018). It features denoising additive auto-encoders with optional NLI hidden state initialization (based on [Infersent](https://github.com/facebookresearch/InferSent)).\n\n![Model architecture](architecture.png)\n\n\nTable of Contents\n=================\n\n  * [Requirements](#requirements)\n  * [Quickstart](#quickstart)\n  \n## Requirements\n\n\n```bash\npip install -r requirements.txt\n```\n\n## Quickstart\n\n### Step 1: Get the data and create the vocabulary\n\nGigaword data can be downloaded from : https://github.com/harvardnlp/sent-summary. Then, extract it (```tar -xzf summary.tar.gz ```). Vocabulary can then be created by running ``` python src/datasets/preprocess.py train_data_file output_voc_file```)\n\n### Step 2: Create an environment configuration file\n\nThis is used to locate the datasets, embeddings, whether you want to use gpu, etc. on your computer. You can see an example configuration at ```env_configs/env_config.json```. You only need to set nli variables if you use InferSent embeddings. \n\nThen setup the variable `NLU_ENV_CONFIG_PATH` to point to that file (e.g: `export NLU_ENV_CONFIG_PATH=\"env_configs/env_config.json\"`). \n\n### Step 3: Train the model\n\nSimply run:\n```bash\npython sample_scripts/dae_json.py runs/default/default.json\n```\n\n### Step 4: Run inference\n\n```bash\npython sample_scripts/simple_inference.py model_path test_data_path [output_data_path]\n```\n\n### Step 5: Evaluate ROUGE scores\n\nTo evaluate for rouge, we use [files2rouge](https://github.com/pltrdy/files2rouge), which itself uses\n[pythonrouge](https://github.com/tagucci/pythonrouge).\n\nInstallation instructions:\n\n```bash\npip install git+https://github.com/tagucci/pythonrouge.git\ngit clone https://github.com/pltrdy/files2rouge.git\ncd files2rouge\npython setup_rouge.py\npython setup.py install\n```\n\nTo run evaluation, simply run:\n```bash\nfiles2rouge summaries.txt references.txt\n```\n\n## FAQ\n\n* **Random seed**: We did not use a random seed nor random restarts for the results in the paper\n* **Teacher forcing**: We used teacher forcing in all of our experiments\n* **Beam search**: We decoded using greedy decoding only, never using beam search\n* **Added noise**: Is done on a sentence-per-sentence basis, not based on the max length in a batch. This is critical for performance","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzphang%2Fusc_dae","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzphang%2Fusc_dae","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzphang%2Fusc_dae/lists"}