{"id":13735423,"url":"https://github.com/notnews/archive_news_cc","last_synced_at":"2025-05-08T11:33:35.999Z","repository":{"id":79658208,"uuid":"112410624","full_name":"notnews/archive_news_cc","owner":"notnews","description":"Closed Caption Transcripts of News Videos from archive.org 2014--2023","archived":false,"fork":false,"pushed_at":"2023-08-20T07:48:21.000Z","size":26464,"stargazers_count":42,"open_issues_count":0,"forks_count":3,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-01-28T23:09:26.677Z","etag":null,"topics":["archive-org","closed-caption-transcripts","closed-captions","news","news-videos","transcript","transcripts"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/notnews.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-11-29T01:32:38.000Z","updated_at":"2024-01-18T21:58:52.000Z","dependencies_parsed_at":"2024-01-06T14:07:56.173Z","dependency_job_id":null,"html_url":"https://github.com/notnews/archive_news_cc","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/notnews%2Farchive_news_cc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/notnews%2Farchive_news_cc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/notnews%2Farchive_news_cc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/notnews%2Farchive_news_cc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/notnews","download_url":"https://codeload.github.com/notnews/archive_news_cc/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224727181,"owners_count":17359532,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archive-org","closed-caption-transcripts","closed-captions","news","news-videos","transcript","transcripts"],"created_at":"2024-08-03T03:01:06.728Z","updated_at":"2025-05-08T11:33:35.990Z","avatar_url":"https://github.com/notnews.png","language":"HTML","funding_links":[],"categories":["HTML"],"sub_categories":[],"readme":"## Closed Captions of News Videos from Archive.org\n\nThe repository provides scripts for downloading the data, and link to two datasets that were built using the scripts:\n\n* [Scripts](https://github.com/notnews/archive_news_cc#downloading-the-data-from-archiveorg)\n* [Data](https://github.com/notnews/archive_news_cc#data)\n\n-------------\n\n### Downloading the Data from Archive.org\n\nDownload closed caption transcripts of nearly 1.3M news shows from [http://archive.org](http://archive.org). \n\nThere are three steps to downloading the transcripts:\n\n1. We start by searching [https://archive.org/advancedsearch.php](https://archive.org/advancedsearch.php) with collection `collection:\"tvarchive\"`. This gets us unique identifiers for each of the news shows. An identifier is a simple string that combines channel_name, show_name, time, and date. The current final list of identifiers (2009--Nov. 2017) is posted [here](data/search.csv). \n\n2. Next, we use the identifier to build a URL where the metadata file and HTML file with the closed captions is posted. The general base URL is http://archive.org/download followed by the identifier.\n\n3. The third script parses the downloaded metadata and HTML closed caption files and creates a CSV along with the meta data.\n\nFor instance, we will go http://archive.org/download/CSPAN_20090604_230000 for identifier `CSPAN_20090604_230000` And from http://archive.org/download/CSPAN_20090604_230000/CSPAN_20090604_230000_meta.xml, we read the link http://archive.org/details/CSPAN_20090604_230000, from which we get the text from HTML file. We also store the meta data from the META XML file.\n\n#### Scripts\n\n1. **Get Show Identifiers**  \n    - [Get Identifiers For Each Show (Channel, Show, Date, Time)](scripts/get_news_identifiers.py)\n    - Produces [data/search.csv](data/search.csv)\n\n2. **Download Metadata and HTML Files**  \n    - [Download the Metadata and HTML Files](scripts/scrape_archive_org.py)\n    - Saves the metadata and HTML files to two separate folders specified in `--meta` and `--html` respectively. The default folder names are `meta` and `html` respectively.\n\n3. **Parse Metadata and HTML Files**  \n    - [Parses metadata and HTML Files and Saves to a CSV](scripts/parse_archive.py)\n    - Produces a CSV. [Here's an example](data/archive-out.csv)\n\n#### Running the Scripts\n\n1. Get all TV Archive identifiers from archive.org.  \n\n    ```\n    python get_news_identifiers.py -o ../data/search.csv\n    ```\n\n2. Download metadata and HTML files for all the shows in the [sample input file](data/search-test.csv)  \n\n    ```\n    python scrape_archive_org.py ../data/search-test.csv\n    ```\n\n    This will create two directories `meta` and `html` by default in the same folder as where the script is. We have included the first [25 metadata](data/meta/) and first 25 [html files](data/html/).  \n\n    You can change the folder for `meta` by using the `--meta` flag. To change the directory for `html`, use the `--html` flag and specify the new directory. For instance,  \n\n    ```\n    python scrape_archive_org.py --meta meta-foxnews --html html-foxnews ../data/search-test.csv\n    ```\n\n    Use `-c/--compress` option to store and parse the downloaded files in compression format (GZip).\n\n3. Parse and extract meta fields and text from [sample metadata](data/meta) and [HTML files](data/html). \n\n    ```\n    python parse_archive.py ../data/search-test.csv\n    ```\n\n    A [sample output file](data/archive-out.csv).\n\n### Data\n\nThe data are hosted on [Harvard Dataverse](https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/OAJJHI)\n\n\n**Dataset Summary:**\n\n1. **500k Dataset from 2014:**\n   - CSV: `archive-cc-2014.csv.xza*` (2.7 GB, split into 2GB files)\n   - HTML: `html-2014.7za*` (10.4 GB, split into 2GB files)\n\n2. **860k Dataset from 2017:**\n   - CSV: `archive-cc-2017.csv.gza*` (10.6 GB, split into 2GB files)\n   - HTML: `html-2017.tar.gza*` (20.2 GB, split into 2GB files)\n   - Meta: `meta-2017.tar.gza*` (2.6 GB, split into 2GB files)\n\n3. **917k Dataset from 2022:**\n   - CSV: `archive-cc-2022.csv.gza*` (12.6 GB, split into 2GB files)\n   - HTML: `html-2022.tar.gza*` (41.1 GB, split into 2GB files)\n   - Meta: `meta-2022.tar.gz` (2.1 GB)\n\n4. **179k Dataset from 2023:**\n   - CSV: `archive-cc-2023.csv.gz` (1.7 GB)\n   - HTML: `html-2023.tar.gza*` (7.3 GB, split into 2GB files)\n   - Meta: `meta-2023.tar.gz` (317 MB)\n\nPlease note that the file sizes and splitting information mentioned above are approximate.\n\n\n### License\n\nWe are releasing the scripts under the [MIT License](https://opensource.org/licenses/MIT).\n\n### Suggested Citation\n\nPlease credit Internet Archive for the data. \n\nIf you wanted to refer to this particular corpus so that the research is reproducible, you can cite it as:\n```\narchive.org TV News Closed Caption Corpus. Laohaprapanon, Suriyan and Gaurav Sood. 2017. https://github.com/notnews/archive_news_cc/     \n```\n\n## 🔗 Adjacent Repositories\n\n- [notnews/lacc_to_csv](https://github.com/notnews/lacc_to_csv) — Los Angeles Closed-Caption Television News Archive Data to CSV\n- [notnews/fox_news_transcripts](https://github.com/notnews/fox_news_transcripts) — Fox News Transcripts 2003--2025\n- [notnews/cnn_transcripts](https://github.com/notnews/cnn_transcripts) — CNN Transcripts 2000--2025\n- [notnews/msnbc_transcripts](https://github.com/notnews/msnbc_transcripts) — MSNBC Transcripts: 2003--2022\n- [notnews/nbc_transcripts](https://github.com/notnews/nbc_transcripts) — NBC transcripts 2011--2014\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnotnews%2Farchive_news_cc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnotnews%2Farchive_news_cc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnotnews%2Farchive_news_cc/lists"}