https://github.com/MaxBittker/nyt-first-said

Tweets when words are published for the first time in the NYT
https://github.com/MaxBittker/nyt-first-said

civic-tech journalism linguistics newsroom politics python scraper twitter

Last synced: 3 months ago
JSON representation

Tweets when words are published for the first time in the NYT

• Host: GitHub
• URL: https://github.com/MaxBittker/nyt-first-said
• Owner: MaxBittker
• License: other
• Fork: true (ecprice/newsdiffs)
• Created: 2017-03-05T00:14:26.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2023-11-30T03:02:37.000Z (7 months ago)
• Last Synced: 2024-01-30T07:41:26.773Z (5 months ago)
• Topics: civic-tech, journalism, linguistics, newsroom, politics, python, scraper, twitter
• Language: Python
• Homepage: https://twitter.com/nyt_first_said
• Size: 2.46 MB
• Stars: 220
• Watchers: 7
• Forks: 19
• Open Issues: 5
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Lists

• awesome-stars - MaxBittker/nyt-first-said - Tweets when words are published for the first time in the NYT (Python)

README

        
# [@NYT_first_said](https://twitter.com/NYT_first_said)

#### [Info website](https://maxbittker.github.io/nyt-first-said/)

A twitter bot to track when the New York Times publishes a word for the first time in history.

running at: [@NYT_first_said](https://twitter.com/NYT_first_said)

It also powers a sibling bot [@NYT_said_where](https://twitter.com/NYT_said_where), which replies to each tweet with a few words of source-text context, and a link to the article.

The code takes some steps to throw away un-interesting words like proper nouns and urls, but still picks up a lot of typos and nonsense, so the sanitization is an ongoing process.

Some points of inspiration are Allison Parrish's @everyword bot, and the [NewsDiffs](http://newsdiffs.org/about/) editorial change archiving software.

## Basic architecture

NYT-first-said is essentially a single script. It's running once an hour as a cron job on a small VPS.

`nyt.py` is a beautifulsoup parser adapted from the [newsdiffs sourcecode](https://github.com/ecprice/newsdiffs). 

`redis` holds a list of scraped URLs and seen words (to reduce load on NYT API). It also holds a count of words tweeted recently to avoid blasting out too many tweets in a short period of time.

`api_check.py` uses the [NYT article_search API](https://developer.nytimes.com/) to check through all digitized NYT history to be confident this is really the first occurrence of a word. It returns weird 500s for some words. If you know why let me know.

`simple_scrape.py` Checks for new article urls, retrieves the article text using `nyt.py`, splits them into words, and then determines whether each word is fit to tweet using (in this order) some heuristics to discard unwanted types of words, uniqueness in our local redis instance, and finally uniqueness against the article_search api. If all of these checks pass, it tweets the word, and replies with the context and link.

Also check out [@nyt-finally-said](https://github.com/uniphil/nyt-finally-said), a cool sibling bot that cross-references these words with the google n-gram dataset!