{"id":20517661,"url":"https://github.com/francisbarton/yearprogress","last_synced_at":"2025-08-19T00:11:17.192Z","repository":{"id":146781121,"uuid":"322719882","full_name":"francisbarton/yearprogress","owner":"francisbarton","description":"Retrieving, tidying and visualising tweets from @year_progress 2019-2020","archived":false,"fork":false,"pushed_at":"2022-03-08T09:05:01.000Z","size":574,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-02T02:55:57.234Z","etag":null,"topics":["ggplot2","httr","purrr","twitter-api"],"latest_commit_sha":null,"homepage":"","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/francisbarton.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":".github/SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"francisbarton"}},"created_at":"2020-12-18T22:38:58.000Z","updated_at":"2022-03-08T09:05:05.000Z","dependencies_parsed_at":null,"dependency_job_id":"646aece3-0998-460d-b4e8-b71416b5b58a","html_url":"https://github.com/francisbarton/yearprogress","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/francisbarton%2Fyearprogress","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/francisbarton%2Fyearprogress/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/francisbarton%2Fyearprogress/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/francisbarton%2Fyearprogress/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/francisbarton","download_url":"https://codeload.github.com/francisbarton/yearprogress/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242123176,"owners_count":20075344,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ggplot2","httr","purrr","twitter-api"],"created_at":"2024-11-15T21:36:39.271Z","updated_at":"2025-03-05T23:41:17.172Z","avatar_url":"https://github.com/francisbarton.png","language":"R","funding_links":["https://github.com/sponsors/francisbarton"],"categories":[],"sub_categories":[],"readme":"---\ntitle: \"Looking at tweets by @year_progress 2019-2020\"\ndate: '`r format(Sys.Date(), \"%B %d, %Y\")`'\nstub: \"year-progress-tweets-viz\"\noutput:\n  github_document:\n    html_preview: true\n    toc: false\n    fig_width: 9\n    fig_height: 5\n    md_extensions: +emoji+bracketed_spans+inline_notes\nkeywords: [lubridate, ggplot, twitter, api, dataviz]\nalways_allow_html: true\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r, setup, include = FALSE}\n\nragg_png = function(..., res = 192) {\n  ragg::agg_png(..., res = res, units = \"in\")\n}\n\n\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  dev = \"ragg_png\",\n  # dev.args = list(ragg_png = list(res = 192)),\n  fig.ext = \"png\",\n  fig.showtext = TRUE,\n  fig.path = \"man/figures/README-\",\n  out.width = \"100%\"\n)\n\nlibrary(dplyr)\nlibrary(extrafont)\nlibrary(ggplot2)\nlibrary(ggthemr)\nlibrary(lubridate)\nlibrary(purrr)\nlibrary(stringr)\nlibrary(tweetrmd)\nlibrary(yearprogress)\n\n```\n\n\u003c!-- badges: start --\u003e\n\u003c!-- badges: end --\u003e\n\n:::{#flow}\n\nI was randomly interested in whether there might be patterns in the amount of interaction with tweets from the **@year_progress** bot during 2020 :scream:,\nand what the shape of the year might look like compared to 2019.\n\n\nThis gave me a chance to review the current landscape of `R` clients for the twitter API :disappointed:.\nAfter doing this review, I decided it would be easier and more satisfying for me to script my own access to [v2 of the API][tw-api].\n\n[tw-api]: https://developer.twitter.com/en/docs/twitter-api/early-access\n\n\nThis meant more practice with `httr`, error handling, working with lists with the frankly brilliant [`purrr` toolkit][purrr], and with writing functions.\nHappy as a :pig: in :poop:, _tbqhwy_.\n\n\nIt's also chance for me to try various other things I haven't really used before:\n\n* Creating a GitHub README as an `.Rmd` file (this very file here),\n  instead of my usual `.md`, so I might incorporate some output charts\n* Using a couple of [Pandoc's Markdown extensions][pdmd] ^[1][1]^\n* GitHub Actions (set up using the `usethis` helper function)\n* Making _a whole actual package_ :package: just for a little bit of fun like\n  this, as opposed to as some bigger project\n* Learning a bit more about testing with `testthat`\n* Yet more practice at package architecture and requirements,\n  and appeasing R CMD check\n* Doing some hopefully nice [dataviz]{.sparkle} :bar_chart: with `ggplot2`, which   I still don't feel like I have really used _well_ yet.\n  My charts usually look way too crappy for my liking.\n\n\n[1]: # \"We've got md_extensions: +emoji+bracketed_spans+inline_notes going on in the yaml here\"\n[purrr]: https://purrr.tidyverse.org/\n[pdmd]: https://pandoc.org/MANUAL.html#pandocs-markdown\n\n\nIf you suspect that this whole idea was essentially driven by pre-Christmas/end-of-term task avoidance, angst and procrastination,\nyou would be correct.\n\n\n```{r, get-dtf, eval = FALSE}\n# Set variables and run overall query -------------------------------------\n\n\n# Obtained originally by:\n# user_id \u003c- get_user_id(username = \"year_progress\")\n# It isn't going to change, so once obtained just hardcode it:\nyear_progress_id \u003c- \"3233484298\"\n\n# This doesn't work with the Twitter API\n# https://github.com/tidyverse/lubridate/issues/941\n# start_2019 \u003c- lubridate::make_datetime(2019) %\u003e% \n#   lubridate::format_ISO8601(usetz = TRUE)\n# so instead:\n\nstart_2019 \u003c- \"2019-01-01T00:00:02Z\"\n\n# see `R/get_user_twetes.R` for how this works\ndtf \u003c- get_user_twetes(user_id = year_progress_id, start = start_2019) %\u003e%\n  purrr::compact() %\u003e%\n  unpack_list()       # \u003c- in `R/helpers.R`\n```\n\n\n\n```{r, load-dtf, include = FALSE}\n# myrmidon::save_it(dtf)\ndtf \u003c- readRDS(here::here(\"rds_data\", \"dtf.Rds\"))\n```\n\nI realised the other day that it would be fine to call a dataframe `dtf` instead of boring old `df` :smirk:...\n\n```{r, dtf-tweet, echo = FALSE, cache = TRUE}\ntweetrmd::include_tweet(\"https://twitter.com/ludictech/status/1338815874770329602?s=20\")\n```\n\n```{r, glimpse-dtf}\nglimpse(dtf)\n\n```\n\n\nLet's lick this dataframe into shape :icecream: so it's ready for visualising.\n\n\n```{r, prepare-dtf}\n\ndtf2 \u003c- dtf %\u003e% \n  # filter out any \"normal\" tweets!\n  filter(str_detect(text, \"^(\\u2591|\\u2593)+\")) %\u003e% \n  mutate(pcnt = as.numeric(str_extract(text, \"[0-9]+\"))) %\u003e% \n  mutate(date_tweeted = case_when(\n    # 100% tweets get tweeted just _after_ midnight on 1st Jan: need correcting\n    # Be better here to check *if* date == January 1 but OK for now\n    pcnt == 100 ~ as_date(created_at) - period(1, \"days\"),\n    TRUE ~ as_date(created_at))) %\u003e% \n  mutate(year = year(date_tweeted)) %\u003e% \n  mutate(yday = yday(date_tweeted)) %\u003e% \n  mutate(wkdy = weekdays(as_date(date_tweeted))) %\u003e% \n  mutate(across(c(year, wkdy), ~ as.factor(.)))\n\n```\n\n```{r, ggthemr-set, echo = FALSE}\nggthemr::ggthemr(palette = \"solarized\", layout = \"clean\", type = \"outer\")\n# ggthemr::ggthemr_reset()\n\ntheme_update(\n  text = element_text(family = \"IBM Plex Sans\", colour = \"#100810\")\n)\n\n```\n\nLet's look at how retweets vary through the year:\n\n```{r, plot-retweets, warning = FALSE, message = FALSE, cache = TRUE}\ndtf2 %\u003e% \n  filter(year == 2019) %\u003e% \n  slice_max(n = 4, order_by = retweet_count) %\u003e% \n  mutate(label = str_glue(\"{format(date_tweeted, '%b %d')}: {pcnt}%\")) %\u003e% \n  left_join(dtf2, .) %\u003e% \n  ggplot(aes(yday, retweet_count)) +\n  geom_line(aes(colour = year), size = 0.6, alpha = 1,) +\n  geom_point(aes(colour = year), size = 1) +\n  geom_label(\n    aes(label = label),\n    vjust = \"outward\",\n    hjust = \"inward\",\n    nudge_y = 0.1,\n    fill = \"aquamarine3\",\n    colour = \"white\",\n    size = 3,\n    fontface = \"bold\") +\n  scale_colour_brewer(\"Year\", palette = \"Set2\") +\n  scale_y_log10() +\n  labs(\n    x = \"Day of the year\",\n    y = \"Number of retweets (log scale)\",\n    title = \"RTs of @year_progress tweets, 2019 vs. 2020\"\n  )\n  \n\n```\n\nCertain tweets are vastly more popular than the rest, with four tweets standing out as peaks - even when a log scale is employed on the y axis.\nThe next tranche of popularity is reserved for the multiples of 10%, apart from 50% which is already one of the top four retweeted.\n\nThere doesn't seem to be a particularly different trend affecting the popularity of tweets or the pattern of retweets as 2020 has progressed, relative to 2019. 2020 numbers are generally a little higher, which is most likely due to the account having acquired more followers over time.\n\nLet's see if the numbers of likes for each tweet show a similar pattern\n(we'd generally expect them to):\n\n```{r, plot-likes, warning = FALSE, message = FALSE}\ndtf3 \u003c- dtf2 %\u003e% \n  filter(year == 2019) %\u003e% \n  slice_max(n = 4, order_by = like_count) %\u003e% \n  mutate(label_2019 = str_glue(\"{format(date_tweeted, '%b %d')}: {pcnt}%\")) %\u003e% \n  left_join(dtf2, .)\n\ndtf3 %\u003e% \n  filter(year == 2020) %\u003e% \n  slice_max(n = 4, order_by = like_count) %\u003e% \n  mutate(label_2020 = str_glue(\"{format(date_tweeted, '%b %d')}: {pcnt}%\")) %\u003e% \n  left_join(dtf3, .) %\u003e% \n  mutate(like_count = like_count/1000) %\u003e% \n  ggplot(aes(yday, like_count)) +\n  geom_line(aes(colour = year), size = 0.6, alpha = 1) +\n  geom_point(aes(colour = year), size = 1) +\n  geom_label(\n    aes(label = label_2019),\n    vjust = \"outward\",\n    hjust = \"inward\",\n    nudge_y = 0.1,\n    fill = \"aquamarine3\",\n    colour = \"white\",\n    size = 3,\n    fontface = \"bold\") +\n  geom_label(\n    aes(label = label_2020),\n    vjust = \"outward\",\n    hjust = \"inward\",\n    nudge_y = 0.1,\n    fill = \"chocolate2\",\n    colour = \"white\",\n    size = 3,\n    fontface = \"bold\") +\n  scale_colour_brewer(\"Year\", palette = \"Set2\") +\n  scale_y_log10() +\n  labs(\n    x = \"Day of the year\",\n    y = \"Number of likes ('000s) (log scale)\",\n    title = \"\\\"Likes\\\" of @year_progress tweets, 2019 vs. 2020\"\n  )\n  \n\n```\n\nAnd quote tweets:\n\n\n```{r, plot-quotes, warning = FALSE, message = FALSE}\ndtf3 \u003c- dtf2 %\u003e% \n  filter(year == 2019) %\u003e% \n  slice_max(n = 4, order_by = quote_count) %\u003e% \n  mutate(label_2019 = str_glue(\"{format(date_tweeted, '%b %d')}: {pcnt}%\")) %\u003e% \n  left_join(dtf2, .)\n\ndtf3 %\u003e% \n  filter(year == 2020) %\u003e% \n  slice_max(n = 4, order_by = quote_count) %\u003e% \n  mutate(label_2020 = str_glue(\"{format(date_tweeted, '%b %d')}: {pcnt}%\")) %\u003e% \n  left_join(dtf3, .) %\u003e% \n  # mutate(quote_count = quote_count/1000) %\u003e% \n  ggplot(aes(yday, quote_count)) +\n  geom_line(aes(colour = year), size = 0.6, alpha = 1) +\n  geom_point(aes(colour = year), size = 1) +\n  geom_label(\n    aes(label = label_2019),\n    vjust = \"outward\",\n    hjust = \"inward\",\n    nudge_y = 0.1,\n    fill = \"aquamarine3\",\n    colour = \"white\",\n    size = 3,\n    fontface = \"bold\") +\n  geom_label(\n    aes(label = label_2020),\n    vjust = \"outward\",\n    hjust = \"inward\",\n    nudge_y = 0.1,\n    fill = \"chocolate2\",\n    colour = \"white\",\n    size = 3,\n    fontface = \"bold\") +\n  scale_colour_brewer(\"Year\", palette = \"Set2\") +\n  scale_y_log10() +\n  labs(\n    x = \"Day of the year\",\n    y = \"Number of quote-tweets (log scale)\",\n    title = \"Quotes of @year_progress tweets, 2019 vs. 2020\"\n  )\n  \n\n```\n\nNow, am I imagining it, or is there a much higher proportional difference between 2019 and 2020 when it comes to the QTs, than was visible with the RTs and the Likes?\nFrom about day 80? There's some quite big gaps in there between the two lines, particularly if you look at the period between day 80 and day 200.\n2020 never drops down below 100 QTs for any tweet like 2019 did.\n\nThe gap is a bit less noticeable in the 70-percents, around day 270-280, when 2019's figures start to pick up as the end of the year approaches.\nAgain, some of this may just be a factor of more people following the account, and RTs and QTs generate more follows, and so it goes round.\n\nI'll be interested to see what happens to 2020's final few data points in the closing days of December.\n:::\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrancisbarton%2Fyearprogress","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffrancisbarton%2Fyearprogress","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrancisbarton%2Fyearprogress/lists"}