{"id":18906728,"url":"https://github.com/oduwsdl/followercounthistory","last_synced_at":"2025-10-06T22:12:39.198Z","repository":{"id":38328745,"uuid":"117931879","full_name":"oduwsdl/FollowerCountHistory","owner":"oduwsdl","description":"Crawler that grabs Twitter follower counts across time via internet archives given account user name","archived":false,"fork":false,"pushed_at":"2022-08-20T18:37:11.000Z","size":13540,"stargazers_count":32,"open_issues_count":2,"forks_count":4,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-28T16:21:24.324Z","etag":null,"topics":["archives","twitter","web"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oduwsdl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-01-18T04:23:00.000Z","updated_at":"2025-03-12T08:43:27.000Z","dependencies_parsed_at":"2022-08-09T03:00:48.942Z","dependency_job_id":null,"html_url":"https://github.com/oduwsdl/FollowerCountHistory","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oduwsdl%2FFollowerCountHistory","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oduwsdl%2FFollowerCountHistory/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oduwsdl%2FFollowerCountHistory/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oduwsdl%2FFollowerCountHistory/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oduwsdl","download_url":"https://codeload.github.com/oduwsdl/FollowerCountHistory/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249006450,"owners_count":21197279,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archives","twitter","web"],"created_at":"2024-11-08T09:18:31.862Z","updated_at":"2025-10-06T22:12:39.071Z","avatar_url":"https://github.com/oduwsdl.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Twitter Follower Count History via Web Archives     [![Build Status](https://travis-ci.org/oduwsdl/FollowerCountHistory.svg?branch=development)](https://travis-ci.org/oduwsdl/FollowerCountHistory)\nFollower Count History is a Python module that collects Twitter follower count from the web archives using [MemGator](https://github.com/oduwsdl/MemGator) for a given Twitter handle. The module parses the follower count by identifying various CSS Selectors that match the follower count element on the historical Twitter pages for almost every major overhaul their page layout has gone through. The program collects all of the memento data points by default.\n\n\u003cb\u003eSince PyPi version 1.0.14, the FCH package is now platform independent. Pre 1.0.14 versions of FCH on PyPi are only compatible with Linux systems.\u003c/b\u003e\n\n[1] Mohammed Nauman Siddique. 2020. Historical Twitter Follower Count Via Web Archives. (August 2020). Retrieved August 05, 2020 from https://ws-dl.blogspot.com/2020/08/2020-08-05-historical-twitter-follower.html\n\n[2] Miranda Smith. 2018. Twitter Follower Count History via the Internet Archive. (March 2018). Retrieved July 25, 2020 from https://ws-dl.blogspot.com/2018/03/2018-03-14-twitter-follower-count.html\n\n## Installation and Usage\n### Dependencies\n* Python 3\n* bs4\n* warcio\n* requests\n* pytest (used for testing purposes)\n* pytz (used for testing purposes)\n* R* (Optional: to create graph)\n\n### Usage\n```shell\n$ git clone https://github.com/oduwsdl/FollowerCountHistory.git\n$ cd FollowerCountHistory\n$ pip install -r requirements.txt\n$  ./fch/__main__.py [-h] [--st] [--et] [--freq] [--out] \u003cTwitter handle/ Twitter URL\u003e\n```\n#### Install from pypi\n```shell\n$ pip install fch\n$  fch [-h] [--st] [--et] [--freq] [--out] \u003cTwitter handle/ Twitter URL\u003e\n```\n\nTo just create the graph from a csv file\n```shell\n$ Rscript twitterFollowerCount.R \u003cCSV file path\u003e\n```\n\n### Docker\n\nWe have published a docker image at [oduwsdl/fch](https://hub.docker.com/r/oduwsdl/fch) with the tag \u003cb\u003e2.0\u003c/b\u003e, which can be used to run this tool as following:\n\n```\n$  docker container run --rm -it   -v \u003cOutput Directory\u003e:/app  -u $(id -u):$(id -g)  oduwsdl/fch:2.0 [options] \u003cTwitter Handle\u003e\n```\n\nExample of output being mapped to the current directory\n\n```\n$  docker container run --rm -it -v $PWD:/app -u $(id -u):$(id -g) oduwsdl/fch:2.0 --out  --st=20200101000000 --et=20200331000000 --freq=2592000  joebiden\n```\n\nExample of docker command for generating follower graph\n\n```\n$ docker container run --rm -it -v $PWD:/app -u $(id -u):$(id -g) --entrypoint /bin/bash oduwsdl/fch:2.0\nI have no name!@736a209b64d6:/app$ ./fch/__main__.py --freq=2592000 joebiden| Rscript twitterFollowerCount.R\n```\n### Options\n```\nFollower Count History (fch)\n\npositional arguments:\n  thandle     Enter a Twitter handle/ URL\n\noptional arguments:\n  -h, --help     show this help message and exit\n  --st           Memento start datetime (YYYYMMDDHHMMSS)\n  --et           Memento end datetime (YYYYMMDDHHMMSS)\n  --freq         Sampling frequency of mementos (in seconds)\n  -f             Output file path (Supported Extensions: JSON and CSV)\n  -v, --version  Report the version of fch\n```\n* --st: Default is set to Twitter birth date (2006-03-21 12:00:00). It accepts the memento datetime in [RFC 8601](https://www.iso.org/iso-8601-date-and-time-format.html) fourteen digit variation.\n* --et: Default is set to the current datetime. It accepts the memento datetime in [RFC 8601](https://www.iso.org/iso-8601-date-and-time-format.html) fourteen digit variation.\n* --freq: Default is set to download all the mementos\n* -f: Accepts JSON and CSV file paths for output. If no value is provided, output is returned to stdout in CSV format.\n\n## Output\n\nThe program can generate output in JSON and CSV format. The -f option directs the output of CSV or JSON files to the supplied file path. By default, the module returns the outut in CSV format to the stdout.  \n\n### Output Fields\n\nField| Description\n---------|------------\nMementoTimestamp |         memento datetime in [RFC 8601](https://www.iso.org/iso-8601-date-and-time-format.html) fourteen digit variation\nURI-M            |         link to the memento\nFollowerCount    |         follower count from the URI-M\nAbsGrowth        |         follower count increase/decrease w.r.t. the first memento\nRelGrowth\t | \t   follower Count increase/decrease w.r.t. the previous memento\nAbsPerGrowth \t |\t   pecentage increase/decrease in follower count w.r.t. the first memento\nRelPerGrowth\t | \t   pecentage increase/decrease in follower count w.r.t. the previous memento\nAbsFolRate \t |\t   daily Twitter follower growth rate w.r.t. the first memento\nRelFolRate\t | \t   daily Twitter follower growth rate w.r.t. the previous memento\n\n### Sample Outputs\nJSON Output\n```json\n[{\n\t\"MementoDatetime\": \"20200101001959\",\n\t\"URIM\": \"https://web.archive.org/web/20200101001959/https://twitter.com/JoeBiden\",\n\t\"FollowerCount\": 4048208\n}, {\n\t\"MementoDatetime\": \"20200131120028\",\n\t\"URIM\": \"https://web.archive.org/web/20200131120028/https://twitter.com/joebiden\",\n\t\"FollowerCount\": 4142510\n}, {\n\t\"MementoDatetime\": \"20200301001210\",\n\t\"URIM\": \"https://web.archive.org/web/20200301001210/https://twitter.com/JoeBiden/\",\n\t\"FollowerCount\": 4202148\n}]\n```\n\nCSV Output\n```csv\nMementoDatetime,URIM,FollowerCount,AbsGrowth,RelGrowth,AbsPerGrowth,RelPerGrowth,AbsFolRate,RelFolRate\n20200101001959,https://web.archive.org/web/20200101001959/https://twitter.com/JoeBiden,4048208,0,0,0,0,0,0\n20200131120028,https://web.archive.org/web/20200131120028/https://twitter.com/joebiden,4142510,94302,94302,2.33,2.33,0.0358,0.0358\n20200301001210,https://web.archive.org/web/20200301001210/https://twitter.com/JoeBiden/,4202148,153940,59638,3.8,1.44,0.0297,0.02339\n```\n### Output to stdout\n\n```shell\n$ fch --st=20200101000000 --et=20200331000000  --freq=2592000 joebiden\n```\n\n### Output to files\n\n**Command to return output to the file path**\n\n```shell\n$ fch --st=20200101000000 --et=20200331000000  --freq=2592000 -f=output/joebiden.csv joebiden\n$ fch --st=20200101000000 --et=20200331000000  --freq=2592000 -f=output/joebiden.json joebiden\n```\n\n**Command to create graphs for each handle**\n\n```shell\n$ Rscript twitterFollowerCount.R \u003cfile path\u003e\n```\n\n* List of Graphs for each Twitter handle:\n\nFile Name| Description\n---------|------------\n`\u003cTwitterhandle\u003e`-follower-count.jpg|                shows Twitter follower growth over time\n`\u003cTwitterhandle\u003e`-follower-growth-relative.jpg|      shows Twitter follower growth w.r.t. previous memento\n`\u003cTwitterhandle\u003e`-follower-growth.jpg|               shows absolute number and pecentage Twitter follower growth w.r.t. to first memento\n`\u003cTwitterhandle\u003e`-follower-perc-growth-relative.jpg| shows Twitter follower growth over time w.r.t. previous memento in percentage\n`\u003cTwitterhandle\u003e`-follower-rate-relative.jpg|        shows new followers added per day w.r.t. previous memento\n`\u003cTwitterhandle\u003e`-follower-rate.jpg|                 shows new followers added per day w.r.t. first memento\n\n## Examples\n\n* Command to find Twitter follower count for a Twitter handle from all the mementos since the account creation up until today\n  * Output to stdout as CSV\n  ```shell\n  $  fch joebiden\n  ```\n  * Output as CSV file\n  ```shell\n  $  fch -f=joebiden.csv joebiden\n  ```\n* Command to find Twitter follower count for a Twitter handle with a monthly sampling of the the mementos since the account creation up until today\n  ```\n  Frequency = 3600*24*30\n  Frequency = 2592000\n  ```\n* Output to stdout as CSV\n  ```shell\n  $  fch --freq=2592000 joebiden\n  ```\n  * Output as CSV file\n  ```shell\n  $  fch -f=joebiden.csv --freq=2592000 joebiden\n  ```\n* Command to find Twitter follower count for a Twitter handle with a monthly sampling of the the mementos within a specified start and end timestamp\n  * Output to stdout as CSV\n  ```shell\n  $  fch --st=20200101000000 --et=20200331000000 --freq=2592000 joebiden\n  ```\n  * Output as CSV file\n  ```shell\n  $  fch -f=joebiden.csv --st=20200101000000 --et=20200331000000 --freq=2592000 joebiden\n  ```\n  ## Fch group script\n * To run the 'fchgrp.sh' script, pass a txt or csv file as a parameter\n   * Output defaults to 'output/' folder\n   * Can use the same parameters as fch to generate the same results\n  ```shell\n  $  ./fchgrp.sh user.txt\n  ```\n * Setting output to 'newOutput/' for csv files using 'fchgrp.sh'\n  ```shell\n  $  ./fchgrp.sh user.txt -f=newOutput/\n  ```\n ## Generate D3 Stub\n * GenerateD3Stub is a python script that uses generateGraph_base.js and index.html and csv data generated by fch.\n  \n * Usage: GenerateD3Stub functions by passing n number of csv filepaths as parameters. The file generated afterwards 'generatedGraphStub.html' will have the embeddable code for use.\n  ```shell\n  $  cd html_generator; python generateD3stub.py csv1path csv2path ... csvnpath\n  ```\n * By default, the GenerateD3Stub will print the data passed to it in the console, so the user can verify the correct data is being passed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foduwsdl%2Ffollowercounthistory","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foduwsdl%2Ffollowercounthistory","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foduwsdl%2Ffollowercounthistory/lists"}