https://github.com/centre-for-humanities-computing/crowdtangle-api-scraper
https://github.com/centre-for-humanities-computing/crowdtangle-api-scraper
Last synced: about 23 hours ago
JSON representation
- Host: GitHub
- URL: https://github.com/centre-for-humanities-computing/crowdtangle-api-scraper
- Owner: centre-for-humanities-computing
- Created: 2021-03-03T21:20:50.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2022-12-06T15:02:42.000Z (over 3 years ago)
- Last Synced: 2025-09-09T23:59:49.054Z (9 months ago)
- Language: JavaScript
- Size: 20.5 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# CrowdTangle Api Scraper
A tool for extracting post objects from the
[CrowdTangle](https://help.crowdtangle.com/en/articles/1189612-crowdtangle-api) api.
## Installation
- Install [node.js](https://nodejs.org/en/download/) version 14.11 or higher
- Clone this repository
- Navigate to the root of the repository and run
```
$ npm install
```
## Usage
- Navigate to the root of the repository and run to see the help for the CLI
```
$ node cli -h
```
### CLI options
- **`-k, --api-credentials `** [required] - The CrowdTangle API key or a file containing KEY
- **`-q, --query `** [required] - The term to search for
- **`-d, --destination `** [required] - The directory where the result should be stored
- **`-p, --filename-prefix `** [optional, default="posts"] - The name of the result file
- **`-f, --from `** [required] - The date to fetch data from in the format "yyyy-mm-dd"
- **`-t, --to `** [required] - The date to fetch data to (included) in the format "yyyy-mm-dd". To fetch a single day this should be the same as "from"
- **`-l, --language `** [optional] - The iso language code to search for. Eg. "da"
- **`-o, --platforms `** [optional] - The platforms to search. Default: "facebook". Possible values are [facebook, instagram, reddit] multiple platforms should be separated by comma
- **`-c, --csv`** [optional] - Generate a csv file along with the json file
- **`-z, --development-mode`** [optional] - Should logging data be printed to the stdout
### Full Example
```
$ node cli -k "API-KEY" -q "election" -d "/data/crowdtangle", -f "2019-01-01" -t "2020-01-01" -l "da" -o "facebook" -c
```