Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/hitomarukonpaku/twspace-crawler
Script to monitor & download Twitter Spaces 24/7
https://github.com/hitomarukonpaku/twspace-crawler
discord ffmpeg nodejs twitter twitter-api twitter-spaces typescript
Last synced: 1 day ago
JSON representation
Script to monitor & download Twitter Spaces 24/7
- Host: GitHub
- URL: https://github.com/hitomarukonpaku/twspace-crawler
- Owner: HitomaruKonpaku
- Created: 2021-10-10T11:34:26.000Z (about 3 years ago)
- Default Branch: master
- Last Pushed: 2024-11-10T20:43:33.000Z (about 1 month ago)
- Last Synced: 2024-12-22T02:03:51.040Z (1 day ago)
- Topics: discord, ffmpeg, nodejs, twitter, twitter-api, twitter-spaces, typescript
- Language: TypeScript
- Homepage:
- Size: 809 KB
- Stars: 210
- Watchers: 12
- Forks: 41
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Funding: .github/FUNDING.yml
Awesome Lists containing this project
README
# twspace-crawler
[](https://www.npmjs.com/package/twspace-crawler)
[](https://www.npmjs.com/package/twspace-crawler)
> **Node.js script & command-line app to automatically monitor & download [Twitter Spaces](https://help.twitter.com/en/using-twitter/spaces).**
> **UPDATE 2023-07-01: SINCE TWITTER CHANGED THEIR APIS, READ [INSTALLATION](INSTALLATION.md#unofficial-api) FIRST**
## Contents
1. [Requirements](#requirements)
1. [Env](#env)
1. [Docker](#docker)
1. [Installation](#installation)
1. [Usage](#usage)
1. [Options](#options)
1. [Commands](#commands)
1. [Webhooks](#webhooks)
1. [Changelog](CHANGELOG.md)
## Requirements
- [Node.js](https://nodejs.org) (>=14)
- [FFMPEG](https://www.ffmpeg.org)
## Env
| Name | Desc
| - | -
| LOG_LEVEL | -
| TWITTER_AUTH_TOKEN | -
| TWITTER_CSRF_TOKEN | -
| SKIP_DOWNLOAD | -
| SKIP_DOWNLOAD_AUDIO | -
| SKIP_DOWNLOAD_CAPTION | -
## Docker
> Recommended
### Monitor users
- See [example](./example/)
- Download [docker-compose.yaml](./example/docker-compose.yaml) & [config.yaml](./example/config.yaml)
- Sign in Twitter
- `Open DevTools (F12)` > `Application` > `Storage` > `Cookies`
- Update docker compose environment
- `TWITTER_AUTH_TOKEN`: Value of `auth_token`
- `TWITTER_CSRF_TOKEN`: Value of `ct0`
- Run
```sh
docker compose up -d
```
### Download Space
```sh
docker run --rm \
-e TWITTER_AUTH_TOKEN= \
-e TWITTER_CSRF_TOKEN= \
-v ./logs:/app/logs \
-v ./download:/app/download \
ghcr.io/hitomarukonpaku/twspace-crawler \
--id 1mnGeRRRwkrJX
```
## Installation
[FULL INSTALLATION](INSTALLATION.md)
### Command-line installation
```bash
npm install --global twspace-crawler
```
### Module installation
```bash
npm install twspace-crawler
```
## Usage
### Monitor user(s) indefinitely, wait for live Space and download when Space ended
```
twspace-crawler --user nakiriayame,LaplusDarknesss
```
```
twspace-crawler --env ./.env --config ./config.json
```
### Monitor & download Space by id
```
twspace-crawler --id 1yoJMWvbybNKQ
```
### Download Space by playlist url
```
twspace-crawler --url https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/1Nq1QFkYTQ4v1X4BTV_aJ_pFeQhYyuYXY7ykz5xB7v5NvGwFMJMKwnRBmxyi9twF4BZ90ZKks5wdGKqESVsjLw...
```
### Download Space by playlist url with additional metadata (if Space url still available)
```
twspace-crawler --id 1yoJMWvbybNKQ --url https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/1Nq1QFkYTQ4v1X4BTV_aJ_pFeQhYyuYXY7ykz5xB7v5NvGwFMJMKwnRBmxyi9twF4BZ90ZKks5wdGKqESVsjLw...
```
## Options
### General options
```
-h, --help Display help
-d, --debug Show debug logs
```
### One-time download options
```
--id Monitor & download live Space with its id
-surl, --space-url Monitor & download live Space with its URL
--force Force download Space when using with --id
--url Download Space using playlist url
```
### User monitoring options
```
--env Path to .env file, default to current working
folder (See .env.example)
--config Path to config file (See config.example.json)
--user Monitor & download live Spaces from users
indefinitely, separate by comma (,)
```
### Additional options
```
--skip-download Do not download anything
--skip-download-audio Do not download audio
--skip-download-caption Do not download caption
--notification Show notification about new live Space
--force-open Force open Space in browser
```
## Commands
Use to manually process audio/captions
### List
- cc
```
download|d Download Space captions, with
- ENDPOINT: Chat endpoint
- TOKEN: Chat access token
extract|e [STARTED_AT] Extract Space captions
```
### Example
```
twspace-crawler cc d 1yoJMWneoZwKQ https://prod-chatman-ancillary-ap-northeast-1.pscp.tv 2Ozpcu2xxqb5wxMdkyodUCygOrbYMLv8rq...
```
```
twspace-crawler cc e /download/sample_cc.jsonl
```
```
twspace-crawler cc e /download/sample_cc.jsonl 1633612289669
```
## Webhooks
Currently support Discord Webooks
Check [config.example.yaml](config.example.yaml) ~~or [config.example.json](config.example.json)~~ for more detail