Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/hitomarukonpaku/twspace-crawler

Script to monitor & download Twitter Spaces 24/7
https://github.com/hitomarukonpaku/twspace-crawler

discord ffmpeg nodejs twitter twitter-api twitter-spaces typescript

Last synced: 22 days ago
JSON representation

Script to monitor & download Twitter Spaces 24/7

Host: GitHub
URL: https://github.com/hitomarukonpaku/twspace-crawler
Owner: HitomaruKonpaku
Created: 2021-10-10T11:34:26.000Z (about 3 years ago)
Default Branch: master
Last Pushed: 2024-10-13T22:54:45.000Z (23 days ago)
Last Synced: 2024-10-14T04:03:38.860Z (22 days ago)
Topics: discord, ffmpeg, nodejs, twitter, twitter-api, twitter-spaces, typescript
Language: TypeScript
Homepage:
Size: 719 KB
Stars: 196
Watchers: 12
Forks: 41
Open Issues: 19
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Funding: .github/FUNDING.yml

Awesome Lists containing this project

README

        # twspace-crawler

[![npm](https://img.shields.io/npm/v/twspace-crawler)](https://www.npmjs.com/package/twspace-crawler)

[![npm](https://img.shields.io/npm/dt/twspace-crawler)](https://www.npmjs.com/package/twspace-crawler)

> **Node.js script & command-line app to automatically monitor & download [Twitter Spaces](https://help.twitter.com/en/using-twitter/spaces).**

> **UPDATE 2023-07-01: SINCE TWITTER CHANGED THEIR APIS, READ [INSTALLATION](INSTALLATION.md#unofficial-api) FIRST**

## Contents

1. [Requirements](#requirements)

1. [Docker](#docker)

1. [Installation](#installation)

1. [Usage](#usage)

1. [Options](#options)

1. [Commands](#commands)

1. [Webhooks](#webhooks)

1. [Changelog](CHANGELOG.md)

## Requirements

- [Node.js](https://nodejs.org) (>=14)

- [FFMPEG](https://www.ffmpeg.org)

## Docker

### Monitor users

- See [example](./example/)

- Download [docker-compose.yaml](./example/docker-compose.yaml) & [config.yaml](./example/config.yaml)

- Sign in Twitter

- `Open DevTools (F12)` > `Application` > `Storage` > `Cookies`

- Update docker compose environment

  - `TWITTER_AUTH_TOKEN`: Value of `auth_token`

  - `TWITTER_CSRF_TOKEN`: Value of `ct0`

- Run

    ```sh

    docker compose up -d

    ```

### Download Space

```sh

docker run --rm \

  -e TWITTER_AUTH_TOKEN= \

  -e TWITTER_CSRF_TOKEN= \

  -v ./logs:/app/logs \

  -v ./download:/app/download \

  ghcr.io/hitomarukonpaku/twspace-crawler \

  --id 1mnGeRRRwkrJX

```

## Installation

[FULL INSTALLATION](INSTALLATION.md)

### Command-line installation

```bash

npm install --global twspace-crawler

```

### Module installation

```bash

npm install twspace-crawler

```

## Usage

### Monitor user(s) indefinitely, wait for live Space and download when Space ended

```

twspace-crawler --user nakiriayame,LaplusDarknesss

```

```

twspace-crawler --env ./.env --config ./config.json

```

### Monitor & download Space by id

```

twspace-crawler --id 1yoJMWvbybNKQ

```

### Download Space by playlist url

```

twspace-crawler --url https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/1Nq1QFkYTQ4v1X4BTV_aJ_pFeQhYyuYXY7ykz5xB7v5NvGwFMJMKwnRBmxyi9twF4BZ90ZKks5wdGKqESVsjLw...

```

### Download Space by playlist url with additional metadata (if Space url still available)

```

twspace-crawler --id 1yoJMWvbybNKQ --url https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/1Nq1QFkYTQ4v1X4BTV_aJ_pFeQhYyuYXY7ykz5xB7v5NvGwFMJMKwnRBmxyi9twF4BZ90ZKks5wdGKqESVsjLw...

```

## Options

### General options

```

  -h, --help                        Display help

  -d, --debug                       Show debug logs

```

### One-time download options

```

  --id                    Monitor & download live Space with its id

  -surl, --space-url     Monitor & download live Space with its URL

  --force                           Force download Space when using with --id

  --url               Download Space using playlist url

```

### User monitoring options

```

  --env                   Path to .env file, default to current working

                                    folder (See .env.example)

  --config             Path to config file (See config.example.json)

  --user                      Monitor & download live Spaces from users

                                    indefinitely, separate by comma (,)

```

### Additional options

```

  --notification                    Show notification about new live Space

  --force-open                      Force open Space in browser

```

## Commands

Use to manually process audio/captions

### List

- cc

```

  download|d     Download Space captions, with

                                            - ENDPOINT: Chat endpoint

                                            - TOKEN: Chat access token

  extract|e  [STARTED_AT]             Extract Space captions

```

### Example

```

twspace-crawler cc d 1yoJMWneoZwKQ https://prod-chatman-ancillary-ap-northeast-1.pscp.tv 2Ozpcu2xxqb5wxMdkyodUCygOrbYMLv8rq...

```

```

twspace-crawler cc e /download/sample_cc.jsonl

```

```

twspace-crawler cc e /download/sample_cc.jsonl 1633612289669

```

## Webhooks

Currently support Discord Webooks

Check [config.example.yaml](config.example.yaml) ~~or [config.example.json](config.example.json)~~ for more detail