https://github.com/xyk2/cloud-transcoder

Transcoding long (>1 hour) videos quickly and cost-effectively on GCP to adaptive HLS and MP4 mezzanine files. Up to 40x $ savings.
https://github.com/xyk2/cloud-transcoder

Last synced: about 13 hours ago
JSON representation

Transcoding long (>1 hour) videos quickly and cost-effectively on GCP to adaptive HLS and MP4 mezzanine files. Up to 40x $ savings.

• Host: GitHub
• URL: https://github.com/xyk2/cloud-transcoder
• Owner: xyk2
• Created: 2018-12-14T20:08:09.000Z (almost 6 years ago)
• Default Branch: master
• Last Pushed: 2018-12-15T10:35:41.000Z (almost 6 years ago)
• Last Synced: 2024-04-21T06:33:31.224Z (7 months ago)
• Language: JavaScript
• Homepage:
• Size: 3.54 MB
• Stars: 12
• Watchers: 3
• Forks: 3
• Open Issues: 0
• Metadata Files:
  • Readme: readme.md

Awesome Lists containing this project

• awesome-video - xyk2/cloud-transcoder - Transcoding long (>1 hour) videos quickly and cost-effectively on GCP to adaptive HLS and MP4 mezzanine files. Up to 40x $ savings. - xyk2/cloud-transcoder (Encoding / Talks Presentations Podcasts)

README

        
# Cloud transcoding adaptive bitrate MP4 and HLS

Transcodes a source video file (MP4, AVI, MPG, M4A, WMV, MOV) to adaptive bitrate MP4 and HLS to be stored in Google Cloud Storage. Cheaper than 3rd party transcoding services by more than 40-50x when running on preemptible instances (compared to AWS Elastic Transcoder, Zencoder, Bitmovin, Cloudinary), and saves time/bandwidth massively by transcoding in the same region as where the files are stored.

### Configuring boot disk template for GCE

```bash

sudo add-apt-repository ppa:jonathonf/ffmpeg-3 # Enter to confirm

sudo apt-get update

sudo apt-get -y upgrade

sudo apt install -y ffmpeg libav-tools x264 x265

sudo curl -sL https://deb.nodesource.com/setup_6.x -o nodesource_setup.sh

sudo bash nodesource_setup.sh

sudo apt-get install nodejs -y

sudo npm install pm2 -g

# Route 8080 to port 80 to allow node to access port 80 without root privileges

sudo iptables -t nat -I PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080

# Create ROOT RSA private key then copy to Github repo (pull access only)

sudo ssh-keygen -t rsa

sudo cat /root/.ssh/id_rsa.pub

```

### Startup script for each new instance

```bash

#! /bin/bash

ssh-keyscan github.com >> ~/.ssh/known_hosts

git clone [email protected]:xyk2/cloud-transcoder.git

cd /cloud-transcoder

npm install

PM2_HOME=/root/.pm2 pm2 kill

PM2_HOME=/root/.pm2 NODE_ENV=production pm2 start app.js

```

### Shutdown script before preemption

* Call localhost endpoint to reset running job in DB queue

* Terminate PM2

*

### Workflow to upload & transcode & tag game videos from YouTube

* `youtube-dl` to download and merge original files

* `gsutil -m cp` to `broadcast-cx-raw-recordings` bucket

* Insert filename, size, and format into `assets_game_footage` table

* Insert type, id, status into `assets_transcode_queue`

* Find game ID of game film, insert into `assets_game_footage` which will be populated

* Find start_record_time of game film, insert into `assets_game_footage`

### Useful snippets

`youtube-dl`

youtube-dl --external-downloader aria2c --external-downloader-args '-x 8' https://www.youtube.com/playlist?list=PLUz3zvlwsLgWJRmJo8fLOTMiXkFC25S61 --playlist-start 46 --playlist-end 150

### ffmpeg errors and possible errors

* `ffmpeg exited with code 1: Conversion failed!` -max_muxing_queue_size 1024