{"id":13617527,"url":"https://github.com/x1wins/CW-OVP","last_synced_at":"2025-04-14T06:34:18.650Z","repository":{"id":38291813,"uuid":"278267172","full_name":"x1wins/CW-OVP","owner":"x1wins","description":"video packaging to *.m3u8 for HLS (HTTP Live Streaming) with FFMPEG on website (ruby on rails)","archived":false,"fork":false,"pushed_at":"2023-03-06T03:38:48.000Z","size":22355,"stargazers_count":27,"open_issues_count":28,"forks_count":10,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-08-01T20:47:49.672Z","etag":null,"topics":["docker","docker-swarm","ffmpeg","hls","hls-live-streaming","hls-video","http-live-streaming","m3u8","m3u8-playlist","m3u8-videos","mp4","ott","ott-streaming","packaging","ruby-on-rails-6","streaming-service","transcoding","video-streaming"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/x1wins.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"ROADMAP.md","authors":null}},"created_at":"2020-07-09T04:59:01.000Z","updated_at":"2024-03-08T14:53:19.000Z","dependencies_parsed_at":"2024-01-03T02:24:30.979Z","dependency_job_id":"7b8365b9-e459-4f21-9a74-f9b734d05bf8","html_url":"https://github.com/x1wins/CW-OVP","commit_stats":null,"previous_names":[],"tags_count":32,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/x1wins%2FCW-OVP","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/x1wins%2FCW-OVP/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/x1wins%2FCW-OVP/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/x1wins%2FCW-OVP/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/x1wins","download_url":"https://codeload.github.com/x1wins/CW-OVP/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223621829,"owners_count":17174766,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-swarm","ffmpeg","hls","hls-live-streaming","hls-video","http-live-streaming","m3u8","m3u8-playlist","m3u8-videos","mp4","ott","ott-streaming","packaging","ruby-on-rails-6","streaming-service","transcoding","video-streaming"],"created_at":"2024-08-01T20:01:43.195Z","updated_at":"2024-11-08T02:30:38.208Z","avatar_url":"https://github.com/x1wins.png","language":"Ruby","funding_links":[],"categories":["HarmonyOS","Open Source Rails Apps"],"sub_categories":["Windows Manager","Articles"],"readme":"[![Build Status](https://travis-ci.com/x1wins/CW-OVP.svg?branch=master)](https://travis-ci.com/x1wins/CW-OVP)\n\n# CW-OVP\n* [Preview](#Preview)\n* [Introduction](#Introduction)\n    * [EN](#EN)\n    * [KO](#KO)\n* [Tech Stack](#Tech-Stack)\n* [Feature](#Feature)\n* [System Structure](#System-Structure)\n* [Getting started](#Getting-started)\n    * [Storage config](#Storage-config)\n    * [How To Run Development mode with Docker-compose](#How-To-Run-Development-mode-with-Docker-compose)\n    * [How To Run Development mode without docker](#How-To-Run-Development-mode-without-docker)\n    * [Sample video file download](#Sample-video-file-download)    \n* [Minimum Requirements for Production](#Minimum-Requirements-for-Production)\n* [Roadmap](#Roadmap)\n* [Deploy](#Deploy)    \n    * [Kubernetes](#kubernetes)    \n    * [Docker Swarm](#Docker-Swarm)    \n* [Webhook](#Webhook)\n\n## Preview\n![show](/screenshot/cw_ovp_show.png)          \n![index](/screenshot/cw_ovp_index.png)\n\n## Introduction\n### EN\nOVP(online video platform) mean that online transcoding, packaging and hosting service with video file upload to online storage or cloud system.\u003cbr/\u003e\nInternet speed was increase and many internet user prefer to streaming service for Mobile, web, OTT than service from over-the-air television networks.\nbut streaming service use OVP or solution for start streaming service with Mobile, web, OTT(over the top). \nsuch as OVP and some solution require many spend money for streaming service.\u003cbr/\u003e\nFamous OVP business company is brightcove.com, mux.com, dacast.com, vimeo.com, Dacast.com, Amazon elastic transcoding and Wowza solution.\u003cbr/\u003e\nCW-OVP project will commit helping for who want ready for streaming service.\u003cbr/\u003e\nThis is why CW-OVP project was built.\n\n### KO\nOVP(online video platform)란\u003cbr/\u003e\n온라인 저장소 또는 클라우드에 비디오 파일을 업로드하여 트랜스코딩과 페키징 처리후 호스팅 서비스를 제공함을 말한다.\u003cbr/\u003e\n인터넷 속도의 발달과 사용자들의 요구로 많은 스트리밍 서비스가 생겨나고 있고, 기존의 방송사들도 흐름에 따라  Mobile, web, OTT(over the top) 셋탑기기의 스트리밍 서비스를 제공하기 위해서 OVP(online video platform)이나 솔류션을 많이 사용하고 있다.\u003cbr/\u003e\n대표적인 해외 OVP는 brightcove.com, mux.com, dacast.com, vimeo.com, Dacast.com, Amazon elastic transcoding 등 서비스들이 존재하며 솔류션으로는 Wowza가 존재한다.\u003cbr/\u003e\n이에 CW-OVP 프로젝트를 오픈소스로 개발하여 기업이나 단체, 개인에 도움이 되고자 한다.\n\n## Tech Stack\n|what use|description|\n|---|---|\n|FFMPEG|video transcoding, packaging|\n|h.264|Codec|\n|HLS|Protocol|\n|Ruby on Rails|web framework, websocket(action cable)|\n|Redis|queue, pub/sub|\n|Sidekiq|background job, queue|\n|Postresql|database|\n|Docker, docker-compose|install environment|\n|Docker Swarm|Clustering|\n|Kubernetes|Clustering|\n- you can choose Docker swarm or Kubernetes(kops or EKS) for Clustering\n\n## Feature\n- Clustering ffmpeg worker with sidekiq on Docker swarm or Kubernetes\n- Transcoding, Packaging with ffmpeg for HLS\n    - Generated multiple m3u8 for Adaptive Streaming\n- Extract 10 random Thumbnail images during video packaging\n- AWS S3 for stored video, thumbnail assets\n- AWS Cloudfront for CDN\n- Real time base on web UI with websocket(Action Cable), ruby on rails\n- Open source (GPL-2.0 License) and Free, but need hardware…\n\n## System Structure\n![cw-ovp-system-structure.png](cw-ovp-system-structure.png)\n- If your node for database that got fault or something wrong, You can lost persistent in database. that's why I recommend AWS RDS for postgresql. \n\n## Getting started\n### Storage config\n* AWS S3 Storage\n    1. How to make s3 bucket for AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY\n        - https://www.changwoo.org/x1wins@changwoo.net/2019-10-23/Upload-file-to-S3-with-AWS-CLI-d12442012c\n    2. How to make cloudfront url for AWS_CLOUDFRONT_DOMAIN \n        - https://www.changwoo.org/x1wins@changwoo.net/2019-10-23/using-cloud-front-with-s3-51d2eb17bb\n    3. Update ```.env.dev.s3``` file for s3, cloudfront\n        1. open ```.env.dev.s3```\n        2. add below of code and update each value                                                              \n            ```\n                AWS_ACCESS_KEY_ID=[Change key id]\n                AWS_SECRET_ACCESS_KEY=[Change access key]\n                REGION=[Change region (us-west-1 or us-west-2 or us-east-1...)]  \n                BUCKET=[Change bucket]\n                CDN_BUCKET=[Change cdn bucket]\n                AWS_CLOUDFRONT_DOMAIN=[Change cdn domain]\n            ```    \n        3. Check config\n            ```\n            docker-compose --env-file .env.dev.s3 config\n            ```\n    4. ignore changed ```.env.dev.s3``` file when ```git commit``` \n       ```\n          git update-index --assume-unchanged .env.dev.s3\n       ```\n\n### How To Run Development mode with Docker-compose\n1. Download source\n    1. ```git clone https://github.com/x1wins/CW-OVP.git```\n    2. ```cd ./CW-OVP```\n2. Build and Run    \n    1. ```docker-compose --env-file .env.dev.s3 up --build -d```\n    2. ```docker-compose run --no-deps web bundle exec rails webpacker:install```\n    2. ```docker-compose --env-file .env.dev.s3 restart web```\n    2. ```docker-compose run --no-deps web bundle exec rake db:create```\n    3. ```docker-compose run --no-deps web bundle exec rake db:migrate```\n    4. ```docker-compose run --no-deps web bundle exec rake db:create RAILS_ENV=test```\n    4. ```docker-compose run --no-deps web bundle exec rake db:migrate RAILS_ENV=test```\n3. Restart for updated code\n    1. ```git fetch origin develop```\n    2. ```git reset --hard origin/develop```\n    3. Restart web, sidekiq\n        ```\n        docker-compose --env-file .env.dev.s3 restart web\n        docker-compose --env-file .env.dev.s3 restart sidekiq\n        ```\n4. Unit testing with rspec\n    ```\n    docker-compose run --no-deps web bundle exec rspec --format documentation\n    ```\n5. Console\n    ```\n    docker-compose run --no-deps web bundle exec rails console\n    ```\n6. Troubleshooting \n    1. when I got bundle error such as ```minemagic```\n        1. Log of minemagic gem error\n            ```\n            Step 10/16 : RUN bundle install\n            ---\u003e Running in 578d62fab6ac\n            Fetching gem metadata from https://rubygems.org/............\n            You have requested:\n             mimemagic ~\u003e 0.4.1\n            The bundle currently has mimemagic locked at 0.3.5.\n            Try running `bundle update mimemagic`\n            If you are updating multiple gems in your Gemfile at once,\n            try passing them all to `bundle update`\n            ERROR: Service 'web' failed to build: The command '/bin/sh -c bundle install' returned a non-zero code: 7\n            The command \"docker-compose build\" failed and exited with 1 during .\n            Your build has been stopped.\n            ```\n        2. Solution\n            ```\n            docker-compose run --no-deps web bundle update mimemagic\n            ```\n    2. Keep latest gem for defense vulnerability\n        ```\n        docker-compose run --no-deps web bundle update\n        docker-compose run --no-deps web yarn upgrade --latest\n        ```\n    3. no space left on device\n        1. https://www.changwoo.org/x1wins@changwoo.net/2021-02-12/write-myapp-tmp-redis-appendonly-aof-9ff1127366\n        1. Log\n            ```\n            % docker-compose --env-file .env.dev.s3 up --build\n            Building web\n            Step 1/16 : FROM ruby:2.6.3\n             ---\u003e 8fe6e1f7b421\n            Step 2/16 : RUN curl -sL https://deb.nodesource.com/setup_12.x | bash - \u0026\u0026     curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \u0026\u0026     echo \"deb https://dl.yarnpkg.com/debian/ stable main\" | tee /etc/apt/sources.list.d/yarn.list \u0026\u0026     apt-get update \u0026\u0026     apt-get install -qq -y build-essential nodejs yarn vim     libpq-dev postgresql-client ffmpeg\n             ---\u003e Using cache\n             ---\u003e 14eb2f07d9e5\n            Step 3/16 : RUN apt-get update \u0026\u0026     apt-get install -y         python3         python3-pip         python3-setuptools         groff         less     \u0026\u0026 pip3 install --upgrade pip     \u0026\u0026 apt-get clean\n             ---\u003e Using cache\n             ---\u003e c57e46926299\n            Step 4/16 : RUN pip3 --no-cache-dir install --upgrade awscli\n             ---\u003e Using cache\n             ---\u003e 7e8e641bfc90\n            Step 5/16 : RUN mkdir /myapp\n             ---\u003e Using cache\n             ---\u003e a0a39c82945a\n            Step 6/16 : RUN mkdir /storage\n             ---\u003e Using cache\n             ---\u003e 66348ad31ef0\n            Step 7/16 : WORKDIR /myapp\n             ---\u003e Using cache\n             ---\u003e 47b601075951\n            Step 8/16 : COPY . /myapp\n            ERROR: Service 'web' failed to build : Error processing tar file(exit status 1): write /myapp/tmp/storage/og/op/ogopd3w9xnrwq0b6kqf8beh726xu: no space left on device\n            ```\n        2. Solution\n            ```\n            % docker system prune\n            WARNING! This will remove:\n              - all stopped containers\n              - all networks not used by at least one container\n              - all dangling images\n              - all dangling build cache\n            \n            Are you sure you want to continue? [y/N] y\n            ```\n       \n\n### How To Run Development mode without docker\n[SETUP_WITHOUT_DOCKER.md](/SETUP_WITHOUT_DOCKER.md)       \n\n### Sample video file download\n- http://distribution.bbb3d.renderfarming.net/video/mp4/bbb_sunflower_1080p_30fps_normal.mp4              \n- https://filesamples.com/samples/video/avi/sample_1920x1080.avi\n- https://filesamples.com/samples/video/ts/sample_1920x1080.ts\n- https://filesamples.com/samples/video/mp4/sample_1920x1080.mp4\n- https://filesamples.com/samples/video/mkv/sample_1920x1080.mkv\n- https://filesamples.com/samples/video/mov/sample_1920x1080.mov\n\n## Minimum Requirements for Production\n- Required AWS S3 for Storage\n- Required AWS Cloudfront for CDN\n- Server Spec\n    - CPU\n        - 8 or more Cpu per server (c5.2xlarge on aws ec2)\n    - Disk\n        - more 10 Gb\n        - if you will upload heavy video or more 10Gb, need more space\n    - Memory\n        - more 2Gb\n    - Number of Server\n        - master : 1 or more\n        - slave : 2 or more\n\n## Roadmap\n[Roadmap](/ROADMAP.md)\n       \n## Deploy\n### kubernetes\n[K8s with kops for Production](/k8s-manifests/README.md)\n\n### Docker Swarm\n[DOCKER-SWARM.md](/DOCKER-SWARM.md)\n\n## Webhook\n[Webhook](/webhook/README.md)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fx1wins%2FCW-OVP","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fx1wins%2FCW-OVP","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fx1wins%2FCW-OVP/lists"}