{"id":25648893,"url":"https://github.com/groovybits/mpegts_to_s3","last_synced_at":"2025-04-15T16:34:39.184Z","repository":{"id":272592034,"uuid":"917120925","full_name":"groovybits/mpegts_to_s3","owner":"groovybits","description":"MpegTS UDP to HLS storage in S3 and playback from S3 replaying back to UDP ","archived":false,"fork":false,"pushed_at":"2025-03-25T21:07:28.000Z","size":311,"stargazers_count":7,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-25T22:22:35.069Z","etag":null,"topics":["broadcast","hls","minio","mpegts","s3","udp","video"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/groovybits.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-01-15T11:51:28.000Z","updated_at":"2025-03-25T21:04:34.000Z","dependencies_parsed_at":"2025-01-15T13:41:02.635Z","dependency_job_id":"322d5331-494b-4bc0-9407-60ef842b7593","html_url":"https://github.com/groovybits/mpegts_to_s3","commit_stats":null,"previous_names":["groovybits/mpegts_to_s3"],"tags_count":33,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/groovybits%2Fmpegts_to_s3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/groovybits%2Fmpegts_to_s3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/groovybits%2Fmpegts_to_s3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/groovybits%2Fmpegts_to_s3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/groovybits","download_url":"https://codeload.github.com/groovybits/mpegts_to_s3/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249108946,"owners_count":21214089,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["broadcast","hls","minio","mpegts","s3","udp","video"],"created_at":"2025-02-23T13:37:37.727Z","updated_at":"2025-04-15T16:34:39.177Z","avatar_url":"https://github.com/groovybits.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# UDP to HLS MpegTS Capture to Cloud Storage\n\nUDP to HLS enables capturing of MPEG-TS UDP multicast streams, segmenting them into time-based HLS segments, creating `.m3u8` playlists, and uploading them to MinIO or S3 storage. The segments and playlists can be signed for secure playback. It runs diskless and doesn't store segments locally, only in memory and upload to S3/MinIO. It has an [API](recording-playback-api/API.md) for Recording, Playback and Storage Pool management. The API server is a Node.js application that manages the recording and playback jobs, storage pools, and assets. It communicates with the agent to start and stop recording/playback jobs.\n\nThere are container image build and compose options using Podman/Docker + *-Compose that set up a local MinIO server and an API Server consisting of a Manager and an Agent that runs the udp-to-hls and hls-to-udp when requested. See  [hls-to-udp](hls-to-udp/README.md) and the API Server in Node.js [recording-playback-api](recording-playback-api).\n\n```mermaid\ngraph LR\n    subgraph \"Input/Output\"\n        UDPIN[\"UDP\u003cbr/\u003eMulticast\u003cbr/\u003eSource\"]\n        UDPOUT[\"UDP\u003cbr/\u003eMulticast\u003cbr/\u003eOutput\"]\n    end\n\n    subgraph \"Manager API (Port 3000)\"\n        MAPI[\"Manager API\u003cbr/\u003eServer\"]\n        REC[\"Recording\u003cbr/\u003eJobs\"]\n        PB[\"Playback\u003cbr/\u003eJobs\"]\n        POOL[\"Storage\u003cbr/\u003ePools\"]\n        ASSET[\"Assets\u003cbr/\u003e(Recordings)\"]\n        \n        MAPI --\u003e |\"Creates/Manages\"| REC\n        MAPI --\u003e |\"Creates/Manages\"| PB\n        MAPI --\u003e |\"Manages\"| POOL\n        POOL --\u003e |\"Contains\"| ASSET\n    end\n\n    subgraph \"Storage\"\n        S3[\"S3/MinIO\u003cbr/\u003eBuckets\"]\n        META[\"Job Metadata\u003cbr/\u003e(JSON in S3)\"]\n    end\n\n    subgraph \"Agent API (Port 3001)\"\n        AAPI[\"Agent API\u003cbr/\u003eServer\"]\n        UHL[\"udp-to-hls\u003cbr/\u003eProcess\"]\n        HLU[\"hls-to-udp\u003cbr/\u003eProcess\"]\n        \n        AAPI --\u003e |\"Spawns/Monitors\"| UHL\n        AAPI --\u003e |\"Spawns/Monitors\"| HLU\n    end\n\n    %% Recording Flow\n    UDPIN --\u003e |\"Captured via libpcap\"| UHL\n    UHL --\u003e |\"MPEG-TS Segments \u0026 m3u8\"| S3\n    \n    %% Playback Flow\n    S3 --\u003e |\"Retrieves Segments \u0026 Playlist\"| HLU\n    HLU --\u003e |\"Timed Playback\"| UDPOUT\n    \n    %% Job Control\n    MAPI --\u003e |\"Forwards Requests\"| AAPI\n    MAPI --\u003e |\"Stores Job Info\"| META\n    AAPI --\u003e |\"Reports Status\"| MAPI\n    AAPI --\u003e |\"Stores Process IDs\"| META\n    \n    REC -.- |\"Initiates\"| MAPI\n    PB -.- |\"Initiates\"| MAPI\n    ASSET -.- |\"Used by\"| PB\n    \n    style UDPIN fill:#b3e0ff,stroke:#0066cc,stroke-width:2px,color:#003366,font-weight:bold\n    style UDPOUT fill:#b3e0ff,stroke:#0066cc,stroke-width:2px,color:#003366,font-weight:bold\n    style UHL fill:#b3ffb3,stroke:#006600,stroke-width:2px,color:#003300,font-weight:bold\n    style HLU fill:#b3b3e6,stroke:#000066,stroke-width:2px,color:#000033,font-weight:bold\n    style S3 fill:#ff99cc,stroke:#cc0066,stroke-width:2px,color:#660033,font-weight:bold\n    style META fill:#e6ccff,stroke:#3300cc,stroke-width:2px,color:#1a0066,font-weight:bold\n    style MAPI fill:#e6b3e6,stroke:#660066,stroke-width:2px,color:#330033,font-weight:bold\n    style AAPI fill:#ffb3ff,stroke:#660066,stroke-width:2px,color:#330033,font-weight:bold\n    style REC fill:#ccffcc,stroke:#006600,stroke-width:2px,color:#003300,font-weight:bold\n    style PB fill:#ccffcc,stroke:#006600,stroke-width:2px,color:#003300,font-weight:bold\n    style POOL fill:#ccffcc,stroke:#006600,stroke-width:2px,color:#003300,font-weight:bold\n    style ASSET fill:#ccffcc,stroke:#006600,stroke-width:2px,color:#003300,font-weight:bold\n```\n\n---\n\n## Quick Start Guide (Containerized)\n```bash\ngit clone https://github.com/groovybits/mpegts_to_s3.git\ncd mpegts_to_s3\n\n# Edit the config.env file to set the desired settings\nls configs/*.env \u0026\u0026 cp -R configs configs_custom \u0026\u0026 vim configs_custom/*.env\n# Then edit the docker-compose_* manager and agent to point to the custom configs\n\n# Start MinIO Server\npodman-compose -f docker-compose_minio.yml up -d --build\n\n# With the API server, Start MinIO, the Mpeg_to_S3 capture, and the API server\npodman-compose -f docker-compose_apiserver.yml up --build\n\n# Now browse to the host IP on port 3000 to access the API server swagger UI\n# webbrowser --url http://localhost:3000/api-docs  # Manager default port is 3000\n# webbrowser --url http://localhost:3001/api-docs  # Agent default port is 3001\n```\n\nThere are official stable release containers available at: \n- `docker.io/groovybits/manager:latest`\n- `docker.io/groovybits/agent:latest`\n\n## Quick Start Guide (Local Build)\n\n### Clone and Build the Project \n```bash\ngit clone https://github.com/groovybits/mpegts_to_s3.git\ncd mpegts_to_s3\n\n# Build the udp-to-hls application in release mode\nmake \u0026\u0026 make install\n```\n\n### Configure and Run the Components\n#### Start MinIO Server (runs on ports 9000 and 9001 like S3)\n```bash\n# Start the MinIO server for local testing (uses ./data/ for storage)\npodman-compose -f docker-compose_minio.yml up --build\n```\n\n#### Capture and Segment UDP Stream to HLS\n```bash\n# Capture multicast stream from udp://224.0.0.200:10001 on interface eth0\n# Segments are uploaded to S3/MinIO in the s3://hls/channel01/* bucket/key\ncd hls \u0026\u0026 ../bin/udp-to-hls \\\n    -n eth0 \\         # Network interface for packet capture\n    -i 224.0.0.200 \\  # Multicast IP to filter\n    -p 10001 \\        # UDP port to filter\n    -b hls \\\n    -o channel01      # Unique Identifier and Output directory for .ts segments\n```\n\n#### Playback from HLS to UDP\n\n- **Playback with hls-to-udp** [hls-to-udp](hls-to-udp/README.md)\n  1. Start the hls-to-udp agent\n     ```bash\n     # Get the last m3u8 hour url in S3 storage to playback to MpegTS UDP via multicast\n     ./bin/hls-to-udp -u `cat hls/index.txt | tail -1` -o 224.0.0.200:10001\n     ```\n  2. Play the relayed stream while being re-broadcasted\n     ```bash\n     mpv udp://224.0.0.200:10001\n     ```\n---\n\n## Prerequisites\n\n- **Rust Toolchain:** Install via [Rustup](https://rustup.rs/).\n- **MinIO/S3 Server:** Ensure MinIO is available locally or via a container.\n- **Dependencies:** Install `libpcap` for packet capture and FFmpeg (optional) for HLS segment generation.\n- **Ports:** Open ports 9000 and 9001 for MinIO and the HTTP server.\n- **SSH Tunneling:** For HTTP access to MinIO, set up SSH forwarding.\n- **LibLTNTSTools (optional, really is important):** Use the `smoother` feature flag to enable the Bitrate Smoother, without it the output will not be broadcast compliant but will playback fine with possibly some jitter.\n\n```bash\n\n---\n\n## Usage\n\n   | Option                                 | Description                                                  | Default                  |\n   |----------------------------------------|--------------------------------------------------------------|--------------------------|\n   | -e, --endpoint \u003cendpoint\u003e              | S3-compatible endpoint                                       | http://127.0.0.1:9000    |\n   | -r, --region \u003cregion\u003e                  | S3 region                                                    | us-east-1                |\n   | -b, --bucket \u003cbucket\u003e                  | S3 bucket name                                               | hls                      |\n   | -i, --udp_ip \u003cudp_ip\u003e                  | UDP multicast IP to filter                                   | 227.1.1.102              |\n   | -p, --udp_port \u003cudp_port\u003e              | UDP port to filter                                           | 4102                     |\n   | -n, --interface \u003cinterface\u003e            | Network interface for pcap                                   | net1                     |\n   | -t, --timeout \u003ctimeout\u003e                | Capture timeout in milliseconds                              | 1000                     |\n   | -o, --output_dir \u003coutput_dir\u003e          | Local dir for HLS output and Channel Name/Key                | channel01                |\n   | --remove_local                         | Remove local .ts/.m3u8 after uploading?                      |                          |\n   | --hls_keep_segments \u003chls_keep_segments\u003e| Max segments kept in ${output_Dir}.m3u8                      | 3                        |\n   | --unsigned_urls                        | Generate unsigned S3 URLs instead of presigned URLs          |                          |\n   | --diskless_ring_size \u003cdiskless_rs\u003e     | Number of segments in memory buffer                          | 1                        |\n   | -v, --verbose \u003cverbose\u003e                | Verbose level                                                | 0                        |\n   | -h, --help                             | Print help                                                   |                          |\n   | -V, --version                          | Print version                                                |                          |\n\n### Environment Variables:\n\n#### udp-to-hls Environment Variables:\n\n  - `CHANNEL_NAME`: Name of the channel (default: `channel01`) used as the subdirectory for HLS ts segments\n  - `SEGMENT_DURATION_MS`: Duration of each segment in milliseconds (default: `1000`), (less than 1 second may not work well)\n  - `FILE_MAX_AGE_SECONDS`: Maximum age of files in seconds to upload (default: `30`)\n  - `URL_SIGNING_SECONDS`: Duration of signed URLs in seconds (default: `31104004`)\n  - `MINIO_ROOT_USER`: S3 username / access key ID (default: `minioadmin`)\n  - `MINIO_ROOT_PASSWORD`: S3 password / secret access key (default: `minioadmin`)\n  - `PCAP_PACKET_COUNT`: Number of packets to capture at a time (default: `7`)\n  - `PCAP_PACKET_SIZE`: Size of mpegts packets to capture (default: `188`)\n  - `PCAP_PACKET_HEADER_SIZE`: Size of mpegts packet ip/eth header (default: `42`)\n  - `PACAP_BUFFER_SIZE`: Size of the pcap buffer (default: `4194304`)\n  - `USE_ESTIMATED_DURATION`: Use estimated duration for manual segmentation (default: `true`)\n  - `CAPTURE_BUFFER_SIZE`: Size of the capture buffer (default: `4194304`)\n  - `PCAP_PACKET_COUNT`: Number of packets to capture at a time (default: `7`)\n  - `USE_ESTIMATED_DURATION`: Use estimated duration for manual segmentation (default: `false`)\n  - `M3U8_LIVE_SEGMENT_COUNT`: Number of segments to keep in the live m3u8 (default: `3`)\n  - `USE_UNSIGNED_URLS`: Generate unsigned S3 URLs instead of presigned URLs (default: `false`)\n\n---\n\n## How It Works\n\n1. **Capture:** The application uses `libpcap` to capture UDP multicast MPEG-TS packets on a specified interface.\n2. **Segment:** It either:\n   - **Automatically segments** streams with FFmpeg\n   - **Manually segments** streams by directly processing MPEG-TS packets\n3. **Upload:** A directory watcher uploads new `.ts` segments and playlists to S3 or MinIO.\n4. **Playback:** The uploaded segments are accessible via signed or unsigned URLs, enabling HLS playback.\n5. **Relay:** The hls-to-udp relay can be used to replay the content back as a multicast stream later or in real-time.\n\n---\n\n## Monitoring and Logs\n- **MinIO Web Interface:** View uploaded files via the MinIO web client.\n- **Segment Logs:** Signed URL logs are stored in `urls.log` in the output directory.\n\n---\n\n## Example of the Storage S3 object key hierarchy\nhls/{recordId}/\n├── index.m3u8\n├── 2025/\n    └── 01/\n        └── 16/\n            └── 06/\n                ├── segment_20250116-060000__0000.ts\n                └── index.m3u8\n```\n---\n\n## Development Notes\n\nAdditional utility scripts for managing MinIO and HTTP servers are available in the `scripts/` folder.\n\nFor questions or issues, refer to the repository's issue tracker.\n\n---\n\n**Author:** CK\n**Date:** March 2025\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgroovybits%2Fmpegts_to_s3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgroovybits%2Fmpegts_to_s3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgroovybits%2Fmpegts_to_s3/lists"}