{"id":23828855,"url":"https://github.com/onitake/restreamer","last_synced_at":"2025-04-11T21:18:56.299Z","repository":{"id":44920900,"uuid":"60090824","full_name":"onitake/restreamer","owner":"onitake","description":"HbbTV 1.0/1.1 HTTP streaming proxy","archived":false,"fork":false,"pushed_at":"2024-10-21T14:25:55.000Z","size":846,"stargazers_count":46,"open_issues_count":14,"forks_count":14,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-04-11T21:18:50.402Z","etag":null,"topics":["hbbtv","hbbtv1","http","mpegts","proxy","streaming"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/onitake.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-05-31T13:15:07.000Z","updated_at":"2024-12-25T10:39:03.000Z","dependencies_parsed_at":"2024-06-19T00:05:34.435Z","dependency_job_id":"7655efdc-9206-4b7f-90a1-c4bf9fa674a1","html_url":"https://github.com/onitake/restreamer","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onitake%2Frestreamer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onitake%2Frestreamer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onitake%2Frestreamer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onitake%2Frestreamer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/onitake","download_url":"https://codeload.github.com/onitake/restreamer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248480438,"owners_count":21110939,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hbbtv","hbbtv1","http","mpegts","proxy","streaming"],"created_at":"2025-01-02T13:55:06.643Z","updated_at":"2025-04-11T21:18:56.271Z","avatar_url":"https://github.com/onitake.png","language":"Go","readme":"[![language](https://img.shields.io/badge/language-go-%2300ADD8.svg?style=flat-square)](https://golang.org/) [![release](https://img.shields.io/github/release/onitake/restreamer.svg?style=flat-square)](https://github.com/onitake/restreamer/releases) [![goreport](https://goreportcard.com/badge/github.com/onitake/restreamer?style=flat-square)](https://goreportcard.com/report/github.com/onitake/restreamer) ![license](https://img.shields.io/github/license/onitake/restreamer?style=flat-square)\n\n![restreamer logo](doc/logo.png)\n\n# restreamer\n\nHTTP transport stream proxy\n\nCopyright © 2016-2024 Gregor Riepl;\nAll rights reserved.\n\nPlease see the LICENSE file for details on permitted use of this software.\n\n\n## Introduction\n\nrestreamer is a proof-of-concept implementation of a streaming proxy\nthat can fetch, buffer and distribute [MPEG transport streams](https://en.wikipedia.org/wiki/MPEG-TS).\n\nIt serves as a non-transcoding streaming proxy for legacy HbbTV applications.\n\nData sources can be local files, remote HTTP servers or raw TCP streams.\nUnix domain sockets are also supported.\n\nThe proxy is stateless: Streams are transported in realtime\nand cached resources are only kept in memory.\nThis makes restreamer perfectly suited for a multi-tiered architecture,\nthat can be easily deployed on containers and expanded or shrunk as\nload demands.\n\n\n## Architecture\n\nrestreamer is written in [Go](https://golang.org/), a very versatile\nprogramming language. The built-in network layer makes it a first-class\nchoice for a streaming server.\n\nThese are the key components:\n* util - a small utility library\n* streaming/client - HTTP getter for fetching upstream data\n* streaming/connection - HTTP server that feeds data to clients\n* streaming/streamer - connection broker and data queue\n* api/api - web API for service monitoring\n* streaming/proxy - static web server and proxy\n* protocol - network protocol library\n* configuration - abstraction of the configuration file\n* metrics - a small wrapper around the Promethus client library\n* metrics/stats - the old, deprecated metrics collector; use Prometheus if possible\n* cmd/restreamer - core program that glues the components together\n\n\n## Compilation\n\nrestreamer is go-gettable.\nJust invoke:\n\n```\nCGO_ENABLED=0 go get github.com/onitake/restreamer/...\n```\n\nDisabling CGO is recommended, as this will produce a standalone binary that\ndoes not depend on libc. This is useful for running restreamer in a bare container.\n\nA makefile is also provided, as a quick build reference.\nSimply invoke `make` to build `restreamer`.\n\nPassing GOOS and/or GOARCH will yield cross-compiled binaries for the\nrespective platform. For example:\n\n```\nGOOS=windows GOARCH=amd64 CGO_ENABLED=0 go build github.com/onitake/restreamer/cmd/restreamer\n```\nor\n```\nmake GOOS=windows GOARCH=amd64\n```\n\nYou can also use `make test` to run the test suite, or `make fmt` to run `go fmt` on all sources.\n\n\n## Releases\n\nRelease builds are automatically done by a GitHub Action whenever a tag is pushed.\n\nMake sure that your tags conform to semantic versioning and begin with a \"v\". \nExample: v0.9.1\n\nNote: The release upload my sometimes fail due to issues with the GitHub asset API.\nRetry the release step until all artifacts have been uploaded.\n\nOnce the upload is complete, open the newly generated release and add the changelog\nto the description.\n\n\n## Configuration\n\nAll configuration is done through a configuration file named `restreamer.json`.\n\nSee `examples/documented/restreamer.json` for a documented example.\n\nThe input and output buffer sizes should be adapted to the expected stream\nbitrates and must account for unstable or slow client-side internet connections.\n\nOutput buffers should cover at least 1-5 seconds of video and the input buffer\nshould be 4 times as much.\nThe amount of packets to buffer, based on the video+audio bitrates, can be\ncalculated with this formula (overhead is not taken into account):\n\n```\nmpegts_packet_size = 188\nbuffer_size_packets = (avg_video_bitrate + avg_audio_bitrate) * buffer_size_seconds / (mpegts_packet_size * 8)\n```\n\nIt is also important to keep the bandwidth of the network interfaces\nin mind, so the connection limit should be set accordingly.\n\nBuffer memory usage is equally important, and can be roughly calculated as follows:\n\n```\nmpegts_packet_size = 188\nmax_buffer_memory = mpegts_packet_size * (number_of_streams * input_buffer_size + max_connections * output_buffer_size)\n```\n\nIt is possible to specify multiple upstream URLs per stream.\nThese will be tested in a round-robin fashion in random order,\nwith the first successful one being used.\nThe list is only shuffled once, at program startup.\n\nIf a connection is terminated, all URLs will be tried again after a delay.\nIf delay is 0, the stream will stay offline.\n\nTo protect against overload, both a soft and a hard limit on the number of\ndownstream connections can be set. When the soft limit is reached, the health\nAPI will start reporting that the server is \"full\". Once the hard limit is\nreached, new connections will be responded with a 404. A 503 would be more\nappropriate, but this is not handled well by many legacy streaming clients.\n\n\n## Logging\n\nrestreamer has a JSON logging module built in.\nOutput can be sent to stdout or written to a log file.\n\nIt is highly recommended to log to stdout and collect logs using journald\nor a similar logging engine.\n\n\n## Metrics\n\nMetrics are exported through the Prometheus client library. Enable a Prometheus\nAPI endpoint and expose it on /metrics to expose them.\n\nSupported metrics are:\n\n* _streaming_packets_sent_\n  Total number of MPEG-TS packets sent from the output queue.\n* _streaming_bytes_sent_\n  Total number of bytes sent from the output queue.\n* _streaming_packets_dropped_\n  Total number of MPEG-TS packets dropped from the output queue.\n* _streaming_bytes_dropped_\n  Total number of bytes dropped from the output queue.\n* _streaming_connections_\n  Number of active client connections.\n* _streaming_duration_\n  Total time spent streaming, summed over all client connections. In nanoseconds.\n* _streaming_source_connected_\n  Connection status, 0=disconnected 1=connected.\n* _streaming_packets_received_\n  Total number of MPEG-TS packets received.\n* _streaming_bytes_received_\n  Total number of bytes received.\n\nAdditionally, the standard process metrics supported by the Prometheus client\nlibrary are exported. Go runtime statistics are disabled, as they can have a\nconsiderable effect on realtime operation. To enable them, you need to turn on\nprofiling.\n\n\n## Optimisation\n\n### Test Stream\n\nUsing the example config and ffmpeg, a test stream can be set up.\n\nCreate a pipe:\n```\nmkfifo /tmp/pipe.ts\n```\nStart feeding data into the pipe (note the -re parameter for real-time streaming):\n```\nffmpeg -re -stream_loop -1 -i test.ts -fflags +genpts -c:a copy -c:v copy -y /tmp/pipe.ts\n```\nStart the proxy:\n```\nbin/restreamer\n```\nStart playing:\n```\ncvlc http://localhost:8000/pipe.ts\n```\n\n### File Descriptors\n\nContinuous streaming services require a lot of open file descriptors,\nparticularly when running a lot of low-bandwidth streams on a powerful\nstreaming node.\n\nMany operating systems limit the number of file descriptors a single\nprocess can use, so care must be taken that servers aren't starved by\nthis artificial limit.\n\nAs a rough guideline, determine the maximum downstream bandwidth your\nserver can handle, then divide by the bandwidth of each stream and\nadd the number of upstream connections.\nReserve some file descriptors for reconnects and fluctuations.\n\nFor example, if your server has a dedicated 10Gbit/s downstream network\nconnection and it serves 10 streams at 8Mbit/s, the required number\nof file descriptors would be:\n\n```\ndescriptor_count = streams_count + downstream_bandwidth / stream_bandwidth * 200%\n=\u003e (10 + 10 Gbit/s / 8 Mbit/s) * 2 = 2520\n```\n\nOn many Linux systems, the default is very low, at 1024 file\ndescriptors, so restreamer would not be able to saturate all bandwidth.\nWith the following systemd unit file, this would be remedied:\n```\n[Unit]\nDescription=restreamer HTTP streaming proxy\nAfter=network.target\n\n[Service]\nType=simple\nExecStart=/usr/bin/restreamer\nRestart=always\nKillMode=mixed\nLimitNOFILE=3000\n\n[Install]\nWantedBy=multi-user.target\n```\n\nIn most cases, it's safe to set a high value, so something like\n64000 (or more) would be fine.\n\n### Memory/CPU/Network Usage\n\nTo test behaviour under heavy network load, it is recommended to run\nmultiple concurrent HTTP connections that only read data occasionally.\n\nThis will force restreamer to buffer excessively, increasing memory usage.\n\nYou should monitor memory usage of the restreamer process closely,\nand change buffer sizes or system memory accordingly.\n\nNote that the Go runtime does not immediately return garbage collected\nmemory to the operating system. This is done only about every 5 minutes\nby a special reclaiming task.\n\nUsing the built-in Go profiler can be more useful to watch memory usage.\nYou should check out the `profiler` branch, it opens a separate web\nserver on port 6060 that Go pprof can connect to. See [net/http/pprof](https://golang.org/pkg/net/http/pprof/)\nfor instructions on its usage.\n\n### Syscall Load\n\nLive streaming services like restreamer require realtime or near-realtime\noperation. Latency should be minimised, highlighting the importance of\nsmall buffers. Unfortunately, small buffers will increase the number of\nsyscalls needed to feed data to the remote clients.\n\nIn light of the recently published [speculative execution exploits\naffecting Intel CPUs](https://meltdownattack.com/), high syscall rates will\nhave a devastating effect on performance on these CPUs due to\nmitigations in the operating system.\n\nFor this reason, restreamer is relying on buffered I/O, with default buffer\nsizes as set in the Go http package. In the current implementation,\na 2KiB and a 4KiB buffer is used. This should give a good compromise\nbetween moderate latency and limited system load.\n\nEven with these buffers, performance loss compared with unmitigated systems\nis considerable. If this is unacceptable, it is highly recommended to run\nrestreamer on a system that does not need/use Meltdown mitigations.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonitake%2Frestreamer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fonitake%2Frestreamer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonitake%2Frestreamer/lists"}