{"id":50304958,"url":"https://github.com/porkcharsui/yt-dlp-feed","last_synced_at":"2026-05-28T15:30:49.474Z","repository":{"id":358764446,"uuid":"1237167409","full_name":"porkcharsui/yt-dlp-feed","owner":"porkcharsui","description":"Podcast RSS feed service backed just-in-time by yt-dlp","archived":false,"fork":false,"pushed_at":"2026-05-19T00:19:37.000Z","size":201,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-19T02:33:11.462Z","etag":null,"topics":["podcast","rss","rust","self-hosted","tailscale","yt-dlp"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/porkcharsui.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-13T00:06:38.000Z","updated_at":"2026-05-19T00:19:40.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/porkcharsui/yt-dlp-feed","commit_stats":null,"previous_names":["porkcharsui/yt-dlp-feed"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/porkcharsui/yt-dlp-feed","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/porkcharsui%2Fyt-dlp-feed","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/porkcharsui%2Fyt-dlp-feed/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/porkcharsui%2Fyt-dlp-feed/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/porkcharsui%2Fyt-dlp-feed/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/porkcharsui","download_url":"https://codeload.github.com/porkcharsui/yt-dlp-feed/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/porkcharsui%2Fyt-dlp-feed/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33615489,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-28T02:00:06.440Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["podcast","rss","rust","self-hosted","tailscale","yt-dlp"],"created_at":"2026-05-28T15:30:48.622Z","updated_at":"2026-05-28T15:30:49.461Z","avatar_url":"https://github.com/porkcharsui.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# yt-dlp-feed\n\n`yt-dlp-feed` is a Rust HTTP server that turns configured yt-dlp-supported sources into podcast-style RSS feeds. Feed items point at stable server URLs that perform just-in-time downloads with `yt-dlp`, stream the audio back to the client, and keep the downloaded media only temporarily.\n\nThe first target service is SoundCloud. A configured SoundCloud source can expose a profile feed, a likes feed, or both.\n\n## Status\n\nThis repository contains the first Rust implementation scaffold: config loading, route wiring, RSS rendering, a simple index page, temporary media cache behavior, optional HTTP Basic auth, and a mockable yt-dlp boundary. Live service behavior depends on yt-dlp, ffmpeg, and the supported extractor behavior for each service.\n\n## Supported Services\n\nThe server is designed around yt-dlp-compatible sources. V1 implements SoundCloud profile and likes feeds; other services can be added later by teaching the app how to fetch their feed metadata and download their media.\n\nFor the broader list of services that yt-dlp may support, see the canonical yt-dlp documentation:\n\n- [yt-dlp supported sites](https://github.com/yt-dlp/yt-dlp/blob/master/supportedsites.md)\n\n## Configuration\n\nBy default, the server looks for `config.yaml`. If no config exists, it uses built-in example SoundCloud defaults. You can also pass a path with `--config` or `YT_DLP_FEED_CONFIG`, and override `cache.data_dir` with `--data-dir` or `YT_DLP_FEED_DATA_DIR`.\n\n```yaml\nserver:\n  bind: \"127.0.0.1:8080\"\n\ncache:\n  data_dir: \"./data\"\n  media_ttl_minutes: 360\n  media_max_megabytes: 10240\n  disconnect_behavior: \"delay_cancel\"\n  disconnect_grace_seconds: 15\n\nmetadata:\n  refresh_interval_hours: 24\n  refresh_recent_grace_minutes: 60\n\ndownloads:\n  max_concurrent: 3\n  probe_timeout_seconds: 300\n\n# Docker enables this with YT_DLP_FEED_PIP_TOOL_UPDATES_ENABLED=true for the\n# pip-managed /opt/yt-dlp install. Leave disabled for local/dev runs.\npip_tool_updates:\n  enabled: false\n  startup_check: true\n  interval_hours: 168\n  pip_package: \"yt-dlp\"\n\n# Docker enables this with YT_DLP_FEED_PIP_TOOL_UPDATES_ENABLED=true for the\n# pip-managed /opt/yt-dlp install. Leave disabled for local/dev runs.\npip_tool_updates:\n  enabled: false\n  startup_check: true\n  interval_hours: 168\n  pip_package: \"yt-dlp\"\n\nauth:\n  enabled: false\n\nusers:\n  - name: \"operator\"\n    services:\n      - kind: \"soundcloud\"\n        name: \"primary\"\n        profile_url: \"https://soundcloud.com/example-profile\"\n        feeds:\n          - profile\n          - likes\n```\n\n## Routes\n\n- `GET /` serves a simple HTML index page with direct RSS links.\n- `GET /index.json` serves the same configured feed/status data in structured JSON.\n- `GET /healthz` returns `ok`.\n- `GET /readyz` returns `ready` only after every configured feed has metadata cached at least once.\n- `GET /users/{user}/soundcloud/{name}/feed.xml` serves the profile feed.\n- `GET /users/{user}/soundcloud/{name}/likes.xml` serves the likes feed.\n- `GET /users/{user}/soundcloud/{name}/items/{item_id}/audio.m4a` downloads or serves cached AAC/M4A audio for a feed item.\n\n## Cache Design\n\nThe app keeps two separate caches:\n\n- Feed metadata is stored as app-owned JSON under `cache.data_dir/feed-metadata`.\n- Downloaded media is stored temporarily under `cache.data_dir/media`.\n\nMetadata cache files are the source of truth for RSS rendering. Each file stores schema version `1`, the feed identity, the source URL, the last successful refresh timestamp, and the normalized item list in the same order returned by SoundCloud/yt-dlp. The app does not sort or deduplicate feed items.\n\nOn startup, the server begins warming any configured feed that has no metadata cache yet. Startup does not block on SoundCloud. If a feed is requested before its first successful metadata fetch, it returns `503 Service Unavailable` with a message that the metadata cache is warming.\n\nScheduled metadata refresh runs every `metadata.refresh_interval_hours`, defaulting to 24 hours. Scheduled work skips feeds refreshed within `metadata.refresh_recent_grace_minutes`, defaulting to 60 minutes. Metadata refreshes run one at a time globally, with manual refreshes taking priority over queued scheduled work. Logs include metadata refresh queue size when work is queued, started, completed, or fails.\n\nAppend `?refresh=1` to an RSS feed URL to manually refresh that feed. Manual refresh waits for the fetch to finish when no refresh is already running. If another refresh for that feed is already running, the response serves the current cache when available and includes `X-Yt-Dlp-Feed-Refresh: refreshing`; if no cache exists yet, it returns `503` with `X-Yt-Dlp-Feed-Refresh: warming`. Successful manual refresh responses include `X-Yt-Dlp-Feed-Refresh: ready`; failed refreshes with an existing cache serve stale metadata with `X-Yt-Dlp-Feed-Refresh: stale`.\n\nMetadata refreshes are atomic: the old cache remains in service until a new fetch succeeds and the JSON file is written with a temp-file-then-rename swap. If SoundCloud or yt-dlp fails and cached metadata exists, RSS continues to render from the last successful metadata. Last refresh errors are kept in memory for `index.json` and logs, not persisted to metadata cache files.\n\nRSS responses set `Last-Modified` and `\u003clastBuildDate\u003e` from the metadata cache's last successful refresh timestamp. Normal feed requests honor `If-Modified-Since` and may return `304 Not Modified`; explicit `?refresh=1` requests always evaluate the refresh path.\n\n`GET /index.json` includes `generated_at`, a summary, and one entry per configured feed with metadata cache state: `missing`, `warming`, `ready`, `refreshing`, `stale`, or `error`. The HTML index keeps things simple: RSS links, quiet per-feed refresh links, state, and `Last fetched (UTC)` timestamps.\n\n`GET /readyz` returns `200 OK` only when every configured feed has at least one successful metadata cache. `GET /healthz` remains a process-alive check and is the endpoint used by the Docker healthcheck.\n\n## Download Behavior\n\nAudio downloads prefer the best available M4A/AAC stream, falling back to yt-dlp's best available audio format when M4A is unavailable. yt-dlp fetches the source audio and ffmpeg transcodes the streamed bytes to AAC in an MP4/M4A container, so the app serves cached media consistently from `.m4a` URLs. If cached media exists and is still inside `media_ttl_minutes`, the server serves it directly. If `media_ttl_minutes` is `null`, existing cached media is considered reusable until another cleanup limit removes it. Otherwise, the first client request starts a new download, and concurrent requests for the same item share the same in-flight job.\n\nNo more than `downloads.max_concurrent` media downloads can run at once. The default is `3`. Requests joining an existing in-flight item do not count as new downloads. If the limit is reached for a new item, the server returns `503 Service Unavailable` with `Retry-After: 30`.\n\nCompleted cached media supports `HEAD`, `If-Modified-Since`, and byte `Range` requests, including `206 Partial Content` and `416 Range Not Satisfiable`. On a cold cache, plain `GET` requests stream a live fragmented MP4/AAC response as `200 OK` so browsers can begin playback while the download is still running. Cold `HEAD` and byte `Range` probes start or join the download and wait up to `downloads.probe_timeout_seconds` for the completed cache file, then return normal cached-file headers. If the probe wait times out, the response is `503 Service Unavailable` with `Retry-After: 30`.\n\nThis behavior targets private Apple Podcasts app use over a private network or Tailscale URL. Apple expects episode enclosures to support `HEAD` and byte-range requests, and it may probe audio before playback. Pocket Casts is explicitly unsupported for tailnet-only feeds because it performs server-side processing of public feeds; its backend cannot reliably fetch private Tailscale URLs.\n\nIf every client disconnects while a download is still in flight, `cache.disconnect_behavior` controls whether the server keeps or cancels the orphaned download:\n\n- `continue` keeps downloading and caches the completed file.\n- `cancel` stops `yt-dlp` immediately and removes the partial `.download.m4a`.\n- `delay_cancel` waits `disconnect_grace_seconds` for a reconnect, then cancels if no client is attached.\n\nCompleted media files live under `cache.data_dir` and are cleaned up by the background cache cleaner every five minutes. This is intentional: clients are expected to cache media after the first successful fetch.\n\nCache cleanup supports either or both of these limits:\n\n- `media_ttl_minutes` removes completed `.m4a` files older than the configured age. The default is `360`, or 6 hours. Set it to `null` to disable age-based cleanup.\n- `media_max_megabytes` keeps completed `.m4a` files under the configured total size, measured in MiB, by deleting the oldest files first. For example, `10240` allows about 10 GiB. Set it to `null` or omit it to disable size-based cleanup.\n\nWhen both limits are configured, TTL cleanup runs first, then the remaining completed media files are trimmed to `media_max_megabytes`. In-progress `.download.m4a` files are not counted against the size limit.\n\n## Security\n\nThe V1 security model is intentionally small and explicit. The server is meant\nfor a single trusted operator on a LAN, private network, VPN, or behind a real\naccess-control layer such as Tailscale or an auth proxy. It is not designed as a\npublic multi-tenant service.\n\nThe `users` section in the config is still meaningful even for a single\noperator. A configured user is the owner of one or more feed sources, and that\nuser name is part of every feed URL:\n\n```text\n/users/{user}/soundcloud/{name}/feed.xml\n```\n\nFor example, this config says that the authenticated app user `operator` owns a\nSoundCloud source labelled `primary`:\n\n```yaml\nusers:\n  - name: \"operator\"\n    services:\n      - kind: \"soundcloud\"\n        name: \"primary\"\n        profile_url: \"https://soundcloud.com/example-profile\"\n```\n\nBuilt-in auth uses one HTTP Basic username/password pair. If `auth.enabled` is\ntrue, the Basic auth username is treated as the configured app user name. A user\nauthenticated as `operator` can access `/users/operator/...`; requests for another\nconfigured user path return `403 Forbidden`. The HTML index and `index.json`\nare also filtered to the authenticated user.\n\n```yaml\nauth:\n  enabled: true\n  username: \"operator\"\n  password: \"change-me\"\n```\n\nIf `auth.enabled` is false, the app trusts the surrounding network or proxy and\ndoes not enforce per-user access. In that mode, anyone who can reach the server\ncan see the index and request any configured feed URL. The app logs a warning\nwhen auth is disabled while binding to `0.0.0.0`.\n\nV1 deliberately does not include browser sessions, OAuth, per-user password\nlists, admin screens, or multi-user isolation. Add those only if the deployment\nmodel changes from \"single trusted operator\" to \"shared service.\"\n\n## Development\n\nThis repo includes a Nix dev shell with Rust tooling:\n\n```sh\nnix develop\ncargo test\ncargo run -- --config config.example.yaml\n```\n\nTo see request traces and yt-dlp wrapper activity in the console, run with debug logging:\n\n```sh\ncargo run -- --debug\n```\n\nYou can also use `YT_DLP_FEED_DEBUG=true` for the same default debug filter, or set\n`RUST_LOG` directly for custom filtering.\n\nNormal tests should mock the yt-dlp boundary. Live SoundCloud tests should be opt-in so CI does not depend on network access or service availability.\n\n## Docker\n\nBuild the image:\n\n```sh\ndocker build -t yt-dlp-feed .\n```\n\nThe image installs `yt-dlp` from PyPI at build time so it gets the current\nupstream extractor code instead of the older Debian package. Docker can cache\nthat pip install layer, so normal rebuilds may keep the same build-time\n`yt-dlp` version. To refresh only the pip install layer without throwing away the\nwhole build cache, pass a new `YT_DLP_PIP_CACHE_BUSTER` value:\n\n```sh\ndocker build \\\n  --build-arg YT_DLP_PIP_CACHE_BUSTER=\"$(date +%Y-%m-%d)\" \\\n  -t yt-dlp-feed .\n```\n\nUse a daily, weekly, or release-specific cache-buster value depending on how\noften you want build-time `yt-dlp` freshness. Runtime startup and weekly pip\nupdate checks still handle normal container freshness after the image is built.\n\nRun the container with the tracked example config, a persistent Docker volume for\nmedia and metadata, and debug logging enabled:\n\n```sh\ndocker run --rm \\\n  -p 8080:8080 \\\n  -v \"$PWD/config.example.yaml:/config/config.yaml:ro\" \\\n  -v yt-dlp-feed-data:/data \\\n  yt-dlp-feed --config /config/config.yaml --data-dir /data --debug\n```\n\nThe image includes the latest build-time `yt-dlp` and Debian `ffmpeg`.\n`--data-dir /data` overrides `cache.data_dir` from the mounted config so cache\nfiles land on the `yt-dlp-feed-data` Docker volume.\n\nFor personal feed lists or secrets, copy the example to the ignored root\n`config.yaml`, edit it, and mount that file instead:\n\n```sh\ncp config.example.yaml config.yaml\ndocker run --rm \\\n  -p 8080:8080 \\\n  -v \"$PWD/config.yaml:/config/config.yaml:ro\" \\\n  -v yt-dlp-feed-data:/data \\\n  yt-dlp-feed --config /config/config.yaml --data-dir /data --debug\n```\n\nThe image includes pip-managed `yt-dlp` and `ffmpeg`. The local-run config binds\n`0.0.0.0:8080`; the Tailscale Compose config binds `127.0.0.1:8080` so raw HTTP\nis only reachable inside the sidecar network namespace. Both configs use `/data`\nfor the media and metadata cache.\n\nIn Docker, `yt-dlp` is installed from pip into the hardcoded virtualenv\n`/opt/yt-dlp`. The image build creates that virtualenv and fails if\n`/opt/yt-dlp/bin/yt-dlp` is not available. Docker sets\n`YT_DLP_FEED_PIP_TOOL_UPDATES_ENABLED=true`, which makes the server check for a\nnewer pip package on startup and then weekly. The matching config\nsection is named `pip_tool_updates` to make clear this is only for the\nDocker/pip-managed `yt-dlp` install, not a general app updater. Runtime update\nfailures are logged and the existing `/opt/yt-dlp` install stays in use. While\nan update runs, new metadata refreshes and uncached audio downloads wait; cached\nmedia, index pages, and health checks remain available.\n\n## Docker Compose With Tailscale\n\n`compose.yaml` runs the app behind a Tailscale sidecar. The app shares the\nsidecar network namespace and listens only on `127.0.0.1:8080`; Tailscale Serve\naccepts tailnet HTTP on port 80 and HTTPS on port 443, then proxies both to the\napp's internal port 8080. Tailscale ACLs provide the tailnet access control\nlayer, so the app's built-in Basic auth stays disabled in the provided container\nconfig.\n\nCreate the ignored runtime config from the tracked template. For Compose, set\n`server.bind: \"127.0.0.1:8080\"` so the app only listens inside the shared\nTailscale network namespace:\n\n```sh\ncp config.example.yaml config.yaml\n```\n\nThen edit `config.yaml` for your private feed list and Compose bind address.\n\nCreate a reusable or ephemeral auth key in Tailscale, then start the stack:\n\n```sh\nexport TS_AUTHKEY=\"tskey-auth-...\"\ndocker compose up -d --build\n```\n\nCompose mounts the ignored root `config.yaml` into the app container at\n`/config/config.yaml` and passes `--data-dir /data` so media and metadata live\non the Docker volume.\n\nOptional environment variables:\n\n- `TAILSCALE_HOSTNAME=yt-dlp-feed` changes the MagicDNS machine name.\n- `TS_EXTRA_ARGS=--advertise-tags=tag:container` is useful when authenticating\n  with an OAuth client secret or a tagged auth key.\n- `RUST_LOG=yt_dlp_feed=debug,yt_dlp=debug,tower_http=debug` enables verbose\n  server logs.\n\nThe Serve config lives at `docker/tailscale/serve.json` and uses\n`${TS_CERT_DOMAIN}` so Tailscale fills in the node's DNS name. It maps both\n`${TS_CERT_DOMAIN}:80` and `${TS_CERT_DOMAIN}:443` to\n`http://127.0.0.1:8080`. It keeps `AllowFunnel` set to `false`, which exposes\nthe service only to your tailnet. Set that value to `true` only if you\nintentionally want public Funnel ingress and your tailnet policy allows it.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fporkcharsui%2Fyt-dlp-feed","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fporkcharsui%2Fyt-dlp-feed","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fporkcharsui%2Fyt-dlp-feed/lists"}