{"id":17280416,"url":"https://github.com/umputun/feed-master","last_synced_at":"2025-04-05T15:02:45.597Z","repository":{"id":24786782,"uuid":"28200463","full_name":"umputun/feed-master","owner":"umputun","description":"Pulls multiple podcast feeds (RSS) and republishes as a common feed, properly sorted and podcast-client friendly.","archived":false,"fork":false,"pushed_at":"2025-03-26T03:05:24.000Z","size":9286,"stargazers_count":125,"open_issues_count":10,"forks_count":26,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-29T14:04:10.078Z","etag":null,"topics":["podcast","rss","rss-aggregator","youtube"],"latest_commit_sha":null,"homepage":"https://feed-master.umputun.dev","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/umputun.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["umputun"]}},"created_at":"2014-12-18T20:33:05.000Z","updated_at":"2025-03-29T12:29:47.000Z","dependencies_parsed_at":"2024-05-10T04:27:39.792Z","dependency_job_id":"0476af94-22e9-47be-a356-00447da57a2b","html_url":"https://github.com/umputun/feed-master","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umputun%2Ffeed-master","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umputun%2Ffeed-master/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umputun%2Ffeed-master/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umputun%2Ffeed-master/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/umputun","download_url":"https://codeload.github.com/umputun/feed-master/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247353729,"owners_count":20925329,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["podcast","rss","rss-aggregator","youtube"],"created_at":"2024-10-15T09:20:32.007Z","updated_at":"2025-04-05T15:02:45.571Z","avatar_url":"https://github.com/umputun.png","language":"Go","funding_links":["https://github.com/sponsors/umputun"],"categories":["Go"],"sub_categories":[],"readme":"# Feed Master [![Build Status](https://github.com/umputun/feed-master/workflows/build/badge.svg)](https://github.com/umputun/feed-master/actions) [![Coverage Status](https://coveralls.io/repos/github/umputun/feed-master/badge.svg?branch=master)](https://coveralls.io/github/umputun/feed-master?branch=master) [![Docker Automated build](https://img.shields.io/docker/automated/umputun/feed-master)](https://hub.docker.com/r/umputun/feed-master)\n\n\nFeed-Master is a service that aggregates and publishes RSS feeds from multiple sources into a single feed. It normalizes the feeds to ensure that they are valid, compatible with podcast clients, and compliant with the RSS 2.0 specification. This allows users to access all of their desired content in a single, easy-to-use feed.\n\nIn addition to aggregating RSS feeds, Feed-Master can also publish updates to social media platforms such as Twitter and Telegram. For Telegram, the actual audio file is published, while for Twitter, a link to the original audio file is included in the tweet along with episode information like the title and description.\n\nFeed-Master also supports extracting audio from YouTube channels and using it to create the final feed. The service uses tools like  [yt-dlp](https://github.com/yt-dlp/yt-dlp) and [ffmpeg](https://www.ffmpeg.org/) to pull videos and extract the audio, respectively. In this mode, Feed-Master serves the audio files in addition to the generated RSS feed, providing users with even more options for accessing and consuming content.\n\n## Run in docker (short version)\n\n- Copy `docker-compose.yml` and adjust exposed port if needed\n- Create `etc/fm.yml` (samples provided in `_example`)\n- Start container with `docker-compose up -d feed-master`\n\n_example of docker-compose.yml available in [_example](https://github.com/umputun/feed-master/tree/master/_example)_\n\n## Main application parameters\n\n| Command line | Environment  | Default               | Description                           |\n|--------------|--------------|-----------------------|---------------------------------------|\n| db           | FM_DB        | `var/feed-master.bdb` | bolt db file                          |\n| conf         | FM_CONF      | `feed-master.yml`     | config file (yml)                     |\n| admin-passwd | ADMIN_PASSWD | `none` (disabled)     | admin password for protected endpoint |\n| dbg          | DEBUG        | `false`               | debug mode                            |\n\n\n## Configuration\n\nUsually, feed-master configuration is stored in `feed-master.yml` file. It is a yaml file with the following structure:\n\n```yaml\nfeeds:\n  yt-example: # feed name, can be repeated for multiple source feeds\n    title: Some cool channels # feed title\n    description: an example of youtube-based podcas # feed description\n    link: http://localhost:8080/feed/yt-example # link to the source site\n    language: \"ru-ru\" # feed language\n    author: \"Someone\" # feed author, default \"Feed Master\"\n    owner_email: \"blah@example.com\" # feed owner email, used in various services (i.e. spotify) to confirm RSS submission\n    image: images/yt-example.png # feed image, used in generated RSS as podcast thumbnail\n    filter: \n      - Title: \"something\" # filter from the feed, can be regexp or string\n      - Invert: true # invert filter (acts as \"only\"), default false\n    sources: # list of sources, each source is a name of and the source RSS feed\n      - {name: \"Точка\", url: http://localhost:8080/yt/rss/PLZVQqcKxEn_6YaOniJmxATjODSVUbbMkd}\n      - {name: \"Живой Гвоздь\", url: http://localhost:8080/yt/rss/UCWAIvx2yYLK_xTYD4F2mUNw}\n      - {name: \"Дилетант\", url: http://localhost:8080/yt/rss/UCuIE7-5QzeAR6EdZXwDRwuQ}\n\n\nyoutube: # youtube configuration, optional\n  base_url: http://localhost:8080/yt/media # base url for youtube media\n  dl_template: yt-dlp --extract-audio --audio-format=mp3 --audio-quality=0 -f m4a/bestaudio \"https://www.youtube.com/watch?v={{.ID}}\" --no-progress -o {{.FileName}} # template for youtube-dl\n  base_chan_url: \"https://www.youtube.com/feeds/videos.xml?channel_id=\" # base url for youtube channel\n  base_playlist_url: \"https://www.youtube.com/feeds/videos.xml?playlist_id=\" # base url for youtube playlist\n  update: 60s # update interval for youtube feeds\n  skip_shorts: 120s # skip videos (and audios) shorter than this value, optional\n  max_per_channel: 2 # max number of the latest videos per yt channel to download and process\n  files_location: ./var/yt # location for downloaded youtube files\n  rss_location: ./var/rss # location for generated youtube channel's RSS\n  channels: # list of youtube channels to download and process\n      # id: channel or playlist id, name: channel or playlist name, type: \"channel\" or \"playlist\", \n      # lang: language of the channel, keep: override default keep value\n      # filter: criteria to include and exclude videos, can be regex\n      - {id: UCWAIvx2yYLK_xTYD4F2mUNw, name: \"Живой Гвоздь\", lang: \"ru-ru\"}\n      - {id: UCuIE7-5QzeAR6EdZXwDRwuQ, name: \"Дилетант\", type: \"channel\", lang: \"ru-ru\", \"keep\": 10}\n      - {id: PLZVQqcKxEn_6YaOniJmxATjODSVUbbMkd, name: \"Точка\", type: \"playlist\", lang: \"ru-ru\", filter: {include: \"ТОЧКА\", exclude: \"STAR'цы Live\"}} \n  ytdlp_update: \n    interval: 24h # update interval for yt-dlp. If not set, yt-dlp will not be updated \n    command: \"pip3 install --break-system-packages -U yt-dlp\" # update yt-dlp command\n\nsystem: # system configuration\n  update: 1m # update interval for checking source feeds\n  http_response_timeout: 30s # http response timeout\n  max_per_feed: 10 # max items per feed to be processed and inclueded in the final RSS\n  max_total: 50 # max total items to be included in the final RSS\n  max_keep: 1000 # max items to be kept in the internal database \n  base_url: http://localhost:8080 # base url for the generated RSS and media files\n```\n\n_see [examples](https://github.com/umputun/feed-master/tree/master/_example/etc) for more details._\n\n### Single-feed configuration\n\nFor a very simple configuration, command-line only configuration is available. In this case only a single source feed is allowed and yt processing is disabled.  The command-line configuration is the following:\n\n| Command line     | Environment         | Default                    | Description                               |\n|------------------|---------------------|----------------------------|-------------------------------------------|\n| feed             | FM_FEED             |                            | single feed, overrides config             |\n| update-interval  | UPDATE_INTERVAL     | `1m`                       | update interval, overrides config         |\n| telegram_chan    | TELEGRAM_CHAN       |                            | single telegram channel, overrides config |\n\nName of the channel or numeric ChatID could be used, [here](https://remark42.com/docs/configuration/telegram/#notifications-for-administrators) are the instructions on obtaining the ChatID. To be able to post messages to the channel, bot must be added as administrator with Post permission.\n\nAll this command-line mode is good for - process a single feed, send a telegram message and send a tweet on each new item.\n\n### Notifications\n\nIn both configuration modes, user can specify a list of telegram and twitter accounts to be notified.\n\n| Command line     | Environment         | Default                    | Description                               |\n|------------------|---------------------|----------------------------|-------------------------------------------|\n| telegram_server  | TELEGRAM_SERVER     | `https://api.telegram.org` | telegram bot api server                   |\n| telegram_token   | TELEGRAM_TOKEN      |                            | telegram token                            |\n| telegram_timeout | TELEGRAM_TIMEOUT    | `1m`                       | telegram timeout                          |\n| consumer-key     | TWI_CONSUMER_KEY    |                            | twitter consumer key                      |\n| consumer-secret  | TWI_CONSUMER_SECRET |                            | twitter consumer secret                   |\n| access-token     | TWI_ACCESS_TOKEN    |                            | twitter access token                      |\n| access-secret    | TWI_ACCESS_SECRET   |                            | twitter access secret                     |\n| template         | TEMPLATE            | `{{.Title}} - {{.Link}}`   | twitter message template                  |\n\n\n## API\n\n_See [requests.http](https://github.com/umputun/feed-master/blob/master/requests.http)_\n\n### public endpoints\n\n- `GET /rss/{name}` - returns feed-set for given feed name\n- `GET /list` - returns list of feed-sets (json)\n- `GET /image/{name}` - returns image for given feed name\n- `GET /feed/{name}/sources` - returns list of sources for given feed name\n- `GET /yt/rss/{channel}` - return RSS feed for given youtube channel\n\n### admin endpoints\n\n- `POST /yt/rss/generate` - regenerate RSS feed for all youtube channels\n- `DELETE /yt/entry/{channel}/{video}` - delete youtube entry from internal database and remove it from RSS feed\n\n## Web UI\n\nWeb UI shows a list of items from generated RSS. It is available on `/feeds` or, for the particular output feed on `/feed/{name}`\n\n## Telegram notifications details\n\nBy default, (with only `TELEGRAM_TOKEN` provided) Telegram notifications will be sent using standard Bot API which has a limit of [50Mb](https://core.telegram.org/bots/api#sending-files) for audio file upload.\n\nYou can provide `TELEGRAM_API_ID` and `TELEGRAM_API_HASH` (from [here](https://my.telegram.org/apps)) to `telegram-bot-api` service in docker-compose.yml and uncomment `TELEGRAM_SERVER` for `feed-master`, then it would use the local bot api server to raise audio file upload limit from 50Mb [to 2000Mb](https://core.telegram.org/bots/api#using-a-local-bot-api-server).\n\nTo use local telegram bot api server, use `docker-compose up -d` command instead of `docker-compose up -d feed-master`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fumputun%2Ffeed-master","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fumputun%2Ffeed-master","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fumputun%2Ffeed-master/lists"}