{"id":20592053,"url":"https://github.com/rien/sfeed","last_synced_at":"2026-04-25T03:25:40.787Z","repository":{"id":95852199,"uuid":"240740194","full_name":"rien/sfeed","owner":"rien","description":"A fork of sfeed (https://git.codemadness.org/sfeed) with some personal customizations","archived":false,"fork":false,"pushed_at":"2020-02-15T15:43:35.000Z","size":544,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-17T00:11:49.110Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rien.png","metadata":{"files":{"readme":"README","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-15T15:43:14.000Z","updated_at":"2020-02-15T15:44:31.000Z","dependencies_parsed_at":"2023-03-04T20:45:22.463Z","dependency_job_id":null,"html_url":"https://github.com/rien/sfeed","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rien%2Fsfeed","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rien%2Fsfeed/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rien%2Fsfeed/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rien%2Fsfeed/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rien","download_url":"https://codeload.github.com/rien/sfeed/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242223024,"owners_count":20092173,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T07:42:38.515Z","updated_at":"2026-04-25T03:25:40.735Z","avatar_url":"https://github.com/rien.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"sfeed\n-----\n\nRSS and Atom parser (and some format programs).\n\nIt converts RSS or Atom feeds from XML to a TAB-separated file. There are\nformatting programs included to convert this TAB-separated format to various\nother formats. There are also some programs and scripts included to import and\nexport OPML and to fetch, filter, merge and order feed items.\n\n\nBuild and install\n-----------------\n\n$ make\n# make install\n\n\nUsage\n-----\n\nInitial setup:\n\n\tmkdir -p \"$HOME/.sfeed/feeds\"\n\tcp sfeedrc.example \"$HOME/.sfeed/sfeedrc\"\n\nEdit the sfeedrc(5) configuration file and change any RSS/Atom feeds. This file\nis included and evaluated as a shellscript for sfeed_update, so it's functions\nand behaviour can be overridden:\n\n\t$EDITOR \"$HOME/.sfeed/sfeedrc\"\n\nor you can import existing OPML subscriptions using sfeed_opml_import(1):\n\n\tsfeed_opml_import \u003c file.opml \u003e \"$HOME/.sfeed/sfeedrc\"\n\nan example to export from an other RSS/Atom reader called newsboat and import\nfor sfeed_update:\n\n\tnewsboat -e | sfeed_opml_import \u003e \"$HOME/.sfeed/sfeedrc\"\n\nan example to export from an other RSS/Atom reader called rss2email (3.x+) and\nimport for sfeed_update:\n\n\tr2e opmlexport | sfeed_opml_import \u003e \"$HOME/.sfeed/sfeedrc\"\n\nUpdate feeds, this script merges the new items, see sfeed_update(1) for more\ninformation what it can do:\n\n\tsfeed_update\n\nFormat feeds:\n\nPlain-text list:\n\n\tsfeed_plain $HOME/.sfeed/feeds/* \u003e \"$HOME/.sfeed/feeds.txt\"\n\nHTML view (no frames), copy style.css for a default style:\n\n\tcp style.css \"$HOME/.sfeed/style.css\"\n\tsfeed_html $HOME/.sfeed/feeds/* \u003e \"$HOME/.sfeed/feeds.html\"\n\nHTML view with the menu as frames, copy style.css for a default style:\n\n\tmkdir -p \"$HOME/.sfeed/frames\"\n\tcd \"$HOME/.sfeed/frames\" \u0026\u0026 sfeed_frames $HOME/.sfeed/feeds/*\n\nTo automatically update your feeds periodically and format them in a way you\nlike you can make a wrapper script and add it as a cronjob.\n\nMost protocols are supported because curl(1) is used by default and also proxy\nsettings from the environment (such as the $http_proxy environment variable)\nare used.\n\nThe sfeed(1) program itself is just a parser that parses XML data from stdin\nand is therefore network protocol-agnostic. It can be used with HTTP, HTTPS,\nGopher, SSH, etc.\n\nSee the section \"Usage and examples\" below and the man-pages for more\ninformation how to use sfeed(1) and the additional tools.\n\n\nDependencies\n------------\n\n- C compiler (C99).\n- libc (recommended: C99 and POSIX \u003e= 200809).\n\n\nOptional dependencies\n---------------------\n\n- POSIX make(1) for Makefile.\n- POSIX sh(1),\n  used by sfeed_update(1) and sfeed_opml_export(1).\n- POSIX utilities such as awk(1) and sort(1),\n  used by sfeed_update(1).\n- curl(1) binary: https://curl.haxx.se/ ,\n  used by sfeed_update(1), but can be replaced with any tool like wget(1),\n  OpenBSD ftp(1) or hurl(1): https://git.codemadness.org/hurl/\n- iconv(1) command-line utilities,\n  used by sfeed_update(1). If the text in your RSS/Atom feeds are already UTF-8\n  encoded then you don't need this. For a minimal iconv implementation:\n  https://git.etalabs.net/cgit/noxcuse/tree/src/iconv.c\n- mandoc for documentation: https://mdocml.bsd.lv/\n\n\nOS tested\n---------\n\n- Linux (glibc+gcc, musl+gcc, clang).\n- OpenBSD (gcc, clang).\n- NetBSD\n- FreeBSD\n- Windows (cygwin gcc, mingw).\n- HaikuOS (using libbsd).\n\n\nArchitectures tested\n--------------------\n\namd64, ARM, aarch64, i386, SPARC64.\n\n\nFiles\n-----\n\nsfeed             - Read XML RSS or Atom feed data from stdin. Write feed data\n                    in TAB-separated format to stdout.\nsfeed_atom        - Format feed data (TSV) to an Atom feed.\nsfeed_frames      - Format feed data (TSV) to HTML file(s) with frames.\nsfeed_gopher      - Format feed data (TSV) to Gopher files.\nsfeed_html        - Format feed data (TSV) to HTML.\nsfeed_opml_export - Generate an OPML XML file from a sfeedrc config file.\nsfeed_opml_import - Generate a sfeedrc config file from an OPML XML file.\nsfeed_mbox        - Format feed data (TSV) to mbox.\nsfeed_plain       - Format feed data (TSV) to a plain-text list.\nsfeed_twtxt       - Format feed data (TSV) to a twtxt feed.\nsfeed_update      - Update feeds and merge items.\nsfeed_web         - Find urls to RSS/Atom feed from a webpage.\nsfeed_xmlenc      - Detect character-set encoding from XML stream.\nsfeedrc.example   - Example config file. Can be copied to $HOME/.sfeed/sfeedrc.\nstyle.css         - Example stylesheet to use with sfeed_html(1) and\n                    sfeed_frames(1).\n\n\nFiles read at runtime by sfeed_update(1)\n----------------------------------------\n\nsfeedrc - Config file. This file is evaluated as a shellscript in\n          sfeed_update(1).\n\nAtleast the following functions can be overridden per feed:\n\n- fetch: to use wget(1), OpenBSD ftp(1) or an other download program.\n- filter: to filter on fields.\n- merge: to change the merge logic.\n- order: to change the sort order.\n\nSee also the sfeedrc(5) man page documentation for more details.\n\nThe feeds() function is called to process the feeds. The default feed()\nfunction is executed concurrently as a background job in your sfeedrc(5) config\nfile to make updating faster. The variable maxjobs can be changed to limit or\nincrease the amount of concurrent jobs (8 by default).\n\n\nFiles written at runtime by sfeed_update(1)\n-------------------------------------------\n\nfeedname     - TAB-separated format containing all items per feed. The\n               sfeed_update(1) script merges new items with this file.\n               The format is documented in sfeed(5).\n\n\nFile format\n-----------\n\nman 5 sfeed\nman 5 sfeedrc\nman 1 sfeed\n\n\nUsage and examples\n------------------\n\nFind RSS/Atom feed urls from a webpage:\n\n\turl=\"https://codemadness.org\"; curl -L -s \"$url\" | sfeed_web \"$url\"\n\noutput example:\n\n\thttps://codemadness.org/blog/rss.xml\tapplication/rss+xml\n\thttps://codemadness.org/blog/atom.xml\tapplication/atom+xml\n\n- - -\n\nMake sure your sfeedrc config file exists, see sfeedrc.example. To update your\nfeeds (configfile argument is optional):\n\n\tsfeed_update \"configfile\"\n\nFormat the feeds files:\n\n\t# Plain-text list.\n\tsfeed_plain $HOME/.sfeed/feeds/* \u003e $HOME/.sfeed/feeds.txt\n\t# HTML view (no frames), copy style.css for a default style.\n\tsfeed_html $HOME/.sfeed/feeds/* \u003e $HOME/.sfeed/feeds.html\n\t# HTML view with the menu as frames, copy style.css for a default style.\n\tmkdir -p somedir \u0026\u0026 cd somedir \u0026\u0026 sfeed_frames $HOME/.sfeed/feeds/*\n\nView formatted output in your browser:\n\n\t$BROWSER \"$HOME/.sfeed/feeds.html\"\n\nView formatted output in your editor:\n\n\t$EDITOR \"$HOME/.sfeed/feeds.txt\"\n\n- - -\n\nExample script to view feed items in a vertical list/menu in dmenu(1). It opens\nthe selected url in the browser set in $BROWSER:\n\n\t#!/bin/sh\n\turl=$(sfeed_plain \"$HOME/.sfeed/feeds/\"* | dmenu -l 35 -i | \\\n\t\tsed -n 's@^.* \\([a-zA-Z]*://\\)\\(.*\\)$@\\1\\2@p')\n\ttest -n \"${url}\" \u0026\u0026 $BROWSER \"${url}\"\n\ndmenu can be found at: https://git.suckless.org/dmenu/\n\n- - -\n\nGenerate a sfeedrc config file from your exported list of feeds in OPML\nformat:\n\n\tsfeed_opml_import \u003c opmlfile.xml \u003e $HOME/.sfeed/sfeedrc\n\n- - -\n\nExport an OPML file of your feeds from a sfeedrc config file (configfile\nargument is optional):\n\n\tsfeed_opml_export configfile \u003e myfeeds.opml\n\n- - -\n\nThe filter function can be overridden in your sfeedrc file. This allows\nfiltering items per feed. It can be used to shorten urls, filter away\nadvertisements, strip tracking parameters and more.\n\n\t# filter fields.\n\t# filter(name)\n\tfilter() {\n\t\tcase \"$1\" in\n\t\t\"tweakers\")\n\t\t\tawk -F '\\t' 'BEGIN { OFS = \"\\t\"; }\n\t\t\t# skip ads.\n\t\t\t$2 ~ /^ADV:/ {\n\t\t\t\tnext;\n\t\t\t}\n\t\t\t# shorten link.\n\t\t\t{\n\t\t\t\tif (match($3, /^https:\\/\\/tweakers\\.net\\/[a-z]+\\/[0-9]+\\//)) {\n\t\t\t\t\t$3 = substr($3, RSTART, RLENGTH);\n\t\t\t\t}\n\t\t\t\tprint $0;\n\t\t\t}';;\n\t\t\"yt BSDNow\")\n\t\t\t# filter only BSD Now from channel.\n\t\t\tawk -F '\\t' '$2 ~ / \\| BSD Now/';;\n\t\t*)\n\t\t\tcat;;\n\t\tesac | \\\n\t\t\t# replace youtube links with embed links.\n\t\t\tsed 's@www.youtube.com/watch?v=@www.youtube.com/embed/@g' | \\\n\n\t\t\tawk -F '\\t' 'BEGIN { OFS = \"\\t\"; }\n\t\t\tfunction filterlink(s) {\n\t\t\t\t# protocol must start with http, https or gopher.\n\t\t\t\tif (match(s, /^(http|https|gopher):\\/\\//) == 0) {\n\t\t\t\t\treturn \"\";\n\t\t\t\t}\n\n\t\t\t\t# shorten feedburner links.\n\t\t\t\tif (match(s, /^(http|https):\\/\\/[^/]+\\/~r\\/.*\\/~3\\/[^\\/]+\\//)) {\n\t\t\t\t\ts = substr($3, RSTART, RLENGTH);\n\t\t\t\t}\n\n\t\t\t\t# strip tracking parameters\n\t\t\t\t# urchin, facebook, piwik, webtrekk and generic.\n\t\t\t\tgsub(/\\?(ad|campaign|pk|tm|wt)_([^\u0026]+)/, \"?\", s);\n\t\t\t\tgsub(/\u0026(ad|campaign|pk|tm|wt)_([^\u0026]+)/, \"\", s);\n\n\t\t\t\tgsub(/\\?\u0026/, \"?\", s);\n\t\t\t\tgsub(/[\\?\u0026]+$/, \"\", s);\n\n\t\t\t\treturn s\n\t\t\t}\n\t\t\t{\n\t\t\t\t$3 = filterlink($3); # link\n\t\t\t\t$8 = filterlink($8); # enclosure\n\n\t\t\t\tprint $0;\n\t\t\t}'\n\t}\n\n- - -\n\nThe fetch function can be overridden in your sfeedrc file. This allows to\nreplace the default curl(1) for sfeed_update with any other client to fetch the\nRSS/Atom data:\n\n\t# fetch a feed via HTTP/HTTPS etc.\n\t# fetch(name, url, feedfile)\n\tfetch() {\n\t\thurl -m 1048576 -t 15 \"$2\" 2\u003e/dev/null\n\t}\n\n- - -\n\nAggregate feeds. This filters new entries (maximum one day old) and sorts them\nby newest first. Prefix the feed name in the title. Convert the TSV output data\nto an Atom XML feed (again):\n\n\t#!/bin/sh\n\tcd ~/.sfeed/feeds/ || exit 1\n\n\tawk -F '\\t' -v \"old=$(($(date +'%s') - 86400))\" '\n\tBEGIN {\tOFS = \"\\t\"; }\n\tint($1) \u003e= old {\n\t\t$2 = \"[\" FILENAME \"] \" $2;\n\t\tprint $0;\n\t}' * | \\\n\tsort -k1,1rn | \\\n\tsfeed_atom\n\n- - -\n\nTo have a \"tail(1) -f\"-like FIFO stream filtering for new unique feed items and\nshowing them as plain-text per line similar to sfeed_plain(1):\n\nCreate a FIFO:\n\n\tfifo=\"/tmp/sfeed_fifo\"\n\tmkfifo \"$fifo\"\n\nOn the reading side:\n\n\t# This keeps track of unique lines so might consume much memory.\n\t# It tries to reopen the $fifo after 1 second if it fails.\n\twhile :; do cat \"$fifo\" || sleep 1; done | awk '!x[$0]++'\n\nOn the writing side:\n\n\tfeedsdir=\"$HOME/.sfeed/feeds/\"\n\tcd \"$feedsdir\" || exit 1\n\ttest -p \"$fifo\" || exit 1\n\n\t# 1 day is old news, don't write older items.\n\tawk -F '\\t' -v \"old=$(($(date +'%s') - 86400))\" '\n\tBEGIN { OFS = \"\\t\"; }\n\tint($1) \u003e= old {\n\t\t$2 = \"[\" FILENAME \"] \" $2;\n\t\tprint $0;\n\t}' * | sort -k1,1n | sfeed_plain | cut -b 3- \u003e \"$fifo\"\n\ncut -b is used to trim the \"N \" prefix of sfeed_plain(1).\n\n- - -\n\nFor some podcast feed the following code can be used to filter the latest\nenclosure url (probably some audio file):\n\n\tawk -F '\\t' 'BEGIN { latest = 0; }\n\tlength($8) {\n\t\tts = int($1);\n\t\tif (ts \u003e latest) {\n\t\t\turl = $8;\n\t\t\tlatest = ts;\n\t\t}\n\t}\n\tEND { if (length(url)) { print url; } }'\n\n- - -\n\nOver time your feeds file might become quite big. You can archive items of a\nfeed from (roughly) the last week by doing for example:\n\n\tawk -F '\\t' -v \"old=$(($(date +'%s') - 604800))\" 'int($1) \u003e old' \u003c feed \u003e feed.new\n\tmv feed feed.bak\n\tmv feed.new feed\n\nThis could also be run weekly in a crontab to archive the feeds. Like throwing\naway old newspapers. It keeps the feeds list tidy and the formatted output\nsmall.\n\n- - -\n\nConvert mbox to separate maildirs per feed and filter duplicate messages using the\nfdm program.\nfdm is available at: https://github.com/nicm/fdm\n\nfdm config file (~/.sfeed/fdm.conf):\n\n\tset unmatched-mail keep\n\n\taccount \"sfeed\" mbox \"%[home]/.sfeed/mbox\"\n\t\t$cachepath = \"%[home]/.sfeed/fdm.cache\"\n\t\tcache \"${cachepath}\"\n\t\t$maildir = \"%[home]/feeds/\"\n\n\t\t# Check if message is in in cache by Message-ID.\n\t\tmatch case \"^Message-ID: (.*)\" in headers\n\t\t\taction {\n\t\t\t\ttag \"msgid\" value \"%1\"\n\t\t\t}\n\t\t\tcontinue\n\n\t\t\t# If it is in the cache, stop.\n\t\t\tmatch matched and in-cache \"${cachepath}\" key \"%[msgid]\"\n\t\t\taction {\n\t\t\t\tkeep\n\t\t\t}\n\n\t\t# Not in the cache, process it and add to cache.\n\t\tmatch case \"^X-Feedname: (.*)\" in headers\n\t\taction {\n\t\t\t# Store to local maildir.\n\t\t\tmaildir \"${maildir}%1\"\n\n\t\t\tadd-to-cache \"${cachepath}\" key \"%[msgid]\"\n\t\t\tkeep\n\t\t}\n\nNow run:\n\n\t$ sfeed_mbox ~/.sfeed/feeds/* \u003e ~/.sfeed/mbox\n\t$ fdm -f ~/.sfeed/fdm.conf fetch\n\nNow you can view feeds in mutt(1) for example.\n\n- - -\n\nRead from mbox and filter duplicate messages using the fdm program and deliver\nit to a SMTP server. This works similar to the rss2email program.\nfdm is available at: https://github.com/nicm/fdm\n\nfdm config file (~/.sfeed/fdm.conf):\n\n\tset unmatched-mail keep\n\n\taccount \"sfeed\" mbox \"%[home]/.sfeed/mbox\"\n\t\t$cachepath = \"%[home]/.sfeed/fdm.cache\"\n\t\tcache \"${cachepath}\"\n\n\t\t# Check if message is in in cache by Message-ID.\n\t\tmatch case \"^Message-ID: (.*)\" in headers\n\t\t\taction {\n\t\t\t\ttag \"msgid\" value \"%1\"\n\t\t\t}\n\t\t\tcontinue\n\n\t\t\t# If it is in the cache, stop.\n\t\t\tmatch matched and in-cache \"${cachepath}\" key \"%[msgid]\"\n\t\t\taction {\n\t\t\t\tkeep\n\t\t\t}\n\n\t\t# Not in the cache, process it and add to cache.\n\t\tmatch case \"^X-Feedname: (.*)\" in headers\n\t\taction {\n\t\t\t# Connect to a SMTP server and attempt to deliver the\n\t\t\t# mail to it.\n\t\t\t# Of course change the server and e-mail below.\n\t\t\tsmtp server \"codemadness.org\" to \"hiltjo@codemadness.org\"\n\n\t\t\tadd-to-cache \"${cachepath}\" key \"%[msgid]\"\n\t\t\tkeep\n\t\t}\n\nNow run:\n\n\t$ sfeed_mbox ~/.sfeed/feeds/* \u003e ~/.sfeed/mbox\n\t$ fdm -f ~/.sfeed/fdm.conf fetch\n\nNow you can view feeds in mutt(1) for example.\n\n- - -\n\nConvert mbox to separate maildirs per feed and filter duplicate messages using\nprocmail(1).\n\nprocmail_maildirs.sh file:\n\n\tmaildir=\"$HOME/feeds\"\n\tfeedsdir=\"$HOME/.sfeed/feeds\"\n\tprocmailconfig=\"$HOME/.sfeed/procmailrc\"\n\n\t# message-id cache to prevent duplicates.\n\tmkdir -p \"${maildir}/.cache\"\n\n\tif ! test -r \"${procmailconfig}\"; then\n\t\techo \"Procmail configuration file \\\"${procmailconfig}\\\" does not exist or is not readable.\" \u003e\u00262\n\t\techo \"See procmailrc.example for an example.\" \u003e\u00262\n\t\texit 1\n\tfi\n\n\tfind \"${feedsdir}\" -type f -exec printf '%s\\n' {} \\; | while read -r d; do\n\t\tname=$(basename \"${d}\")\n\t\tmkdir -p \"${maildir}/${name}/cur\"\n\t\tmkdir -p \"${maildir}/${name}/new\"\n\t\tmkdir -p \"${maildir}/${name}/tmp\"\n\t\tprintf 'Mailbox %s\\n' \"${name}\"\n\t\tsfeed_mbox \"${d}\" | formail -s procmail \"${procmailconfig}\"\n\tdone\n\nProcmailrc(5) file:\n\n\t# Example for use with sfeed_mbox(1).\n\t# The header X-Feedname is used to split into separate maildirs. It is\n\t# assumed this name is sane.\n\n\tMAILDIR=\"$HOME/feeds/\"\n\n\t:0\n\t* ^X-Feedname: \\/.*\n\t{\n\t\tFEED=\"$MATCH\"\n\n\t\t:0 Wh: \"msgid_$FEED.lock\"\n\t\t| formail -D 1024000 \".cache/msgid_$FEED.cache\"\n\n\t\t:0\n\t\t\"$FEED\"/\n\t}\n\nNow run:\n\n\t$ procmail_maildirs.sh\n\nNow you can view feeds in mutt(1) for example.\n\n\nLicense\n-------\n\nISC, see LICENSE file.\n\n\nAuthor\n------\n\nHiltjo Posthuma \u003chiltjo@codemadness.org\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frien%2Fsfeed","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frien%2Fsfeed","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frien%2Fsfeed/lists"}