{"id":16275284,"url":"https://github.com/vijinho/feedex","last_synced_at":"2025-10-29T15:31:16.645Z","repository":{"id":71221127,"uuid":"153395808","full_name":"vijinho/feedex","owner":"vijinho","description":"PHP CLI and REST tool to search URLs for RSS/ATOM feeds and extract them, optionally as JSON or OPML.","archived":false,"fork":false,"pushed_at":"2018-10-19T16:53:51.000Z","size":144,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2024-12-22T06:22:14.500Z","etag":null,"topics":["atom-feed","atom-feed-scraper","cli","json","json-api","opml","opml-to-json","php","php-cli","rest-api","rss","rss-feed","rss-feed-parsing","rss-feed-scraper","rss-feeds","webservice"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vijinho.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-17T04:34:01.000Z","updated_at":"2023-02-14T01:05:30.000Z","dependencies_parsed_at":"2023-05-23T18:45:28.083Z","dependency_job_id":null,"html_url":"https://github.com/vijinho/feedex","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vijinho%2Ffeedex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vijinho%2Ffeedex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vijinho%2Ffeedex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vijinho%2Ffeedex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vijinho","download_url":"https://codeload.github.com/vijinho/feedex/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238845335,"owners_count":19540326,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["atom-feed","atom-feed-scraper","cli","json","json-api","opml","opml-to-json","php","php-cli","rest-api","rss","rss-feed","rss-feed-parsing","rss-feed-scraper","rss-feeds","webservice"],"created_at":"2024-10-10T18:32:46.102Z","updated_at":"2025-10-29T15:31:16.224Z","avatar_url":"https://github.com/vijinho.png","language":"PHP","readme":"# feedex\n\nPHP CLI and WWW tool to extract and save feeds from URL(s)\n\n- Full-resolves URL before attempting to extract feed URLs\n- Extracts the URLs of RSS (1.0 and 2.0) and ATOM feeds associated to a page, as well as OPML outline documents with [nicolus/picofeed](https://github.com/nicolus/picoFeed)\n- Runs on the command-line\n- Can be called as a stand-alone webservice using the php command line built-in server\n- All messages when running with `--debug` or `--verbose` are to *stderr* to avoid interference with *stdout*\n- Can output the result if successful to *stdout*\n- Output file can be used as input when using options 'txt' and '[json](https://json.org/)', 'opml' and URLs can be re-checked with --force option.\n- Can output found feeds as an [opml](http://dev.opml.org/) file.\n- Can output data to markdown .md for easy sharing of podcast information (output-only --format)\n- Errors are output in JSON as 'errors' with just a bunch of strings\n\n```\n{\n    \"errors\": [\n        \"Unable to parse --date: next sunsaday\"\n    ]\n}\n```\n\n## Installation\n\n- `composer require nicolus/picofeed`\n- `composer dump`\n\n## Command-line options\n\n```\nUsage: php feedex.php\nExtract and save feeds from URL(s)\n(Specifying any other unknown argument options will be ignored.)\n\n        -h,  --help                        Display this help and exit\n        -v,  --verbose                     Run in verbose mode\n        -d,  --debug                       Run in debug mode (implies also -v, --verbose)\n        -u,  --url=\u003curl\u003e                   (Required or -i) URL to check for feeds)\n        -i   --input={filename}            (Required or -u) Text file of URLs, one-per-line to read in and process.\n        -c,  --clear                       (Optional) Clear-out URLs which have no feeds before writing output file.\n        -e,  --echo                        (Optional) Echo/output the result to stdout if successful\n        -f   --format={txt|json|php|opml|md}  (Optional) Output format for screen and filename: txt (default)|json|php(serialized)|opml|markdown\n             --filename={output}           (Optional) Filename for output data from operation\n             --force-check                 (Optional) Forcibly check URLs, even for those which already have feeds in the input file.\n             --skip-opml                   (Optional) Skip opml processing, just go to output, e.g. for generating markdown\n```\n\n## Example output Format\n\nThe script can take-in previous txt format output and re-use it as input, checking only urls where there are no existing feeds.\n\n### txt\n\nOnly stand-alone URL lines without following \u003cTAB\u003e feed lines are searched for feeds, unless `--force-check` is used which forces all URLs to be checked in the text file.\n\n\n```\nhttp://campaigntoabolishthebbc.blogspot.com/\n        http://campaigntoabolishthebbc.blogspot.com/feeds/posts/default\n        http://campaigntoabolishthebbc.blogspot.com/feeds/posts/default?alt=rss\n        https://www.blogger.com/feeds/7418166530762317285/posts/default\n\nhttp://cash-is-cool.com/\n        http://cash-is-cool.com/rss/articles.php\n        http://cash-is-cool.com/rss/twitter.php\n\nhttp://datastori.es/\n        http://datastori.es/comments/feed/\n        http://datastori.es/feed/\n        http://datastori.es/feed/m4a/\n        http://datastori.es/feed/mp3/\n        http://datastori.es/feed/podcast/\n\nhttp://eurofolkradio.com/\n        http://eurofolkradio.com/comments/feed/\n        http://eurofolkradio.com/feed/\n        http://eurofolkradio.com/feed/podcast\n\nhttp://grahamhancock.com/blog/\n        https://grahamhancock.com/blog/feed/\n```\n\n### json\n\n```\n{\n    \"http:\\/\\/campaigntoabolishthebbc.blogspot.com\\/\": [\n        \"http:\\/\\/campaigntoabolishthebbc.blogspot.com\\/feeds\\/posts\\/default\",\n        \"http:\\/\\/campaigntoabolishthebbc.blogspot.com\\/feeds\\/posts\\/default?alt=rss\",\n        \"https:\\/\\/www.blogger.com\\/feeds\\/7418166530762317285\\/posts\\/default\"\n    ],\n    \"http:\\/\\/cash-is-cool.com\\/\": [\n        \"http:\\/\\/cash-is-cool.com\\/rss\\/articles.php\",\n        \"http:\\/\\/cash-is-cool.com\\/rss\\/twitter.php\"\n    ],\n    \"http:\\/\\/datastori.es\\/\": [\n        \"http:\\/\\/datastori.es\\/comments\\/feed\\/\",\n        \"http:\\/\\/datastori.es\\/feed\\/\",\n        \"http:\\/\\/datastori.es\\/feed\\/m4a\\/\",\n        \"http:\\/\\/datastori.es\\/feed\\/mp3\\/\",\n        \"http:\\/\\/datastori.es\\/feed\\/podcast\\/\"\n    ],\n    \"http:\\/\\/eurofolkradio.com\\/\": [\n        \"http:\\/\\/eurofolkradio.com\\/comments\\/feed\\/\",\n        \"http:\\/\\/eurofolkradio.com\\/feed\\/\",\n        \"http:\\/\\/eurofolkradio.com\\/feed\\/podcast\"\n    ],\n    \"http:\\/\\/grahamhancock.com\\/blog\\/\": [\n        \"https:\\/\\/grahamhancock.com\\/blog\\/feed\\/\"\n    ],\n    \"http:\\/\\/greatgameindia.com\\/\": [\n        \"http:\\/\\/greatgameindia.com\\/comments\\/feed\\/\",\n        \"http:\\/\\/greatgameindia.com\\/feed\\/\",\n        \"http:\\/\\/greatgameindia.com\\/homepage\\/feed\\/\"\n    ],\n}\n```\n\n### OPML file to Markdown\n\n`php feedex.php --input=audio.opml -e -fmd --filename=podcasts.md`\n\nExample output, slightly modified: [urunu.com/blog/2018-10-18-podcasts](http://www.urunu.com/blog/2018-10-18-podcasts)\n\n# Examples\n\nFind feeds for URL http://example.com/blog/public outputting as 'txt'.\n\n`php feedex.php --url=http://example.com/blog/public --echo --format=txt --debug`\n\noutput:\n\n```\nhttp://example.com/blog/public\n        http://example.com/feed/\n        http://example.com/comments/feed/\n```\n\noutput as json, saving to a filename 'urls.json' and debugging enabled:\n\n`php feedex.php --url=http://example.com/blog/public --filename=urls.json --format=json`\n\noutput with full-debugging:\n\n```\n[D 1/1] OPTIONS:\nArray\n(\n    [debug] =\u003e 1\n    [echo] =\u003e 1\n    [help] =\u003e 0\n    [input] =\u003e 0\n    [url] =\u003e 1\n    [verbose] =\u003e 1\n)\n[V 1/1] OUTPUT_FORMAT: txt\n[D 1/1] Using dir: /Users/vijay/tmp\n[D 1/1] Using input filename:\n[D 1/1] Checking URL:\n        http://example.com/blog/public\n[D 1/2] Feeds found for URL:\n        http://example.com/blog/public/\nArray\n(\n    [0] =\u003e http://example.com/feed/\n    [1] =\u003e http://example.com/comments/feed/\n)\nhttp://example.com/blog/public\n        http://example.com/feed/\n        http://example.com/comments/feed/\n[D 1/2] Memory used (1/2) MB (current/peak).\n```\n\n## Example of reading in file of URLs\n\nRead in file of URLs and process, outputting txt, piping screen output (stderr and stdout) to less\n\n`php feedex.php --input=urls.txt --filename=results.txt --verbose --echo --clear 2\u003e\u00261 | less`\n\n```\n[V] OUTPUT_FORMAT: txt\n[V] Found 1 valid URL(s) in input file:\n        urls.txt\nArray\n(\n    [0] =\u003e http://example.com/blog/public/\n\n)\nhttp://example.com/blog/public/\n        http://example.com/feed/\n        http://example.com/comments/feed/\n```\n\n## Example of creating OPML file from extracted subscriptions\n\nReads in 'urls.txt', extracts feeds and writes to 'results.opml', debug mode piping all output including *stderr* to *stdout* into *less* viewer\n\n`php feedex.php --input=urls.txt --filename=results.opml --echo --format=opml --debug 2\u003e\u00261 | less`\n\nContents of 'results.opml':\n\n```\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003copml version=\"1.0\"\u003e\n  \u003chead\u003e\n    \u003ctitle\u003eFeedEx\u003c/title\u003e\n  \u003c/head\u003e\n  \u003cbody\u003e\n    \u003coutline type=\"rss\" text=\"Stories by RT Digital on Medium\" xmlUrl=\"https://medium.com/feed/@rtdublindigital\" title=\"Stories by RT Digital on Medium\" description=\"Stories by RT Digital on Medium\" htmlUrl=\"https://medium.com/@rtdublindigital?source=rss-dbca3f85c5a3------2\"/\u003e\n    \u003coutline type=\"rss\" text=\"ChinaPower Project\" xmlUrl=\"https://chinapower.csis.org/feed/\" title=\"ChinaPower Project\" description=\"Unpacking the complexity of China's rise\" htmlUrl=\"https://chinapower.csis.org/\"/\u003e\n\u003c!-- SNIP! --\u003e\n  \u003c/body\u003e\n\u003c/opml\u003e    \n```\n\n## Running as a webservice\n\n### Starting the service\n\n1. Start the PHP webserver with `php -S 127.0.0.1:12312`\n2. Browse the URL: http://127.0.0.1:12312/feedex.php with GET/POST parameters\n\nAccepted request input parameters: 'url=', 'format='\n\n### Webservice Example\n\nThis will only check a single URL.\n\n`http://127.0.0.1:12312/feedex.php?url=http://example.com/blog/public\u0026format=json`\n\nResult:\n\n```\n{\n    \"http:\\/\\/example.com\\/blog\\/public\": [\n        \"http:\\/\\/example.com\\/feed\\/\",\n        \"http:\\/\\/example.com\\/comments\\/feed\\/\"\n    ]\n}\n```\n\nUsing `format=text`\n\n`http://127.0.0.1:12312/feedex.php?url=http://example.com/blog/public\u0026format=txt`\n\n```\nhttp://example.com.com/\n\thttp://example.com.com/comments/feed/\n\thttp://example.com.com/feed/\n\thttp://example.com.com/home/feed/\n```\n\n----\nvijay@yoyo.org\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvijinho%2Ffeedex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvijinho%2Ffeedex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvijinho%2Ffeedex/lists"}