Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/shouya/rss-funnel

RSS multi-tool
https://github.com/shouya/rss-funnel

rss rss-aggregator rss-feed yahoo-pipes

Last synced: 2 months ago
JSON representation

RSS multi-tool

Awesome Lists containing this project

README 

        
* RSS Funnel



/This project is in the early stage of development. Please expect bugs and frequent breaking changes./



RSS Funnel is a modular RSS processing pipeline system. It can manipulate RSS/Atom sources in various ways.



- Fetch full-text content

- Generate an RSS feed from an HTML page

- Remove unwanted elements or texts

- Filter articles matching keywords

- Highlight keywords

- Split aggregation (e.g. top 10 links of day) into individual articles

- Combine multiple feeds into one

- Convert between feed formats

- Run arbitrary JavaScript transformation

- Apply image proxy

- [[https://github.com/shouya/rss-funnel/wiki/Filter-config][And more!]]



The tool comes with a web UI for inspecting the feed and the result of each filter, all within a single statically-built binary with no external dependencies.



[[https://rss-funnel-demo.fly.dev/][Try out the live demo!]]



** Installation



You can use the Docker image ([[https://github.com/shouya/rss-funnel/pkgs/container/rss-funnel][latest version]]) in your =docker-compose.yaml=:



#+begin_src yaml

version: "3.8"

services:

  rss-funnel:

    image: ghcr.io/shouya/rss-funnel:latest

    ports:

      - 4080:4080

    volumes:

      - ./funnel.yaml:/funnel.yaml

    environment:

      RSS_FUNNEL_CONFIG: /funnel.yaml

      RSS_FUNNEL_BIND: 0.0.0.0:4080

#+end_src



The web UI will be available at =http://localhost:4080=.



Alternatively, you can build it directly from source:



#+begin_src bash

git clone https://github.com/shouya/rss-funnel.git

cd rss-funnel



# First, build the front-end assets

cd inspector && npm i && npm run build && cd ..



# Then, build the binary

cargo build --release

#+end_src



If you prefer not to build from source, you can download the pre-built artifacts from the [[https://github.com/shouya/rss-funnel/releases][release page]].



** Usage



To use =rss-funnel=, you need to supply a configuration file in YAML format. Here's an example configuration:



#+begin_src yaml

endpoints:

  - path: /tokio-blog.xml

    note: Full text of Tokio blog

    source: https://tokio.rs/_next/static/feed.xml

    filters:

      - full_text: {}

      - simplify_html: {}



  - path: /hackernews.xml

    note: Full text of Hacker News

    source: https://news.ycombinator.com/rss

    filters:

      - full_text:

          simplify: true

          append_mode: true

#+end_src



Save the above file as =/path/to/funnel.yaml= and run the following command:



#+begin_src

rss-funnel -c /path/to/funnel.yaml server

#+end_src



You can optionally specify the bind address and port (default =127.0.0.1:4080=). Detailed usage can be found in the =--help= output.



The endpoints like =http://127.0.0.1:4080/tokio-blog.xml= should serve the filtered feeds.



** Documentation



- [[https://github.com/shouya/rss-funnel/wiki/Config-syntax][Config syntax]] contains information about each component in the configuration file.

- [[https://github.com/shouya/rss-funnel/wiki/Filter-config][Filter config]] details the configuration for all filters.

- [[https://github.com/shouya/rss-funnel/wiki/Cookbook][Cookbook]] contains snippets on how to use =rss-funnel= for several practical applications.

- [[https://github.com/shouya/rss-funnel/wiki/Deployment][Deployment]] includes examples of docker-compose and kubernetes definitions.