{"id":18878249,"url":"https://github.com/uriel1998/agaetr","last_synced_at":"2025-04-14T18:32:09.685Z","repository":{"id":52788900,"uuid":"232619571","full_name":"uriel1998/agaetr","owner":"uriel1998","description":"Modular scripts to take text, images, and links from RSS feeds and push to social media","archived":false,"fork":false,"pushed_at":"2025-03-21T19:49:55.000Z","size":1418,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-21T20:28:11.801Z","etag":null,"topics":["python","rss","shell","social-media"],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uriel1998.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-08T17:28:06.000Z","updated_at":"2025-03-21T19:49:59.000Z","dependencies_parsed_at":"2025-03-14T11:24:18.222Z","dependency_job_id":"d0f99fe4-bfc8-473d-b78a-cad999174d3e","html_url":"https://github.com/uriel1998/agaetr","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uriel1998%2Fagaetr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uriel1998%2Fagaetr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uriel1998%2Fagaetr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uriel1998%2Fagaetr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uriel1998","download_url":"https://codeload.github.com/uriel1998/agaetr/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248936850,"owners_count":21186115,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python","rss","shell","social-media"],"created_at":"2024-11-08T06:25:24.140Z","updated_at":"2025-04-14T18:32:09.675Z","avatar_url":"https://github.com/uriel1998.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# agaetr\n\nA modular system to take a list of RSS feeds, process them, and send them to \nsocial media with images, content warnings, and sensitive image flags when \navailable. \n\n![agaetr logo](https://raw.githubusercontent.com/uriel1998/agaetr/master/agaetr-open-graph.png \"logo\")\n\n## Contents\n 1. [About](#1-about)\n 2. [License](#2-license)\n 3. [Prerequisites](#3-prerequisites)\n 4. [Installation](#4-installation)\n 5. [Services Setup](#5-services-setup)\n 6. [Feeds Setup](#6-feeds-setup)\n 7. [Feed Preprocessing](#7-feed-preprocessing)\n 8. [Feed Options](#8-feed-options)\n 9. [Advanced Content Warning](#9-advanced-content-warning)\n 10. [Usage](#10-usage)\n 11. [Other Files](#11-other-files)\n 12. [TODO](#12-todo)\n\n***\n\n## 1. About\n\n`agaetr` is a modular system made up of several small programs designed to take \ninput (particularly RSS feeds) and then share them to various social media outputs.\n\nThis system is designed for *single user* use, as API keys are required.\n\nTested with feeds from:\n\n* [dlvr.it](https://dlvrit.com/) \n* [shaarli](https://github.com/shaarli/Shaarli) instances (see note below)\n* [Wordpress](https://wordpress.org/) (with preprocessing script)\n* [TT-RSS](https://tt-rss.org/) (with preprocessing script)\n* [Trakt.tv](https://trakt.tv) \n* [DeviantArt](https://www.deviantart.com)\n* [YouTube](https://youtube.com) (particularly public playlists, like favorites)\n* [UPI](https://rss.upi.com/news/news.rss)\n\n`agaetr` can also *deobfuscate* incoming links and optionally shorten outgoing links.\n\nThis was created because pay services are expensive, and other options are \neither limited or subject to frequent bitrot.\n\nThe modular structure is specifically designed so that it should be easy to \ncreate a new module for additional services, as it relies on other programs \nto do most of the posting. Therefore, if one posting tool dies, another can be \nfound and (relatively) easily swapped in without changing your whole setup.\n\n`agaetr` is an anglicization of ágætr, meaning \"famous\".\n\nSpecial thanks to Alvin Alexander's [whose post](https://alvinalexander.com/python/python-script-read-rss-feeds-database) got me on the right track.\n\n## 2. License\n\nThis project is licensed under the Apache License. For the full license, see `LICENSE`.\n\n## 3. Prerequisites\n\nThese are probably already installed or are easily available from your distro on\nlinux-like distros:  \n \n* [python3](https://www.python.org)  \n* [bash](https://www.gnu.org/software/bash/)  \n* [wget](https://www.gnu.org/software/wget/)  \n* [gawk](http://www.gnu.org/software/gawk/manual/gawk.html)  \n* [grep](http://en.wikipedia.org/wiki/Grep)  \n* [curl](http://en.wikipedia.org/wiki/CURL)  \n* [sed](https://en.wikipedia.org/wiki/Sed)\n* [detox](https://linux.die.net/man/1/detox)\n* [xmlstarlet](https://xmlstar.sourceforge.net/)\n* [imagemagick](https://www.imagemagick.org/)\n* [lynx](https://lynx.invisible-island.net/)\n* [pandoc](https://pandoc.org/)\n* [html-xml-utils](https://www.w3.org/Tools/HTML-XML-utils/README)\n* [pipx](https://pipx.pypa.io/stable/installation/)\n\nOn Debian/Ubuntu systems, you should be able to snag all these with:\n\n`sudo apt install xmlstarlet html-xml-utils pandoc lynx imagemagick detox python3 bash wget gawk grep curl python3`\n\n### Python dependencies\n\nIt is recommended that you use `pipx` and your package installer's python packages. \nIf you do not, you should create a virtualenv for this project, as there are a number \nof python dependencies.  \n\n* `sudo apt install python3-appdirs python3-configargparse python3-requests python3-feedparser python3-bs4`\n\nOR\n\n* `pip install -r requirements.txt` \n\nOR\n\n* `pip install appdirs`  \n* `pip install configparser`  \n* `pip install beautifulsoup4`  \n* `pip install feedparser`  \n* `pip install requests`\n\n## 4. Installation\n\n### Manual installation\n\nYou will need some variety of posting mechanism and optionally an URL \nshortening mechanism. See [Services Setup](#5-services-setup) for details.\n\n* `mkdir -p $HOME/.config/agaetr`\n* `mkdir -p $HOME/.local/agaetr`\n* Edit `agaetr.ini` (see instructions below)\n* `cp $PWD/agaetr.ini $HOME/.config/agaetr`\n* `sudo chmod +x $PWD/agaetr_parse.py`\n* `sudo chmod +x $PWD/agaetr_send.sh`\n* `sudo chmod +x $PWD/agaetr.sh`\n* `sudo chmod +x $PWD/muna.sh`\n\n\n\nAny service you would like to use needs to have a symlink made from the \"avail\" \ndirectory to the \"enabled\" directory. For example:\n\n* `ln -s $PWD/short_avail/yourls.sh $PWD/short_enabled/yourls.sh`\n\nYou may use as many \"out\" options as you care to; choose 0 or 1 shortening \nservices.\n\n## 5. Services Setup\n\n### Services Not Covered Here\n\nOne of the reason there are multiple different example service wrappers \n(and that they are written in pretty straightforward BASH scripting) \nis so that future users (including myself) can use them as templates or \nexamples for other tools or new services with as little fuss as possible \nand without requiring a great deal of knowledge on the part of the user. \n\nIf you create one for another service, please contact me so I can merge it in \n(this repository is mirrored multiple places).\n\n\n### Shorteners and Archivers\n\n#### YOURLS  \n\nGo to your already functional [YOURLS](https://yourls.org/) instance.  Get the \nAPI key (secret signature token) from the `Tools` page of your admin interface. \nPlace the URL of your instance and API key into `agaetr.ini`.  \n\n`yourls_api =`  \n`yourls_site =`  \n\n#### ARCHIVE.IS\n\nInstall the `archiveis` cli tool from [https://github.com/palewire/archiveis](https://github.com/palewire/archiveis), \nor if you have pipx, by `pipx install archiveis`.\n\nFind the location of the binary by typing `which archiveis`, then place that in \nthe ini file. *Placing the binary location turns on archiving all links*.\n\n#### WAYBACK MACHINE\n\nInstall the `waybackpy` cli tool from [https://pypi.org/project/waybackpy/](https://pypi.org/project/waybackpy/), \nor if you have pipx, by `pipx install waybackpy`.\n\nFind the location of the binary by typing `which waybackpy`, then place that in \nthe ini file. *Placing the binary location turns on archiving all links*.\n\n#### All Archivers\n\nPlace into `agaetr.ini` whether your want archived links to `replace` the description, to `append` them to the description, or to `ignore` them.\n\n`ArchiveLinks = append`\n\n### Outbound parsers\n\n* Shaarli \n* Wallabag\n* Mastodon\n* Bluesky\n* Pixelfed\n* Tumblr\n* RSS     \n* Email            \n* Daily Post\n\nNote that each service has its own line in `agaetr.ini`.  Leave blank any \nyou are not using; adding additional services should follow the pattern shown.  \n\n### Shaarli (output)\n\nInstall and set up the [Shaarli-Client](https://github.com/shaarli/python-shaarli-client). \nIf you already have pipx, this can be as simple as `pipx install shaarli-client`.\nMake sure you set up the configuration file for the client properly. Place the \nlocation of the binary into `agaetr.ini`.\n\nIf no configuration is specified in the ini, the default config in `$XDG_DATA_HOME/shaarli/client.ini` will be used.  \n\n#### Wallabag (output)\n\nInstall and set up [Wallabag-cli](https://github.com/Nepochal/wallabag-cli). \nIf you already have pipx, this can be as simple as `pipx install wallabag-client`.\nPlace the location of the binary into `agaetr.ini`.\n\nNote that shorteners and wallabag don't get along all the time.\n\n#### Mastodon via toot  \n\nInstall and set up [toot](https://github.com/ihabunek/toot/).  \nIf you already have pipx, this can be as simple as `pipx install toot`.\nPlace the location of the binary into `agaetr.ini`.\n\nSpecify the account to use (see all accounts with `toot auth`) in `agaetr.ini`:\n\n`mastodon = username@mastodon.example.com`\n\n#### Bsky via bsky \n\nWe use [bsky](https://github.com/mattn/bsky) for Bluesky. You can download the \nbinary from the [releases](https://github.com/mattn/bsky/releases) page.\n\nInstall as per the directions, place the location of the binary into `agaetr.ini`.\n\nNote that if you're specifying an alternate (self-hosted) AT host, that should go *before* \nthe handle and password when performing the `login` command.\n\n#### Pixelfed via toot  \n\nInstall and set up [toot](https://github.com/ihabunek/toot/).  \nIf you already have pipx, this can be as simple as `pipx install toot`.\nPlace the location of the binary into `agaetr.ini` if you have not already for \nMastodon.  Create a login for pixelfed as well (`toot login`).  Note the pixelfed \naccount name to send to using `toot auth`.  Place this in `agaetr.ini` like so:\n\n`pixelfed = username@pixelfed.example.com`\n\nThis sender will *only* send if there is an image retrieved. Content warnings \nand the like are applied.\n\n#### RSS via XMLStarlet\n\nInstall [XMLStarlet](https://xmlstar.sourceforge.net/) which may be as easy as \n`sudo apt install xmlstarlet` on Debian/Ubuntu.\nIn `agaetr.ini` specify the path for the resulting xml file and the link where it \nwill eventually be accessed from:\n```\nrss_output_path = /full/path/including/filename.xml\nself_link = https://location.of.xml.example.com/output.xml\n\n```\n#### Email\n\nFill in the appropriate bits in `agaetr.ini`.  The field `email_from` should be \none valid email address, the field `email_to` may contain multiple addresses separated \nby a comma.\n\nsmtp_server =\nsmtp_port =\nsmtp_username =\nsmtp_password = \nemail_from = \nemail_to = \n\n#### Tumblr \n\n* IMPORTANT: This module requires `go` and `npm` for `gotumblr` and `picgo`, respectively. \n\nInstall [gotumblr](https://github.com/admacro/gotumblr) by installing go and \ndownloading the repository.  Get the appropriate keys as per its README.  Put the \nfull path to `gotumblr.go` and `text.md` in `agaetr.ini`. Please note that these \ntwo files should be in **the same** directory.\n\n* IMPORTANT: If you are wanting to post locally-hosted images in your posts (e.g. if \nyou're using `hooty`, below, or something similar), you will need to install \n[picgo](https://github.com/PicGo/PicGo-Core) as well.  `gotumblr` only posts \ntext posts, so we have to host the image elsewhere. Put the full path to `picgo` \nin `agaetr.ini`.\n\n```\ngotumblr = /path/to/gotumblr.go\ntextmd = /path/to/text.md\npicgo = /path/to/picgo\n```\n\nAdditionally, in `agaetr.ini` you will need to set up these values (see the documentation \nfor `gotumblr` for the values).\n\n```\nTUMBLR_BLOG_NAME=blogname\nTUMBLR_CONSUMER_KEY=see_readme_for_gotumblr\nTUMBLR_CONSUMER_SECRET=see_readme_for_gotumblr\nTUMBLR_OAUTH_TOKEN=see_readme_for_gotumblr\nTUMBLR_OAUTH_TOKEN_SECRET=see_readme_for_gotumblr\n```\n\n* IMPORTANT: If you want to use hashtags, you will need to drop my replacement of `gotumblr`, \n`gotumblr_ss.go`, *alongside* the original and update `agaetr.ini` appropriately.  \n\nIt makes the second line of the text file the hashtags of the post. It currently adds \nand empty hashtag, and I don't know why.\n\n#### Daily Post\n\nIn `agaetr.ini` set up your *directory* for daily posts.\n\n`daily_post = /path/to/directory`\n\nIt will create a markdown formatted text file of your links for each day, e.g.\n\n`/path/to/dailypost/YYYYMMDD.md`\n\nAdditional processing and formatting is up to you. If you want YAML frontmatter or the \nlike, you'll need to edit the sending script.\n\n\n## 6. Feeds Setup\n\nInformation about your feeds goes into `agaetr.ini`.  Each feed is marked by a\nheader line `[Feed#]` with a different number for each feed. \n\nIf a feed is being preprocessed (see below) or you have the RSS as an \nXML file, you can put the filename directly into `agaetr.ini`, **RELATIVE TO `$XDG_CONFIG_HOME/agaetr`**.  \n\nThe options are explained in [Feed Options](#8-feed-options) below.\n\nFor example:\n\n```\n[Feed1]\nurl = /relative_path_to_xml_file/my_xml_file.xml\nsensitive = yes\nContentWarning = no\nGlobalCW = \n\n[Feed2]\nurl = https://ideatrash.net/feed\nsensitive = yes\nContentWarning = yes\nGlobalCW = ideatrash\n\n```\n\n## 7. Feed Preprocessing\n\nWhile RSS is *supposed* to be a standard... it isn't. Too often there are \nunusual or irregular elements in an RSS feed.\n\nWhile I've tried to make some of the more popular \"odd\" feeds - like YouTube \nand DeviantArt - work properly inside of `agaetr_parse.py`, I cannot check \nor code for every possibility. \n\nIf you have a feed with some unruly elements - such as the \"Read more...\" that \nWordpress loves to put in my own feed, or how the \"published articles\" feed from \ntt-rss uses `\u003cupdated\u003e` instead of `\u003cpubDate\u003e`, there is an option to put in a `sed`\nscript or the like in `agaetr.ini`.  In this case, `src` is where the feed \noriginally comes from, and `url` is where the processed feed goes to be picked up\nby `agaetr_parse.py`.  These three must be in this order: `src`, `cmd`, `url`, one per line, as below.\n\n```\n[Feed4]\nsrc = https://ideatrash.net/feed\ncmd = sed 's/\u003cdiv class=\"more-link-wrapper\"\u003e.*\\]\\]\u003e\u003c\\/description\u003e/\\]\\]\\\u003e\u003c\\/description\u003e/g'\nurl = /relative/path/to/xml/filename.xml\n```\n\nAgain, you can specify the output filename for the feed location in \n`agaetr.ini`. This allows the use of the preprocessor without changing \nanything else.\n\nThis isn't meant to be a comprehensive \"fix\" so much as an example to \nhelp get you started with your own unruly feeds.\n\n### Note about Shaarli feeds\n\nPlease note that if you're importing a Shaarli feed, you will probably want to \ntoggle \"RSS direct links\" in the Preferences menu, otherwise it links directly \nto your Shaarli, not to the thing your Shaarli is pointing at.\n\n## 8. Feed Options\n\nThere are two places to configure feed options in `agaetr.ini`. \n\nIn the default block, you can define the (duh) default options. For \nsocial media accounts that support content warnings and sensitive image \nmarkers (like Mastodon) you can configure if images are \"sensitive\" by \ndefault, whether the posts from `agaetr` are marked with content warning \nby default, and what strings (in the post title or tags) will *always* \ntrigger the content warning.\n\n*Note*: Images are marked as sensitive if the content warning is triggered.\n\n```\nSensitive = no\nContentWarning = no\nGlobalCW = RSS-fed\n# These ALWAYS trigger a content warning\nfilters =\n#filters = politics blog sex bigot supremacist nazi climate\n```\n\nIn each feed's configuration, you can choose the default for *that feed*. \nFor example, in *Feed1* below, images are marked sensitive, but there is *not* \na content warning for any items in the feed.  \n\nIn *Feed2* below, all images are marked sensitive and all posts are marked with a \ncontent warning of \"ideatrash\".  It will also mark the content warning with \nany other tags the post may have.\n\nIn *Feed3* below, images are only marked sensitive if they are triggered by a \ncontent warning (from the \"filter\" line in the *Default* section), otherwise \nthere are no content warnings and images are presented normally.\n\n```\n[Feed1]\nurl = /feeds/ideatrash_parsed.xml\nsensitive = yes\nContentWarning = no\nGlobalCW = \n\n[Feed2]\nurl = https://ideatrash.net/feed\nsensitive = yes\nContentWarning = yes\nGlobalCW = ideatrash\n\n[Feed3]\nurl = https://ideatrash.net/feed\nsensitive = no\nContentWarning = yes\nGlobalCW = \n\n# If a path, it must be relative to $XDG_DATA_HOME/agaetr and begin with a slash\n# $XDG_DATA_HOME *is* different if you're using Flatpak!\n# note the source and command. \n[Feed4]\nsrc = https://ideatrash.net/feed\ncmd = sed 's/\u003cdiv class=\"more-link-wrapper\"\u003e.*\\]\\]\u003e\u003c\\/description\u003e/\\]\\]\\\u003e\u003c\\/description\u003e/g'\nurl = /relative/path/to/xml/filename.xml\n\n```\n\n## 9. Advanced Content Warning\n\nIf you need ideas for what tags/terms make good content warnings, the file \n`cwlist.txt` is included for your convenience. Because of how it matches, a \nfilter of \"abuse\" should catch \"child abuse\" and \"sexual abuse\", etc. However, \nit matches whole words, so \"war\" should *not* catch \"bloatware\" or \"warframe\".\n\nThe advanced content warning system is configured in the `agaetr.ini` as \nwell, following a similar format to the feeds:\n\n```\n[CW9]\nkeyword = social-media\nmatches = facebook twitter mastodon social-media online\n```\n\nThe \"keyword\" is what is outputted as the content warning, the space-separated \nline after matches is what strings will trigger that keyword as a content \nwarning.  This will work on *all* feeds where `ContentWarning = yes` is \nconfigured. \n\n### The keyword should **NOT** be a potentially sensitive word itself.\n\n## 10. Usage\n\n\nStandalone: /path/to/agaetr.sh [options]\n\n*  --help:  show help \n* --locations: print config and data locations\n* --readme: display the README on the console\n* --version: report version  \n* --pull: draw in configured RSS sources\n* --push: push out from queue\n* --muna [URL]: unredirect a URL \n* --url [URL] --description [text]: add single url to outbound queue \n\n### IMPORTANT NOTE ABOUT CRON\n\n**If you run `agaetr` as a cron job, ensure that the cron job is \nrun as the user (and with the environment) you used to set up the \nonline services.**  \n\n\nBecause it's all modular, it's possible to run independent elements, or to run from the \ncontrol file `agaetr.sh`.  It's also possible to source the senders and archivers \nto pass additional (global) variables to them:\n\n\"$pubtime\" \"$title\" \"$description\" \"$link\" \"$hashtags\"\n\"$cw\"  \"$imgurl\" \"ALT_TEXT\"\n\n`hooty.sh` is an example of this.\n\n## 11. Other files\n\nThere are other files in this repository:\n\n* `hooty.sh` - a rewrite of `patootie` to use `agaetr`'s framework to interactively \nsend a post to (optionally multiple) services immediately.\n* `muna.sh` - Used by `agaetr` to remove redirections and shortening.  Exactly the same as [muna](https://github.com/uriel1998/muna).  \n\n\n## 12. TODO\n\n### Roadmap:\n\n* Other services, eg. linkedin, medium?\n\n### Someday/Maybe:\n\n* timeout for archivers?\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Furiel1998%2Fagaetr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Furiel1998%2Fagaetr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Furiel1998%2Fagaetr/lists"}