{"id":18878278,"url":"https://github.com/uriel1998/orindi","last_synced_at":"2026-02-19T15:30:16.962Z","repository":{"id":147177356,"uuid":"241971780","full_name":"uriel1998/orindi","owner":"uriel1998","description":"Transform your e-mail newsletters into webpages and an RSS feed. Uses pico as a front end.","archived":false,"fork":false,"pushed_at":"2021-04-25T17:08:04.000Z","size":128,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-12-31T02:40:12.727Z","etag":null,"topics":["email","pico","rss"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uriel1998.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-20T19:30:29.000Z","updated_at":"2021-04-25T17:08:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"c0486f1c-8135-442a-aedd-4bd3440cbc48","html_url":"https://github.com/uriel1998/orindi","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uriel1998%2Forindi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uriel1998%2Forindi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uriel1998%2Forindi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uriel1998%2Forindi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uriel1998","download_url":"https://codeload.github.com/uriel1998/orindi/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239841731,"owners_count":19705981,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["email","pico","rss"],"created_at":"2024-11-08T06:25:36.931Z","updated_at":"2026-02-19T15:30:16.916Z","avatar_url":"https://github.com/uriel1998.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# orindi\n\nTransform your e-mail newsletters into webpages and an RSS feed. Uses pico as a front end.\n`orindi` is an anglicization of ørindi, meaning \"message\".\n\n\n![orindi logo](https://raw.githubusercontent.com/uriel1998/orindi/master/orindi-icon.png \"logo\")\n\n## Contents\n 1. [About](#1-about)\n 2. [License](#2-license)\n 3. [Prerequisites](#3-prerequisites)\n 4. [Installation](#4-installation)\n 5. [Usage](#5-usage)\n 6. [TODO](#6-todo)\n\n***\n\n## 1. About\n\nTransform your e-mail newsletters into webpages and an RSS feed. Uses pico as a front end.\n`orindi` is an anglicization of ørindi, meaning \"message\".\n\nThe idea is that this should be able to slip into your existing workflow, \nregardless of what that is or what you're using.  And if it doesn't, it's meant \nto be as simple as possible to customize the code.\n\n### Why Pico and Procmail?\n\nI'd previously used [Kill The Newsletter](https://www.kill-the-newsletter.com/) \nwhich did a bangup job. But when the author changed it from using a cloud service \nto exim, I had a problem. I don't use exim, and the configuration was in the code \nthat I (quite frankly) don't have the skill to understand.\n\nTherefore, I wanted something that would rely on [procmail](https://linux.die.net/man/5/procmailrc), which can work with \npretty much any mail transfer agent seamlessly, including after delivery if \nmail is in the maildir format.  (See [TODO](#6-todo) if you're about to say I \nshould use something different like `courier-maildrop`.)\n\n[Pico](http://picocms.org/) provides a good front-end for similar reasons. \nIt's lightweight, does not require one to disrupt their existing setup, deals \nwith plain text files, and can generate RSS feeds fairly easily.\n\n\n## 2. License\n\nThis project is licensed under the MIT License. For the full license, see `LICENSE`.\n\n## 3. Prerequisites\n\n* [Pandoc](https://pandoc.org/)  \n* [Pico](http://picocms.org/)   \n* [procmail](https://linux.die.net/man/5/procmailrc)  \n\n\n## 4. Installation\n\nIdeally, you should create a virtualenv for this project, as there are a number \nof python dependencies.  Instructions on how to do that are beyond the scope of \nthis document.  It is assumed that you have created and activated the \nvirtualenv henceforth.\n\n### Install MDA / procmail  \n\n* This should be handled by your package manager.\n\n### Install pandoc\n\n* Follow the instructions at [Pandoc's documentation](https://pandoc.org/installing.html).  \n\n### Install pico \n\nThe instructions and program as written presume that `pico` is local to `orindi`. \nIf the files need to be uploaded elsewhere by a script after processing, you will \nneed a local mirror for the files to be written into. The user that runs `orindi` \nshould be part of the same user group as pico (www-data, probably). \n\n* NOTE:  Tested with php7.3; when I had older versions of PHP the program ran \njust fine, but the output (particularly of the RSS feed) was sometimes mis-parsed.\n\n\n* Follow the installation instructions in [pico's documentation](http://picocms.org/docs/).  \n* Copy `orindi-index.twig` to pico's `/themes/default` directory.\n* Copy `index.md` to pico's `/content` directory.\n* Change permissions on pico's `/content` and `/themes/default` directory to 775 and\nuse the sticky bit (`chmod -t DIRECTORY`) to reduce the possibility of permission \nissues.  \n* Optionally, if your setup will allow it, you may install pico somewhere else \n(such as in the same directory as `orindi`) and create a symlink to /var/www/html \nor wherever your webserver can see it.\n* If you have problems with serving HTTPS, you may need to manually set \n`themes_url`, `base_url`, `assets_url`, `plugins_url`, and `rewrite_url` \nin `pico/config/config.yml`.  \n\n\n### Install python modules and application\n\n* `pip install -r requirements.txt` or `pip3 install -r requirements.txt` if you didn't make a venv and still have python2 installed.\n* `mkdir -p $HOME/.config/orindi`\n* Edit `orindi.ini` (see instructions below)\n* `cp $PWD/orindi.ini $HOME/.config/orindi`\n* Edit `orindi_procmailrc` (see instructions below)\n* `cp $PWD/orindi_procmailrc $HOME/.config/procmail`\n* `sudo chmod +x $PWD/orindi_parse.py`\n\n\n### orindi_procmailrc setup  \n\nThere is only one thing to change in the procmail rc file:\n```\n:0Wc:\n| /path/to/orindi/orindi_parse.py\n```\n\nType in the path to orindi_parse.py on the second line, making sure you keep \nthe pipe character.  This will pass a *copy* of each email on to `orindi`, and \nthen allow it to be delivered normally.  \n\nYou *may* try letting `procmail` not wait between each email to process it; if \nyou do, change the first line to `:0c:`.  \n\nThere is an alternate way to use procmail to match mails instead of using `orindi`, \nbut since `orindi` must also have that list, you'll end up having to keep \nmultiple lists synchronized... \n\n**Integrating procmail with your MTA is an exercise for the user.** \n\n### INI files setup\n\n`orindi` can have one or many ini files; it's all up to how you want to \norganize things ... and how many things you're filtering for.  You *must* have \n`orindi.ini` in the configuration directory. The main `orindi.ini` file \nmust have these lines (with the appropriate values):  \n\n```\n[DEFAULT]  \nBaseDir = /var/www/html/pico/content  \nBaseThemeDir = /var/www/html/pico/themes/default  \nAppDir = /where/orindi/is/installed/to  \n```\n\nAfter that you can configure any number of feeds and the criteria that they \nmatch on.  You can put them all in one file, or in separate files in \n`$HOME/.config/orindi`.  For each feed, there are four elements.\n\n* The label (the bit in brackets).  They should be in the format *feed###*, and each should be unique across *all* ini files.\n* The keyword. This is how it will be organized under `pico`. \n* A list (indented if multiline) of strings that will be tested against the \"from\" field\n* A list (comma separated) of strings that will be tested against the \"subject\" field.\n\nExample: \n\n```\n[feed5]\nkeyword = crowdfunding\nfrom = @indiegogo.com\n    bingo@patreon.com\n    no-reply@patreon.com\n    messages@gofundme.com\n    gfm-community@gofundme.com\n    hello@gofundme.com\n    news@gofundme.com\n    no-reply@kickstarter.com    \nsubject = Thanks for becoming a backer,Project Update,launched\n\n```\n\nAny mail that `orindi` gets that matches either the \"from\" condition or the \n\"subject\" condition will be put into the \"crowdfunding\" subdirectory and \n\"crowdfunding\" feed.\n\n### Security  \n\nThere is no default security set up with `orindi`.  This is not ideal, for \nobvious reasons.\n\nThe easiest way is to use *htaccess* basic authentication.  Instructions on \nsetting up *htaccess* for [Apache](https://www.elated.com/password-protecting-your-pages-with-htaccess/) and [Nginx](https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-http-basic-authentication/) are available on the web (or the links in this sentence).\n\nOnce you've set up *htaccess* authentication, the generated RSS feeds are \naccessible at:\n\n`https://USERNAME:PASSWORD@domain.for.your.site/feed-link`\n\n\n## 5. Usage\n\n`orindi` takes an email from either stdin or as a filename from the first \ncommand line variable. This provides additional flexibility in inserting it into \nyour already existing workflow. \n\nIt even works for already existent mail that is in the MAILDIR format.  For \nexample, if you were to use `offlineimap` and `getmail` to pull down your \nemail to a single directory, you could then use this bit of bash in a script \nto process the files:\n\n``\nfor i in `find $MAILDIR -type f \\( -iname \"*.*\" ! -iname \"dovecot*\" ! -name \"subscriptions\" \\) `; do\n  cat \"$i\" | procmail $PROCMAILD/orindi_procmailrc\ndone\n``\n\nor just as validly, use this bit of bash to do the same thing:\n\n``\nfor i in `find $MAILDIR -type f \\( -iname \"*.*\" ! -iname \"dovecot*\" ! -name \"subscriptions\" \\) `; do\n  $PATH/TO/orindi_parse.py \"$i\"\ndone\n``\n\n(If you wished to remove the files, obviously just add `rm \"$i\"` to the loop.)\n\nA warning with the latter; the filenames that `dovecot` uses (at least) will \nsometimes completely pooch the process.\n\nOr you could even use the same to process a different mailbox after some other \nprogram has had a crack at it.  \n\n\n### Cleaning up  \n\nWhen processing, `orindi` will set the outputted file to the same date/time \nas the mail header, which will help with maintenance and housekeeping.  \n\nThe easiest way to clean up old files is through using a script like this, \nsubstituting your pico content directory where appropriate:  \n\n```\nfor i in `find /var/www/html/pico/content/ -type d | grep -v -e \"content/$\" | xargs realpath`; do \n    find \"$i\" -mtime +30 | xargs rm -f\n    find \"$i\" -atime +30 | xargs rm -f\ndone\n\n```\n\nThe reverse grep is needed so you don't accidentally delete your subject index \nfiles in the main content directory.  Of course, if you use pico for other \nsites, you will want to edit the script appropriately.\n\n### Viewing the output\n\nIf `pico` is installed in the subdirectory `/pico` to your domain, then you \ncan view it at `yourdomain.com/pico`.  Each keyword's output is viewable at \n`yourdomain/pico/?keyword` and the RSS feed is available at `yourdomain/pico/?keyword-feed`. \n\nTo use the example above, that means I could see an index of the emails at \n`yourdomain/pico/?crowdfunding` and get the RSS feed at `yourdomain/pico/?crowdfunding-feed`.\n\nMOST emails look pretty much how they're supposed to.  Linkedin emails seem to\nget mangled to hell.  Some extra spaces and linefeeds, but I'm not sure I want \nto keep trying to parse them in the main program...\n\n## 6. TODO\n\n\n* Customize pico's theme a little bit\n* Which plugins should be added in (recommendations, anyway):\n    https://github.com/alejandroliu/ForceHttpsPlugin\n* Per section output chooser - (or rather, first attempt one) so that we're not tied to pandoc so hard  \n* Clean output further - remove empty paragraphs, maybe a tidy library? (Beautiful Soup did a LOT, though)\n* Try use of dehtml instead of pypandoc\n* Remove tracking beacons completely.  Not sure how other than to look for \n  img tags with small pixel sizes or img style=\"overflow: hidden\"\n```\n\u003cimg style=\"overflow: hidden;position: fixed;visibility: hidden !important;display: block !important;height: 1px !important;width: 1px !important;border: 0 !important;margin: 0 !important;padding: 0 !important;\" src=\"https://connectednation.cmail20.com/t/j-o-chklljl-yuiyjkttht/o.gif\" width=\"1\" height=\"1\" border=\"0\" alt=\"\"\u003e\n```\n* Enable use of [courier-maildrop](http://www.courier-mta.org/maildrop/) instead of procmail.\n* See if procmail has to wait between deliveries\n* Use errorcodes (sys.exit(3) for example) to indicate non-match?\n\n\n### Roadmap:\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Furiel1998%2Forindi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Furiel1998%2Forindi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Furiel1998%2Forindi/lists"}