{"id":19565515,"url":"https://github.com/swaldman/feedletter","last_synced_at":"2026-05-17T14:34:48.960Z","repository":{"id":204342904,"uuid":"711254968","full_name":"swaldman/feedletter","owner":"swaldman","description":"A service that converts RSS feeds into e-mail newsletters and notification bridges.","archived":false,"fork":false,"pushed_at":"2026-05-09T20:28:25.000Z","size":607,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-05-09T20:41:01.025Z","etag":null,"topics":["e-mail-lists","mailing-list","newsletter","rss","self-hosted"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/swaldman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-10-28T17:15:39.000Z","updated_at":"2026-05-09T18:58:00.000Z","dependencies_parsed_at":null,"dependency_job_id":"fbd273f3-f0ee-4c81-aa1f-3a23482f9a23","html_url":"https://github.com/swaldman/feedletter","commit_stats":null,"previous_names":["swaldman/feedletter"],"tags_count":24,"template":false,"template_full_name":null,"purl":"pkg:github/swaldman/feedletter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swaldman%2Ffeedletter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swaldman%2Ffeedletter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swaldman%2Ffeedletter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swaldman%2Ffeedletter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/swaldman","download_url":"https://codeload.github.com/swaldman/feedletter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swaldman%2Ffeedletter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33142274,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-17T09:28:26.183Z","status":"ssl_error","status_checked_at":"2026-05-17T09:27:52.702Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["e-mail-lists","mailing-list","newsletter","rss","self-hosted"],"created_at":"2024-11-11T05:27:22.045Z","updated_at":"2026-05-17T14:34:48.938Z","avatar_url":"https://github.com/swaldman.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# feedletter\n\n**Turn any RSS feed into a newsletter or notification bot**\n\n## Introduction\n\n**feedletter** is a service that\n\n- **watches RSS (or Atom!) feeds with great care**\n   * works great with feeds generated by static-site generators!\n   * distinguishes between new items and older stuff or stuff already seen that flakily reappears\n   * awaits \"finalization\" of items, meaning their stabilization (and nondeletion) over specified time intervals\n- lets you **define a wide variety of subscriptions** to those feeds\n   * Over different media\n     - e-mail\n     - Post to Mastodon\n     - SMS (coming soon!)\n     - etc\n   * In different arrangements\n     - each item as newsletter\n     - daily or weekly digests\n     - compilations of every `n` posts\n     - etc\n- which are **formatted via rich, customizable [`untemplates`](https://github.com/swaldman/untemplate-doc#readme)**\n- which are **managed via a web API** for easy subscription, confirmation, and unsubscription by users\n\n## Prerequisites\n\nThe application requres a Java 17+ JVM and a Postgres database.\n\nTypical installations will proxy the web API behind e.g. `nginx`, and run the daemon as a `systemd` service.\n\n## Getting Started\n\nA (very) detailed tutorial on setting up, configuring, and customizing a _feedletter_ instance is available [here](https://tech.interfluidity.com/2024/01/29/feedletter-tutorial/index.html).\n\n## Developer Notes\n\n### Lifecycle\n\n1. A `feed` is added\n2. One or more `subscribable`s is defined against the feed\n3. One or more `destination`s subscribe to the feed.\n4. `item`s are observed in the feed, and are added in the `Unassigned` state\n5. Each `item` is assigned, in a single transaction, to all the collections (`assignable`s) to which they will ever belong.\n\n(Steps 4 and 5 can repeat arbitrarily as new items come in.)\n\n6. Separately, collections (`assignable`s) are periodically marked \"complete\"\n   and, in the same transaction forwarded to subscribers.\n7. Complete `assignable`s are deleted, along with their `assignment`s\n8. `item`s that are...\n   * Already assigned\n   * No longer belong to not-yet-completed `assignables` can drop their cached contents,\n     and then move into the `Cleared` state.\n\n### Database Schema\n\nI want to sketch the not-so-obvious db schema I've adopted for this project while\nI still understand it.\n\n#### feed\n\nFirst there are feeds:\n\n```sql\nCREATE TABLE feed(\n  id                          INTEGER,\n  url                         VARCHAR(1024),\n  min_delay_minutes           INTEGER NOT NULL,\n  await_stabilization_minutes INTEGER NOT NULL,\n  max_delay_minutes           INTEGER NOT NULL,\n  assign_every_minutes        INTEGER NOT NULL,\n  added                       TIMESTAMP NOT NULL,\n  last_assigned               TIMESTAMP NOT NULL,     -- we'll start at added\n  PRIMARY KEY(id)\n)\n```\n\nFeeds must be defined before subscriptions can be created against them.\nThey are defined by a URL, and they define what it means for a feed to\n\"finalize\", in the sense of being ready for notification.\n\nFeeds are permanent and basically unchanging until (when someday I implement\nthis) they are manually removed. \n\n(`last_assigned` changes, but so far it's\njust informational, has no role in the application.)\n\n#### item\n\nNext there are items:\n\n```sql\nCREATE TABLE item(\n  feed_id          INTEGER,\n  guid             VARCHAR(1024),\n  single_item_rss  TEXT,\n  content_hash     INTEGER, -- ItemContent.contentHash\n  link             VARCHAR(1024),\n  first_seen       TIMESTAMP NOT NULL,\n  last_checked     TIMESTAMP NOT NULL,\n  stable_since     TIMESTAMP NOT NULL,\n  assignability    ItemAssignability NOT NULL,\n  PRIMARY KEY(feed_id, guid),\n  FOREIGN KEY(feed_id) REFERENCES feed(id)\n)\"\"\".stripMargin\n```\n\n* `feed_id` and `guid` identify an item.\n* `single_item_rss` _caches_ the RSS item.\n   We want to cache this, in case by the time we get around to notifying, the item is no longer available in the feed.\n* `content_hash` is a hash based on the prior five fields. We use it to identify whether an item has changed.\n* `link` may eventually be used as a neurotic double-check so we never notify the same human-perceived item twice\n* `first_seen`, `last_checked`, and `stable_since` are pretty self-explanatory timestamps, We use these to\n  calculate whether an item has stabilized and so can be \"assigned\". (See below.)\n* `assignability`: items can be in one of four states\n    * `Unassigned` — The item has not yet been assigned to the collections (including single member collections)\n      to which it will eventually belong, but is eligible for assignment.\n    * `Assigned` — The item _hash_ been assigned to _all_ the collections (including single member collections)\n      to which it will eventually belong. The application may not be done assigning to those collections, and the\n      items may not yet be distributed to subscribers.\n    * `Cleared` — This is the terminal state for an item. The item has been assigned to all collections, and have\n      already been distributed to subscribers. The cache fields (`title`, `author`, `article`, `publication_date`, and `link`)\n      should all be cleared in this state. `Cleared` items are not deleted, but retained indefinitely, so that we don't\n      renotify if the item (an item with the same `guid`) reappears in the feed.\n    * `Excluded` — Items which are marked to always be ignored. Items are marke `Excluded` only upon initial insert.\n      Items can be manually updated from `Excluded` to `Unassigned` (timestamps should be reset to the tie of the update),\n      to cause `Excluded` posts to be published.\n\n#### subscribable (subscription definition)\n\nNext there is `subscribable`, which represents the definition of a subscription by which parties will be\nnotified of items or collections of items.\n\n```sql\nCREATE TABLE subscribable(\n  subscribable_name         VARCHAR(64),\n  feed_id                   INTEGER       NOT NULL,\n  subscription_manager_json JSONB         NOT NULL,\n  last_completed_wti        VARCHAR(1024),\n  PRIMARY KEY (subscribable_name),\n  FOREIGN KEY (feed_id) REFERENCES feed(id)\n)\n```\n\nA subscribable maps a name to a feed and a `SubscriptionManager`. For our purposes here,\nthe main role of a `SubscriptionManager` (a serialization of a Scala ADT) is to\n\n1. Generate for items a `within_type_id`, which is really just a collection identifier.\n   All items in a collection of items that will be distributed will share the same `within_type_id`.\n2. Determine whether a collection (identified by its `within_type_id`) is \"complete\" — that is,\n   no further items need by assigned the same `within_type_id`.\n3. When a collection has been notified, it is deleted from the database. However, some\n   `SubscriptionManagers` need to maintain a sequence of `within_type_id` identifiers.\n   So for each subscribable, the `last_completed_wti` is retained.\n\n`SubscriptionManager` determines how collections are compiled, to what kind of destination (e-mail,\nMastodon, mobile message, whatever) notifications will be sent, and how they will be formatted.\n\nNames are scoped on a per-feed-URL basis. Users subscribe to a `(feed_url, subscribable_name)`\npair.\n\n#### assignable (a collection of items)\n\nNext there is `assignable`, which represents a collection. They essentially map\n`subscribables` (subscription definitions) to `within_type_id`s (the collections\ngenerated by the subscription definition and notified to subscribers).\n\n```sql\nCREATE TABLE assignable(\n  subscribable_name VARCHAR(64),\n  within_type_id    VARCHAR(1024),\n  opened            TIMESTAMP NOT NULL,\n  PRIMARY KEY(subscribable_name, within_type_id),\n  FOREIGN KEY(subscribable_name) REFERENCES subscribable(subscribable_name)\n)\n```\n\n`opened` is the timestamp of the first assignment to the collection.\n\nOnce an assignable has been notified (\"completed\"), it is simply deleted from the database.\nFor each subscribable, the `within_type_id` of only the most recently completed\nassignable is retained (see `subscribable` table above).\n\n#### assignment (an item in a collection)\n\nNext there is `assignment`, which represents an item in an `assignable`, i.e. a collection.\nIt's pretty self-explanatory I think.\n\n```sql\nCREATE TABLE assignment(\n  subscribable_name VARCHAR(64),\n  within_type_id    VARCHAR(1024),\n  guid              VARCHAR(1024),\n  PRIMARY KEY( subscribable_name, within_type_id, guid ),\n  FOREIGN KEY( subscribable_name, within_type_id ) REFERENCES assignable( subscribable_name, within_type_id )\n)\n```\n\n#### subscription\n\nNext there is `subscription`, which just maps a destination to a `subscribable`.\nthe destination is JSON blob that can refer to a variety of things: e-mail addresses, SMS numbers, mastodon instances, etc.\nEach `SubscriptionManager` works with a destination subtype. \n\n```sql\nCREATE TABLE subscription(\n  subscription_id    BIGINT,\n  destination_json   JSONB         NOT NULL,\n  destination_unique VARCHAR(1024) NOT NULL,\n  subscribable_name  VARCHAR(64)   NOT NULL,\n  confirmed          BOOLEAN       NOT NULL,\n  added              TIMESTAMP     NOT NULL,\n  PRIMARY KEY( subscription_id ),\n  FOREIGN KEY( subscribable_name ) REFERENCES subscribable( subscribable_name )\n)\n```\n\nSince destinations can have ornamentation (an e-mail address,\nfor example, might have a personal part (e.g. Buffy in \"Buffy \u003cb@slayers.org\u003e\"), it's not sufficient to prevent multiple\nsubscriptions to insist that the JSON entities be unique. So destinations declare a unique core, whose uniqueness within\na subscription the database enforces:\n\n```sql\nCREATE UNIQUE INDEX destination_unique_subscribable_name ON subscription(destination_unique, subscribable_name)\n```\n\nThat's it for the base schema! There are also tables that convert destinations specific to subscription\ntypes into their various queues for notification. I'm omitting those for now.\n\n### Untemplates vs Templates\n\nThere are two layers of templating in _feedletter_:\n\nMany notifications are rendered via [untemplates](https://github.com/swaldman/untemplate-doc#readme).\nHowever, what untemplates render goes to all subscribers of a subscribable. We generate one \"form letter\"\nfor all recipients, and store it only once.\n\nBut since we may want to customize our notifications in a per-recipient basis, the _output_ of the untemplates\ncan take the form of a [trivial template](https://github.com/swaldman/feedletter/blob/main/src/com/mchange/feedletter/trivialtemplate/TrivialTemplate.scala)\nwith case-insensitive, percentage-delimited `%Fields%` that get filled in separately for each recipient.\n\nWe try to refer to the former, initial, shared templates as _untemplates_ (because that's the technology\nthat underlies them), and the last-minute substitution templates that are generated by the untemplates as\nmere templates.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fswaldman%2Ffeedletter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fswaldman%2Ffeedletter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fswaldman%2Ffeedletter/lists"}