{"id":21162854,"url":"https://github.com/ferdinandyb/maildir-rank-addr","last_synced_at":"2025-07-09T15:34:50.744Z","repository":{"id":62865067,"uuid":"555986745","full_name":"ferdinandyb/maildir-rank-addr","owner":"ferdinandyb","description":"Creates a ranked list of email addresses from local email files, which can be used for address completion for example in aerc.","archived":false,"fork":false,"pushed_at":"2025-01-27T10:32:39.000Z","size":260,"stargazers_count":28,"open_issues_count":3,"forks_count":4,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-05T02:12:15.784Z","etag":null,"topics":["address-book","addressbook","aerc","email","maildir","mbox","ranking-algorithm"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ferdinandyb.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-22T20:05:11.000Z","updated_at":"2025-02-19T16:19:46.000Z","dependencies_parsed_at":"2023-11-25T09:37:16.278Z","dependency_job_id":"861f368e-df6d-4586-bfe3-8971e4f05955","html_url":"https://github.com/ferdinandyb/maildir-rank-addr","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/ferdinandyb/maildir-rank-addr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ferdinandyb%2Fmaildir-rank-addr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ferdinandyb%2Fmaildir-rank-addr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ferdinandyb%2Fmaildir-rank-addr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ferdinandyb%2Fmaildir-rank-addr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ferdinandyb","download_url":"https://codeload.github.com/ferdinandyb/maildir-rank-addr/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ferdinandyb%2Fmaildir-rank-addr/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264486392,"owners_count":23616038,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["address-book","addressbook","aerc","email","maildir","mbox","ranking-algorithm"],"created_at":"2024-11-20T13:34:31.211Z","updated_at":"2025-07-09T15:34:50.719Z","avatar_url":"https://github.com/ferdinandyb.png","language":"Go","funding_links":[],"categories":["\u003ca name=\"email\"\u003e\u003c/a\u003eEmail"],"sub_categories":[],"readme":"# Description\n\nGenerates a ranked addressbook from your locally available email. It can be\nused in MUA's like [aerc](http://aerc-mail.org) or [mutt](http://www.mutt.org/)\nby grepping the list.\n\nWhy? No need to manually edit an address book, yet the cached ranking is\navailable extremely fast.\n\n### Features:\n\n- scans all your emails\n- support maildir or similar one-email-per-file formats, and mbox\n- ranks based on both recency and frequency of addresses\n- collects from To, Cc, Bcc, From, Sender and Reply-To fields\n- ranks addresses explicitly emailed by you higher\n- configurable output via go templates\n- uses the most frequent non-empty display name for each email\n- display name can be unicode normalized for search purposes\n- filters common \"no reply\" addresses, additional filters can be added via regexes\n- normalizes emails to lower case\n- ability to add additional email addresses from a command\n- \"blazingly fast\"\u003csup\u003e\\*\u003c/sup\u003e: crunch time for 270k emails is 7s on my machine, grepping from the output is instantaneous\n\n\u003csup\u003e\\*\u003c/sup\u003e: compared to original python implementation for crunching (see Behind the scenes below) and compared to using notmuch query for address completion\n\n[Changelog](CHANGELOG.md)\n\n# Installation\n\nThe easiest way to install is running:\n\n```\ngo install github.com/ferdinandyb/maildir-rank-addr@latest\n```\n\nArch users can also install it from AUR, either the `maildir-rank-addr` or the `maildir-rank-addr-git` package.\n\n# Usage\n\nAt the minimum, you need to specify where your maildir formatted email are:\n\n```\nmaildir-rank-addr --maildir=~/.mail\n```\n\nFor most use cases, it likely only needs to be run once or twice a day (cronjob\nor systemd timer).\n\nSupported flags:\n\n```\n      --addr-book-add-unmatched   flag to determine if you want unmatched addressbook contacts to be added to the output\n      --addr-book-cmd string      optional command to query addresses from your addressbook\n      --addresses strings         comma separated list of your email addresses (regex possible)\n      --config string             path to config file\n      --filters strings           comma separated list of regexes to filter\n      --list-template string      list name template\n      --maildir strings           comma separated list of paths to maildir folders\n      --outputpath string         path to output file\n      --template string           output template\n```\n\n**maildir**\n\nThe paths to the folders that will be scanned. No default is set for this.\nThese actually do NOT need to be maildir format folders, it will try to read\nall files as an email or an mbox (it will skip any hidden files and anything\nthat is in a folder called `tmp` or `.notmuch`).\n\n**outputpath**\n\nBy default results are output to\n`$HOME/.cache/maildir-rank-addr/addressbook.tsv\"`. Specifing `-` as the\noutputpath will print to STDOUT.\n\n**addresses**\n\nList of your own email addresses. If you do not provide your own addresses,\nclassification based on your explicit sends will not be possible!\n\n**template**\n\nUses go's `text/template` to configure output for each address (one line per address).\nAvailable keys:\n\n```\n\tAddress\n\tName\n\tNormalizedName: same as Name, but unicode normalized\n\tNames\n\tClass\n\tFrequencyRank\n\tRecencyRank\n\tTotalRank\n\tClassCount\n\tClassDate\n\tListName: based on list-id header if applicable\n\tListId: based on list-id header if applicable\n```\n\nDefault: `{{.Address}}\\t{{.Name}}`\n\n**list-template**\n\nIf we detect a mailinglist, based on the list-id header, then in the above\ntemplate `{{.Name}}` will be set to the output of this template. Set it to\n`DISABLELIST` to disable this feature.\n\nDefault `{{.ListName}}`\n\n**filters**\n\nList of regexes. If an address is matched against a regex, it will be excluded\nfrom the output. The regex is matched against the entire email address.\n\nNote that we already filter out addresses, where the local part (the part\nbefore the @) matches any of these strings:\n\n```\n\t\"do-not-reply\",\n\t\"donotreply\",\n\t\"no-reply\",\n\t\"bounce\",\n\t\"noreply\",\n\t\"no.reply\",\n\t\"no_reply\",\n\t\"nevalaszolj\",\n\t\"nincsvalasz\",\n```\n\n**addr-book-cmd**\n\nOptional command to fetch email addresses and names, the output it returns must have\nan email address first, followed by a tab space and and the desired name, the name\nmust end in a tab space or a newline for the command to work, this can be\nuseful for integrating with command line addressbooks such as abook or khard\n\n```\nabook --mutt-query \"s\"\nkhard email -p --remove-first-line\n```\n\n**config**\n\nPath to a config file to be loaded instead of the defaults (see below).\n\n## config file\n\nBesides the flags, toml formatted configuration file is also possible. It's\nfirst looked for at `$HOME/.config/maildir-rank-addr/config` and then the\ncurrent working directory.\n\nComplete example configuration with the default (aerc compatible) template:\n\n```\nmaildir = \"~/.mail\"\naddresses = [\n    \"address1@example.com\",\n    \"address2@otherexample.com\"\n]\nfilters = [\"@spam.(com|org)\"]\noutputpath = \"~/.mail/addressbook\"\ntemplate = \"{{.Address}}\\t{{.Name}}\\t{{.NormalizedName}}\"\n```\n\n## Integration\n\n### aerc\n\nPut something like this in your aerc config (using your favourite grep):\n\n```\naddress-book-cmd=\"ugrep -jP -m 100 --color=never %s /home/[myuser]/.cache/maildir-rank-addr/addressbook.tsv\"\n```\n\n(`-j` is smart case insensitive, and needs to be combined with `-P` for UTF-8).\n\nSince aerc only uses the first two of the tab separated columns any other\ncolumn can be added to help with search or to combine with external tools. For\nexample adding `NormalizedName` as the third column will allow you to type\n\"arpad\", and still find and use the entry for \"Árpád X\" who uses accents in his\nname properly, and \"Arpad Y\" who conformed to ASCII for some reason.\n\nNote that `address-book-cmd` is not executed in the shell, so you need to hard\ncode the path without shell expansion.\n\nIf you are using aerc with `[compose].edit-headers=true` you need integrate\nwith your editor (e.g. with vim), instead of the above.\n\n### vim\n\nThis is an example using `fzf` and the `fzf.vim` plugin. Add it to for example\n`~/.vim/after/ftplugin/mail.vim` to load it only for eml files. The example has\nbeen optimized for `aerc` with `[compose].edit-headers=true`. To insert\naddresses in the `To:` field, take your cursor to the line containing `To:` and\npress `\u003cleader\u003ea`. Use tab to select multiple addresses in the pop-up.\n\n```vimscript\nfunction! InsertAddressAerc()\n    call fzf#run(fzf#wrap(\"insertaddress\", {\n    \\ 'source':'cat ~/.cache/maildir-rank-addr/addressbook.tsv',\n    \\ 'sink*': function(\"InsertContactsLine\"),\n    \\ 'options': '--no-sort -i --multi'\n    \\}))\nendfunction\n\nfunction! InsertAddress()\n    call fzf#run(fzf#wrap(\"insertaddress\", {\n    \\ 'source':'cat ~/.cache/maildir-rank-addr/addressbook.tsv',\n    \\ 'sink': function(\"InsertContact\"),\n    \\ 'options': '--no-sort -i'\n    \\}))\nendfunction\n\nfunction! InsertContactsLine(names) abort\n    for name in a:names\n        call InsertContactLine(name)\n    endfor\nendfunction\n\nfunction! InsertContactLine(name) abort\n    let [address, name; rest] = split(a:name,\"\\t\")\n    call append(line('.'), '    ' . name . \" \u003c\" . address . \"\u003e,\")\nendfunction\n\nfunction! InsertContact(name) abort\n    let [address, name; rest] = split(a:name,\"\\t\")\n    exec 'normal! a'  . name . \" \u003c\" . address . \"\u003e\\\u003cEsc\u003e\"\nendfunction\n\nnnoremap \u003cleader\u003ea :call InsertAddressAerc()\u003cCR\u003e\nnnoremap \u003cleader\u003eA :call InsertAddress()\u003cCR\u003e\n\n```\n\n# Behind the scenes\n\n## Ranking\n\nRanking is actually done by first classifying and then ranking within class.\n\n### Classifying addresses\n\nFirst we go through each email found in your maildir and for each address found\nin any of the address headers we assign a class, based on whether the sender is\nyou or not and which type of header the address was found in:\n\n- 2: from address is yours, address found in To, or Bcc,\n- 1: from address is yours, address found in Cc,\n- 0: From fields and anything else.\n\nFor each _unique_ address seen, we record a class dates (the date of the latest\nemail in which that address was assigned class X) and class counts (the number\nof times in which that address was assigned class X). The unique address itself\nalso get assigned a class, which is the highest class it was seen in.\n\n#### Example\n\nBob writes Alice 3 letters and Alice answers one of them. When Alice runs the\nsoftware both email addresses will receive 3 counts to class 0 and have the\nclass date for 0 at the latest email. Alice now answers one of Bob's emails and\nCc-s Eve. Alice's address receives another count for class 0 and the date is\nalso updated. Eve's address receive a count for class 1 and Bob's email\nreceives a count for class 2 with the class 2 date being set to this latest\nemail. This puts Bob's email address as class 2 as that is the highest class\nit has, Eve's at class 1 and Alice's own address in class 0.\n\n### Ranking\n\nThe addresses are then ranked by their highest class based on only the highest\nclass's count and date. The output will be structured so class 2 emails are on\nthe top, then class 1 email and class 0 emails are at the bottom. This solves\ntwo things: firstly, all mailing-lists and marketing which you get but only\nread will be available to send to, but are guaranteed to be at the very bottom\nof search results. Secondly, even if you accidentally replied to a newsletter\nemail which you get daily, it's class count will be just 1, so even though it's\na bit higher up now due to the reply, it will still feature at the bottom of\nit's class and not get conflated by having received hundreds of email from the\naddress pretty recently.\n\n**Frequency rank:** The emails are ordered according to the class count and\ntheir frequency rank becomes their place in this ordered list, with the highest\ncount receiving a rank of 0. In case of equal counts the order is the\nalphabetical order of the email addresses.\n\n**Recency rank:** Similar, the emails are ordered again, now based on their\nclass date, where the most recent email receives the rank of 0. In case of\nequal dates the order is the alphabetical order of the email addresses.\n\n**Total rank = Frequency rank + Recency Rank**\n\nThe output is then generated by printing class 2 address from lowest to highest\nrank, then class 1 addresses from lowest to highest and finally class\n0 addresses from lowest to highest. In case the total ranks are equal the order\nis the alphabetical order of the email addresses.\n\n## Statistics\n\nThe amount of email I have seems to grow approximately linearly and the amount\nof email addresses also more-or-less, but with a much-much smaller coefficient.\nCompared to needing to grep the email headers caching the unique address leads\nto a 250x compression. Since grep retains ordering of results in a file, it\nalso makes sense encoding rankings by simply ordering the addresses.\n\nYou can generate these images for yourself using the python script `stats/generateEmailStatistics.py`.\n\n![Number of emails and address over time](stats/date-address.svg)\n![Ratio of address to email](stats/email-address.svg)\n\nThe `stats` folder also includes the original PoC implementation of this in\npython (`stats/generateAddressbookMaildir.py`) which takes a whopping 36\n_minutes_ to complete the same task, compared to this implementation's 10\n_seconds_.\n\n# Contribution\n\nPlease see [contribution guidelines](https://github.com/ferdinandyb/maildir-rank-addr/blob/master/CONTRIBUTING.md).\n\n# Similar Projects\n\n- [maildir2addr](https://github.com/BourgeoisBear/maildir2addr): somewhat similar address book generator\n- [notmuch-addrlookup-c](https://github.com/aperezdc/notmuch-addrlookup-c): address lookup from notmuch\n- [addr-book-combine](https://jasoncarloscox.com/creations/addr-book-combine/): for combining generated addressbooks with hand currated ones, like [khard](https://github.com/lucc/khard)\n\n# Acknowledgments\n\nSome functions for parsing email was taken from [aerc](http://aerc-mail.org).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fferdinandyb%2Fmaildir-rank-addr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fferdinandyb%2Fmaildir-rank-addr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fferdinandyb%2Fmaildir-rank-addr/lists"}