{"id":16137412,"url":"https://github.com/purarue/HPI","last_synced_at":"2025-10-23T20:32:23.362Z","repository":{"id":41857721,"uuid":"290394554","full_name":"seanbreckenridge/HPI","owner":"seanbreckenridge","description":"Human Programming Interface - a way to unify, access and interact with all of my personal data [my modules]","archived":false,"fork":false,"pushed_at":"2024-06-07T06:47:56.000Z","size":1413,"stargazers_count":68,"open_issues_count":3,"forks_count":6,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-06-07T07:44:54.559Z","etag":null,"topics":["data","gdpr","history","lifelogging","personal-api","quantified-self"],"latest_commit_sha":null,"homepage":"https://beepb00p.xyz/hpi.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/seanbreckenridge.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-26T04:25:51.000Z","updated_at":"2024-06-07T06:47:59.000Z","dependencies_parsed_at":"2024-06-07T07:44:13.134Z","dependency_job_id":"556a802a-3d8e-424b-af04-93431a399105","html_url":"https://github.com/seanbreckenridge/HPI","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seanbreckenridge%2FHPI","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seanbreckenridge%2FHPI/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seanbreckenridge%2FHPI/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seanbreckenridge%2FHPI/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/seanbreckenridge","download_url":"https://codeload.github.com/seanbreckenridge/HPI/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219867752,"owners_count":16555813,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","gdpr","history","lifelogging","personal-api","quantified-self"],"created_at":"2024-10-09T23:26:56.425Z","updated_at":"2025-10-23T20:32:23.356Z","avatar_url":"https://github.com/seanbreckenridge.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"**TLDR**: I'm using `HPI`(Human Programming Interface) package as a means of unifying, accessing and interacting with all of my personal data.\n\nIt's a Python library (named `my`), a collection of modules for:\n\n- social networks: posts, comments, favorites, searches\n- shell/program histories (zsh, bash, python, mpv, firefox)\n- programming (github/commits)\n- instant messaging\n- media histories (movies, TV shows, music, video game achievements/history); see \u003chttps://purarue.xyz/feed/\u003e\n\n[_Why?_](https://github.com/karlicoss/HPI#why)\n\nThis is built on top of [`karlicoss/HPI`](https://github.com/karlicoss/HPI). These are all additional modules which aren't present in that repository - this is installed alongside the upstream repository (meaning _you can use both modules from upstream and here simultaneously_), see [#install](#install)\n\n### My Modules\n\n- `my.zsh` and `my.bash`, access to my shell history w/ timestamps\n- `my.mail.imap` and `my.mail.mbox` to parse local IMAP sync's of my mail/mbox files -- see [doc/MAIL_SETUP.md](doc/MAIL_SETUP.md)\n- `my.mpv.history_daemon`, accesses movies/music w/ activity/metadata that have played on my machine, facilitated by a [mpv history daemon](https://github.com/purarue/mpv-history-daemon)\n- `my.discord.data_export`, parses ~1,000,000 messages/events from the discord data export, parser [here](https://github.com/purarue/discord_data)\n- `my.todotxt.active` to parse my current [todo.txt](https://github.com/todotxt/todo.txt-cli) file; `my.todotxt.git_history` tracks my history using backups of those files in [`git_doc_history`](https://github.com/purarue/git_doc_history)\n- `my.rss.newsboat`, keeps track of when I added/removed RSS feeds (for [`newsboat`](https://newsboat.org/))\n- `my.ipython`, for timestamped python REPL history\n- `my.ttt`, to parse shell/system history tracked by [`ttt`](https://github.com/purarue/ttt)\n- `my.activitywatch.active_window`, to parse active window events (what application I'm using/what the window title is) using [`window_watcher`](https://github.com/purarue/aw-watcher-window) and [activitywatch](https://activitywatch.net/) on android\n- `my.chess.export`, to track my [chess.com](https://www.chess.com)/[lichess.org](https://lichess.org/) games, using [`chess_export`](https://github.com/purarue/chess_export)\n- `my.trakt.export`, providing me a history/my ratings for Movies/TV Show (episodes) using [`traktexport`](https://github.com/purarue/traktexport)\n- `my.listenbrainz.export`, exporting my music listening history from [ListenBrainz](https://listenbrainz.org/) (open-source Last.fm) using [`listenbrainz_export`](https://github.com/purarue/listenbrainz_export)\n- `my.offline.listens`, for offline music listen history, using [offline_listens](https://github.com/purarue/offline_listens)\n- `my.mal.export`, for anime/manga history using [`malexport`](https://github.com/purarue/malexport)\n- `my.grouvee.export`, for my video game history/backlog using [`grouvee_export`](https://github.com/purarue/grouvee_export)\n- `my.runelite.screenshots`, parses data from the [automatic runelite screenshots](https://github.com/runelite/runelite/wiki/Screenshot)\n- `my.minecraft.advancements`, parses advancement (local achievement data) from the `~/.minecraft` directory\n- `my.project_euler`, when I solved [Project Euler](https://projecteuler.net/) problems\n- `my.linkedin.privacy_export`, to parse the [privacy export](https://www.linkedin.com/help/linkedin/answer/50191/downloading-your-account-data?lang=en) from linkedin\n- `my.scramble.history` for merged (timed) rubiks cube solves from multiple sources, using [scramble_history](https://github.com/purarue/scramble-history)\n\n#### 'Historical' Modules\n\nThese are modules to parse GDPR exports/data from services I used to use, but don't anymore. They're here to provide more context into the past.\n\n- `my.apple.privacy_export`, parses Game Center and location data from the [apple privacy export](https://privacy.apple.com/)\n- `my.facebook.gdpr`, to parse the GDPR export from Facebook\n- `my.league.export`, gives League of Legends game history using [`lolexport`](https://github.com/purarue/lolexport)\n- `my.steam.scraper`, for steam achievement data and game playtime using [`steamscraper`](https://github.com/purarue/steamscraper)\n- `my.piazza.scraper`, parsing [piazza](https://piazza.com/) (university forum) posts using [`piazza-scraper`](https://github.com/purarue/piazza-scraper)\n- `my.blizzard.gdpr`, for general battle.net event data [parsed from a GDPR export](https://github.com/purarue/blizzard_gdpr_parser)\n- `my.skype.gdpr` to parse a couple datetimes from the Skype GDPR export (seems all my data from years ago is long gone)\n- `my.spotify.gdpr`, to parse the GDPR export from Spotify, mostly to access songs from my playlists from years ago\n- `my.twitch`, merging the [data export](https://www.twitch.tv/p/en/legal/privacy-choices/#user-privacy-requests) and my messages parsed from the [overrustle logs dump](https://github.com/purarue/overrustle_parser)\n\nSee [here](https://github.com/purarue/dotfiles/blob/master/.config/my/my/config/__init__.py) for my `HPI` config\n\n[Promnesia `Source`s for these `HPI` modules](https://github.com/purarue/promnesia)\n\nI also have some more personal scripts/modules in a separate repo; [`HPI-personal`](https://github.com/purarue/HPI-personal)\n\n### In-use from [karlicoss/HPI](https://github.com/karlicoss/HPI)\n\n- `my.browser`, to parse browser history using [`browserexport`](https://github.com/purarue/browserexport)\n- `my.google.takeout.parser`, parses lots of (~500,000) events (youtube, searches, phone usage, comments, location history) from [google takeouts](https://takeout.google.com/), using [`google_takeout_parser`](https://github.com/purarue/google_takeout_parser)\n- `my.coding.commits` to track git commits across the system\n- `my.github` to track github events/commits and parse the GDPR export, using [`ghexport`](https://github.com/karlicoss/ghexport)\n- `my.reddit`, get saved posts, comments. Uses [`rexport`](https://github.com/karlicoss/rexport) to create backups of recent activity periodically, and [`pushshift`](https://github.com/purarue/pushshift_comment_export) to get old comments.\n- `my.smscalls`, exports call/sms history using [SMS Backup \u0026 Restore](https://play.google.com/store/apps/details?id=com.riteshsahu.SMSBackupRestore\u0026hl=en_US)\n- `my.stackexchange.stexport`, for stackexchange data using [`stexport`](https://github.com/karlicoss/stexport)\n\n#### Partially in-use/with overrides:\n\n- `my.location`, though since I also have some locations from `apple.privacy_export`, I have a [`my.location.apple`](./my/location/apple.py) which I then merge into `my.location.all` in my overridden [`all.py`](https://github.com/purarue/HPI-personal/blob/master/my/location/all.py) file on my personal repo\n- similarly, I do use `my.ip` and `my.location.via_ip` from upstream, but I have [overridden `all.py` and module files here](https://github.com/purarue/HPI/tree/master/my/ip)\n\n'Overriding' an `all.py` file means replacing the `all.py` from upstream repo (this means it can use my sources here to grab more locations/ips, since those don't exist in the upstream). For more info see [reorder_editable](https://github.com/purarue/reorder_editable#editable-namespace-packages), and the [module design](https://github.com/karlicoss/HPI/blob/master/doc/MODULE_DESIGN.org#adding-new-modules) docs for HPI, but you might be able to get the gist by comparing:\n\n- [my.location.all](https://github.com/karlicoss/HPI/blob/master/my/location/all.py) in `karlicoss/HPI`\n- [my.location.all](https://github.com/purarue/HPI-personal/blob/master/my/location/all.py) in `purarue/HPI-personal`\n\nSince I've mangled my `PYTHONPATH` (see [reorder_editable](https://github.com/purarue/reorder_editable#editable-namespace-packages)), it imports from my repo instead of `karlicoss/HPI`. `all.py` files tend to pretty small -- so overriding/changing a line to add a source is the whole point.\n\nSee [here](https://github.com/purarue/reorder_editable#custom-editablepth-files) for an explanation of how that works.\n\n### Companion Tools/Libraries\n\nDisregarding tools which actively collect data (like [`ttt`](https://github.com/purarue/ttt)/[`window_watcher`](https://github.com/purarue/aw-watcher-window)) or repositories which have their own exporter/parsers which are used here, there are a couple other tools/libraries I've created for this project:\n\n- [`ipgeocache`](https://github.com/purarue/ipgeocache) - for any IPs gathered from data exports, provides geolocation info, so I have partial location info going back to 2013\n- [`sqlite_backup`](https://github.com/purarue/sqlite_backup) - to safely copy/backup application sqlite databases that may currently be in use\n- [`git_doc_history`](https://github.com/purarue/git_doc_history) - a bash script to copy/backup files into git history, with a python library to help traverse and create a history/parse diffs between commits\n- [`HPI_API`](https://github.com/purarue/HPI_API) - automatically creates a JSON API/server for HPI modules\n- [`url_metadata`](https://github.com/purarue/url_metadata) - caches youtube subtitles, url metadata (title, description, image links), and a html/plaintext summary for any URL\n\nI also use this in [`my_feed`](https://github.com/purarue/my_feed), which creates a feed of media/data using `HPI`, live at \u003chttps://purarue.xyz/feed/\u003e\n\n### Ad-hoc and interactive\n\nSome basic examples.\n\nWhen was I most using reddit?\n\n```python\nimport collections, my.reddit.all, pprint\n\npprint.pprint(collections.Counter([c.created.year for c in my.reddit.all.comments()]))\nCounter({2016: 3288, 2017: 801, 2015: 523, 2018: 209, 2019: 65, 2014: 4, 2020: 3})\n```\n\nMost common shell commands?\n\n```python\nimport collections, pprint, my.zsh\n\n# lots of these are git-related aliases\npprint.pprint(\n    collections.Counter([c.command for c in my.zsh.history()]).most_common(10)\n)\n[\n    (\"ls\", 51059),\n    (\"gst\", 11361),\n    (\"ranger\", 6530),\n    (\"yst\", 4630),\n    (\"gds\", 3919),\n    (\"ec\", 3808),\n    (\"clear\", 3651),\n    (\"cd\", 2111),\n    (\"yds\", 1647),\n    (\"ga -A\", 1333),\n]\n```\n\nWhat websites do I visit most?\n\n```python\nimport collections, pprint, my.browser.export, urllib\n\npprint.pprint(\n    collections.Counter(\n        [urllib.parse.urlparse(h.url).netloc for h in my.browser.export.history()]\n    ).most_common(5)\n)\n[\n    (\"github.com\", 20953),\n    (\"duckduckgo.com\", 10146),\n    (\"www.youtube.com\", 10126),\n    (\"discord.com\", 8425),\n    (\"stackoverflow.com\", 2906),\n]\n```\n\nSong I've listened to most?\n\n```python\nimport collections, my.mpv.history_daemon\n\ncollections.Counter([m.path for m in my.mpv.history_daemon.history()]).most_common(1)[\n    0\n][0]\n\"/home/username/Music/JPEFMAFIA/JPEGMAFIA - LP! - 2021 - V0/JPEGMAFIA - LP! - 05 HAZARD DUTY PAY!.mp3\"\n```\n\nMovie I've watched most?\n\n```python\nimport my.trakt\nfrom collections import Counter\n\nCounter(e.media_data.title for e in my.trakt.history()).most_common(1)\n[(\"Up\", 92)]  # (the pixar movie)\n```\n\n`hpi` also has a JSON query interface, so I can do quick computations using shell tools like:\n\n```bash\n# how many calories have I eaten today (from https://github.com/purarue/ttally)\n$ hpi query ttally.__main__.food --recent 1d -s | jq -r '(.quantity)*(.calories)' | datamash sum 1\n2258.5\n```\n\n### Install\n\nFor the basic setup, I recommend you clone and install both directories as editable installs:\n\n```bash\n# clone and install upstream as an editable package\ngit clone https://github.com/karlicoss/HPI ./HPI-karlicoss\npython3 -m pip install --user -e ./HPI-karlicoss\n\n# clone and install my repository as an editable package\ngit clone https://github.com/purarue/HPI ./HPI-pura\npython3 -m pip install --user -e ./HPI-pura\n```\n\nEditable install means any changes to python files reflect immediately, which is very convenient for debugging and developing new modules. To update, you can just `git pull` in those directories.\n\nIf you care about [overriding modules](https://github.com/purarue/HPI#partially-in-usewith-overrides), to make sure your `easy-install.pth` is ordered correctly:\n\n```bash\npython3 -m pip install --user reorder_editable\npython3 -m reorder_editable reorder ./HPI-pura ./HPI-karlicoss\n```\n\nThen, you likely need to run `hpi module install` for any modules you plan on using -- this can be done incrementally as you setup new modules. E.g.:\n\n- `hpi module install my.trakt.export` to install dependencies\n- Check the [stub config](./tests/my/my/config/__init__.py) or [my config](https://github.com/purarue/dotfiles/blob/master/.config/my/my/config/__init__.py) and setup the config block in your HPI configuration file\n- Run `hpi doctor my.trakt.export` to check for any possible config issues/if your data is being loaded properly\n\n(The [install](./install) script does that for all my modules, but you likely don't want to do that)\n\nIts possible to install both `my` packages because `HPI` is a namespace package. For more information on that, and some of the complications one can run into, see [reorder_editable](https://github.com/purarue/reorder_editable#editable-namespace-packages), and the [module design](https://github.com/karlicoss/HPI/blob/master/doc/MODULE_DESIGN.org#adding-new-modules) docs for HPI.\n\nIf you're having issues installing/re-installing, check the [TROUBLESHOOTING_INSTALLS.md](doc/TROUBLESHOOTING_INSTALLS.md) file.\n\nIf you recently updated and it seems like something has broke, check the [CHANGELOG](CHANGELOG.md) for any possible breaking changes\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpurarue%2FHPI","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpurarue%2FHPI","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpurarue%2FHPI/lists"}