{"id":13910795,"url":"https://github.com/jakubvalenta/automatic-diary","last_synced_at":"2025-07-28T04:31:57.072Z","repository":{"id":45442430,"uuid":"175301898","full_name":"jakubvalenta/automatic-diary","owner":"jakubvalenta","description":"Create one timeline from various digital sources","archived":false,"fork":false,"pushed_at":"2024-11-08T17:24:59.000Z","size":590,"stargazers_count":12,"open_issues_count":2,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-25T19:41:38.386Z","etag":null,"topics":["calendar","csv","diary","journal","log","productivity"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jakubvalenta.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-12T21:53:38.000Z","updated_at":"2024-11-08T17:25:04.000Z","dependencies_parsed_at":"2024-11-25T19:45:34.678Z","dependency_job_id":null,"html_url":"https://github.com/jakubvalenta/automatic-diary","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jakubvalenta%2Fautomatic-diary","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jakubvalenta%2Fautomatic-diary/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jakubvalenta%2Fautomatic-diary/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jakubvalenta%2Fautomatic-diary/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jakubvalenta","download_url":"https://codeload.github.com/jakubvalenta/automatic-diary/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227866561,"owners_count":17831811,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["calendar","csv","diary","journal","log","productivity"],"created_at":"2024-08-07T00:01:45.758Z","updated_at":"2024-12-03T06:28:19.312Z","avatar_url":"https://github.com/jakubvalenta.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Automatic Diary\n\nAutomatic Diary is a script that creates one timeline from various digital\nsources describing your life.\n\n![Automatic Diary screenshot](./automatic_diary_screenshot.png)\n\n_On the screenshot above, you can see: calendar events (blue), completed todo\nlist items (yellow), software development work (gray), sport activity (green),\nand films watched (red)._\n\n## Why\n\nAutomatic Diary can be useful to those who:\n\n(a) have bad memory,\\\n(b) want to remember what they did,\\\n(c) but don't have the time to write a real diary.\n\n## What is collected\n\n- [X] Calendar events (via CalDAV service and iCalendar file reading)\n- [X] Emails sent and received (via Maildir reading)\n- [X] Sport activity (via spreadsheet parsing)\n- [X] Facebook and Twitter posts (via Facebook/Twitter data archive parsing)\n- [X] Software development work (via Git repository log)\n- [X] Completed todo list items (via todo.txt parsing)\n- [X] Films watched (via ČSFD website parsing and Trakt.tv)\n- [X] Custom diary notes (via plain-text and Org-mode files parsing)\n\n## What is not collected\n\nAutomatic Diary could be extended to collect also the following data but right\nnow it is considered low priority:\n\n- [ ] Custom notes and ideas\n- [ ] Articles and blog posts written\n- [ ] Phone calls and text messages (SMS)\n- [ ] Social media private messages and messaging apps\n- [ ] Browser history\n- [ ] Music listened (via Last.fm)\n- [ ] OpenStreetMap contributions\n- [ ] Money transfers\n- [ ] News headlines\n- [ ] Weather\n\n## Installation\n\n### Mac\n\n``` shell\n$ brew install python\n$ pip install poetry\n$ pip install --user --upgrade .\n```\n\n### Arch Linux\n\n``` shell\n# pacman -S python-poetry libsecret\n$ pip install --user --upgrade .\n```\n\n### Other systems\n\nInstall these dependencies manually:\n\n- Python \u003e= 3.9\n- poetry\n- libsecret (Linux keyring) -- required by the `caldav` provider\n\nThen run:\n\n``` shell\n$ pip install --user --upgrade .\n```\n\n## Configuration\n\nBefore you run Automatic Diary, you need to configure all the providers (sources\nfrom which the data for your timeline will be read). The following providers are\nsupported:\n\n- [caldav](#caldav)\n- [csfd](#csfd)\n- [csv](#csv)\n- [facebook](#facebook)\n- [git](#git)\n- [icalendar](#icalendar)\n- [maildir](#maildir)\n- [orgmode](#orgmode)\n- [orgmodelist](#orgmodelist)\n- [todotxt](#todotxt)\n- [trakt](#trakt)\n- [twitter](#twitter)\n- [txt](#txt)\n\nAll providers are configured using a single `config.json` file. Use\n[config-sample.json](./config-sample.json) as a template for your own\nconfiguration.\n\n### caldav\n\n- Input: CalDAV server\n\n- Output: Names and locations of calendar events\n\n- Configuration:\n\n    ``` json\n    {\n        \"url\": \"\u003cserver url\u003e\",\n        \"username\": \"\u003cserver authentication username\u003e\",\n        \"password_key\": \"\u003cserver authentication password -- libsecret key\u003e\",\n        \"password_val\": \"\u003cserver authentication password -- libsecret value\u003e\",\n        \"cache_dir\": \"\u003ccache directory path\u003e\"\n    }\n    ```\n\n### csfd\n\n- Input: User profile on [ČSFD](https://www.csfd.cz/) (film database website, something\n  like IMDB)\n\n- Output: Titles of films rated\n\n- Configuration:\n\n    ``` json\n    {\n        \"profile_url\": \"\u003ccsfd.cz profile url\u003e\",\n        \"cache_dir\": \"\u003ccache directory path\u003e\"\n    }\n    ```\n\n### csv\n\n- Input: CSV spreadsheet (.csv) file\n\n- Output: Rows formatted using a template\n\n- Configuration\n\n    ``` json\n    {\n        \"path\": \"\u003ccsv file path\u003e\",\n        \"date_source\": \"{{\u003ccolumn name\u003e}}\",\n        \"date_format\": \"\u003cstrptime date format\u003e\",\n        \"text_source\": \"\u003ctemplate string in the Mustache format\u003e\"\n    }\n    ```\n\n### facebook\n\n- Input: Downloaded Facebook archive\n\n- Output: Texts of statuses\n\n- Configuration\n\n    ``` json\n    {\n        \"path\": \"\u003cpath to wall.htm or timeline.htm\u003e\",\n        \"username\": \"\u003cfacebook username\u003e\"\n    }\n    ```\n\n### git\n\n- Input: Directory with checked-out Git repositories and an author name\n\n- Output: Commit messages by the author from all repositories\n\n- Configuration\n\n    ``` json\n    \"config\": {\n        \"base_path\": \"\u003cpath to directory - will be searched recursively for git repos\u003e\",\n        \"author\": \"\u003cauthor name\u003e\",\n        \"cache_dir\": \"\u003ccache directory path\u003e\",,\n        \"max_depth\": \"\u003cmax directory depth to search - 5 is a good default value\u003e\"\n    }\n    ```\n\n### icalendar\n\n- Input: Calendar events stored offline in iCalendar (.ics) files\n\n- Output: Names and locations of calendar events\n\n- Configuration:\n\n    ``` json\n    {\n        \"paths\": [\n            \"\u003cpath to an .ics file\u003e\",\n            ...\n        ]\n    }\n    ```\n\n    Not that events from all the listed .ics files will be merged -- duplicate\n    events removed.\n\n### maildir\n\n- Input: Emails stored offline in the Maildir format\n\n- Output: Subjects of emails\n\n- Configuration:\n\n    ``` json\n    {\n        \"received_pathname\": \"\u003cglob pathname of directories with received emails\u003e\",\n        \"sent_pathname\": \"\u003cglob pathname of directories with sent emails\u003e\"\n    }\n    ```\n\n### orgmode\n\n- Input: Emacs Org-mode (.org) file in format:\n\n    ``` org\n    * \u003c2019-01-17 Thu\u003e\n\n    Lorem ipsum\n    foo.\n\n    bar\n\n    * \u003c2019-01-18 Fri\u003e\n\n    spam spam\n    ...\n    ```\n\n- Output: Example:\n\n    ``` csv\n    2019-01-17,Lorem ipsum foo.\n    2019-01-17,bar\n    2019-01-18,spam spam\n    ```\n\n- Configuration:\n\n    ``` json\n    {\n        \"path\": \"\u003cpath to the .org file\u003e\"\n    }\n    ```\n\n### orgmodelist\n\n- Input: Emacs Org-mode (.org) file in format:\n\n        - Lorem ipsum foo. \u003c2019-01-17 Thu\u003e\n        - bar \u003c2019-01-18 Fri 11:30\u003e\n        ...\n\n- Output: Example:\n\n    ``` csv\n    2019-01-17,Lorem ipsum foo.\n    2019-01-18T11:00:00+02:00,bar\n    ```\n\n- Configuration:\n\n    ``` json\n    {\n        \"path\": \"\u003cpath to the .org file\u003e\"\n    }\n    ```\n\n### todotxt\n\n- Input: Todo.txt completed tasks file (done.txt)\n\n- Output: Texts of completed tasks\n\n- Configuration\n\n    ``` json\n    {\n        \"path\": \"\u003cdone.txt file path\u003e\"\n    }\n    ```\n\n### trakt\n\n- Input: [Manually create an application within Trakt.tv](https://trakt.tv/oauth/applications/new). Manual login with OAuth when running this provider. A recommended `Redirect URI` is `urn:ietf:wg:oauth:2.0:oob`\n\n- Output: Texts of watched films and shows\n\n- Configuration\n\n    ``` json\n    {\n        \"key_id\": \"\u003cTrakt.tv app Key ID\u003e\",\n        \"key_secret\": \"\u003cTrakt.tv app Secret\u003e\",\n        \"app_id\": \"\u003cTrakt.tv app ID (numeric)\u003e\"\n    }\n    ```\n\n\n\n### twitter\n\n- Input: Downloaded Twitter archive\n\n- Output: Texts of tweets\n\n- Configuration\n\n    ``` json\n    {\n        \"path\": \"\u003cpath to twitter archive directory\u003e\"\n    }\n    ```\n\n### txt\n\n- Input: Plain text (.txt) file in format:\n\n    ```\n    2015-12-02 St\n        Some\n        Text\n        Lorem\n            Ipsum\n                Hierarchy\n                Spam\n    2015-12-03 Čt\n        Foobar\n        ...\n    ```\n\n- Output: Example:\n\n    ``` csv\n    2015-12-02,Some\n    2015-12-02,Text\n    2015-12-02,Lorem: Ipsum: Hierarchy\n    2015-12-02,Lorem: Ipsum: Spam\n    2015-12-03,Foobar\n    ```\n\n- Configuration\n\n    ``` json\n    {\n        \"path\": \"\u003cpath to the .txt file\u003e\"\n    }\n    ```\n\n## Usage\n\n### Generating CSV\n\nThe basic output of Automatic Diary is a CSV file. Generate it by running:\n\n``` shell\n$ automatic-diary \u003cconfig path\u003e \u003coutput csv path\u003e\n```\n\nExample:\n\n``` shell\n$ automatic-diary ~/.config/automatic-diary/config.json ~/Desktop/automatic_diary.csv\n```\n\nThe CSV output is in format:\n\n``` csv\n\u003cdatetime\u003e,\u003cprovider\u003e,\u003csubprovider\u003e,\u003ctext\u003e\n```\n\nExample CSV output:\n\n``` csv\n2019-01-23T15:31:54-08:00,git,human-activities,model: Log exceptions while scanning\n2019-01-24T09:00:00+01:00,caldav,https://dav.mailbox.org/caldav/da39a3ee5e6b,DHL Packstation\n2019-01-25,todotxt,done.txt,Opravit Ondrovi kolo\n```\n\nSee the help for all command line options:\n\n``` shell\n$ automatic-diary --help\n```\n\n### Visualization\n\nThe output CSV file can also be rendered as an HTML document which looks kind of\nlike a calendar. See the screenshot above. Generate this HTML document by\nrunning:\n\n``` shell\n$ automatic-diary-visualize \u003ccsv path\u003e \u003coutput html path\u003e\n```\n\nExample:\n\n``` shell\n$ automatic-diary-visualize ~/Desktop/automatic_diary.csv ~/Desktop/automatic_diary.html\n```\n\nBy default, this command visualizes only those CSV items that happened within\nthe latest year in the CSV file. To visualize **all items**, use the `-a` /\n`--all-years` option. Then pass a directory path as `\u003coutput html path\u003e`\n(instead of passing a file path). Example:\n\n``` shell\n$ automatic-diary-visualize --all-years ~/Desktop/automatic_diary.csv ~/Desktop/automatic_diary\n```\n\nThe result will be a directory with an HTML document for each year:\n\n```\nautomatic_diary\n├── 2022.html\n├── 2023.html\n└── 2024.html\n```\n\nFor **custom styling** of the HTML document, you can pass an additional CSS file\nusing the `-c` / `--css-url` option. The passed CSS URL must be absolute or\nrelative to the output path. Example:\n\n``` shell\n$ automatic-diary-visualize --css-url ./my_stylesheet.css ~/Desktop/automatic_diary.csv ~/Desktop/automatic_diary.html\n```\n\nIn your file, you can use for example the `data-subprovider` selector. Example:\n\n``` css\n.csv[data-subprovider=\"running.csv\"] {\n  color: #008080;\n}\n```\n\nYou can also use the `-t` / `--tags-file` option to add **additional CSS\nclasses** to the diary items based on provided regexes. For example when you\ncreate the following tags JSON file:\n\n``` json\n{\n  \"^Sport: \": \"sport\",\n  \"^Idea: \": \"idea\",\n}\n```\n\nAnd pass it to the command like this:\n\n``` shell\n$ automatic-diary-visualize --css-url ./my_stylesheet.css --tags-file ./my_tags.json  ~/Desktop/automatic_diary.csv ~/Desktop/automatic_diary.html\n```\n\nThen the command will add a CSS class `tag-sport` to each diary item that starts\nwith \"Sport: \" and a CSS class `tag-idea` to each diary item that starts with\n\"Idea: \". Then you can use these classes in your custom CSS (see the `-c` /\n`--css-url` option).\n\nSee the **help** for all command line options:\n\n``` shell\n$ automatic-diary-visualize --help\n```\n\n## Development\n\n### Installation\n\n``` shell\n$ make setup\n```\n\n### Testing and linting\n\n``` shell\n$ make test\n$ make lint\n```\n\n### Help\n\n``` shell\n$ make help\n```\n\n## Contributing\n\n__Feel free to remix this project__ under the terms of the GNU General Public\nLicense version 3 or later. See [COPYING](./COPYING) and [NOTICE](./NOTICE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjakubvalenta%2Fautomatic-diary","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjakubvalenta%2Fautomatic-diary","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjakubvalenta%2Fautomatic-diary/lists"}