{"id":16528383,"url":"https://github.com/danvk/personal-archive","last_synced_at":"2025-04-10T05:30:29.438Z","repository":{"id":5039686,"uuid":"6199433","full_name":"danvk/personal-archive","owner":"danvk","description":"Organize your digital debris","archived":false,"fork":false,"pushed_at":"2012-10-13T03:42:56.000Z","size":195,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-24T06:51:31.009Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"govfresh/govpress","license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/danvk.png","metadata":{"files":{"readme":"README","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-10-13T03:34:41.000Z","updated_at":"2020-12-31T01:26:03.000Z","dependencies_parsed_at":"2022-09-21T14:12:46.541Z","dependency_job_id":null,"html_url":"https://github.com/danvk/personal-archive","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danvk%2Fpersonal-archive","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danvk%2Fpersonal-archive/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danvk%2Fpersonal-archive/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danvk%2Fpersonal-archive/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/danvk","download_url":"https://codeload.github.com/danvk/personal-archive/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248162895,"owners_count":21057831,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T17:40:08.812Z","updated_at":"2025-04-10T05:30:29.364Z","avatar_url":"https://github.com/danvk.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Personal Archive Project\n\nThis project aims to collect all your digital debris and make it useful. This\ncould be your gmails, chat logs on old hard drives, foursquare checkins, photos\nor journals.\n\n\n----------------------------------------------------\n                     Quick start\n----------------------------------------------------\n\necho \"~/Dropbox/Personal Archive\" \u003e personal-archive.config\n./import_ims.py path/to/an/adium/logs/dir\n./viewer/viewer.py\n\nVisit http://localhost:8080/\n\nFor more types of data that you can import, see \"Specific Importers\", below.\n\n\n\n----------------------------------------------------\n                     Overview\n----------------------------------------------------\n\nThere are three main parts to the project:\n\n1. Archival Storage\n2. Data Import\n3. Data Exploration\n\n\nArchival Storage\n----------------\n\nAt its most basic level, the Personal Archive Project is a directory structure\nand a file format. It looks something like:\n\n[Personal Archive Folder]\n  - 2000\n    - 07\n      - 01\n        + chats.json\n        - chats\n          + chat1.txt\n          + chat2.txt\n        + sent-mail.json\n        - sent-mail\n          + message1.txt\n          + message2.txt\n\nThere's a folder for each year. These contain folders for each month. And these\ncontain folders for each day.\n\nWithin each day's folder, there are entries for each Maker.\n\nA Maker is something which puts data into the personal archive.\n\nFor each day, a Maker produces a summary of the day's activity in JSON format.\nIt looks something like this:\n\n{\"summary\": \"02:08 PM: Joe \u0026 Amy's wedding\"}\n\nThis is always stored in a file named [maker].json. If there's additional\ninformation that the maker would like to store (perhaps full emails or chat\ntranscripts), then it may do so inside a folder named [maker]. Hence\n\"chats.json\" and \"chats/chat[12].txt\" in the example above.\n\nThis format is designed to be simple and easily grep'able. It is also\ncompatible with manual exploration, whether in the Finder/Windows Explorer or\nin the Dropbox application (should you choose to put your Personal Archive on\nDropbox).\n\nThe Personal Archive Project includes some utilities for generating data in\nthis format. See below.\n\n\nData Import\n-----------\n\nThe Personal Archive Project includes some scripts to import common data\nformats into its own directory \u0026 file formats. For example:\n\n./download_gmail_folder.py \"[Gmail]/Sent Mail\" staging/sent-msgs.gmail.json\n./import_gmail_folder.py --maker 'sent-gmail' staging/sent-msgs.gmail.json\n\nThis will download all of your Gmail sent messages (i.e. the things you\nactually typed) and import them under the 'sent-gmail' maker.\n\nSee below for info on how to import common formats.\n\nEveryone has their own unique data sources. If you'd like to write your own\nimporter, check out utils.py.\n\n\n\nData Exploration\n----------------\n\nThe Personal Archive Project includes a web viewer to help you explore the data\nyou've imported. To use it, run:\n\n./viewer/viewer.py\n\nThen visit http://localhost:8080/\n\nYou can visit http://localhost:8080/coverage to get a sense for the coverage of\nindividual makers.\n\n\n\n----------------------------------------------------\n                 Specific Importers\n----------------------------------------------------\n\n\nGMail Chat\n----------\n\nCreate a file called \"gmail-credentials.js\", and fill it out like so:\n\n{\n  \"email\": \"youremail@gmail.com\",\n  \"password\": \"yourpassword\"\n}\n\nThen run\nchmod 600 gmail-credentials.js\n\nTo make sure no one else can read your password.\n\nNow follow the instructions here to enable chat downloading over IMAP: \nhttp://dataliberation.blogspot.com/2011/09/gmail-liberates-recorded-chat-logs-via.html\n\nFinally, download all the messages to your staging dir and import them:\n\nmkdir staging\n./download_gmail_folder.py '[Gmail]/Chats' staging/chats.gmail.json\n./import_gmail_chats.py staging/chats.gmail.json\n\n\nGoogle Voice\n------------\n\nInstall pygooglevoice (http://code.google.com/p/pygooglevoice/).\nInstall this patch: http://productforums.google.com/forum/#!topic/voice/OS-abSdgz-k\n\nRun:\n./sms2csv.py\n./import_google_voice.py staging/GVsms\\ 2012-09-08\\ 12-34-56.csv\n\nYou'll likely have to modify the import_google_voice.py script to point at the\nright location and include names for a few extra numbers.\n\n\nAdium Chat Logs\n---------------\n\nCopy all your IM Logs (either in the old .adiumLog format or the newer .html\nformat) into some subdirectory of the staging directory.\n\nThen run:\n./import_ims.py staging/adium-logs\n\nThis will crawl the staging/adium-logs directory, looking for all chat logs. It\nwill import them under the \"chats\" maker.\n\n\nmbox files\n----------\n\nFind all your sent mail mbox files and convert them to JSON:\nfind (root) -name 'mbox' -iregex '.*sent.*' -print0 | xargs -0 ./scan-mbox.py --output_json \u003e staging/sent-mbox.json\n\nImport from the JSON:\n./import_sent_mbox.py staging/sent-mbox.json\n\n\nMaildir\n-------\n\nYou'll have to tar and gzip your maildir before importing it:\n\ntar -czf staging/maildir.tgz path/to/maildir\n./import_tgz_maildir.py staging/maildir.tgz\n\n\nPersonal Journals\n-----------------\n\nYou'll have to get your journal into a very specific format to use the\nimport_journal.py script. The format is described in that file. Then you can\nrun:\n\n./import_journal.py journal path/to/your/journal.txt\n\n\n\n\nTODOs:\n- Standardize command-line arguments for import scripts\n  -\u003e --dry_run\n  -\u003e --maker\n- Remove danvk-specific importers\n- Filter by source in viewer.\n- Support structured times inside summaries \u0026 update existing importers.\n- Split up the \"import_ims\" script into individual importers.\n- Link directly to dropbox for originals\n- Support multiple entries per day for a single maker\n- Write a \"X year(s) ago today\" mailer\n- Fix up encoding stuff.\n  -\u003e ex broken URL: http://localhost:8080/day/2003/01/02\n- Make \"More\" link more prominent \u0026 open in new tab.\n- Add direct links to each Maker's content on the one-day page.\n- Integrate the /coverage page into the main display.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanvk%2Fpersonal-archive","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdanvk%2Fpersonal-archive","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanvk%2Fpersonal-archive/lists"}