https://github.com/danvk/personal-archive
Organize your digital debris
https://github.com/danvk/personal-archive
Last synced: over 1 year ago
JSON representation
Organize your digital debris
- Host: GitHub
- URL: https://github.com/danvk/personal-archive
- Owner: danvk
- Created: 2012-10-13T03:34:41.000Z (almost 14 years ago)
- Default Branch: master
- Last Pushed: 2012-10-13T03:42:56.000Z (almost 14 years ago)
- Last Synced: 2025-03-24T06:51:31.009Z (over 1 year ago)
- Language: Python
- Size: 190 KB
- Stars: 5
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README
Awesome Lists containing this project
README
Personal Archive Project
This project aims to collect all your digital debris and make it useful. This
could be your gmails, chat logs on old hard drives, foursquare checkins, photos
or journals.
----------------------------------------------------
Quick start
----------------------------------------------------
echo "~/Dropbox/Personal Archive" > personal-archive.config
./import_ims.py path/to/an/adium/logs/dir
./viewer/viewer.py
Visit http://localhost:8080/
For more types of data that you can import, see "Specific Importers", below.
----------------------------------------------------
Overview
----------------------------------------------------
There are three main parts to the project:
1. Archival Storage
2. Data Import
3. Data Exploration
Archival Storage
----------------
At its most basic level, the Personal Archive Project is a directory structure
and a file format. It looks something like:
[Personal Archive Folder]
- 2000
- 07
- 01
+ chats.json
- chats
+ chat1.txt
+ chat2.txt
+ sent-mail.json
- sent-mail
+ message1.txt
+ message2.txt
There's a folder for each year. These contain folders for each month. And these
contain folders for each day.
Within each day's folder, there are entries for each Maker.
A Maker is something which puts data into the personal archive.
For each day, a Maker produces a summary of the day's activity in JSON format.
It looks something like this:
{"summary": "02:08 PM: Joe & Amy's wedding"}
This is always stored in a file named [maker].json. If there's additional
information that the maker would like to store (perhaps full emails or chat
transcripts), then it may do so inside a folder named [maker]. Hence
"chats.json" and "chats/chat[12].txt" in the example above.
This format is designed to be simple and easily grep'able. It is also
compatible with manual exploration, whether in the Finder/Windows Explorer or
in the Dropbox application (should you choose to put your Personal Archive on
Dropbox).
The Personal Archive Project includes some utilities for generating data in
this format. See below.
Data Import
-----------
The Personal Archive Project includes some scripts to import common data
formats into its own directory & file formats. For example:
./download_gmail_folder.py "[Gmail]/Sent Mail" staging/sent-msgs.gmail.json
./import_gmail_folder.py --maker 'sent-gmail' staging/sent-msgs.gmail.json
This will download all of your Gmail sent messages (i.e. the things you
actually typed) and import them under the 'sent-gmail' maker.
See below for info on how to import common formats.
Everyone has their own unique data sources. If you'd like to write your own
importer, check out utils.py.
Data Exploration
----------------
The Personal Archive Project includes a web viewer to help you explore the data
you've imported. To use it, run:
./viewer/viewer.py
Then visit http://localhost:8080/
You can visit http://localhost:8080/coverage to get a sense for the coverage of
individual makers.
----------------------------------------------------
Specific Importers
----------------------------------------------------
GMail Chat
----------
Create a file called "gmail-credentials.js", and fill it out like so:
{
"email": "youremail@gmail.com",
"password": "yourpassword"
}
Then run
chmod 600 gmail-credentials.js
To make sure no one else can read your password.
Now follow the instructions here to enable chat downloading over IMAP:
http://dataliberation.blogspot.com/2011/09/gmail-liberates-recorded-chat-logs-via.html
Finally, download all the messages to your staging dir and import them:
mkdir staging
./download_gmail_folder.py '[Gmail]/Chats' staging/chats.gmail.json
./import_gmail_chats.py staging/chats.gmail.json
Google Voice
------------
Install pygooglevoice (http://code.google.com/p/pygooglevoice/).
Install this patch: http://productforums.google.com/forum/#!topic/voice/OS-abSdgz-k
Run:
./sms2csv.py
./import_google_voice.py staging/GVsms\ 2012-09-08\ 12-34-56.csv
You'll likely have to modify the import_google_voice.py script to point at the
right location and include names for a few extra numbers.
Adium Chat Logs
---------------
Copy all your IM Logs (either in the old .adiumLog format or the newer .html
format) into some subdirectory of the staging directory.
Then run:
./import_ims.py staging/adium-logs
This will crawl the staging/adium-logs directory, looking for all chat logs. It
will import them under the "chats" maker.
mbox files
----------
Find all your sent mail mbox files and convert them to JSON:
find (root) -name 'mbox' -iregex '.*sent.*' -print0 | xargs -0 ./scan-mbox.py --output_json > staging/sent-mbox.json
Import from the JSON:
./import_sent_mbox.py staging/sent-mbox.json
Maildir
-------
You'll have to tar and gzip your maildir before importing it:
tar -czf staging/maildir.tgz path/to/maildir
./import_tgz_maildir.py staging/maildir.tgz
Personal Journals
-----------------
You'll have to get your journal into a very specific format to use the
import_journal.py script. The format is described in that file. Then you can
run:
./import_journal.py journal path/to/your/journal.txt
TODOs:
- Standardize command-line arguments for import scripts
-> --dry_run
-> --maker
- Remove danvk-specific importers
- Filter by source in viewer.
- Support structured times inside summaries & update existing importers.
- Split up the "import_ims" script into individual importers.
- Link directly to dropbox for originals
- Support multiple entries per day for a single maker
- Write a "X year(s) ago today" mailer
- Fix up encoding stuff.
-> ex broken URL: http://localhost:8080/day/2003/01/02
- Make "More" link more prominent & open in new tab.
- Add direct links to each Maker's content on the one-day page.
- Integrate the /coverage page into the main display.