{"id":13557345,"url":"https://github.com/karlicoss/rexport","last_synced_at":"2025-04-04T19:08:45.906Z","repository":{"id":40946084,"uuid":"206428112","full_name":"karlicoss/rexport","owner":"karlicoss","description":"Reddit takeout: export your account data as JSON: comments, submissions, upvotes etc.  🦖 ","archived":false,"fork":false,"pushed_at":"2024-10-30T22:05:15.000Z","size":83,"stargazers_count":169,"open_issues_count":0,"forks_count":12,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-28T18:08:43.514Z","etag":null,"topics":["backup","data-liberation","export","praw","reddit","takeout"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/karlicoss.png","metadata":{"files":{"readme":"README.org","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-04T22:45:43.000Z","updated_at":"2025-03-26T20:58:18.000Z","dependencies_parsed_at":"2024-11-18T01:20:06.151Z","dependency_job_id":null,"html_url":"https://github.com/karlicoss/rexport","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karlicoss%2Frexport","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karlicoss%2Frexport/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karlicoss%2Frexport/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karlicoss%2Frexport/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/karlicoss","download_url":"https://codeload.github.com/karlicoss/rexport/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247234921,"owners_count":20905854,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","data-liberation","export","praw","reddit","takeout"],"created_at":"2024-08-01T12:04:17.654Z","updated_at":"2025-04-04T19:08:45.883Z","avatar_url":"https://github.com/karlicoss.png","language":"Python","readme":"#+begin_src python :dir src :results drawer :exports results\nimport rexport.export as E; return E.make_parser().prog\n#+end_src\n\n#+RESULTS:\n:results:\nExport your personal Reddit data: saves, upvotes, submissions etc. as JSON.\n:end:\n\n* Setting up\n1. The easiest way is =pip3 install --user git+https://github.com/karlicoss/rexport=.\n\n   Alternatively, use =git clone --recursive=, or =git pull \u0026\u0026 git submodule update --init=. After that, you can use =pip3 install --editable .=.\n2. To use the API, you need to register a [[https://www.reddit.com/prefs/apps][custom 'personal script' app]] and get =client_id= and =client_secret= parameters.\n \n   See more [[https://praw.readthedocs.io/en/latest/getting_started/authentication.html][here]].\n3. To access user's personal data (e.g. saved posts/comments), Reddit API also requires =username= and =password=.\n\n   Yes, unfortunately it wants your plaintext Reddit password, you can read more about it [[https://praw.readthedocs.io/en/latest/getting_started/quick_start.html#authorized-reddit-instances][here]].\n   \n\n* Exporting\n\n#+begin_src python :dir src :results drawer :exports results\nimport rexport.export as E; return E.make_parser().epilog\n#+end_src\n\n#+RESULTS:\n:results:\n\nUsage:\n\n*Recommended*: create =secrets.py= keeping your api parameters, e.g.:\n\n\n: username = \"USERNAME\"\n: password = \"PASSWORD\"\n: client_id = \"CLIENT_ID\"\n: client_secret = \"CLIENT_SECRET\"\n\nIf you have two-factor authentication enabled, append the six-digit 2FA token to the password, separated by a colon:\n\n: password = \"PASSWORD:343642\"\n\nThe token will, however, be short-lived.\n\nAfter that, use:\n\n: python3 -m rexport.export --secrets /path/to/secrets.py\n\nThat way you type less and have control over where you keep your plaintext secrets.\n\n*Alternatively*, you can pass parameters directly, e.g.\n\n: python3 -m rexport.export --username \u003cusername\u003e --password \u003cpassword\u003e --client_id \u003cclient_id\u003e --client_secret \u003cclient_secret\u003e\n\nHowever, this is verbose and prone to leaking your keys/tokens/passwords in shell history.\n\n\nYou can also import ~export.py~ as a module and call ~get_json~ function directly to get raw JSON.\n\n\nI *highly* recommend checking exported files at least once just to make sure they contain everything you expect from your export. If not, please feel free to ask or raise an issue!\n\n:end:\n\n* API limitations\n\n*WARNING*: reddit API [[https://www.reddit.com/r/redditdev/comments/61z088/sample_more_than_1000_submissions_within_subreddit][limits your queries to 1000 entries]].\n\nI *highly* recommend to back up regularly and keep old exports. Easy way to achieve it is command like this: \n\n: python3 -m rexport.export --secrets /path/to/secrets.py \u003e\"export-$(date -I).json\"\n\nOr, you can use [[https://github.com/karlicoss/arctee][arctee]] that automates this.\n\n# TODO link to exports post?\n# TODO link how DAL part can merge them together\n\nCheck out these links if you're interested in getting older data that's inaccessible by API:\n\n- [[https://www.reddit.com/r/DataHoarder/comments/d0hjs7/reddit_takeout_export_your_account_data_as_json/ezbbcxe][comment]] by /u/binkarus\n- [[https://www.reddit.com/r/ideasfortheadmins/wiki/faq#wiki_can_we_have_a_way_to_download_our_entire_history_even_though_reddit_cuts_off_at_a_certain_point.3F][Reddit admis]] say that the rationale behind the API limitation is performance and caching\n- perhaps you can request all of your data under [[https://www.reddit.com/r/DataHoarder/comments/d0hjs7/reddit_takeout_export_your_account_data_as_json/eza0nsx][GDPR]]? I haven't tried that personally though.\n- [[https://pushshift.io][pushshift]] can help you retrieve old data. You can use [[https://github.com/purarue/pushshift_comment_export][purarue/pushshift_comment_export]] to get the data.\n\n  \n* Example output\nSee [[file:example-output.json][example-output.json]], it's got some example data you might find in your data export. I've cleaned it up a bit as it's got lots of different fields many of which are probably not relevant.\n\nHowever, this is pretty API dependent and changes all the time, so better check with [[https://www.reddit.com/dev/api][Reddit API]] if you are looking to something specific.\n\n\n* Using the data\n  \n#+begin_src python :dir src  :results drawer :exports results\nimport rexport.exporthelpers.dal_helper as D; return D.make_parser().epilog\n#+end_src\n\n#+RESULTS:\n:results:\n\nYou can use =rexport.dal= (stands for \"Data Access/Abstraction Layer\") to access your exported data, even offline.\nI elaborate on motivation behind it [[https://beepb00p.xyz/exports.html#dal][here]].\n\n- main usecase is to be imported as python module to allow for *programmatic access* to your data.\n\n  You can find some inspiration in [[https://beepb00p.xyz/mypkg.html][=my.=]] package that I'm using as an API to all my personal data.\n\n- to test it against your export, simply run: ~python3 -m rexport.dal --source /path/to/export~\n\n- you can also try it interactively: ~python3 -m rexport.dal --source /path/to/export --interactive~\n\n:end:\n\nExample output:\n\n: Your most saved subreddits:\n: [('orgmode', 50),\n:  ('emacs', 36),\n:  ('QuantifiedSelf', 33),\n:  ('AskReddit', 33),\n:  ('selfhosted', 29)]\n\n\n\n","funding_links":[],"categories":["Python","others"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkarlicoss%2Frexport","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkarlicoss%2Frexport","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkarlicoss%2Frexport/lists"}