Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/karlicoss/stexport

Export and access your Stackexchange data
https://github.com/karlicoss/stexport

backup data-liberation export stackexchange stackoverflow takeout

Last synced: 5 days ago
JSON representation

Export and access your Stackexchange data

Awesome Lists containing this project

README 

        
#+begin_src python :dir src :results drawer :exports results

import stexport.export as E; return E.make_parser().prog

#+end_src



#+RESULTS:

:results:

Export your personal Stackexchange data

:end:



* Setting up

1. The easiest way is =pip3 install --user git+https://github.com/karlicoss/stexport=.



   Alternatively, use =git clone --recursive=, or =git pull && git submodule update --init=. After that, you can use =pip3 install --editable=.

2. Get API =key=

   Register a new Oauth app [[https://meta.stackexchange.com/questions/261829/where-i-can-get-my-access-token-key-for-the-api][this]] for information.

   Once you've registered, find your app in [[https://stackapps.com/apps/oauth][the list]] and take note of

   - 'Key' -- this is your =key= parameter

   - 'Client Id' and 'Oauth Domain/Redirect URI' -- this might be useful for the next step

3. Get API =access_token=

   This is optional, since in principle you can run the script without access token.

   But some functionality won't work (e.g. =--user-sites=), and you might miss some of your private data, so it's recommended to do.



   Visit the following link, but



   - instead of =12345=, substitute 'Client Id' from step 3

   - instead of =http://example.com=, substitute 'Redirect URI' from step 3



   : https://stackoverflow.com/oauth/dialog?client_id=12345&redirect_uri=http://example.com&scope=private_info+no_expiry

   # https://api.stackexchange.com/docs/authentication#scope

   After that you should be authorised and redirected. Copy the =access_token= somewhere, you'll need it later.



* Exporting



NOTE: even if you decided not to use =access_token=, you still need to pass the parameter, just set it to empty string.

# TODO ugh. it doesn't capture --all-sites or --user-sites or anything. ugh

# really need a more literate approach somehow..



#+begin_src python :dir src :results drawer :exports results

import stexport.export as E; return E.make_parser().epilog

#+end_src



#+RESULTS:

:results:



Usage:



*Recommended*: create =secrets.py= keeping your api parameters, e.g.:



: key = "KEY"

: access_token = "ACCESS_TOKEN"



After that, use:



: python3 -m stexport.export --secrets /path/to/secrets.py



That way you type less and have control over where you keep your plaintext secrets.



*Alternatively*, you can pass parameters directly, e.g.



: python3 -m stexport.export --key  --access_token 



However, this is verbose and prone to leaking your keys/tokens/passwords in shell history.



I *highly* recommend checking exported files at least once just to make sure they contain everything you expect from your export. If not, please feel free to ask or raise an issue!



:end:



* Using data

  

#+begin_src python :dir src  :results drawer :exports results

import stexport.exporthelpers.dal_helper as D; return D.make_parser().epilog

#+end_src



#+RESULTS:

:results:



You can use =stexport.dal= (stands for "Data Access/Abstraction Layer") to access your exported data, even offline.

I elaborate on motivation behind it [[https://beepb00p.xyz/exports.html#dal][here]].



- main usecase is to be imported as python module to allow for *programmatic access* to your data.



  You can find some inspiration in [[https://beepb00p.xyz/mypkg.html][=my.=]] package that I'm using as an API to all my personal data.



- to test it against your export, simply run: ~python3 -m stexport.dal --source /path/to/export~



- you can also try it interactively: ~python3 -m stexport.dal --source /path/to/export --interactive~



:end: