Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/karlicoss/pockexport

Export/access your Pocket data, including highlights!
https://github.com/karlicoss/pockexport

backup data-liberation export pocket takeout

Last synced: about 11 hours ago
JSON representation

Export/access your Pocket data, including highlights!

Awesome Lists containing this project

README 

        
#+begin_src python :dir src :results drawer :exports results

import pockexport.export as E; return E.make_parser().prog

#+end_src



#+RESULTS:

:results:

Export your personal Pocket data, *including highlights* as JSON.

:end:



* Setting up

1. The easiest way is =pip3 install --user git+https://github.com/karlicoss/pockexport=.



   Alternatively, use =git clone --recursive=, or =git pull && git submodule update --init=. After that, you can use =pip3 install --editable=.

2. To use the API you need a =consumer_key=. You've got three alternatives here:

   - Lazy way: just use =78809-9423d8c743a58f62b23ee85c= as the consumer key. 



     This seems to be the hardcoded key Pocket uses for their web app. It's public and shared between all users, so there is no problem with sharing it here.

   - Chaotic way: get API key directly from web app. 



     The benefit of doing this is that the API gives away more data, *including highlights* (unlike the consumer key you get by registering your own app).



     Note that this will likely just give you the same key as above, although I'm not 100% sure.

     E.g. [[https://github.com/karlicoss/pockexport/issues/6#issuecomment-1071883685][some people]] reported this method didn't work for them, unlike just simply using the hardcoded key.

     

     # TODO link to exports?

     To do that, go to [[https://app.getpocket.com][Pocket web app]], open Network Monitor from your browser dev tools 

     (e.g. [[https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor#UI_overview][firefox]]), and refresh the page.

     

     You can find =consumer_key= in 'Request URL' for any of =json= requests.

     

     If you still struggle with these steps, I recommend a comprehensive [[https://willschenk.com/articles/2019/reverse_engineering_apis_using_chrome/#select-the-network-tab][tutorial]].

     

     This is sort of hacky, but only way I know of extracting highlights. I tried registering apps targeting other platforms (e.g. web/extension), but still nothing, seems that Pocket's consumer key is hardcoded in backend code or something.

   - Lawful good way: register an app [[https://getpocket.com/developer/apps/new][here]] with 'Retrieve' permissions and type 'Desktop (other)'.



     Then open app in the list and take note of =consumer_key=.

     

3. Follow [[https://github.com/tapanpandita/pocket#oauth][these]] instructions to retrieve an API token using =consumer_key= you got on the previous step. 



   You can use anything as =redirect_uri=, e.g. =https://example.com=. You should get =access_token= after that.

   

   You only need to do this step once, after that you can use =access_token= straightaway.



* Exporting



#+begin_src python :dir src :results drawer :exports results

import pockexport.export as E; return E.make_parser().epilog

#+end_src



#+RESULTS:

:results:



Usage:



*Recommended*: create =secrets.py= keeping your api parameters, e.g.:



: consumer_key = "CONSUMER_KEY"

: access_token = "ACCESS_TOKEN"



After that, use:



: python3 -m pockexport.export --secrets /path/to/secrets.py



That way you type less and have control over where you keep your plaintext secrets.



*Alternatively*, you can pass parameters directly, e.g.



: python3 -m pockexport.export --consumer_key  --access_token 



However, this is verbose and prone to leaking your keys/tokens/passwords in shell history.



You can also import ~pockexport.export~ as a module and call ~get_json~ function directly to get raw JSON.



I *highly* recommend checking exported files at least once just to make sure they contain everything you expect from your export. If not, please feel free to ask or raise an issue!



:end:



* Limitations

I'm not aware of any limits on number of old entries you can retrieve through API; it doesn't even have pagination. If you know of them, please let me know or open PR!



I *highly* recommend to back up regularly and keep old exports. Easy way to achieve it is command like this: 



: python3 -m pockexport.export --secrets /path/to/secrets.py >"export-$(date -I).json"



Or, you can use [[https://github.com/karlicoss/arctee][arctee]] that automates this.



# TODO link to exports post?

# TODO could add this to epilog of export script if api is restrictive?



* Example output

See [[file:example-output.json][./example-output.json]], it's got some example data you might find in your data export.



* Using data

  

#+begin_src python :dir src  :results drawer :exports results

import ghexport.exporthelpers.dal_helper as D; return D.make_parser().epilog

#+end_src



#+RESULTS:

:results:



You can use =ghexport.dal= (stands for "Data Access/Abstraction Layer") to access your exported data, even offline.

I elaborate on motivation behind it [[https://beepb00p.xyz/exports.html#dal][here]].



- main usecase is to be imported as python module to allow for *programmatic access* to your data.



  You can find some inspiration in [[https://beepb00p.xyz/mypkg.html][=my.=]] package that I'm using as an API to all my personal data.



- to test it against your export, simply run: ~python3 -m ghexport.dal --source /path/to/export~



- you can also try it interactively: ~python3 -m ghexport.dal --source /path/to/export --interactive~



:end: