https://github.com/knguyenanhoa/cli-arxiv

CLI tool for exploring arXiv (inspired by karpathy's brilliant ArXiv Sanity Preserver)
https://github.com/knguyenanhoa/cli-arxiv

Last synced: 3 months ago
JSON representation

CLI tool for exploring arXiv (inspired by karpathy's brilliant ArXiv Sanity Preserver)

• Host: GitHub
• URL: https://github.com/knguyenanhoa/cli-arxiv
• Owner: knguyenanhoa
• License: gpl-2.0
• Created: 2019-12-17T13:56:26.000Z (almost 5 years ago)
• Default Branch: master
• Last Pushed: 2021-05-22T14:43:27.000Z (over 3 years ago)
• Last Synced: 2024-05-02T20:13:50.475Z (6 months ago)
• Language: Python
• Size: 82 KB
• Stars: 38
• Watchers: 3
• Forks: 5
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# cli-arxiv

CLI tool for exploring arXiv (inspired by karpathy's brilliant ArXiv Sanity

Preserver)

The script will create data/pdf/, data/to-read/, data/txt/ and

data/summary/ directories to hold files downloaded from arXiv. I am also

aware that this is a rather stupid way to implement a datastore but DBs

seem a bit over the top. Text from PDFs are auto-converted on downloaded

and are used to suggest future articles to the user. Downloading

articles is idempotent.

It uses tfidf with cosine similarity to recommend articles to you. Frequency

sum ranking from the tfidf matrix is also implemented and can be switched to by

editing the ml/main\_ml.py file (set metric="sum\_freq"). Please do add other

algorithms as you see fit.

If you'd like to alter this code for any reason, please fork it because

I will be pushing to this repo from time to time. Cheers!

# INSTALL

- Clone the repo (it uses python 3).

- Install dependencies.

```

pip install -r requirements.txt

```

- Start the script.

```

python main.py

```

# ISSUES

- There may be a few seconds of network delay when browsing new articles for the first

  time as the script has to go fetch the article list from arXiv. The same

  holds for first-time article downloads. The script does cache article lists

  for a while before asking arXiv for an updated list. It's currently

  not possible to force a refresh of the list ad hoc.

- I should've used an available arXiv API wrapper.

# DISCLAIMER

- This script hits the arXiv web API to function so please do not alter it to

  spam the arXiv site (though I don't think one can meaningfully impact

  their servers with this). I take no responsibility for its misuse.

# TODO

- ML should store learnt model somewhere to use later

- Recommended articles should use normalized rank metric

- Add ability to fetch new data from arXiv on demand