Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/cschreib/flexible-arxiv-rss

Suite of tools to discover new articles on the arXiv, filter them, and broadcast them as an RSS feed, for your own use or for others.
https://github.com/cschreib/flexible-arxiv-rss

arxiv rss-generator

Last synced: 30 days ago
JSON representation

Suite of tools to discover new articles on the arXiv, filter them, and broadcast them as an RSS feed, for your own use or for others.

Awesome Lists containing this project

README 

          
Install instructions

====================



*) Install a simple web server (nginx):

        sudo apt-get install nginx-common nginx-light



   Note: you do not need root permission to do any of the following.



*) Build and install the arxiv-transform tools. From the source directory:

        mkdir build && cd build

        cmake ../

        make

        make install



*) Run the web server: (note: the path has to be absolute here)

        nginx -c /home//.local/share/arxiv-transform/nginx.conf



   It may print an error about being unable to open a log file. You can ignore this error. If

   instead you get errors about "98: Adress already in use", this means that nginx is already

   running, so it is not necesary to run this command again. To make sure everything is fine, you

   can "killall nginx" and run the command once more. Ideally, this command should be added to

   your list of commands to run at startup as well, so you don't have to manually call it.



*) Then we need to setup the daily probe of the arXiv. If you leave your computer running during

   the whole week without turning it off, you will want to use a chron job. For example, open your

   cron table with:

        crontab -e



   Then add this line to the file to probe the arXiv every day at 7am:

        00 07 * * * /home//.local/share/arxiv-transform/poll_arxiv_oai.sh



   You can choose this update time yourself: you should choose a time that is a) before you

   usually arrive to work, and b) after the arXiv daily update to remain on top of the flow...



   If you shutdown your computer every day, then you can instead simply add the command to your

   list of startup programs:

        /home//.local/share/arxiv-transform/poll_arxiv_oai.sh



   It does not matter if this command is executed before or after the web server is started.



*) To continue we need a first probe of the arXiv to work with.

   Run the script manually (just this one time):

        /home//.local/share/arxiv-transform/poll_arxiv_oai.sh



*) Then open your favorite RSS reader and use the following RSS feed URL:

        http://localhost:8080/arxiv/rss/feed.rss



   If you are using Thunderbird, you need a "Blog and news" account. If you don't have one already,

   go to "Edit > Account Settings > Account Actions > Add Feed account...". This will add a new

   entry alongside your inbox.



   Right click on that entry and choose "Subscribe...", then write the above URL in the "Feed URL"

   edit box. Then press "Add".



*) And that's it! Now you will get an RSS feed from the arXiv without missing any article if you go

   traveling without internet connection for a week. The first time the program is ran, it will

   fetch the new articles from the past 5 days, and will accumulate papers with up to 2 weeks of

   history.



*) Note that this setup will only work locally, one one computer. If you need to access this RSS

   feed from multiple computers, or if you wish to use a feed aggregator, you need to make the

   website public, and use the public address instead of "localhost" when registering to the feed.

   This may require you to:

    1) Have a static public IP address (depending on your internet provider, you can have either a

       fixed or dynamic address). NB: you can see your ip by Googleing "what is my ip".

    2) Have a static local IP address (192.168.0.1, for example).

    3) Configure your router to forward the traffic through port 8080 toward your computer.



   Obviously this requires you to have admin access to your network. If this is not possible, an

   alternative solution is to build (or pay for) a dedicated server on which the code will run.



   The filters will be shared among all computers, but each computer can have a different read

   history.



*) You can also edit the preferences.xml file to setup some filtering to reduce the amount of

   papers that appear in the RSS feed. See next section.



Customizing RSS feed

====================



Open the file:

    /home//.local/share/arxiv-transform/preferences.xml



This file controls, among other things, the various filters that can be applied to the arXiv

entries before they are sent to the RSS feed. Below is a description of all these options.



*) keep_history

This defines the amount of time a paper will remain in the RSS feed. Papers older than this will

be removed. You should set this amount of time to the longest time you may expect your computer to

be turned off (or without internet access). The syntax is that of the "date" command line tool, so

you can have "3 days ago" or "1 month ago". For example:

    2 weeks ago



*) star_include

This is a list of comma-separated words or expressions that will be searched for in each paper's

title and abstract. If at least one of these is found, the paper will appear with a little star in

its title in your RSS feed. These are case-insensitive. For example:

    galaxy, main sequence, main-sequence, bulges



*) content_exclude

Same as above, except that a paper is deleted if any of these words or expressions is found in the

title or abstract.

    chemistry, neutralino, comet



*) categories_must_include/categories_exclude

These two options work exactly the same as star_include and content_exclude, but will be applied

to the arXiv category string instead. The category string contains all the categories the paper

has been attributed to. There can be only one category (usually), or several if the paper has been

crossposted. For example:

    astro-ph.GA, astro-ph.CO

    gr-qc, astro-ph.SR

... will only display articles listed in astro-ph.GA or astro-ph.CO, and will exclude those that

are also listed in gr-qc and astro-ph.SR.



*) allow_crosspost

Set this to "0" to hide all entries that are listed in more than one arXiv category.



*) allow_replacements

Set this to "0" to only show papers when they appear on arXiv for the first time, and hide

announcements for updates and revisions.



*) only_accepted

Set this to "1" to only display the papers that contain the word "accepted" in their comment

field. The goal is to only list papers that are accepted for publication, rather than just

submitted. This option will take precedence over "allow_replacement", so you may see the same

paper appearing twice if the authors update the arXiv version multiple times after the paper

has been accepted.