https://github.com/vindarel/cl-torrents

Searching torrents on popular trackers - CLI, readline, GUI, web client. Tutorial and binaries (issue tracker on https://gitlab.com/vindarel/cl-torrents/)
https://github.com/vindarel/cl-torrents

1337 1337x common-lisp pirate-bay torrents tutorial web-scraping

Last synced: 3 days ago
JSON representation

Searching torrents on popular trackers - CLI, readline, GUI, web client. Tutorial and binaries (issue tracker on https://gitlab.com/vindarel/cl-torrents/)

• Host: GitHub
• URL: https://github.com/vindarel/cl-torrents
• Owner: vindarel
• Created: 2017-10-03T22:33:05.000Z (about 7 years ago)
• Default Branch: master
• Last Pushed: 2024-04-08T15:10:16.000Z (9 months ago)
• Last Synced: 2024-10-28T05:59:42.522Z (about 2 months ago)
• Topics: 1337, 1337x, common-lisp, pirate-bay, torrents, tutorial, web-scraping
• Language: Common Lisp
• Homepage: https://vindarel.github.io/cl-torrents/tutorial.html
• Size: 1.07 MB
• Stars: 104
• Watchers: 6
• Forks: 4
• Open Issues: 0
• Metadata Files:
  • Readme: README.org

Awesome Lists containing this project

README

        
[[https://gitlab.com/vindarel/cl-torrents][file:https://gitlab.com/vindarel/cl-torrents/badges/master/pipeline.svg]]

  This is a little app to  search for torrents on popular trackers and

  to open them with a local or a remote client.

  It comes as:

- a *lisp library*,

- a *command line* interface,

- a terminal *readline interface*,

- a simple *GUI*.

- The *[[https://github.com/vindarel/cl-torrents-web][Web UI]]* works too.

  We currently  scrape [[http://1337x.to][1337x.to]]  (since v0.9), [[https://www.torrentdownloads.me][torrentdownloads.me]]

  (v0.10) and [[https://torrent-paradise.ml][torrents-paradise]] (v0.11) and present the results sorted by seeders.

  Download for GNU/Linux 64 bits: see [[https://gitlab.com/vindarel/cl-torrents/tags][Releases]] (/23MB, self-contained

  executable/).

NOTE: torrents-paradise has a rich collection, a simple website as in KISS, it works on IPFS and has

it offers an API… frankly, I want to use its web service and click on its adds.

#+html: 


#+html: 


#+html: 


With the new default Yaru theme (and [[https://lisp-journey.gitlab.io/blog/pretty-gui-in-common-lisp-with-nodgui-tk-themes/][there are more]]):

#+html: 


** Installation

   The library is on Quicklisp (february 2019) and [[http://ultralisp.org/][Ultralisp]].

: (ql:quickload "torrents")

: (torrents:search-torrents "tears of steel")

Note: the master branch needs [[https://github.com/libre-man/cl-transmission/][cl-transmission]] which was added

to the Quicklisp distribution of december, 2019.

   See the download link of the binary above.  It is a self-contained executable for

   GNU/Linux  64-bits.    You  do  /not/   need  to  install   a  Lisp

   implementation. It's a 23MB binary.

To build the readline app, do:

: make build

To build the Tk GUI:

: make build-gui

then run

: ./torrents --help

: ./torrents-tk &

_Troubleshooting_

The Tk GUI needs [[https://notabug.org/cage/nodgui/][nodgui]] newer than march, 2019.

If a search doesn't work with a =CRYPTO_num_locks= error message:

#+BEGIN_EXPORT ascii

> ./torrents matrix

searching '(matrix)' on 1337... no results.

error: The alien function "CRYPTO_num_locks" is undefined.

searching '(matrix)' on DOWNLOADSME... no results.

error: The alien function "CRYPTO_num_locks" is undefined.

#+END_EXPORT

You might need to upgrade your OpenSSL version. See [[https://gitlab.com/vindarel/cl-torrents/issues/5][this issue]].

** Usage

   We can  use this little app  both in a  Lisp REPL (Slime) and  in a

   terminal, as a command line tool or from its embedded interactive prompt.

   Results   are  cached   in  =~/.cl-torrents/cache/=.   Delete  this

   directory if you want new results.

*** Lisp REPL

   Search for torrents:

#+BEGIN_SRC text

(use-package :torrents)

(search-torrents "matrix")

  [...]

  6: The Matrix Revolutions (2003) BRRip [Dual Audio] [Hindi+Eng]

  5: Matrix (1999)Blu-Ray 720p Dublado PT-BR - mo93438

  4: The Matrix Trilogy (1999-2003) BDRip 1080p Dual Audio [ Hind

  3: The Matrix Trilogy (1999-2003) BluRay BDRip 1080p AC3

  2: The Matrix Trilogy (1999-2003) + Extras 1080p BluRay x264 Du

  1: The Matrix Reloaded (2003) FullHD, Dual Audio: English + Spa

  0: Matrix FRENCH DVDRIP 1999 COOL

#+END_SRC

  To help finding the most  relevant results (the website sometimes is

  a bit relaxed  with our search terms), each keyword  is printed in a

  different color:

#+html: 


  (you need  to enable ansi  colors in Slime,  see [[https://github.com/enriquefernandez/slime-repl-ansi-color][slime-repl-ansi-color]].

  Disable this feature with =(setf cl-ansi-text:*enabled* nil)=)

  We get a magnet link with:

#+BEGIN_SRC lisp

(magnet 0)

;; "magnet:?xt=urn:btih:40eca43690cf1b99b0a4d485ebf4855d20b0bac5"

#+END_SRC

  We can open it with a torrent client (transmission-gtk by default):

#+BEGIN_SRC lisp

(download 0)

#+END_SRC

  Download it with a remote client (only transmission-remote so far):

  you need settings, see below. TODO

#+BEGIN_SRC lisp

;; in ~/.config/torrents.lisp

(setf *remote-client-url* "my.url.without.port")

(setf *remote-client-username* "transmission")

(setf *remote-client-password* "password")

;; port is 9091 by default.

#+END_SRC

#+BEGIN_SRC lisp

(remote-download 0)

#+END_SRC

  And voilà :)

  We may add more features, eventually.  Our goal is rather to write a

  tutorial to show you diverse Common Lisp topics.

**** Parameters

=*cache-p*=: if nil, don't use the cache system.

*** Command line

Use the  options given below,

#+BEGIN_SRC text

$ torrents -h

CL-torrents. Usage:

Available options:

  -d, --details            print more details (like the torrent's url)

  -h, --help               print this help text

  -i, --interactive        enter an interactive repl

  -m, --magnet ARG         get the magnet link of the given search result.

  -n, --nb ARG             set the maximum number of results to print.

  -o, --open INDEX         download with a torrent client (transmission-gtk by default)

  -v, --version            print the version

#+END_SRC

Example: below we  search for "matrix…", we display only  1 result and

we get the magnet link of the first result.

: $ torrents -n 1 matrix french 1999 cool -m 0

#+html: 


*** Readline interface

Start it with =torrents -i=.

See the =help= and the help of each command.

The available commands are (use the auto-completion):

- =search=: search for torrents, print the results sorted by number of seeders.

- =magnet =, where /i/ is a result index,

- =open= or =firefox =: open the given torrent page in a web browser

- =download  =:   open  a  magnet   link  with  a   torrent  client

  (transmission-gtk by default)

- =filter  =:  show  results  that   have  this  text  in  their

  title. This reduces the tab-completion of ids for all commands.

- =highlight=  (and  =unhighlight=):  highlight  the  given  words  in

  yellow, for this session.

- =url =

- =nb-results= =n= to set the number of results to print to /n/

- =details=: toggle the display of details

- =quit= (or =C-c=)


And the available settings to change with =set= are (use =set='s autocompletion):

- =*details*=: if  true, print more  details below each row  (like the

  url). Defaults to false.

- =*nb-results*=: how many results to print. Defaults to 20.

- =*browser*=: the browser name to use to open urls. Defaults to Firefox.

- =*torrent-client*=: the desktop torrent client name. Defaults to Transmission.

- =*cache-p*=: if true (the default), use the cached results.

You can also set them in a configuration file.

Note that we are in  a regular [[https://tiswww.case.edu/php/chet/readline/readline.html#SEC9][readline]] interface, so the  usual keybindings and

options are available. For example, Readline obeys the =~/.inputrc=

startup file. You can change the way the TAB key does completion:

: TAB: menu-complete

if you add this, the first press on TAB will insert the first

completion candidate (VS listing all the choices, and requiring a

second keypress). For cl-torrents, it is convenient because we can

enter the first result quickly: I typically do =search foo= then

=download TAB=.

Note: I found out that =C-x C-r= re-reads the inputrc file, so you can

try without quitting cl-torrents.

#+html: 


*** Configuration files (in development)

    =cl-torrents= will  read two configuration files.  An ini-style one,

    and a lispy one.

**** Ini-style config

    First, it will search for a =~.torrents.conf= file

    from =~/.config/.torrents.conf= and  =~/.torrents.conf=.  The last

    one takes precedence.

    For example, those are the default values:

#+BEGIN_SRC conf

[default]

scrapers = 1337 DOWNLOADSME

browser = firefox

nb-results = 20

cache-p = true  # use the cache system.

#+END_SRC

Moreover, you  can set parameters related to the repl:

#+BEGIN_SRC text

[default]

# again, default values:

confirm-exit = true  # ask with a yes/no prompt before quiting the app.

verbose = false

history = true  # use the repl history.

write-history = true  # read the history, but don't write to it.

#+END_SRC

  You can set  them for all replic-based apps  in =~/.replic.conf=, or

  override them in =.torrents.conf=.

  /Note: we only use a "default" profile for now./

**** Lisp init file

If the  file =~/.torrents.lisp=  exists, it will  be loaded  after the

=.conf= one and before the command line arguments.

The option =--no-userinit= prevents it from loading.

You can  write whatever you  want there, it is  just a lisp  file that

will be  =load='ed at startup.

/Note/:  what we  will eventually  do then  is to  expose cl-torrents'

mechanism via hooks.

**** Extending the app, creating commands

One thing you can do is define  additional commands to be found at the

repl. We do so by following [[https://github.com/vindarel/replic/][replic]]'s mechanism:

- define functions inside the =torrents.user= package

- define the completion choices for some functions

- =export=   the    functions   you   want   to    see   turned   into

  commands. =cl-torrents=, with the help of the =replic= library, will

  automatically turn  them into commands available  at the application

  prompt.

You  can  copy the  example  below  in =~/.torrents.lisp=  and  re-run

=torrents -i= to try this out.

#+BEGIN_SRC lisp

(in-package :torrents.user)

(defparameter *names* '()

  "List of names (string) given to `hello`. Will be autocompleted by `goodbye`.")

(defun hello (name)

  "Takes only one argument. Adds the given name to the global

  `*names*` global variable, used to complete arguments of `goodbye`. "

  (format t "hello ~a~&" name)

  (push name *names*))

(defun goodbye (name)

  "Says goodbye to name, where `name` should be completed from what was given to `hello`."

  (format t "goodbye ~a~&" name))

;; Custom completion for goodbye:

(replic.completion:add-completion "goodbye" (lambda () *names*))

;; and export the two functions to find them as commands.

(export '(hello

          goodbye))

#+END_SRC

** Ideas, todos

- [X] cache (on files) v0.3

- [X] CI

- use a local copy of TPB

- [X] build  with  ECL. Used  [[https://hub.docker.com/r/daewok/lisp-devel/][the  lisp-devel]]  docker

  image. Result: a 52Mo  executable with a runtime error.

- [X] open an url

- download the torrent file

- [X] ini config file, to parse with [[https://common-lisp.net/project/py-configparser/][py-configparser]]

- [X] add torrent to a local torrent client

- [X]  add  the   torrent  to  a  remote   transmission  client,  with

  [[https://github.com/libre-man/cl-transmission/][cl-transmission]]. v0.12

- [X] self-contained web app ([[https://lispcookbook.github.io/cl-cookbook/scripting.html][doc]]). POC: [[https://github.com/vindarel/cl-torrents-web][cl-torrents-web]].

- [X] interactive prompt, with completion of commands.

** Release notes

[[https://gitlab.com/vindarel/cl-torrents/tags][https://gitlab.com/vindarel/cl-torrents/tags]]

*** WIP

- added:  a simple  Tk GUI  (search,  open in  browser, download  with

  torrent desktop client).

*** v0.12

- june, 2021: added [[https://torrent-paradise.ml][torrents-paradise]] API.

*** v0.11

- november, 2019: added connection to transmission-remote

- added rarbg scraper (needs new tests)

- june, 2019: changed the location of the lisp init file from

  =~/.torrents.lisp= to =~/.config/torrents.lisp=.

- added:  a =filter=  command,  to only  display  results whose  title

  contains a given  string. It reduces the TAB-completion  of ids (but

  doesn't constrain it).

- added: all functions can now TAB-complete the list of ids.

- added: scrape and display torrents' size.

- fixed =-d= cli arg.

- added:  load  =~/.torrents.lisp=,  create   new  commands  from  the

  =torrents.user= package.

- added   the    ability   to   read   an    ini-like   config   file,

  =~/.torrents.conf=. All exported variables  from the =:torrent= package

  can be overriden. They are written without earmuffs:

#+BEGIN_SRC lisp

           :*nb-results*

           :*browser*

           :*torrent-client*

           :*cache-p*

#+END_SRC

gives

#+BEGIN_SRC text

# ~/.torrents.conf

# those are the defaults.

[default]

nb-results = 20

browser = firefox

torrent-client = firefox

cache-p = true

#+END_SRC

  See more on replic.

  (all parameters need more tests)

- added missing =-o= option to download with a torrent client (transmission-gtk).

- =cl-torrents.conf= configuration file:

  - choose the list of scrapers.

- =download= command (open magnet link with transmission by default).

- use the [[https://github.com/vindarel/replic/][replic]] library to build the repl, the commands and their completions.

  - -> new builtin =highlight= and =unhighlight= commands,

  - -> better error handling (don't exit the repl on an error).

- fix cli usage without a keyword search, print help.

*** v0.10

- added [[https://www.torrentdownloads.me][torrentdownloads.me]] scraper.

- new parameter, =*cache-p*=: if nil, don't use the cache.

*** v0.9

- added [[http://1337x.to][1337x.to]] scraper.

- removed piratebay.to, katcr.co, torrent.cd, they're down :/

*** v0.8

- added a readline-based repl  with some commands: =search=, =magnet=,

  =open=, with their completion.  See =help=.  Built with [[https://github.com/mrkkrp/cl-readline][cl-readline]],

  but might use [[https://github.com/vindarel/replic][replic]] anytime soon.

*** v0.7

- fix cache not created by binary

- create cache in =~/.cl-torrents/cache/=

* Dev

Clone cl-torrents in =~/quicklisp/local-projects=,

now you can =(ql:quickload :torrents)=.

We use our [[https://github.com/vindarel/replic/][replic]] library to automatically build the readline repl.

Unit tests:

: make test

End-to-end tests (checking that the websites respond and our scrapers still work):

: make test-end2end

Try the Ltk GUI: load =gui-tk.lisp= and run =(main)=.

* Resources

  Don't miss these good resources:

- http://lisp-lang.org/

- [[Https://github.com/CodyReichert/awesome-cl][Awesome-cl]]

- [[https://lispcookbook.github.io/cl-cookbook/][the Common Lisp Cookbook]]

- http://quickdocs.org/

- (my) [[https://lisp-journey.gitlab.io/][lisp-journey]]: introductory posts, lists of resources, selection

  of screencasts.

** Tutorial

  *update, 2021*: this tutorial is one of the first things I wrote

  when discovering CL and it probably bitrot a bit. I'd recommend the

  Cookbook now as I ported most of its useful content there.

   Writing   a   little   web   scraper   like   this   one   is   not

   difficult. However, I had to spend  some time to find out the right

   libraries and resources.  It is also  not trivial at first to start

   a Lisp project.  So the first thing  we did is write a tutorial. It

   is a mix of hopefully useful stuff:

- web scraping,

- async web scraping,

- trying out things at the REPL,

- where to find documentation,

- creating and loading a project,

- basic data structures and gotchas,

- some useful libraries,

- unit tests, with mocks, running tests from the shell, continuous integration,

- parsing command line arguments, creating executables, continuous delivery,

- basics of error handling,

- ...

  It will eventually  tackle more topics and  features (more settings,

  working with a local copy of TPB…) but in the meanwhile, read

  the [[https://vindarel.github.io/cl-torrents/tutorial.html][tutorial]] !

  It was built with https://github.com/fniessen/org-html-themes.

** Bypassing ISP blocking

Some Internet Service Providers block access to torrent sites, as it's

the case in France. They currently don't block the HTTP version of

torrents-paradise, but that one works on IPFS anyways. You can bypass

them by using other DNS servers than your ISP's.

Here's how to do it on Ubuntu: https://doc.ubuntu-fr.org/dns (french)

You can use [[https://www.opennic.org/][OpenNIC]] addresses or again the ones of the [[https://www.fdn.fr/actions/dns/][French Data Network]] association.

* Licence

MIT.