Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/eikek/dlm
download manager for the command line
https://github.com/eikek/dlm
Last synced: 14 days ago
JSON representation
download manager for the command line
- Host: GitHub
- URL: https://github.com/eikek/dlm
- Owner: eikek
- License: gpl-3.0
- Created: 2015-05-17T22:27:08.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2016-10-27T22:22:30.000Z (about 8 years ago)
- Last Synced: 2023-03-12T07:48:13.229Z (over 1 year ago)
- Language: Common Lisp
- Homepage: https://eknet.org/main/projects/dlm/
- Size: 64.5 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.org
- License: LICENSE
Awesome Lists containing this project
README
#+title: dlm - a download manager
#+language: en
#+SETUPFILE: ~/.emacs.d/org-templates/eknet-3.org
#+OPTIONS: toc:t num:nil
* COMMENT Emacs setup :noexport:
This is some setup code for emacs.
Execute the following to add font-lock stuff to higlight some of the
macros defined here. This is copied from the [[https://github.com/magnars/dash.el][dash]] library.
#+begin_src emacs-lisp :tangle yes :results none
(require 'dash)
(defun dlm-enable-font-lock ()
"Add syntax highlighting to dash functions, macros and magic values."
(eval-after-load "lisp-mode"
'(progn
(let ((new-keywords '("defcommand" "when-let" "->" "cond->" "with-current-dir"))
(special-variables '("user-error")))
(font-lock-add-keywords 'lisp-mode
`((,(concat "\\_<" (regexp-opt special-variables 'paren) "\\_>")
1 font-lock-variable-name-face)) 'append)
(font-lock-add-keywords 'lisp-mode
`((,(concat "(\\s-*" (regexp-opt new-keywords 'paren) "\\_>")
1 font-lock-keyword-face)) 'append))
(--each (buffer-list)
(with-current-buffer it
(when (and (eq major-mode 'lisp-mode)
(boundp 'font-lock-mode)
font-lock-mode)
(font-lock-refresh-defaults)))))))
(dlm-enable-font-lock)
#+end_src
Execute the following to start slime using a clean environment as
generated by nix. After evaluating this, the function ~dlm/connect~
can reload/restart the connection and load dlm. Also ~dlm/run-tests~
runs all tests, which is bound to =C-c M-,= in lisp buffers.
#+begin_src emacs-lisp :tangle yes :results silent
(require 'slime)
(defvar dlm/project-dir (file-name-directory (buffer-file-name)))
(setq inferior-lisp-program (concat "nix-shell " dlm/project-dir "/build --pure --command sbcl"))
(defun dlm/connect ()
"Connect or reconnect slime to dlm project. The project is
loaded after connecting."
(interactive)
(if (slime-connected-p)
(slime-restart-inferior-lisp)
(slime))
(while (not (slime-connected-p))
(sleep-for 2))
(slime-eval `(cl-user::load ,(format "%s/build/build.lisp" dlm/project-dir)))
(message "dlm loaded."))
(defun dlm/run-tests ()
"Run the tests and report in the minibuffer."
(interactive)
(unless (slime-connected-p)
(user-error "Please connect first."))
(if (slime-eval '(cl-user::test-dlm))
(message "All tests passed.")
(message "Tests failed!")))
(define-key lisp-mode-map (kbd "C-c M-,") 'dlm/run-tests)
(dlm/connect)
#+end_src
The following function generates the usage section from the help
menu. It generates the text, erases everything between the =# --
usage-start= and =# -- usage-end= markers and inserts the new text.
#+begin_src emacs-lisp :results none
(defvar dlm/bin "./build/dlm-0.0.2")
(unless (file-exists-p dlm/bin)
(user-error "Please make the dlm executable first."))
(defun dlm/command-list ()
(with-temp-buffer
(shell-command (concat dlm/bin " help") (current-buffer))
(goto-char (point-min))
(search-forward "Commands:")
(delete-region (point-min) (point))
(insert "'(")
(while (search-forward ":" nil t)
(backward-char 1)
(let ((beg (point)))
(end-of-line)
(delete-region beg (point)))
(insert "\"")
(back-to-indentation)
(insert "\""))
(end-of-buffer)
(insert ")")
(goto-char (point-min))
(eval (read (current-buffer)))))
(defun dlm/get-help (cmd)
(with-temp-buffer
(shell-command (concat dlm/bin " help " cmd) (current-buffer))
(goto-char (point-min))
(insert "** ")
(when (search-forward "Usage: " nil t)
(insert "\n#+begin_example\n")
(forward-line 1)
(insert "#+end_example"))
(when (search-forward "Options are:" nil t)
(insert "\n#+begin_example")
(search-forward-regexp "^$")
(insert "#+end_example" "\n"))
(buffer-string)))
(defun dlm/get-all-help ()
(let ((cmds (dlm/command-list)))
(with-temp-buffer
(erase-buffer)
(dolist (c cmds)
(insert (dlm/get-help c))
(insert "\n"))
(buffer-string))))
(defun dlm/update-usage ()
(let ((text (dlm/get-all-help)))
(save-excursion
(goto-char (point-min))
(search-forward-regexp "^\\* Usage$")
(search-forward "# -- usage-start")
(let ((beg (point)))
(search-forward "# -- usage-end")
(forward-line -1)
(delete-region beg (point))
(goto-char beg)
(insert "\n" text)))))
(dlm/update-usage)
#+end_src
* What and why
Dlm is a basic download manager for the command line: a wrapper around
tools like [[http://curl.haxx.se/][curl]] or [[https://rg3.github.io/youtube-dl/][youtube-dl]]. It maintains a [[https://sqlite.org/][sqlite]] database with
metadata of files that have been downloaded with dlm. It is written in
Common Lisp on a Linux platform. It might work on other platforms, but
it's not tested.
This metadata includes a sha hash of the file, its length, its
location on disk and custom metadata like the url and a lifetime. This
lifetime specifies after which time (default is 30 days) dlm can
delete the file from disk. The metadata will not be deleted so files
can be redownloaded again if needed.
Dlm wants to address the ever growing download folder. But instead of
just deleting all old files, dlm keeps track of the metadata,
including the source url. This allows to run consitency checks and
also to re-download a file that has been deleted already.
Dlm keeps track of the following metadata:
- remote url
- path to the file
- file size
- sha256 checksum of the file
- download time
- keep, a flag indicating to dlm that it should never delete this file
- lifetime: duration after which the file can be deleted
- re-downloads: the number of times this file has been downloaded
* Installation
Checkout the code:
#+begin_src bash
git clone https://github.com/eikek/dlm
#+end_src
and [[*Building][build it]]. There is a prebuild binary for linux can be downloaded
[[https://eknet.org/main/projects/dlm/dlm-0.0.1][here]].
* Usage
Dlm accepts commands which may take options. You can find out what is
available using =dlm help= and =dlm help =. If the command is
not found, a default command is used, which is (by default) =fetch=.
# -- usage-start
** The 'info' command
Usage:
#+begin_example
dlm [--]info [OPTIONS] FILES
#+end_example
Options are:
#+begin_example
-r, --raw-values Print raw db data values instead of human readable form.
-s, --structure Print information as a lisp data structure.
-h, --help Prints this summary
#+end_example
Show information about downloaded files.
For a given file the information from the database is displayed in
key-value form.
** The 'pending' command
Usage:
#+begin_example
dlm [--]pending [OPTIONS] ACTION
#+end_example
Options are:
#+begin_example
-k, --keep Search files that are specified to not be deleted.
-H, --no-header Don't print search parameters header.
-s, --source= Search in the url.
-l, --limit= Apply the action to the first 'n' items only.
-h, --help Prints this summary
#+end_example
Show pending downloads.
This queries the pending downloads. All downloads that have not
finished yet are listed separately with this command. You can use few
search options and actions:
Each search option that takes an argument is compared against the
corresponding field. You can use % character as a wildcard (matching
many characters).
Actions are then applied to each file in the result. Actions are
- list (the default): print it to stdout
- fetch: resume the download. This can be used if a download has been
cancelled. Note that it does *not* check whether a current process
is running regarding this download. It just starts a new one.
- clear: deletes the pending download entry from the db
Action names can be abbreviated to the shortest non-ambiguous
name. Only one action can be given at a time.
** The 'query' command
Usage:
#+begin_example
dlm [--]query [OPTIONS] ACTION
#+end_example
Options are:
#+begin_example
-k, --keep Search files that are specified to not be deleted.
-e, --existing Show only files that exists on disk.
-r, --nonexisting Show files that do not exist on disk.
-c, --valid Show files where its metadata (sha256, size, lastmod
timestamp) matches the db entry.
-C, --invalid Show files where its metadata (sha256, size, lastmod
timestamp) does not match the db entry.
-H, --no-header Don't print search parameters header.
-n, --name= Search in the filename.
-s, --source= Search in the url.
-l, --limit= Apply the action to the first 'n' items only.
-h, --help Prints this summary
#+end_example
Query the database for files that have been downloaded.
Each search option that takes an argument is compared against the
corresponding field. You can use % character as a wildcard (matching
many characters).
Actions are then applied to each file in the result. Actions are
- list: (the default) print information to stdout. By default output
is ansi-colored; this can be suppressed by setting the environment
variable DLM_COLOR=0.
- short-list: print only the local filename to stdout
- structure: print a lisp data structure
- fetch: download the file again, if the file exists and the metadata
matches the db, it is not downloaded again
- delete: deletes the file on disk, but not the record in the database
- prune: deletes the file and removes the record from the database
- clear: deletes the record from the db but leaves the file on disk
- keep: set the keep flag to true
- nokeep: set the keep flag to false
- move [dir]: moves the file to the given directory
- set-lifetime [secs]: set a new lifetime in seconds or use "2d10M"
strings (you can use m,d,h and M)
Action names can be abbreviated to the shortest non-ambiguous
name. Only one action can be given at a time. Note that checking file
metadata (the -C|c option) involves computing a checksum of the file
which may take some time depending on its size.
** The 'collect-garbage' command
Usage:
#+begin_example
dlm [--]collect-garbage [OPTIONS]
#+end_example
Options are:
#+begin_example
-s, --silent Don't print info messages.
-d, --dry Don't actually perform the action.
-h, --help Prints this summary
#+end_example
Deletes expired files.
Checks the last access time of each file with expired lifetime in the
db and deletes it, if it is older than the specified lifetime for this
file.
** The 'fetch' command
Usage:
#+begin_example
dlm [--]fetch [OPTIONS] URLS
#+end_example
Options are:
#+begin_example
-k, --keep Flag the file to never be deleted by dlm
-t, --target=./ The target directory
-l, --lifetime= The lifetime to set for this file. Default is 1.0m.
-u, --user= The username to authenticate with.
-p, --pass= The password to authenticate with.
-h, --help Prints this summary
#+end_example
Download a file at some url.
This will call other download programs like curl in order to download
a file at the given url. If multiple urls are specified they are
downloaded sequentially. What download program to use is determined by
the url. If, for example, the youtube-dl tool is available, certain
urls to video portals are downloaded using this tool. Otherwise curl
is the default fallback (unless configured otherwise).
The '--user' and '--pass' options are simply delegated to the real
download programs. Thus it depends on whether the program supports
these options.
If URL is a path to a local file, it is simply added to the database
and the source and location properties are both set to the same
path. Those files don't have a remote source and are treated a little
different. First, 'deleteing' them causes the db entry to be removed,
too. Then, obviously, they cannot be redownloaded and will be skipped
if tried.
** The 'help' command
Usage:
#+begin_example
dlm [--]help [OPTIONS] COMMAND
#+end_example
Options are:
#+begin_example
-h, --help Prints this summary
#+end_example
Shows a short help for this program.
If called without arguments, a little help is shown, listing all
commands with a short description. If a command is given, a more
detailed help to this command is shown (if provided).
# -- usage-end
* Configuring
Dlm reads a configuration file at =$HOME/.config/dlm/config.lisp=. It
is a normal lisp file that is loaded at the beginning. You can specify
another configuration file by setting the environment variable
=DLM_CONFIG= to another file.
** Settings
A few variables can be set:
The database is usually at =$HOME/.config/dlm/dlm.db=, which can be
overriden with:
#+begin_src lisp
(setq *database* "/path/to/a/file")
#+end_src
If you like to change the default command:
#+begin_src lisp
(setq *default-command* "query")
#+end_src
The default lifetime of a file is 30 days or you set it (in seconds)
via:
#+begin_src lisp
(setq *file-lifetime* (* 60 60)) ;; 1 hour, or
(setq *file-lifetime* (parse-duration "5d10h"))
#+end_src
The curl program is used to download files. If it's not in your path
you can set it:
#+begin_src lisp
(setq *fetch-default-bin* "/path/to/curl")
#+end_src
The options to curl are stored in ~*fetch-default-args*~, it is
=-O#C - ~a= where =~a= is replaced with the url. This string must
contain one =~a=.
Likewise the =youtube-dl= and =scp= program is configured:
#+begin_src lisp
(setq *youtube-dl-bin* "/path/to/youtube-dl")
(setq *youtube-dl-args "--option1 ~a")
(setq *scp-bin* "scp")
(setq *scp-args* "~a .")
#+end_src
As with curl the single =~a= in the line is replaced with the url.
If the =--target= options is not specified, the file is downloaded to
the current directory. You can specify a folder instead:
#+begin_src lisp
(setq *default-target* "/home/eike/Downloads")
#+end_src
For more settings look at the beginning of the =cli.lisp= and
=dlm.lisp= files.
** Extending
*** customize fetching files
The variable =*fetch-configs*= is a list of ~fetch-config~ objects. A
~fetch-config~ contains two functions, where the first is a predicate
that given an url returns true if the second function should be used
to download the file. This can be used to customize how certain urls
are downloaded. For example, the youtube-dl tool is setup using this
mechanism.
A config file may look like this:
#+begin_src lisp :result source
(defun my-program? (url)
;; return true if this url can be downloaded with the function below
...)
(defun my-program-download (url &optional user pass)
;; download and return the filename
...)
(add-fetch-config
:can-fetch? #'my-program?
:fetch #'my-program-download)
#+end_src
By default, this list contains a fetch-config for the =youtube-dl=
command, for the =scp= command and one for =curl=.
*** be notified when download is done
The variable ~*download-notify-hook*~ can contain a list of functions
that are all called with the metadata of a newly downloaded file. A
second boolean argument specifies whether this file was downloaded or
already existed. You can add functions like for example:
#+begin_src lisp
(push (lambda (md existed)
(when (and (not (getf md :error)) (not existed))
(external-program:run
"stumpish"
`("echo" ,(format nil "Download ~a finished." (getf md :source))))))
*download-notify-hook*)
#+end_src
The package ~external-program~ is available that you can use to call
out to other programs. The example raises a notification in the
stumpwm windowmanager.
Or you could run garbage collection after each download:
#+begin_src lisp
(push (lambda (md existed)
(declare (ignore md) (ignore existed))
(dlm-collect-garbage))
*download-notify-hook*)
#+end_src
*** custom query actions
The =query= command applies actions to each item in the result set. An
action is a function of two arguments: the metadata plist and the db
handle. There are several actions provided, but you can add your
own.
You need to create a factory function that, given a number of
arguments, creates the action function. The arguments are those given
after the action on the command line. For example, the =move= action
needs a target directory where to move each file to:
#+begin_src bash
dlm query move /new/path
#+end_src
This last argument is passed to the factory function which then
creates the action function to use for each item in the result.
Add this factory function to the variable ~*query-actions-alist*~.
#+begin_src lisp
(push (lambda (&rest args)
...)
,*query-actions-alist*)
#+end_src
To ease this a little, the function ~make-action-fn~ can be
used. First define your custom action function by declaring all extra
arguments at the beginning and the ~metadata~ and ~db~ argument at
last and then use ~make-action-fn~ to create the factory function.
#+begin_src lisp
(defun my-custom-action (arg1 arg2 md db)
...)
(push `("myaction" . ,(make-action-fn #'my-custom-action))
*query-actions-alist*)
#+end_src
The ~make-action-fn~ function creates a factory function that curries
~my-custom-action~ with its given arguments -- which are two. This
creates another function that only takes the remaining two arguments.
If there are no arguments given to the factory, it just returns the
~my-custom-action~ function. For this to work reliably the factory
function must always be called with exactly two arguments, such that
the currying leaves us with a correct action function. How to achieve
this is described below. Other cases must use a custom factory
function that checks the argument list itself.
To allow the user of the custom command to specify arguments in the
first place, you need to add a hint to ~*query-actions-argn*~ alist
defining how many the action expects:
#+begin_src lisp
(push '("myaction" . 2)
*query-actions-argn*)
#+end_src
This tells dlm to check for 2 arguments and it prints an error message
if there are not exactly two specified on the command line. If there
is no entry (or ~nil~) in ~*query-actions-argn*~ for some action, it
is assumed to not take any arguments. Instead of a plain number, you
can specify a function that takes the argument list and would return
~nil~ to signal an error. With the example above in place, a call to
the custom action would look like this:
#+begin_src bash
dlm query myaction arg1 arg2
#+end_src
The same is used for actions to the =pending= command. Just use the
variables ~*pending-actions-alist*~ and ~*pending-actions-args*~
instead.
* Building
#+attr_html: style="display:inline;"
https://travis-ci.org/eikek/dlm.svg
** Using Nix
There are some lisp packages needed, which must be installed. Using
the [[https://nixos.org/nix][nix]] package manager, this happens automatically:
#+begin_src bash
nix-build build
#+end_src
This creates an fresh environment with the required packages and sbcl
installed to build dlm. The resulting executable can be found
following the newly created =result= link.
To build it manually but use the provided environment from nix, you
can run:
#+begin_src bash
nix-shell --pure build
#+end_src
This drops you in the same shell that =nix-build= uses when
building. Thus you can =cd= into the =build= directory and run =make=
to build the executable and =make test= to run the tests.
** Manually
See the =dlm.asd= file for the dependencies. It should be [[https://common-lisp.net/project/fiveam/][fiveam]],
[[https://common-lisp.net/project/cl-sqlite/][sqlite]], [[https://github.com/froydnj/ironclad][ironclad]], [[https://common-lisp.net/project/external-program/][external-program]], [[https://github.com/astine/unix-options][unix-options]] and
[[https://github.com/pnathan/cl-ansi-text][cl-ansi-text]]. These must be installed (fiveam only for running the
tests). Then you can use the =build/build.lisp= lisp file. Load it and
call for example ~(make-image)~ to create the executable.
All dependencies can be installed via [[http://quicklisp.org][quicklisp]]. Install it and
execute:
#+begin_src shell
$ sbcl --eval '(ql:quickload (quote (:unix-options \
:ironclad :external-program \
:sqlite :cl-ansi-text :fiveam)))'
#+end_src
* Browser integration
** Conkeror
I use the following in my =.conkerorrc=:
#+begin_src js
function dlm_fetch (url, open) {
var cmd = "dlm fetch '" + url + "'";
if (open) {
cmd = cmd + " && xdg-open $(dlm query -Hs '" + url + "' short)";
}
shell_command_blind(cmd);
};
function dlm_download (open) {
return function(I) {
var mb = I.window.minibuffer;
bo = yield read_browser_object(I);
link = load_spec_uri_string(load_spec(bo));
mb.message("Downloading " + link);
dlm_fetch(link, open);
};
}
interactive("dlm-download",
"Download the url using dlm.",
alternates(dlm_download(true), dlm_download(false)),
$browser_object = browser_object_links);
define_key(content_buffer_normal_keymap, "C-d", "dlm-download");
#+end_src
Instead of =s= I now press =C-d= to download and open the file and
=C-u C-d= to download without opening afterwards.
** Firefox
Use the addon [[http://flashgot.net][flashgot]] that lets you easily add a custom download
manager.
** TODO Chromium
* Feedback
… is always most welcome. You can use email
={firstname.lastname}@posteo.de=, the issue tracker or pull requests.
* License
Copyrighted by me 2015-, distributed under GPLv3 or later.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3, or (at your option)
any later version.
This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.
You should have received a copy of the GNU General Public License
along with GNU Emacs; see the file COPYING. If not, write to the Free
Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
02110-1301, USA.