{"id":16726332,"url":"https://github.com/brunobonacci/dragonfiles","last_synced_at":"2025-04-10T10:24:16.835Z","repository":{"id":31021389,"uuid":"34579901","full_name":"BrunoBonacci/dragonfiles","owner":"BrunoBonacci","description":"A Clojure scriptable file processor (awk on steroids)","archived":false,"fork":false,"pushed_at":"2021-02-11T09:22:47.000Z","size":25,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-24T09:11:38.343Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BrunoBonacci.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-04-25T18:10:29.000Z","updated_at":"2021-04-18T10:17:42.000Z","dependencies_parsed_at":"2022-09-08T17:22:49.223Z","dependency_job_id":null,"html_url":"https://github.com/BrunoBonacci/dragonfiles","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BrunoBonacci%2Fdragonfiles","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BrunoBonacci%2Fdragonfiles/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BrunoBonacci%2Fdragonfiles/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BrunoBonacci%2Fdragonfiles/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BrunoBonacci","download_url":"https://codeload.github.com/BrunoBonacci/dragonfiles/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248199088,"owners_count":21063641,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-12T22:52:54.664Z","updated_at":"2025-04-10T10:24:16.817Z","avatar_url":"https://github.com/BrunoBonacci.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dragonfiles\n\nA Clojure scriptable file processor.\nYou can think of it as \"`awk` on steroids\". \n\n```\n                                                                                \n                                                                                \n   ...........                                                     ..........   \n  0MWolcclllllc;;,,'.                  .'.                  .,,,;:cllllcccl0MW; \n   ;OO;             ........          xMMW,          .......             .oKo.  \n     .oKx,                  .....     .x0l      .....                 .c0O;     \n        :0Xo.                         locx.                         ;ONd.       \n          .:c:;;,,''''...........     k: O,    ............''',,,;:cc;          \n                 ':oc;;,,''........   ;WKX    ........'',,;:ol,.                \n             .OXd;.           .....    0M:    ....           .,c0Kc             \n              .cddl,. ..,;,,'..        .:         .'',,;'.  ':odo,              \n                  'ldl:'.             'Ocx              .;cod;.                 \n                                      ;x.O                                      \n                                      ,O,k                                      \n                                      '0cd                                      \n                                      .Nkl                                      \n                                       WM:          -= dragonfiles =-                            \n                                       XM;                                  \n                                       0M'                                      \n                                       kM.                                      \n                                       dX                                       \n                                       ck                                       \n                                       'c                                       \n                                                                                \n                                                                                \n\n```\n\n## Description\n\n`dragonfiles` is a scriptable file processor where scripts can be written\nin Clojure. It is somewhat similar to `awk` although it is not an attempt\nto clone or copy all the `awk` features.\nThe idea is that when processing files there is a number of activities\nwhich are more concerning the management of the input/output\nrather than the actual processing. So this tool is attempting to\ntake care of these 'housekeeping' activities while you can concentrate\nto write the logic of the actual processing is a modern data oriented\nlanguage such as Clojure.\n\nIf I had to express with one line what this tools does I would say:\n\n```bash\ndgf -s /input/directory -o /output/directory 's/upper-case'\n```\n\nThe above script takes all the files contained in `/input/directory`, it\napplies the function `s/upper-case` to a the lines of the file and\nstores the output into `/output/directory`.  The script string contains\nan arbitrary Clojure's expressions which must evaluate to a function\nwhich will be applied to each line.  The script can be arbitrary complex\nand you can use everything is available in Clojure.\n\n```bash\ndgf -s /input/directory -o /output/directory '(comp (partial s/join \" \")  #(s/split % #\"\\W+\") s/upper-case)'\n```\n\n## Installation\n\nTo install simply drop the `dgf` file in your `$PATH`. It requires `java` in the `$PATH` as well.\n\n```bash\nsudo wget \"https://github.com/BrunoBonacci/dragonfiles/releases/download/v0.1.0/dgf\" -O /usr/local/bin/dgf\nchmod -x /usr/local/bin/dgf\n```\n\n## Usage\n\n```\nSYNOPSIS\n\n       dgf -s PATH -o PATH   SCRIPT\n\n\n  -s, --source PATH           A file or directory containing text file to process\n  -o, --output PATH           A file or directory which will contains the output files.\n  -x, --extension EXT         Use the given extension for output files\n  -f, --file-mode             Rather than processing line-by-line the function expects a file-in file-out\n  -p, --parallel              Process files in parallel\n  -i, --init-script SCRIPT    a script which is executed before the first file is processed.\n  -e, --end-script SCRIPT     a script which is executed after the last file is processed\n  -m, --module-script SCRIPT  A script with function definitions to load. (repeatable)\n  -L, --load-library LIBRARY  A library to load from Maven/Clojars in Leiningen dep. format (repeatable)\n  -q, --quiet                 Less verbose output\n  -v, --version               Just print the version\n  -h, --help                  This help\n```\n\n\n  * `-s, --source PATH` (**REQUIRED**) \n  \n    This can be either a file or a directory and it represents the\n    file(s) to process.  The source path or file must exists.\n    \n  * `-o, --output PATH` (**REQUIRED**)\n  \n    This can be either a file or a path and it represents the where the\n    processed files must be written. The specified file or directory\n    MUST NOT exists.  If the source path (`-s`) points to a file, this\n    value will be considered to be the output filename. While if the\n    source path is a directory, this value will be considered to be a\n    folder. Files written in this directory will have the same directory\n    structure of the files in the source directory.\n    \n  * `-x, --extension`\n  \n    If present and the source path is pointing to a directory to process\n    multiple files it will replace the output file name extension with\n    the specified `EXT`. For example if your source is called\n    `/tmp/source` which contains files like `myfile.json` and your\n    output is `/tmp/output` and you specify `-x tsv`, your output file\n    will be called `/tmp/output/myfile.tsv`\n    \n  * `-f, --file-mode`\n  \n    The default processing mode is by line, which mean that the `SCRIPT`\n    must be a function [`(fn [line] ,,,)`] which takes only one\n    parameter which is the line and it must return a string containing\n    the output line. When specifying `-f` instead of processing line by\n    line the script must be able to accept two parameters, respectively\n    the input file and the output file [`(fn [input output] ,,,)`] it is\n    then the user responsibility to make sure that the input file is\n    parsed properly and written back to the output file in the correct\n    format.\n    \n  * `-p, --parallel`\n  \n    When this option is specified *dragonfiles* will process each file\n    in a separate using a thread-pool. If the script uses a global\n    state, it is the user responsibility to ensure the processing is\n    thread-safe.  In Clojure this can be easily achieved by using global\n    `atoms`.\n    \n  * `-i, --init-script SCRIPT`\n\n    `SCRIPT` is a Clojure's expression which is evaluated before the\n    first file is processed. It can be used to `require` additional\n    namespaces, define functions or global var.\n\n    SCRIPT can be also a file, in this case the file name need to be\n    prefixed with `@`. For exmaple: `@./myscript.clj`\n    \n  * `-e, --end-script SCRIPT`\n\n    `SCRIPT` is a Clojure's expression which is evaluated after the last\n    file has been processed and before the termination. This option is\n    often used in conjunction with `--init-script` but not necessarily.\n    It can be used to \"summarise\" the total of the processing and print\n    it out.\n\n    SCRIPT can be also a file, in this case the file name need to be\n    prefixed with `@`. For exmaple: `@./myscript.clj`\n\n  * `-m, --module-script SCRIPT`\n\n    A module is a normal Clojure file with other function definition.\n    The purpose of a module if to reuse functions across different\n    projects. Modules are loaded before the init-script, and you can\n    repeat the option as many times as you require in order to load\n    multiple modules (ex: `-m @mod1.clj -m @mod2.clj` etc). Modules will\n    be loaded and initialised in the order which they are passed.\n\n    SCRIPT can be also a file, in this case the file name need to be\n    prefixed with `@`. For exmaple: `@./myscript.clj`\n\n  * `-L, --load-library LIBRARY`\n\n    Loads a third-party library from an external repository such as\n    Maven or Clojars. Once downloaded it will load in the classpath so\n    that you can use the library in the script. For example, to load a\n    CSV library you could add: `-L [clojure-csv/clojure-csv \"2.0.1\"]`\n    and then in your init script require the namespace:\n    `-i \"(require '[clojure-csv.core :refer :all])\"` and finally\n    you can parse a csv line `parse-csv`. The option can be repeated\n    to load additionally libraries.\n\n  * `-q, --quiet`\n  \n    When this option is specified the information output is reduced\n    to only errors.\n\n\n  * `SCRIPT` (**REQUIRED**)\n  \n    This must be a valid Clojure expression which must evaluate to a\n    function. If the processing is in `line-mode` (default) then the\n    function must take one `String` parameter that is the line of the\n    file to process and return another `String` which represents the\n    transformed line. You can use any of the the core functions, and\n    functions in other namespaces which are shipped with Clojure default\n    distribution.\n    \n    When working on `file-mode` (`-f`) then `SCRIPT` must evaluate to a\n    function which accept two parameters: `input-file` and\n    `output-file`.  and be responsible for writing the output file in\n    the appropriate format.\n    \n    SCRIPT can be also a file, in this case the file name need to be\n    prefixed with `@`. For exmaple: `@./myscript.clj`\n\n    \n## Examples\n\nI will try to list a few examples which will give a glimpse of\n*dragonfiles* power.\n\n  * just copy\n  \n  `dgf -s '/tmp/input' -o '/tmp/output' 'identity'`\n  \n  This script will run the `identity` function to every line of every\n  file present in the directory `/tmp/input` and write it out in a file\n  with the same name under `/tmp/output`\n  \n  \n  * \"Upper-case all the things\"\n  \n  `dgf -s '/tmp/input' -o '/tmp/output' 's/upper-case'`\n  \n  The content of every file in input will be turned upper case in the output.\n  \n**TODO: more examples to come.**\n\n## TODO list\n\n  * support for input without output\n  * support for `stdin` and `stdout`\n  * support TCP sockets\n  * add processing statistics\n\n## License\n\nCopyright © 2015 Bruno Bonacci\n\nDistributed under the MIT License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrunobonacci%2Fdragonfiles","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrunobonacci%2Fdragonfiles","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrunobonacci%2Fdragonfiles/lists"}