{"id":19426615,"url":"https://github.com/oscaro/tools-io","last_synced_at":"2025-04-24T17:31:10.289Z","repository":{"id":34886943,"uuid":"172092496","full_name":"oscaro/tools-io","owner":"oscaro","description":"Oscaro’s generic I/O tools collection","archived":false,"fork":false,"pushed_at":"2025-04-17T09:30:01.000Z","size":143,"stargazers_count":8,"open_issues_count":0,"forks_count":4,"subscribers_count":17,"default_branch":"devel","last_synced_at":"2025-04-17T21:27:53.148Z","etag":null,"topics":["clojure-library","tools"],"latest_commit_sha":null,"homepage":"https://cljdoc.org/d/com.oscaro/tools-io/0.3.17/doc/readme","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"epl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oscaro.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-22T15:50:35.000Z","updated_at":"2025-04-17T09:30:04.000Z","dependencies_parsed_at":"2024-11-10T14:08:23.914Z","dependency_job_id":"067fc7b4-bcbe-459a-8ef7-97708a451898","html_url":"https://github.com/oscaro/tools-io","commit_stats":{"total_commits":80,"total_committers":9,"mean_commits":8.88888888888889,"dds":0.5125,"last_synced_commit":"98ee2640ed1f95abfd81705954340a3d1eb796fd"},"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oscaro%2Ftools-io","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oscaro%2Ftools-io/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oscaro%2Ftools-io/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oscaro%2Ftools-io/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oscaro","download_url":"https://codeload.github.com/oscaro/tools-io/tar.gz/refs/heads/devel","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250674280,"owners_count":21469190,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure-library","tools"],"created_at":"2024-11-10T14:08:20.753Z","updated_at":"2025-04-24T17:31:10.266Z","avatar_url":"https://github.com/oscaro.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# oscaro-tools-io\n\nOscaro’s generic I/O tools collection.\n\n[![Clojars Project](https://img.shields.io/clojars/v/com.oscaro/tools-io.svg)](https://clojars.org/com.oscaro/tools-io)\n\n[![cljdoc badge](https://cljdoc.org/badge/com.oscaro/tools-io)](https://cljdoc.org/d/com.oscaro/tools-io/CURRENT)\n\n- [Usage](#usage)\n  - [io](#io)\n    - [join-path](#join-path)\n    - [basename](#basename)\n    - [parent](#parent)\n    - [splitext](#splitext)\n    - [read-text-file](#read-text-file)\n    - [read-jsons-file](#read-jsons-file)\n    - [read-csv-file](#read-csv-file)\n    - [read-edns-file](#read-edns-file)\n    - [list-files](#list-files)\n    - [list-dirs](#list-dirs)\n    - [load-config-file](#load-config-file)\n    - [copy](#copy)\n    - [rm-rf](#rm-rf)\n    - [with-tempfile](#with-tempfile)\n    - [with-tempdir](#with-tempdir)\n    - [slurp](#slurp)\n    - [spit](#spit)\n    - [exists?](#exists)\n  - [core](#core)\n    - [file-reader](#file-reader)\n- [About compression](#about-compression)\n- [clj-kondo](#clj-kondo)\n- [License](#license)\n\n## Usage\n\n### `io`\n\n#### `join-path`\n\nJoin multiple parts of a path, like `os.path.join` in Python.\n\n```clojure\n(join-path \"foo\" \"bar\") ; =\u003e \"foo/bar\"\n(join-path \"foo/\" \"bar\") ; =\u003e \"foo/bar\"\n(join-path \"gs://mybucket\" \"bar\") ; =\u003e \"gs://mybucket/bar\"\n```\n\n#### `basename`\n\n```clojure\n(basename \"/var/log/mysql/\") ; =\u003e \"mysql\"\n(basename \"http://www.google.com/index.html\") ; =\u003e \"index.html\"\n```\n\n#### `parent`\n\n```clojure\n(parent \"/var/log/mysql/\") ; =\u003e \"/var/log\"\n(parent \"http://www.google.com/index.html\") ; =\u003e \"http://www.google.com\"\n```\n\n#### `splitext`\n\n```clojure\n(splitext \"http://www.google.com/index.html\") ; =\u003e [\"http://www.google.com/index\" \"html\"]\n(splitext \"archive.tar.gz\") ; =\u003e [\"archive.tar\" \"gz\"]\n```\n\n#### `read-text-file`\n\nreturn a lazy seq of string from a [protocol://]jsons[.gz] file.\n*warning*: the seq must be entirely consumed before the file is closed.\n\n**arguments**:\n- filename: string\n\n**returns**: an lazy seq of string\n\n#### `read-jsons-file`\n\nreturn a lazy seq of parsed json objects from a [protocol://]jsons[.gz] file.\n*warning*: the seq must be entirely consumed before the file is closed.\n\n**arguments**:\n- filename: string\n\n**returns**: an lazy seq of parsed objects\n\nexample\n```clojure\n(doall (map println (read-jsons-file \"sample.jsons.gz\")))\n```\n\n#### `read-edns-file`\n\nreturn a lazy seq of parsed edn objects from a [protocol://]edn[.gz] file.\n*warning*: the seq must be entirely consumed before the file is closed.\n\n**arguments**:\n- filename: string\n\n**returns**: an lazy seq of parsed objects\n\n#### `read-csv-file`\n\nreturn a lazy seq of parsed csv row as vector from a [protocol://]file.csv[.gz] file.\nsee http://clojure.github.io/data.csv/ for options.\n*warning*: the seq must be entirely consumed before the file is closed.\n\n**arguments**:\n- filename: string\n- args: options for data.csv  `{:separator (default \\,) :quote (default \\\")}`\n\n**returns**: an lazy seq of parsed objects\n\n#### `read-jsons-files`\n\nreturn a lazy seq of parsed json objects from [protocol://]jsons[.gz] files.\n*warning*: the seq must be entirely consumed before every files are closed.\n\n**arguments**:\n- [filenames]\n\n**returns**: an lazy seq of parsed objects\n\nexample\n```clojure\n(doall (map println (read-jsons-files [\"part1.jsons.gz\" \"part1.jsons.gz\"])))\n```\n\n#### `read-edns-files`\n\nreturn a lazy seq of parsed json objects from [protocol://]jsons[.gz] files.\n*warning*: the seq must be entirely consumed before every files are closed.\n\n**arguments**:\n- [filenames]\n\n**returns**: an lazy seq of parsed objects\n\n#### `list-files`\n\nreturn a seq of filenames beginning with provided path.\n\n**arguments**:\n- path\n- [options]\n\n**returns**: seq of string\n\nexamples\n```clojure\n(doall (map println (list-files \"gs://my-bucket/dir/20160902/animals\")))\n;-\u003e output:\n;gs://my-bucket/dir/20160902/animals-aaaaaaaaaa.jsons.gz\n;gs://my-bucket/dir/20160902/animals-aaaaaaaaab.jsons.gz\n;gs://my-bucket/dir/20160902/animals-aaaaaaaaac.jsons.gz\n;gs://my-bucket/dir/20160902/animals-aaaaaaaaad.jsons.gz\n;gs://my-bucket/dir/20160902/animals-aaaaaaaaae.jsons.gz\n\n\n(doall (map println (list-files \"/home/alice/dir/20160902/animals\")))\n;-\u003e output:\n;/home/alice/dir/20160902/animals-aaaaaaaaaa.jsons.gz\n;/home/alice/dir/20160902/animals-aaaaaaaaab.jsons.gz\n;/home/alice/dir/20160902/animals-aaaaaaaaac.jsons.gz\n;/home/alice/dir/20160902/animals-aaaaaaaaad.jsons.gz\n;/home/alice/dir/20160902/animals-aaaaaaaaae.jsons.gz\n```\n\n#### `list-dirs`\n\nreturn a seq of directory under the path directory.\n\n**arguments**:\n- path\n- [options]\n\n**returns**: seq of string\n\nexamples\n```clojure\n(doall (map println (list-dirs \"gs://my-bucket/dir/\")))\n;gs://my-bucket/dir/20160902/\n;gs://my-bucket/dir/20160902/\n;gs://my-bucket/dir/20160902/\n;gs://my-bucket/dir/20160902/\n;gs://my-bucket/dir/20160902/\n```\n\n#### `load-config-file`\n\nread and parse a configuration file.\nedn, clj, json, js, yaml, yml supported.\n\n*note*: if filename is a string, its searched first in resources, then locally\n\n**arguments**:\n- filename  (string or io/resource or io/file)\n\n**returns**: an object\n\n\n#### `copy`\n\nCopy file from source to destination.\n\n**arguments**\n- from\n- from-opts\n- to\n- to-opts\n- [copy-opts]: by defaults copy-opts = {buffer-size 1024}\n\nexample\n\nText file:\n```clojure\n(copy\n  \"/tmp/windows-file.csv\"\n  {:encoding \"windows-1252\"}\n  \"gs://my-bucket/dir/uf8-file.csv\"\n  {:encoding \"UTF-8\" :mime-type \"text/csv\"}\n  {:buffer-size 2048})\n```\n\nFor binary file, you must use an 8-bits encoding:\n```clojure\n(def byte-encoding \"ISO-8859-1\")\n(copy\n  \"/tmp/workbook.xlsx\"\n  {:encoding byte-encoding}\n  \"gs://my-bucket/dir/uf8-file.csv\"\n  {:encoding byte-encoding :mime-type \"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet\"}\n  {:buffer-size 2048})\n```\n\n#### `rm-rf`\n\nRecursively remove a directory.\n\n```clojure\n(rm-rf \"/path/to/my/directory\")\n```\n\n#### `with-tempfile`\n\nCreate a temporary file and remove it at the end of the body.\n\n```clojure\n(with-tempfile [filename]\n  (println \"There's a file called\" filename \".\"))\n(println \"The file is now gone.\")\n```\n\n#### `with-tempdir`\n\nCreate a temporary directory and remove it at the end of the body.\n\n```clojure\n(with-tempdir [dirname]\n  (println \"There's a directory called\" dirname \".\"))\n(println \"The directory is now gone.\")\n```\n\n#### `slurp`\n\n#### `spit`\n\n#### `exists?`\n\nTest if a file exists.\n\n```clojure\n(exists? \"https://oscaro.com\") ;=\u003e true\n(exists? \"local-file-that-do-not-exists\") ;=\u003e false\n```\n\n### `core`\n\n#### `file-reader`\n\nreturn a file as map like `{:stream clojure.java.io/reader}`, with support for [protocol://]file and file.gz.\nyou need to call (close! file) when you done.\n\n**arguments**:\n- filename\n- [options]: by default options = {encoding \"UTF-8\"}\n\n**returns**: an map with a `:stream` key\n\n## About compression\n\nBy default, `tools.io` supports _[gzip]_, _[bzip2]_ and _[framed lz4][]_\ncompression algorithms and can be extended by implementing a custom protocol\n(see sources).\n\nIt also supports the following formats if you provide the required dependencies.\n- _[xz]_ with `org.tukaani/xz` provided\n- _[zstd]_ with `com.github.luben/zstd-jni` provided\n\n[gzip]: https://en.wikipedia.org/wiki/Gzip\n[bzip2]: https://en.wikipedia.org/wiki/Bzip2\n[xz]: https://en.wikipedia.org/wiki/XZ_Utils\n[zstd]: https://en.wikipedia.org/wiki/Zstd\n[framed lz4]: https://en.wikipedia.org/wiki/LZ4_(compression_algorithm)\n\n## clj-kondo\n\nAn exported clj-kondo with hooks is provided. You can import it using\n\n```sh\nclj-kondo --lint \"$(clojure -Spath)\" --copy-configs --skip-lint\n```\n\nor, if using lein,\n\n```sh\nclj-kondo --lint \"$(lein classpath)\" --copy-configs --skip-lint\n```\n\n## License\n\nCopyright © 2016-2025 Oscaro.com\n\nDistributed under the Eclipse Public License either version 1.0 or (at your\noption) any later version.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foscaro%2Ftools-io","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foscaro%2Ftools-io","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foscaro%2Ftools-io/lists"}