{"id":21721835,"url":"https://github.com/bgpkit/oneio","last_synced_at":"2025-08-10T07:08:23.286Z","repository":{"id":51768518,"uuid":"520566159","full_name":"bgpkit/oneio","owner":"bgpkit","description":"OneIO is a Rust library that provides unified simple IO interface for reading and writing to and from data files from different sources and compressions.","archived":false,"fork":false,"pushed_at":"2025-06-06T20:36:31.000Z","size":367,"stargazers_count":17,"open_issues_count":2,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-05T09:16:10.120Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bgpkit.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-02T16:10:29.000Z","updated_at":"2025-06-06T20:17:36.000Z","dependencies_parsed_at":"2023-11-24T20:27:07.455Z","dependency_job_id":"c832ab61-e091-4161-96be-ff56edfb7be6","html_url":"https://github.com/bgpkit/oneio","commit_stats":{"total_commits":40,"total_committers":3,"mean_commits":"13.333333333333334","dds":0.125,"last_synced_commit":"5cbd0cc867f8985a503ebf4f74eed2b2aaf68dac"},"previous_names":[],"tags_count":34,"template":false,"template_full_name":null,"purl":"pkg:github/bgpkit/oneio","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bgpkit%2Foneio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bgpkit%2Foneio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bgpkit%2Foneio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bgpkit%2Foneio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bgpkit","download_url":"https://codeload.github.com/bgpkit/oneio/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bgpkit%2Foneio/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269688859,"owners_count":24459543,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-10T02:00:08.965Z","response_time":71,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-26T02:19:06.975Z","updated_at":"2025-08-10T07:08:23.277Z","avatar_url":"https://github.com/bgpkit.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OneIO - all-in-one IO library for Rust\n\n[![Rust](https://github.com/bgpkit/oneio/actions/workflows/rust.yml/badge.svg)](https://github.com/bgpkit/oneio/actions/workflows/rust.yml)\n[![Crates.io](https://img.shields.io/crates/v/oneio)](https://crates.io/crates/oneio)\n[![Docs.rs](https://docs.rs/oneio/badge.svg)](https://docs.rs/oneio)\n[![License](https://img.shields.io/crates/l/oneio)](https://raw.githubusercontent.com/bgpkit/oneio/main/LICENSE)\n\nOneIO is a Rust library that provides a unified IO interface for synchronously reading and writing\nto and from data files from different sources and compressions.\n\n### Usage and Feature Flags\n\nEnable all compression algorithms and handle remote files (default)\n\n```toml\noneio = \"0.18\"\n```\n\nSelect from supported feature flags\n\n```toml\noneio = { version = \"0.18\", default-features = false, features = [\"remote\", \"gz\"] }\n```\n\nDefault flags include `lib-core` and `rustls`.\n\n#### Core features: `lib-core`\n\n`lib-core` core features include:\n\n- `remote`: allow reading from remote files, including http(s) and ftp\n    - `http`: support reading from http(s) remote files using `reqwest` crate\n    - `ftp`: support reading from ftp remote files using `suppaftp` crate\n- `compressions`: support all compression algorithms\n    - `gz`: support `gzip` files using `flate2` crate\n    - `bz`: support `bzip2` files using `bzip2` crate\n    - `lz`: support `lz4` files using `lz4` crate\n    - `xz`: support `xz` files using `xz2` crate (requires xz library installed)\n    - `zstd`: support `zst` files using `zstd` crate\n- `json`: allow reading JSON content into structs with `serde` and `serde_json`\n\n#### TLS choice: `rustls` or `native-tls`\n\nUsers can choose between `rustls` or `native-tls` as their TLS library. We use `rustls` as the basic library.\n\nUsers can also choose to accept invalid certificates (not recommending) by setting `ONEIO_ACCEPT_INVALID_CERTS=true`\nenvironment variable.\n\n#### Optional features: `cli`, `s3`, `digest`\n\n- `s3`: allow reading from AWS S3 compatible buckets\n- `cli`: build commandline program `oneio`, uses the following features\n    - `lib-core`, `rustls`, `s3` for core functionalities\n    - `clap`, `tracing` for CLI basics\n- `digest` for generating SHA256 digest string\n\n#### Selecting some compression algorithms\n\nUsers can also manually opt-in to specific compression algorithms. For example, to work with only local `gzip`\nand `bzip2` files:\n\n```toml\noneio = { version = \"0.18\", default-features = false, features = [\"gz\", \"bz\"] }\n```\n\n### Use `oneio` commandline tool\n\nOneIO comes with a commandline tool, `oneio`, that opens and reads local/remote files\nto terminal and handles decompression automatically. This can be useful if you want to\nread some compressed plain-text files from a local or remote source.\n\n```\noneio reads files from local or remote locations with any compression\n\nUsage: oneio [OPTIONS] [FILE] [COMMAND]\n\nCommands:\n  s3      S3-related subcommands\n  digest  Generate SHA256 digest\n  help    Print this message or the help of the given subcommand(s)\n\nArguments:\n  [FILE]  file to open, remote or local\n\nOptions:\n  -d, --download                 download the file to current directory, similar to run `wget`\n  -o, --outfile \u003cOUTFILE\u003e        output file path\n      --cache-dir \u003cCACHE_DIR\u003e    cache reading to specified directory\n      --cache-force              force re-caching if local cache already exists\n      --cache-file \u003cCACHE_FILE\u003e  specify cache file name\n  -s, --stats                    read through the file and only print out stats\n  -h, --help                     Print help\n  -V, --version                  Print version\n```\n\nYou can specify a data file location after `oneio`. The following command\nprints out the raw HTML file from \u003chttps://bgpkit.com\u003e.\n\n```bash\noneio https://bgpkit.com\n```\n\nHere is another example of using `oneio` to read a remote compressed JSON file,\npipe it to `jq` and count the number of JSON objects in the array.\n\n```bash\n$ oneio https://data.bgpkit.com/peer-stats/as2rel-latest.json.bz2 | jq '.|length'\n802861\n```\n\nYou can also directly download a file with the `--download` (or `-d`) flag.\n\n```bash\n$ oneio -d https://archive.routeviews.org/route-views.amsix/bgpdata/2022.11/RIBS/rib.20221107.0400.bz2\nfile successfully downloaded to rib.20221107.0400.bz2\n\n$ ls -lh rib.20221107.0400.bz2\n-rw-r--r--  1 mingwei  staff   122M Nov  7 16:17 rib.20221107.0400.bz2\n\n$ monocle parse rib.20221107.0400.bz2 |head -n5\nA|1667793600|185.1.167.24|3214|0.0.0.0/0|3214 1299|IGP|185.1.167.24|0|0|3214:3001|NAG||\nA|1667793600|80.249.211.155|61955|0.0.0.0/0|61955 50629|IGP|80.249.211.155|0|0||NAG||\nA|1667793600|80.249.213.223|267613|0.0.0.0/0|267613 1299|IGP|80.249.213.223|0|0|5469:6000|NAG||\nA|1667793600|185.1.167.62|212483|1.0.0.0/24|212483 13335|IGP|152.89.170.244|0|0|13335:10028 13335:19000 13335:20050 13335:20500 13335:20530 lg:212483:1:104|NAG|13335|108.162.243.9\nA|1667793600|80.249.210.28|39120|1.0.0.0/24|39120 13335|IGP|80.249.210.28|0|0|13335:10020 13335:19020 13335:20050 13335:20500 13335:20530|AG|13335|141.101.65.254\n```\n\n### Use OneIO Reader as a Library\n\nThe returned reader implements BufRead, and handles decompression from the following types:\n\n- `gzip`: files ending with `gz` or `gzip`\n- `bzip2`: files ending with `bz` or `bz2`\n- `lz4`: files ending with `lz4` or `lz`\n- `xz`: files ending with `xz` or `xz2`\n- `zstd`: files ending with `zst` or `zstd`\n\nIt also handles reading from remote or local files transparently.\n\n#### Examples\n\nRead all into string:\n\n```rust\nconst TEST_TEXT: \u0026str = \"OneIO test file.\nThis is a test.\";\n\nlet mut reader = oneio::get_reader(\"https://spaces.bgpkit.org/oneio/test_data.txt.gz\").unwrap();\nlet mut text = \"\".to_string();\nreader.read_to_string(\u0026mut text).unwrap();\nassert_eq!(text.as_str(), TEST_TEXT);\n```\n\nRead into lines:\n\n```rust\nuse std::io::BufRead;\n\nconst TEST_TEXT: \u0026str = \"OneIO test file.\nThis is a test.\";\n\nlet lines = oneio::read_lines(\"https://spaces.bgpkit.org/oneio/test_data.txt.gz\").unwrap()\n.map(|line| line.unwrap()).collect::\u003cVec\u003cString\u003e\u003e();\nassert_eq!(lines.len(), 2);\nassert_eq!(lines[0].as_str(), \"OneIO test file.\");\nassert_eq!(lines[1].as_str(), \"This is a test.\");\n```\n\n### Use OneIO Writer as a Library\n\n[get_writer] returns a generic writer that implements [std::io::Write], and handles decompression from the following types:\n\n- `gzip`: files ending with `gz` or `gzip`\n- `bzip2`: files ending with `bz` or `bz2`\n\n**Note: lz4 writer is not currently supported.**\n\n#### Example\n\n##### Common IO operations\n\n```rust\nlet to_read_file = \"https://spaces.bgpkit.org/oneio/test_data.txt.gz\";\nlet to_write_file = \"/tmp/test_write.txt.bz2\";\n\n// read text from remote gzip file\nlet mut text = \"\".to_string();\noneio::get_reader(to_read_file).unwrap().read_to_string(\u0026mut text).unwrap();\n\n// write the same text to a local bz2 file\nlet mut writer = oneio::get_writer(to_write_file).unwrap();\nwriter.write_all(text.as_ref()).unwrap();\ndrop(writer);\n\n// read from the newly generated bz2 file\nlet mut new_text = \"\".to_string();\noneio::get_reader(to_write_file).unwrap().read_to_string(\u0026mut new_text).unwrap();\n\n// compare the decompressed content of the remote and local files\nassert_eq!(text.as_str(), new_text.as_str());\nstd::fs::remove_file(to_write_file).unwrap();\n```\n\n##### Read remote content with custom headers\n\n```rust\nuse std::collections::HashMap;\nuse reqwest::header::HeaderMap;\n\nlet client = oneio::create_client_with_headers([(\"X-Custom-Auth-Key\", \"TOKEN\")]).unwrap();\nlet mut reader = oneio::get_http_reader(\n  \"https://SOME_REMOTE_RESOURCE_PROTECTED_BY_ACCESS_TOKEN\",\n  Some(client),\n).unwrap();\nlet mut text = \"\".to_string();\nreader.read_to_string(\u0026mut text).unwrap();\nprintln!(\"{}\", text);\n```\n\n##### Download remote file to local directory\n\n```rust\noneio::download(\n    \"https://data.ris.ripe.net/rrc18/2022.11/updates.20221107.2325.gz\",\n    \"updates.gz\",\n    None\n).unwrap();\n```\n\n##### S3-related operations (needs `s3` feature flag)\n\n```rust\nuse oneio::s3::*;\n\n// upload to S3\ns3_upload(\"oneio-test\", \"test/README.md\", \"README.md\").unwrap();\n\n// read directly from S3\nlet mut content = String::new();\ns3_reader(\"oneio-test\", \"test/README.md\")\n    .unwrap()\n    .read_to_string(\u0026mut content)\n    .unwrap();\nprintln!(\"{}\", content);\n\n// download from S3\ns3_download(\"oneio-test\", \"test/README.md\", \"test/README-2.md\").unwrap();\n\n// get S3 file stats\nlet res = s3_stats(\"oneio-test\", \"test/README.md\").unwrap();\ndbg!(res);\n\n// error if file does not exist\nlet res = s3_stats(\"oneio-test\", \"test/README___NON_EXISTS.md\");\nassert!(res.is_err());\n\n// copy S3 file to a different location\nlet res = s3_copy(\"oneio-test\", \"test/README.md\", \"test/README-temporary.md\");\nassert!(res.is_ok());\nassert_eq!(\n    true,\n    s3_exists(\"oneio-test\", \"test/README-temporary.md\").unwrap()\n);\n\n// delete temporary copied S3 file\nlet res = s3_delete(\"oneio-test\", \"test/README-temporary.md\");\nassert!(res.is_ok());\nassert_eq!(\n    false,\n    s3_exists(\"oneio-test\", \"test/README-temporary.md\").unwrap()\n);\n\n// list S3 files\nlet res = s3_list(\"oneio-test\", \"test/\", Some(\"/\".to_string()), false).unwrap();\n\nassert_eq!(\n    false,\n    s3_exists(\"oneio-test\", \"test/README___NON_EXISTS.md\").unwrap()\n);\nassert_eq!(true, s3_exists(\"oneio-test\", \"test/README.md\").unwrap());\n```\n\n## Built with ❤️ by BGPKIT Team\n\n\u003ca href=\"https://bgpkit.com\"\u003e\u003cimg src=\"https://bgpkit.com/Original%20Logo%20Cropped.png\" alt=\"https://bgpkit.com/favicon.ico\" width=\"200\"/\u003e\u003c/a\u003e\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbgpkit%2Foneio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbgpkit%2Foneio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbgpkit%2Foneio/lists"}