{"id":15168548,"url":"https://github.com/antononcube/raku-data-importers","last_synced_at":"2026-02-23T15:30:58.027Z","repository":{"id":229364444,"uuid":"776497200","full_name":"antononcube/Raku-Data-Importers","owner":"antononcube","description":"Various data importing routines with a unified interface (data-import, slurp).","archived":false,"fork":false,"pushed_at":"2024-10-17T07:56:13.000Z","size":152,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-06T04:51:13.428Z","etag":null,"topics":["data","data-ingestion","raku","rakulang","slurp"],"latest_commit_sha":null,"homepage":"https://raku.land/zef:antononcube/Data::Importers","language":"Raku","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"artistic-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/antononcube.png","metadata":{"files":{"readme":"README-work.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-23T17:12:10.000Z","updated_at":"2024-11-19T12:08:35.000Z","dependencies_parsed_at":"2024-03-30T11:32:33.642Z","dependency_job_id":"cc314f06-6aff-45d7-ba6f-152e11613de2","html_url":"https://github.com/antononcube/Raku-Data-Importers","commit_stats":{"total_commits":42,"total_committers":1,"mean_commits":42.0,"dds":0.0,"last_synced_commit":"8dec283d05df9897a3283331deb8fb15f3d3edf0"},"previous_names":["antononcube/raku-data-slurps","antononcube/raku-data-importers"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/antononcube/Raku-Data-Importers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antononcube%2FRaku-Data-Importers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antononcube%2FRaku-Data-Importers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antononcube%2FRaku-Data-Importers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antononcube%2FRaku-Data-Importers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/antononcube","download_url":"https://codeload.github.com/antononcube/Raku-Data-Importers/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antononcube%2FRaku-Data-Importers/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270946068,"owners_count":24672890,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-18T02:00:08.743Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","data-ingestion","raku","rakulang","slurp"],"created_at":"2024-09-27T06:22:18.626Z","updated_at":"2025-10-25T03:13:02.506Z","avatar_url":"https://github.com/antononcube.png","language":"Raku","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data::Importers\n\n[![Actions Status](https://github.com/antononcube/Raku-Data-Importers/actions/workflows/linux.yml/badge.svg)](https://github.com/antononcube/Raku-Data-Importers/actions)\n[![Actions Status](https://github.com/antononcube/Raku-Data-Importers/actions/workflows/macos.yml/badge.svg)](https://github.com/antononcube/Raku-Data-Importers/actions)\n\n[![](https://raku.land/zef:antononcube/Data::Importers/badges/version)](https://raku.land/zef:antononcube/Data::Importers)\n[![License: Artistic-2.0](https://img.shields.io/badge/License-Artistic%202.0-0298c3.svg)](https://opensource.org/licenses/Artistic-2.0)\n\n## In brief\n\nThis repository is for a Raku package for the import and export of different types of data\nfrom both URLs and files. Automatically deduces the data type from extensions.\n\n**Remark:** The built-in subs `slurp` and `spurt` are overloaded by definitions of this package.\nThe corresponding functions `data-import` and `data-export` can be also used.\n\nThe format of the data of the URLs or files can be specified with the named argument \"format\".\nIf `format =\u003e Whatever` then the format of the data is implied by the extension of the given URL or file name.\n\n(Currently) the recognized formats are: CSV, HTML, JSON, Image (png, jpeg, jpg), PDF, Plaintext, Text, XML.\n\nThe subs `slurp` and `data-import` can work with:\n\n- CSV \u0026 TSV files if [\"Text::CSV\"](https://raku.land/zef:Tux/Text::CSV), [HMBp1], is installed\n\n- PDF files if [\"PDF::Extract\"](https://raku.land/zef:Tux/PDF::Extract), [SRp1], is installed\n\nThe subs `spurt` and `data-export` can work with CSV \u0026 TSV files if [\"Text::CSV\"](https://raku.land/zef:Tux/Text::CSV), [HMBp1], is installed.\n\n**Remark:** Since \"Text::CSV\" is a \"heavy\" to install package, it is not included in the dependencies of this one.\n\n**Remark:** Similarly, \"PDF::Extract\" requires additional, non-Raku installation, and it targets only macOS (currently.)\nThat is why it is not included in the dependencies of \"Data::Importers\".\n\n----\n\n## Installation\n\nFrom Zef' ecosystem:\n\n```\nzef install Data::Importers\n```\n\nFrom GitHub:\n\n```\nzef install https://github.com/antononcube/Raku-Data-Importers.git\n```\n\n-----\n\n## File examples\n\nIn order to use the `slurp` definitions of this package the named argument \"format\" \nhas to be specified:  \n\n### JSON file\n\n```perl6\nuse Data::Importers;\n\nslurp($*CWD ~ '/resources/simple.json', format =\u003e 'json')\n```\n\nInstead of `slurp` the function `data-import` can be used (no need to use \"format\"):\n\n```perl6\ndata-import($*CWD ~ '/resources/simple.json')\n```\n\n### CSV file\n\n```perl6\nslurp($*CWD ~ '/resources/simple.csv', format =\u003e 'csv', headers =\u003e 'auto')\n```\n\n-----\n\n## URLs examples\n\n## JSON URLs\n\nImport a JSON file:\n\n```perl6\nmy $url = 'https://raw.githubusercontent.com/antononcube/Raku-LLM-Prompts/main/resources/prompt-stencil.json';\n\nmy $res = data-import($url, format =\u003e Whatever);\n\n$res.WHAT;\n```\n\nHere is the deduced type:\n\n```perl6\nuse Data::TypeSystem;\n\ndeduce-type($res);\n```\n\nUsing `slurp` instead of `data-import`:\n\n```perl6\nslurp($url)\n```\n\n### Image URL\n\nImport an [image](https://raw.githubusercontent.com/antononcube/Raku-WWW-OpenAI/main/resources/ThreeHunters.jpg):\n\n```perl6\nmy $imgURL = 'https://raw.githubusercontent.com/antononcube/Raku-WWW-OpenAI/main/resources/ThreeHunters.jpg';\n\ndata-import($imgURL, format =\u003e 'md-image').substr(^100)\n```\n\n**Remark:** Image ingestion is delegated to \n[\"Image::Markup::Utilities\"](https://raku.land/zef:antononcube/Image::Markup::Utilities), [AAp1].\nThe format value 'md-image' can be used to display images in Markdown files or Jupyter notebooks.\n\n### CSV URL\n\nHere we ingest a CSV file and show a table of a 10-rows sample:\n\n```perl6, results=asis\nuse Data::Translators;\n\n'https://raw.githubusercontent.com/antononcube/Raku-Data-ExampleDatasets/main/resources/dfRdatasets.csv'\n==\u003e slurp(headers =\u003e 'auto') \n==\u003e { $_.pick(10).sort({ $_\u003cPackage Item\u003e }) }()\n==\u003e data-translation(field-names =\u003e \u003cPackage Item Title Rows Cols\u003e)\n```\n\n### PDF URL\n\nHere is an example of importing a PDF file into plain text:\n\n```perl6\nmy $txt = slurp('https://pdfobject.com/pdf/sample.pdf', format=\u003e'text');\n\nsay text-stats($txt);\n```\n\n**Remark:** The function `text-stats` is provided by this package, \"Data::Importers\". \n\nHere is a sample of the imported text:\n\n```perl6\n$txt.lines[^6].join(\"\\n\")\n```\n\n\n-----\n\n## TODO\n\n- [X] DONE Development\n  - [X] DONE PDF ingestion\n    - [X] DONE Files \n    - [X] DONE URLs\n  - [ ] TODO Export to:\n    - [X] DONE JSON files\n    - [X] DONE text, Markdown, org, HTML, XML files\n    - [X] DONE CSV/TSV files\n    - [ ] TODO PDF files\n    - [ ] TODO Image files\n- [ ] TODO Unit tests\n  - [ ] TODO PDF ingestion\n    - Some initial tests are put in.\n\n----- \n\n## References\n\n[AAp1] Anton Antonov,\n[Image::Markup::Utilities Raku package](https://github.com/antononcube/Raku-Image-Markup-Utilities),\n(2023),\n[GitHub/antononcube](https://github.com/antononcube).\n\n[HMBp1] H. Merijn Brand,\n[Text::CSV Raku package](https://github.com/Tux/CSV),\n(2015-2023),\n[GitHub/Tux](https://github.com/Tux).\n\n[SRp1] Steve Roe,\n[PDF::Extract Raku package](https://github.com/librasteve/raku-PDF-Extract),\n(2023),\n[GitHub/librasteve](https://github.com/librasteve).   ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fantononcube%2Fraku-data-importers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fantononcube%2Fraku-data-importers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fantononcube%2Fraku-data-importers/lists"}