{"id":14067388,"url":"https://github.com/bcgov/dipr","last_synced_at":"2025-07-30T00:33:22.513Z","repository":{"id":39852808,"uuid":"242242685","full_name":"bcgov/dipr","owner":"bcgov","description":"R package for working in the Data Innovation Program","archived":false,"fork":false,"pushed_at":"2022-05-24T19:18:04.000Z","size":1611,"stargazers_count":11,"open_issues_count":3,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-12-16T17:14:21.923Z","etag":null,"topics":["bcstats","citz","dsp","r","r-package","rstats"],"latest_commit_sha":null,"homepage":"https://bcgov.github.io/dipr/","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bcgov.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-21T22:36:53.000Z","updated_at":"2024-08-05T09:57:51.000Z","dependencies_parsed_at":"2022-08-30T21:00:35.925Z","dependency_job_id":null,"html_url":"https://github.com/bcgov/dipr","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/bcgov/dipr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bcgov%2Fdipr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bcgov%2Fdipr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bcgov%2Fdipr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bcgov%2Fdipr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bcgov","download_url":"https://codeload.github.com/bcgov/dipr/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bcgov%2Fdipr/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267785809,"owners_count":24144122,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-29T02:00:12.549Z","response_time":2574,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bcstats","citz","dsp","r","r-package","rstats"],"created_at":"2024-08-13T07:05:34.200Z","updated_at":"2025-07-30T00:33:21.801Z","avatar_url":"https://github.com/bcgov.png","language":"R","funding_links":[],"categories":["R"],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/README-\",\n  out.width = \"100%\"\n)\n```\n\n```{r make-sticker, include=FALSE}\nif (!file.exists(\"man/figures/logo.png\") \u0026\u0026 \n    !identical(Sys.getenv(\"USERDOMAIN\"), \"POPDATA\")) {\n  source(\"inst/sticker/sticker.R\")\n}\n```\n\n# dipr \u003cimg src='man/figures/logo.png' align=\"right\" height=\"139\" /\u003e\n\n\u003c!-- badges: start --\u003e\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n[![R build status](https://github.com/bcgov/dipr/workflows/R-CMD-check/badge.svg)](https://github.com/bcgov/dipr)\n[![img](https://img.shields.io/badge/Lifecycle-Experimental-339999)](https://github.com/bcgov/repomountie/blob/master/doc/lifecycle-badges.md)\n\u003c!-- badges: end --\u003e\n\n\nThe `dipr` package is an R package that loads and provides means of caching of `.dat.gz` files. Functions are also provided to convert from `.dat.gz` files to Apache Arrow formats.\n\n## Installation\nWhile this package is intended for us within the SRE you can install the package from GitHub:\n\n```{r, eval=FALSE}\nif(!requireNamespace(\"remotes\")) install.packages(\"remotes\")\nremotes::install_github(\"bcgov/dipr\")\n```\n\n\n\n## Installation inside the SRE\nInstallation of `dipr` is slightly different than typical R package installation. To begin with you will need to clone the dipr repo. To do this you need access to the SRE instance of GitLab. \n\nGo to this website: https://projectsc.popdata.bc.ca/\n\nScroll down and click on \"Sign in with Pop Data Sso\"\n\nYou will need to do this step everytime you log into the SRE.\n\nNext go to this site: https://projectsc.popdata.bc.ca/profile/personal_access_tokens\n\nCreate a new Personal Access Token enabling each scope and typing in your username. This should create a new personal access token. Those are your \"git credentials\". Save those on your \"U\" drive along with your user name. \n\n### Install from R\n\n```{r eval=FALSE}\n## First you need to store your credentials created in the step above. This is a one time only thing\ntoken \u003c- credentials::git_credential_ask(\"https://projectsc.popdata.bc.ca\")$password\n\n## These next two lines download the package to a temporary folder and install it. \nresp \u003c- httr::GET(\"https://projectsc.popdata.bc.ca/api/v4/projects/90/repository/archive.tar.gz\",\n                  query = list(sha = \"1.3.1\"),\n                  httr::write_disk(tempfile(fileext = \".tar.gz\")),\n                  config = httr::add_headers(`Private-token` = token))\nremotes::install_local(resp[[\"content\"]])\n```\n\n\n\n\n## Usage\nThe `dipr` package comes with some sample _fake_ data to illustrate the usage of the package and facilitate unit tests. Specifically, the package provides a fake data dictionary and a compressed data file. Example fake data can be seen like this:\n```{r}\nlibrary(dipr)\ndipr_examples()\n```\n\nIndividual paths can be extracted like this:\n```{r}\ndata_dict_path \u003c- dipr_example(\"starwars-dict.txt\")\ndict \u003c- read.table(data_dict_path)\ndict\ndat_path \u003c- dipr_example(\"starwars-fwf.dat.gz\")\n```\n\nThe main function in the package is `read_dat()` which allows you to directly read `.dat.gz files:\n```{r}\nraw \u003c- read_dat(data_path = dat_path,\n                data_dict = dict)\nhead(raw)\n```\n\nIf two files share a common data dictionary you can supply a vector of filenames to the `data_path` argument and `read_dat` will combine them for you into one dataframe:\n\n```{r}\ndat_path2 \u003c- dipr_example(\"starwars-fwf2.dat.gz\")\nraw_two_files \u003c- read_dat(\n    data_path = c(dat_path, dat_path2),\n    data_dict = dict,\n    id = \"file\"\n  )\n```\n\nWorking with the data.table package for data manipulation is also possible in dipr using the `read_dat_dt` function:\n```{r}\nraw_dt \u003c- read_dat_dt(\n    data_path = dat_path,\n    data_dict = dict\n  )\n```\n\n\n### Data Dictionary\nData Dictionaries need to be in a data frame format that includes 'name`, 'start', 'stop', and 'length' columns (in any order):\n\n```r\n\u003e dict\n# A tibble: 64 x 4\n   start  stop length name                      \n   \u003cdbl\u003e \u003cdbl\u003e  \u003cdbl\u003e \u003cchr\u003e   \n```\n\n\n## Project Status\nUnder development\n\n## License\n\n\n    Copyright 2019 Province of British Columbia\n\n    Licensed under the Apache License, Version 2.0 (the \"License\");\n    you may not use this file except in compliance with the License.\n    You may obtain a copy of the License at \n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n    Unless required by applicable law or agreed to in writing, software\n    distributed under the License is distributed on an \"AS IS\" BASIS,\n    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n    See the License for the specific language governing permissions and\n    limitations under the License.\n   \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbcgov%2Fdipr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbcgov%2Fdipr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbcgov%2Fdipr/lists"}