{"id":26275962,"url":"https://github.com/eblondel/rsdmx","last_synced_at":"2025-12-12T02:29:38.265Z","repository":{"id":8916294,"uuid":"10642895","full_name":"eblondel/rsdmx","owner":"eblondel","description":"Tools for reading SDMX data and metadata in R","archived":false,"fork":false,"pushed_at":"2025-02-27T12:28:25.000Z","size":1977,"stargazers_count":105,"open_issues_count":15,"forks_count":33,"subscribers_count":28,"default_branch":"master","last_synced_at":"2025-02-27T17:39:48.575Z","etag":null,"topics":["api","datastructures","dsd","r","read","readsdmx","sdmx","sdmx-format","sdmx-provider","sdmx-standards","statistics","timeseries","web-services"],"latest_commit_sha":null,"homepage":"https://github.com/opensdmx/rsdmx/wiki","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"robertsdionne/neural-network-papers","license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eblondel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-06-12T13:01:55.000Z","updated_at":"2025-02-27T12:28:28.000Z","dependencies_parsed_at":"2025-02-27T17:40:01.210Z","dependency_job_id":"edc3d6df-ae21-4810-8019-27d2b88d0ccc","html_url":"https://github.com/eblondel/rsdmx","commit_stats":{"total_commits":455,"total_committers":11,"mean_commits":41.36363636363637,"dds":0.05054945054945059,"last_synced_commit":"5ad5b65f5be3d89490673c9e008fbd80d85ef452"},"previous_names":["eblondel/rsdmx"],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eblondel%2Frsdmx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eblondel%2Frsdmx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eblondel%2Frsdmx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eblondel%2Frsdmx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eblondel","download_url":"https://codeload.github.com/eblondel/rsdmx/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247399885,"owners_count":20932880,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","datastructures","dsd","r","read","readsdmx","sdmx","sdmx-format","sdmx-provider","sdmx-standards","statistics","timeseries","web-services"],"created_at":"2025-03-14T10:19:21.508Z","updated_at":"2025-12-12T02:29:33.214Z","avatar_url":"https://github.com/eblondel.png","language":"R","funding_links":[],"categories":["Open Data and Statistics"],"sub_categories":[],"readme":"# rsdmx \u003ca href=\"https://github.com/eblondel/rsdmx\"\u003e\u003cimg src='https://github.com/eblondel/rsdmx/blob/master/doc/rsdmx.png?raw=true' align=\"right\" height=\"139\" /\u003e\u003c/a\u003e\n\n[![Build Status](https://github.com/eblondel/rsdmx/actions/workflows/r-cmd-check.yml/badge.svg?branch=master)](https://github.com/eblondel/rsdmx/actions/workflows/r-cmd-check.yml)\n[![CRAN_Status_Badge](https://www.r-pkg.org/badges/version/rsdmx)](https://cran.r-project.org/package=rsdmx)\n[![cran checks](https://badges.cranchecks.info/worst/rsdmx.svg)](https://cran.r-project.org/web/checks/check_results_rsdmx.html)\n[![Github_Status_Badge](https://img.shields.io/badge/Github-0.6--6-blue.svg)](https://github.com/eblondel/rsdmx)\n[![R-Universe](https://eblondel.r-universe.dev/badges/rsdmx)](http://eblondel.r-universe.dev/#package:rsdmx)\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.592404.svg)](https://doi.org/10.5281/zenodo.592404)\n\n** Tools for reading SDMX data and metadata documents in R ** \n\n## Overview\n\n``rsdmx`` is a package to parse/read SDMX data and metadata in R. It provides:\n* a set of classes and methods to read data and metadata documents exchanged through the Statistical Data and Metadata Exchange (SDMX) framework. The package currently focuses on the SDMX XML standard format (SDMX-ML).\n* an interface to SDMX web-services for a list of well-known data providers, such as EUROSTAT, OECD, and others \n[Learn more](https://github.com/eblondel/rsdmx/wiki#package_overview).\n\n### Citation\n\nWe thank in advance people that use ``rsdmx`` for citing it in their work / publication(s). For this, please use the citation provided at this link [![DOI](https://zenodo.org/badge/5183/eblondel/rsdmx.svg)](https://doi.org/10.5281/zenodo.592404)\n\n### Collating scattered SDMX data sources\n\nIn spite they are some R package initiatives relying on ``rsdmx`` that aim to provide a wrapper for a _single data source_ (e.g. OECD, EUROSTAT), it is strongly recommended to rely directly on ``rsdmx``. Indeed, one main objective of ``rsdmx`` is to **promote and facilitate collating scattered data** from a growing number of SDMX data providers, whatever the organization.\n\nIt is already possible to query well-known datasources, using the embedded [helpers](https://github.com/eblondel/rsdmx/blob/master/vignettes/quickstart.Rmd#using-the-helper-approach). Pull requests are welcome to support additional data providers by default in ``rsdmx``.\n\n### SDMX standards compliance\n\n[![SDMX_Compliance_Badge_1.0](https://img.shields.io/badge/SDMX--ML-1.0-blue.svg)](https://github.com/eblondel/rsdmx)\n[![SDMX_Compliance_Badge_2.0](https://img.shields.io/badge/SDMX--ML-2.0-blue.svg)](https://github.com/eblondel/rsdmx)\n[![SDMX_Compliance_Badge_2.1](https://img.shields.io/badge/SDMX--ML-2.1-blue.svg)](https://github.com/eblondel/rsdmx)\n\n### Status\nAt now, the package allows to read:\n* Datasets (``GenericData``, ``CompactData``, ``StructureSpecificData``, ``StructureSpecificTimeSeriesData``, ``CrossSectionalData``, ``UtilityData`` and ``MessageGroup`` SDMX-ML types)\n* Concepts (``Concept``, ``ConceptScheme`` and ``Concepts`` SDMX-ML types)\n* Codelists (``Code``, ``Codelist`` and ``Codelists`` SDMX-ML types)\n* DataStructures / KeyFamilies - with all subtypes\n* Data Structure Definitions (DSDs) - with all subtypes\n\n### Fundings\n\n``rsdmx`` is looking for **[sponsors](https://github.com/eblondel/rsdmx/wiki#package_development_funding)**. You have been using ``rsdmx`` and you wish to support its development? Please help us to make the package growing!\n\n### Author\n\nCopyright (C) 2014  Emmanuel Blondel\n\n### Contributors\n\n* Matthieu Stigler\n* Eric Persson\n\n### Distribution\n\n#### on CRAN\n``rsdmx`` is available on the Comprehensive R Archive Network (CRAN). See the R CRAN check results at: [https://cran.r-project.org/web/checks/check_results_rsdmx.html](https://cran.r-project.org/web/checks/check_results_rsdmx.html)\n\nPlease note that following a new submission to CRAN, or eventually a modification of CRAN policies, the package might be temporarily archived, and removed from CRAN. In case you notice that the package is not back in few time, please contact me.\n\n#### on R-Universe\n``rsdmx`` is available on the R-Universe public cloud server. The package version corresponds to the ongoing revision (master branch in Github). See [https://eblondel.r-universe.dev/#package:rsdmx](https://eblondel.r-universe.dev/#package:rsdmx)\n\n## Quickstart\n\n[rsdmx](https://cran.r-project.org/package=rsdmx) offers a low-level set of tools to read **data** and **metadata** in SDMX format. Its strategy is to make it very easy for the user. For this, a unique function named ``readSDMX`` has to be used, whatever it is a ``data`` or ``metadata`` document, or if it is ``local`` or ``remote`` datasource.\n\nIt is important to highlight that one of the major benefits of ``rsdmx`` is to focus first on the SDMX **format** specifications (acting as format abstraction library). This allows ``rsdmx`` reading SDMX data from _remote_ datasources, or from _local_ SDMX files. For accessing _remote_ datasources, it also means that ``rsdmx`` does not bound to SDMX **service** specifications, and can read a wider ranger of datasources.\n\n\n### Install rsdmx\n\n``rsdmx`` can be installed from CRAN\n```{r, echo = FALSE}\ninstall.packages(\"rsdmx\")\n```\n\nor from its development repository hosted in Github (using the ``devtools`` package):\n\n```{r, echo = FALSE}\ndevtools::install_github(\"eblondel/rsdmx\")\n```\n\n### Load rsdmx\n\nTo load rsdmx in R, do the following:\n\n```{r, echo = FALSE}\nlibrary(rsdmx)\n```\n\n### readSDMX \u0026 helper functions\n\n\n#### readSDMX as low-level function\n\nThe ``readSDMX`` function is then first designed at low-level so it can take as parameters a _url_ (``isURL=TRUE`` by default) or a _file_. So wherever is located the SDMX document, ``readSDMX`` will allow you to read it, as follows:\n\n```{r, echo = FALSE}\n\n  #read a remote file\n  sdmx \u003c- readSDMX(file = \"someUrl\")\n  \n  #read a local file\n  sdmx \u003c- readSDMX(file = \"somelocalfile\", isURL = FALSE)\n\n```\n\nIn addition, in order to facilitate querying datasources, ``readSDMX`` also providers helpers to query well-known remote datasources. This allows not to specify the entire URL, but rather specify a simple provider ID, and the different parameters to build a SDMX query (e.g. for a dataset query: operation, key, filter, startPeriod and endPeriod). \n\nThis is made possible as a list of SDMX service providers is embedded within ``rsdmx``, and such list provides all the information required for ``readSDMX`` to build the SDMX request (url) before accessing the datasource.\n\n\n#### get list of SDMX service providers\n\nThe list of known SDMX service providers can be queried as follows:\n\n```{r, echo = FALSE}\n\nproviders \u003c- getSDMXServiceProviders()\nas.data.frame(providers)\n\n```\n\n#### create/add a SDMX service provider\n\nIt also also possible to create and add a new SDMX service providers in this list (so ``readSDMX`` can be aware of it). A provider can be created with the ``SDMXServiceProvider``, and is made of various parameters:\n* ``agencyId`` (provider identifier)\n* ``name``\n* ``scale`` (international or national)\n* ``country`` ISO 3-alpha code (if national)\n* ``builder``\n\nThe request builder can be created with ``SDMXRequestBuilder`` which takes various arguments: \n* ``regUrl``: URL of the service registry endpoint\n* ``repoUrl``: URL of the service repository endpoint (Note that we use 2 different\narguments for registry and repository endpoints, since some providers use different\nURLs, but in most cases those are identical)\n* ``formatter`` list of functions to format the request params (one function per\ntype of resource, e.g. \"dataflow\", \"datastructure\", \"data\")\n* ``handler`` list of functions which will allow to build the web request\n*``compliant`` logical parameter (either the request builder is compliant with some web-service specifications)\n\n``rsdmx`` yet provides common builders, that can be customized if needed, by overriding\neither the ``formatter`` or the ``handler`` functions:\n* ``SDMXREST20RequestBuilder``: connector for SDMX REST 2.0 web-services\n* ``SDMXREST21RequestBuilder``: connector for SDMX REST 2.1 web-services\n* ``SDMXDotStatRequestBuilder``: connector for SDMX .Stat (\"DotStat\") web-services implementations\n\nLet's see it with an example:\n\nFirst create a request builder for our provider:\n\n```{r, echo = FALSE}\n\nmyBuilder \u003c- SDMXRequestBuilder(\n  regUrl = \"http://www.myorg.org/sdmx/registry\",\n  repoUrl = \"http://www.myorg.org/sdmx/repository\",\n  formatter = list(\n    dataflow = function(obj){\n      #format each dataflow id with some prefix\n      obj@resourceId \u003c- paste0(\"df_\",obj@resourceId)\n      return(obj)\n    },\n    datastructure = function(obj){\n      #do nothing\n      return(obj)\n    },\n    data = function(obj){\n      #format each dataset id with some prefix\n      obj@flowRef \u003c- paste0(\"data_\",obj@flowRef)\n      return(obj)\n    }\n  ),\n  handler = list(\n    dataflow = function(obj){\n      req \u003c- sprintf(\"%s/dataflow\",obj@regUrl)\n      return(req)\n    },\n    datastructure = function(obj){\n      req \u003c- sprintf(\"%s/datastructure\",obj@regUrl)\n      return(req)\n    },\n    data = function(obj){\n      req \u003c- sprintf(\"%s/data\",obj@regUrl)\n      return(req)\n    }\n  ),\n  compliant = FALSE\n)\n```\n\nAs you can see, we built a custom ``SDMXRequestBuilder`` that will be able to \ncreate SDMX web-requests for the different resources of a SDMX web-service.\n\nWe can create a provider with the above request builder, and add it to the list \nof known SDMX service providers:\n\n```{r, echo = FALSE}\n\n#create the provider\nprovider \u003c- SDMXServiceProvider(\nagencyId = \"MYORG\",\nname = \"My Organization\",\nbuilder = myBuilder\n)\n\n#add it to the list\naddSDMXServiceProvider(provider)\n\n#check provider has been added\nas.data.frame(getSDMXServiceProviders())\n\n\n```\n\n#### find a SDMX service provider\n\nA another helper allows you to interrogate ``rsdmx`` if a specific provider is \nknown, given an id:\n\n```{r, echo = FALSE}\noecd \u003c- findSDMXServiceProvider(\"OECD\")\n```\n\n#### readSDMX as helper function\n\nNow you know how to add a SDMX provider, you can consider using ``readSDMX`` \nwithout having to specifying a entire URL, but just by specifying the ``agencyId``\nof the provider, and the different query parameters to reach your SDMX document:\n\n```{r, echo = FALSE}\nsdmx \u003c- readSDMX(providerId = \"MYORG\", providerKey = NULL resource = \"data\", flowRef=\"MYSERIE\",\n                 key = \"all\", key.mode = \"SDMX\", start = 2000, end = 2015)\n```\n\nFor embedded service providers that require a user authentication/subscription key or token,\nit is possible to specify it in ``readSDMX`` with the ``providerKey`` argument. If provided,\nand that the embedded provider requires a specific key parameter, the latter will be appended\nto the SDMX web-request.\n\n\n\nThe following sections will show you how to query SDMX documents, by using ``readSDMX`` \nin different ways: either for _local_ or _remote_ files, using ``readSDMX`` as low-level \nor with the helpers (embedded service providers).\n\n### Read dataset documents\n\nThis section will introduce you on how to read SDMX *dataset* documents.\n\n#### Read _remote_ datasets\n\n\nThe following code snipet shows you how to read a dataset from a remote data source, taking as example the [OECD StatExtracts portal](https://data-explorer.oecd.org/): [https://sdmx.oecd.org/public/rest/data/DSD_PRICES@DF_PRICES_N_CP01/GRC......./all/?startPeriod=2020\u0026endPeriod=2020](https://sdmx.oecd.org/public/rest/data/DSD_PRICES@DF_PRICES_N_CP01/GRC......./all/?startPeriod=2020\u0026endPeriod=2020)\n\n```{r, echo = FALSE}\nmyUrl \u003c- \"https://sdmx.oecd.org/public/rest/data/DSD_PRICES@DF_PRICES_N_CP01/GRC......./all/?startPeriod=2020\u0026endPeriod=2020\"\ndataset \u003c- readSDMX(myUrl)\nstats \u003c- as.data.frame(dataset) \n```\n\nYou can try it out with other datasources, such as:\n* [**EUROSTAT portal**](https://ec.europa.eu/eurostat/web/main/home): [https://ec.europa.eu/eurostat/SDMX/diss-web/rest/data/nama_10_gdp/.CLV10_MEUR.B1GQ.BE/?startperiod=2005\u0026endPeriod=2011)\n* [**European Central Bank (ECB)**](https://sdw-wsrest.ecb.europa.eu): [https://sdw-wsrest.ecb.europa.eu/service/data/DD/M.SE.BSI_STF.RO.4F_N](https://sdw-wsrest.ecb.europa.eu/service/data/DD/M.SE.BSI_STF.RO.4F_N)\n\nThe online rsdmx documentation also provides a list of data providers, either from international or national institutions.\n\nNow, the service providers above mentioned are known by ``rsdmx`` which let users using ``readSDMX`` with the helper parameters. It may also be the case for a provider that\nyou register in rsdmx.\n\nLet's see how it would look like for querying an ``OECD`` datasource:\n\n```{r, message = FALSE}\nsdmx \u003c- readSDMX(providerId = \"OECD\", resource = \"data\", flowRef = \"DSD_PRICES@DF_PRICES_N_CP01\",\n                 key = list(\"GRC\", NULL, NULL, NULL, NULL, NULL, NULL, NULL), start = 2020, end = 2020)\ndf \u003c- as.data.frame(sdmx)\nhead(df)\n```\n\nIt is also possible to query a dataset together with its \"definition\", handled\nin a separate SDMX-ML document named ``DataStructureDefinition`` (DSD). It is \nparticularly useful when you want to enrich your dataset with all labels. For this, \nyou need the DSD which contains all reference data.\n\nTo do so, you only need to append ``dsd = TRUE`` (default value is ``FALSE``), \nto the previous request, and specify ``labels = TRUE`` when calling ``as.data.frame``,\nas follows:\n\n```{r, message = FALSE}\nsdmx \u003c- readSDMX(providerId = \"OECD\", resource = \"data\", flowRef = \"DSD_PRICES@DF_PRICES_N_CP01\",\n                 key = list(\"GRC\", NULL, NULL, NULL, NULL, NULL, NULL, NULL), start = 2020, end = 2020,\n                 dsd = TRUE)\ndf \u003c- as.data.frame(sdmx, labels = TRUE)\nhead(df)\n```\n\nNote that in case you are reading SDMX-ML documents with the native approach (with\nURLs), instead of the embedded providers, it is also possible to associate a DSD\nto a dataset by using the function ``setDSD``. Let's try how it works:\n\n```{r, message = FALSE}\n#data without DSD\nsdmx.data \u003c- readSDMX(providerId = \"OECD\", resource = \"data\", flowRef = \"DSD_PRICES@DF_PRICES_N_CP01\",\n                 key = list(\"GRC\", NULL, NULL, NULL, NULL, NULL, NULL, NULL), start = 2020, end = 2020)\n\n#DSD\nsdmx.dsd \u003c- readSDMX(providerId = \"OECD\", resource = \"datastructure\", resourceId = \"DSD_PRICES\")\n\n#associate data and dsd\nsdmx.data \u003c- setDSD(sdmx.data, sdmx.dsd)\n```\n\n\n\n#### Read _local_ datasets\n\nThis example shows you how to use ``rsdmx`` with _local_ SDMX files, previously downloaded from [EUROSTAT](https://ec.europa.eu/eurostat).\n\n```{r, echo = FALSE}\n#bulk download from Eurostat\ntf \u003c- tempfile(tmpdir = tdir \u003c- tempdir()) #temp file and folder\ndownload.file(\"https://ec.europa.eu/eurostat/estat-navtree-portlet-prod/BulkDownloadListing?sort=1\u0026file=data%2Frd_e_gerdsc.sdmx.zip\", tf)\nsdmx_files \u003c- unzip(tf, exdir = tdir)\n\nsdmx \u003c- readSDMX(sdmx_files[2], isURL = FALSE)\nstats \u003c- as.data.frame(sdmx)\nhead(stats)\n\n```\n\nBy default, ``readSDMX`` considers the data source is remote. To read a local file, add ``isURL = FALSE``.\n\n### Read metadata documents\n\nThis section will introduce you on how to read SDMX **metadata** complete ``data structure definitions`` (DSD)\n\n```\n\n#### Data Structure Definition (DSD)\n\nThis example illustrates how to read a complete DSD using a [OECD StatExtracts portal](https://data-explorer.oecd.org/) data source.\n\n```{r, echo = FALSE}\ndsdUrl \u003c- \"https://sdmx.oecd.org/public/rest/datastructure/all/DSD_PRICES/latest/?references=children\"\ndsd \u003c- readSDMX(dsdUrl)\n```\n\n``rsdmx`` is implemented in object-oriented way with ``S4`` classes and methods. The properties of ``S4`` objects are named ``slots`` and can be accessed with the ``slot`` method. The following code snippet allows to extract the list of ``codelists`` contained in the DSD document, and read one codelist as ``data.frame``.\n\n```{r, echo = FALSE}\n#get codelists from DSD\ncls \u003c- slot(dsd, \"codelists\")\ncodelists \u003c- sapply(slot(cls, \"codelists\"), function(x) slot(x, \"id\")) #get list of codelists\ncodelist \u003c- as.data.frame(slot(dsd, \"codelists\"), codelistId = \"CL_TRANSFORMATION\") #get a codelist\n```\n\nIn a similar way, the ``concepts`` of the dataset can be extracted from the DSD and read as ``data.frame``.\n\n```{r, echo = FALSE}\n#get concepts from DSD\nconcepts \u003c- as.data.frame(slot(dsd, \"concepts\"))\n```\n\n### Save \u0026 Reload SDMX R objects\n\nIt is possible to save SDMX R objects as RData file (.RData, .rda, .rds), to then\nbe able to reload them into the R session. It could be of added value for users that\nwant to keep their SDMX objects in R data files, but also for fast loading of large SDMX\nobjects (e.g. DSD objects) for use in statistical analyses and R-based web-applications.\n\nTo save a SDMX R object to RData file:\n\n```{r, echo = FALSE}\nsaveSDMX(sdmx, \"tmp.RData\")\n```\n\nTo reload a SDMX R object from RData file:\n\n```{r, echo = FALSE}\nsdmx \u003c- readSDMX(\"tmp.RData\", isRData = TRUE)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feblondel%2Frsdmx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feblondel%2Frsdmx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feblondel%2Frsdmx/lists"}