{"id":26957215,"url":"https://github.com/wardle/trud","last_synced_at":"2025-06-30T07:34:41.376Z","repository":{"id":54802348,"uuid":"332673837","full_name":"wardle/trud","owner":"wardle","description":"Tools to automate downloads from the UK TRUD (Technology Reference data Update Distribution)","archived":false,"fork":false,"pushed_at":"2024-11-28T20:17:14.000Z","size":140,"stargazers_count":8,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-04T18:47:35.415Z","etag":null,"topics":["health","nhs","nhs-digital","ods","organisations","reference-data","snomed","terminology"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"epl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wardle.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-01-25T08:07:55.000Z","updated_at":"2024-11-28T20:16:17.000Z","dependencies_parsed_at":"2023-10-11T08:48:17.194Z","dependency_job_id":null,"html_url":"https://github.com/wardle/trud","commit_stats":{"total_commits":71,"total_committers":1,"mean_commits":71.0,"dds":0.0,"last_synced_commit":"7473331d6046c7f95901208c06134111170c4a89"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wardle%2Ftrud","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wardle%2Ftrud/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wardle%2Ftrud/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wardle%2Ftrud/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wardle","download_url":"https://codeload.github.com/wardle/trud/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246933359,"owners_count":20857052,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["health","nhs","nhs-digital","ods","organisations","reference-data","snomed","terminology"],"created_at":"2025-04-03T03:38:49.549Z","updated_at":"2025-04-03T03:38:50.095Z","avatar_url":"https://github.com/wardle.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# trud\n\n[![Scc Count Badge](https://sloc.xyz/github/wardle/trud)](https://github.com/wardle/trud/)\n[![Scc Cocomo Badge](https://sloc.xyz/github/wardle/trud?category=cocomo\u0026avg-wage=100000)](https://github.com/wardle/trud/)\n[![CircleCI](https://circleci.com/gh/wardle/trud.svg?style=svg)](https://circleci.com/gh/wardle/trud)\n[![Clojars Project](https://img.shields.io/clojars/v/com.eldrix/trud.svg)](https://clojars.org/com.eldrix/trud)\n\nA clojure library to automate downloads from the UK TRUD (Technology Reference data Update Distribution).\n\n## Introduction\n\nThe source of reference data in the UK is TRUD. However, until 25/1/2021, users needed to manually download distribution files from a web portal. NHS Digital used to provide an ftp server. NHS Digital have now released an API providing metadata on each reference data release, as well as links to the distribution files themselves.\n\n## Getting started\n\n### 1. Obtain a TRUD API key, and subscribe to reference data.\n\nYou will need to register as a user of TRUD and get an API key.\n\nLogin here [https://isd.digital.nhs.uk/trud3/user/guest/group/0/login/form](https://isd.digital.nhs.uk/trud3/user/guest/group/0/login/form).\n\nChoose your products and request a subscription using their portal. \n\nThere is no API for this part. Login to TRUD and get the item identifiers\nfor the distributions you want.\n\n\n### 2. Using from command-line\n\nWhile not primarily designed to be used from the command-line, it is possible\nto use this as a tool to automatically download multiple distributions from\nthe NHS Digital service to a directory of your choosing.\n\nHere, include the right API key and distributions 101 and 105 will be downloaded\ninto the archive directory specified.\n\nFrom source code (you need to escape data literals when invoked this way):\n\n```shell\nclj -X:download :progress 'true' :api-key '\"xxx\"' :cache-dir '\"/tmp/trud\"' :items '[101 105]'\n```\n\nParameters:\n- :progress - whether to show download progress (useful when used interactively)\n- :api-key - the API key for the NHS Digital TRUD service (the key, not a filename of a file containing the key)\n- :cache-dir - a directory to use as a cache, will be created if it doesn't exist\n\nIf you ran the above command at intervals, the cache directory would be \npopulated with the latest releases of the items you specify. \nI have used this approach in a cronjob, for example. Any tools that use `trud`\nfor downloads can then easily share the same cache directory. \n\nIf there is interest, it would be straightforward to make a simple command-line \ntool that can be run as a jar file. Raise an issue if you need this, but I \nnow increasingly simply run from source code, even in production systems.\n\nFor example, here I run a command to download the latest versions of items 101 and 105:\n```shell\nmark@jupiter trud % clj -X:download :api-key '\"***MY API KEY***\"' :cache-dir '\"cache\"' :items '[101 105]'\n12:22:36.095 [main] INFO com.eldrix.trud.core - Processing item 101\n12:22:41.992 [main] INFO com.eldrix.trud.cache - Item already in cache {:itemIdentifier 101, :archiveFileName \"uk_sct2cl_33.0.0_20220511000001Z.zip\", :releaseDate #object[java.time.LocalDate 0x24615687 \"2022-05-18\"]}\n12:22:41.996 [main] INFO com.eldrix.trud.core - Latest for item 101 : \"uk_sct2cl_33.0.0_20220511000001Z.zip\" {:archiveFilePath #object[sun.nio.fs.UnixPath 0x7f4f1536 \"cache/101--2022-05-18--uk_sct2cl_33.0.0_20220511000001Z.zip\"], :archiveFileSizeBytes 843903155}\n12:22:41.996 [main] INFO com.eldrix.trud.core - Processing item 105\n12:22:44.644 [main] INFO com.eldrix.trud.cache - Item already in cache {:itemIdentifier 105, :archiveFileName \"uk_sct2dr_33.0.0_20220511000001Z.zip\", :releaseDate #object[java.time.LocalDate 0x1ad1b737 \"2022-05-18\"]}\n12:22:44.644 [main] INFO com.eldrix.trud.core - Latest for item 105 : \"uk_sct2dr_33.0.0_20220511000001Z.zip\" {:archiveFilePath #object[sun.nio.fs.UnixPath 0x342a1f84 \"cache/105--2022-05-18--uk_sct2dr_33.0.0_20220511000001Z.zip\"], :archiveFileSizeBytes 393105290}\n```\n\nYou can of course use `trud` interactively if you wish, but this approach also\nlends itself to automated scripting. For automatic scripts, you'll likely not\nwant to show download progress reporting.\n\n### 3. Include the trud library in your project\n\n[![Clojars Project](https://img.shields.io/clojars/v/com.eldrix/trud.svg)](https://clojars.org/com.eldrix/trud)\n\ne.g. when using deps.edn, you can reference the library using git or maven coordinates. \n\nFor git coordinates, make sure you use the latest commit hash from [https://github.com/wardle/trud](https://github.com/wardle/trud):\n\n```clojure\n com.eldrix/trud                {:git/url \"https://github.com/wardle/trud.git\"\n                                 :sha     \"xxx\"}\n```\n\nFor a maven coordinate, simply specify the latest version from clojars.\n\n```clojure\ncom.eldrix/trud {:mvn/version \"xxx\"}\n```\n\nSee [clojars](https://clojars.org/com.eldrix/trud) for the latest release.\n\n### 4. Use the trud library \n\nBy default, archive files are stored in a cache directory. \nHere I use `\"/tmp/trud\"`:\n\n```clojure\n(require '[com.eldrix.trud.core :as trud])\n(def api-key \"xxx\")\n(def latest (trud/get-latest api-key \"/tmp/trud\" 341))\n```\n\nThe result will be a map of data direct from the TRUD API for item `341`.\nThe archive file will have been downloaded and available via `:archiveFilePath`.\nIt will have had some integrity checks made, including checks on file size\nand message digest (checksumming).\n\nResult:\n\n(note: this result includes URLs generated using one of my old API keys. \nYour URLs will be different as they will include your API key and should not\nbe publicly shared.)\n\n```clojure\n{:checksumFileLastModifiedTimestamp #object[java.time.Instant 0x72d19fd2 \"2021-01-29T13:28:21Z\"],\n:publicKeyFileSizeBytes 1736,\n:checksumFileSizeBytes 187,\n:signatureFileName \"trud_hscorgrefdataxml_data_1.0.0_20210129000001.sig\",\n:name \"Release 1.0.0\",\n:signatureFileLastModifiedTimestamp #object[java.time.Instant 0x3b6f0a6f \"2021-01-29T13:28:24Z\"],\n:itemIdentifier 341,\n:releaseDate #object[java.time.LocalDate 0x6cdace1f \"2021-01-29\"],\n:checksumFileName \"trud_hscorgrefdataxml_data_1.0.0_20210129000001.xml\",\n:archiveFileLastModifiedTimestamp #object[java.time.Instant 0x738f5264 \"2021-01-29T13:26:23Z\"],\n:publicKeyFileUrl \"https://isd.digital.nhs.uk/api/v1/keys/7daa48e2a26f3afeef6f6c2a2feb00b62bcbe68b/files/public-keys/trud-public-key-2013-04-01.pgp\",\n:publicKeyFileName \"trud-public-key-2013-04-01.pgp\",\n:archiveFileUrl \"https://isd.digital.nhs.uk/api/v1/keys/7daa48e2a26f3afeef6f6c2a2feb00b62bcbe68b/files/ODS/1.0.0/HSCORGREFDATAXML_DATA/hscorgrefdataxml_data_1.0.0_20210129000001.zip\",\n:archiveFileSizeBytes 26688464,\n:id \"hscorgrefdataxml_data_1.0.0_20210129000001.zip\",\n:signatureFileUrl \"https://isd.digital.nhs.uk/api/v1/keys/7daa48e2a26f3afeef6f6c2a2feb00b62bcbe68b/files/ODS/1.0.0/HSCORGREFDATAXML_DATA/trud_hscorgrefdataxml_data_1.0.0_20210129000001.xml.asc\",\n:checksumFileUrl \"https://isd.digital.nhs.uk/api/v1/keys/7daa48e2a26f3afeef6f6c2a2feb00b62bcbe68b/files/ODS/1.0.0/HSCORGREFDATAXML_DATA/trud_hscorgrefdataxml_data_1.0.0_20210129000001.xml\",\n:publicKeyId 6,\n:signatureFileSizeBytes 488,\n:archiveFileName \"hscorgrefdataxml_data_1.0.0_20210129000001.zip\",\n:needsUpdate? true,\n:archiveFilePath #object[sun.nio.fs.UnixPath\n0xdde6cc8\n\"/tmp/trud/341--2021-01-29--hscorgrefdataxml_data_1.0.0_20210129000001.zip\"]}\n```\n\n### 5. Processing your zip file (optional)\n\nOnce you have the zip file, you can unzip to a temporary directory and\nprocess, as necessary For convenience, you can use the utility functions in\n`com.eldrix.trud.zip`.\n\nHere we are looking at the NHS ODS XML distribution, which always contains\ntwo nested zip files \"archive.zip\" and \"fullfile.zip\". Here we extract \nany .xml files using a regexp in our nested query:\n\n```clojure\n(require '[com.eldrix.trud.zip :refer [unzip2 delete-paths]])\n(def ods-xml-files [(:archiveFilePath latest)\n                    [\"archive.zip\" #\"\\w+.xml\"]\n                    [\"fullfile.zip\" #\"\\w+.xml\"]])\n(def results (unzip2 ods-xml-files))\n(get-in results [1 1])    ;; sequence of any XML files in archive zip\n(get-in results [2 1])    ;; sequence of any XML files in fullfile.zip\n(delete-paths results)\n```\n\n\n# Developer information (optional)\n\nIdentify outdated dependencies:\n```shell\nclj -M:outdated\n```\n\nRun compilation checks\n\n```shell\nclj -M:check\n```\n\nRun linting\n\n```shell\nclj -M:lint/eastwood\nclj -M:lint/kondo\n\n```\n\nTo build a library jar and publish to local maven repository:\n```shell\nclj -T:build install\n```\n\nTo build a library jar and publish to [clojars](https://clojars.org)\n\n```shell\nclj -T:build deploy\n```\n\nThe Circle CI badge indicates the results of automated tests - including a live test against the NHS Digital TRUD service. \nIf these tests fail, it may be because the service is down or there has been a breaking versionc change.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwardle%2Ftrud","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwardle%2Ftrud","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwardle%2Ftrud/lists"}