{"id":23918056,"url":"https://github.com/starlink/hds-v5","last_synced_at":"2026-06-30T20:30:17.661Z","repository":{"id":20395325,"uuid":"23671188","full_name":"Starlink/hds-v5","owner":"Starlink","description":"HDS API implemented using HDF5","archived":false,"fork":false,"pushed_at":"2023-03-24T03:55:45.000Z","size":761,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-01-05T13:12:16.533Z","etag":null,"topics":["astronomy","c","file-format"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Starlink.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-09-04T16:56:29.000Z","updated_at":"2022-02-04T12:33:02.000Z","dependencies_parsed_at":"2025-01-05T13:12:22.824Z","dependency_job_id":"3826bac7-b6b1-46f7-b69f-cb57b4d94654","html_url":"https://github.com/Starlink/hds-v5","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Starlink%2Fhds-v5","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Starlink%2Fhds-v5/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Starlink%2Fhds-v5/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Starlink%2Fhds-v5/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Starlink","download_url":"https://codeload.github.com/Starlink/hds-v5/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240371754,"owners_count":19790888,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["astronomy","c","file-format"],"created_at":"2025-01-05T13:12:16.077Z","updated_at":"2026-06-30T20:30:15.573Z","avatar_url":"https://github.com/Starlink.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# HDS re-implemented on top of HDF5\n\nThe Starlink Hierarchical Data System is a hierarchical data format\ninitially developed in 1982. It is the data format that underlies the\n4 million lines of code in the Starlink software collection. HDS was\nnever adopted outside of the Starlink community and associated\ntelescopes and is now a niche data format. HDS has undergone numerous\nupdates over the years as it was rewritten in C from BLISS and ported\nfrom VMS to Unix. It works well but there is no longer any person\ncapable of supporting it or willing to learn the details of the\ninternals. HDS is also an impediment to wider adoption of the\nN-dimensional Data Format (NDF) data model.\n\nDespite being developed in the late 1980s, HDF (and currently HDF5)\nhas been adopted much more widely in the scientific community and is\nnow a de facto standard for storing scientific data in a\nhierarchical manner.\n\n## libhdsh5\n\nThis library is an attempt to reimplement the HDS API in terms of the\nHDF5 API. The idea is that this library could be dropped in as a\ndirect replacement to HDS and allow the entire Starlink software\ncollection to run, using HDF5 files behind the scenes which can then\nbe read by other general purpose tools such as h5py.\n\n## Migration\n\nI'm not yet worrying about migration of HDS v4 files to HDF5. Should\nthere be a wrapper HDS library that manages to forward calls to either\nthe native HDS or HDS/HDF5 based on the file that is being opened? Or\nshould we just write a conversion tool to copy structures from HDS\nclassic to HDF5?\n\n## Authors\n\nTim Jenness\nCopyright (C) Cornell University.\n\nBSD 3 clause license.\n\nCurrently some high level HDS files are included that use a GPL\nlicense and the original CLRC copyright. These implement functionality\nusing public HDS API and so work with no modifications. Their\ncontinued use will cause some rethinking on license but that's for a\nlater time.\n\n## Porting notes\n\n### Error handling\n\nHDF5 maintains its own error stack and HDS (and Starlink) uses EMS. All calls to HDF5\nare wrapped by an EMS status check and on error the HDF5 stack is read and stored\nin EMS error stack.\n\n### datFind\n\ndatFind does not know whether the component to find is a structure or primitive\nso it must query the type.\n\n### datName\n\nH5Iget_name returns full path so HDS layer needs to look at string to\nextract lowest level.\n\n### TRUE and FALSE is used in the HDF5 APIs\n\nbut not defined in HDF5 include files.\n\n### Dimensions\n\nHDS uses hdsdim type which currently is an 32 bit int. HDF5\nuses hsize_t which is an unsigned 64bit int. In theory HDS\ncould switch to that type in the C interface and this will\nwork so long as people properly use hdsdim.\n\nMore critically, HDF5 uses C order for dimensions and\nHDS uses Fortran order (even in the C layer). HDF5\ntransposes the dimensions in the Fortran interface to\nHDF5 and we have to do the same in HDS.\n\n### Memory mapping\n\nNot supported by HDF5. Presumably because of things like\ncompression filters and data type conversion.\n\ndatMap must therefore mmap and anonymous memory area to\nreceive the data and then datUnmap must copy the data back\nto the underlying dataset. Must also be careful to ensure\nthat `datGet`/`datPut` can notice that the locator is memory\nmapped and at minimum must complain.\n\nWill require that `datMap` and `datUnmap` lose the `const` in\nthe API as HDSLoc will need to be updated. Also need `datAnnul`\nto automatically do the copy.\n\nMust work out what happens if the program ends before the\ndatUnmap call has been done and also how `hdsClose` (or\n`datAnnul` equivalent on the file root) can close all the mmapped\narrays.\n\nWhat happens if the primitive is mapped twice? With different\nlocators? What happens in HDS?\n\n### H5LTfind_dataset\n\nStrange name because it also finds groups.\n\n### _LOGICAL\n\nIn HDS (presumably Fortran) a `_LOGICAL` type seems to be\na 4 byte int. This seems remarkably wasteful so in this\nlibrary I am using a 1 byte bitfield type.\n\nI have changed the C interface to use hdsbool_t boolean type\nso as to be explicit about the types.\n\nCurrently hdsbool_t is a 32-bit integer. Internally a 1 byte type is used.\nand externally a 4 byte type is used. This means that the routines\nthat query the HDF5 file for type have to do an additional check\nto see what the in memory type is meant to be.\n\n### datLen vs datPrec??\n\nHow do these routines differ?\n\nSG/4 says Primitive precision vs Storage precision\n\n`datPrec` doesn't seem to be used in any of Starlink.\n\n`datLen` is called but in some cases seemingly as an alias\nfor `datClen`.\n\n### Chunking and dataset creation\n\nHDS allows primitives to be reshaped but HDF5 only allows that\nif you have specified the maximum size in advance. For now\nH5S_UNLIMITED is used as the maximum size but this requires that\nwe define chunking. In the absence of any ability for the HDS API\nto provide guidance the library currently chunks based on the size\nof the array being created.\n\n### datRenam\n\nNeed to test whether `datRenam` or `datMove` break locators\nthat are previously associated with the objects being moved/renamed.\nH5Lmove indicates that they should still work fine.\n\n### datWhere\n\n... is probably not going to work (and is only visible in the Fortran\nAPI).\n\n### datSlice / datCell\n\n`datSlice` and `datCell` (for primitive types) are both attempting to\nselect a subset of a dataspace.\n\nThe `datasapce_id` element in the `HDSLoc` must be treated as the requested\ndataspace by the user and not be confused by the dataspace associated with\nthe primitive datset itself. `datVec` has a similar issue in that the\nlocator from a `datVec` call is must to be rank 1 regardless of the underlying\ndataset and if queried by `datShape` it should return the vectorized dimensions.\nCurrently we do not change the dataspace for datVec and datShape is short-circuited\nto understand the vectorized special case.\n\nHDS only supports the concept of 'sub-setting' and does not support\nsampling or scatter-gather.\n\nSlicing of a vectorized data structure requires extra work because a contiguous\nsubset of the N-D array has to be selected. Currently this is done by selecting points\nfrom the dataspace.\n\n### datPrmry\n\nNot sure if this is possible in HDF5. HDS has a distinction between a primary\nand secondary locator and annulling a secondary locator will not affect\nwhether the file is kept open or not. If HDF5 does not support this then\nwe can simply try to assert that all locators are primary locators, or else\ntry to keep track of this internally in the wrapper.\n\n### Incompatibilies\n\nHDF5 does not support an array of structures. Structures must be supported\nby adding an explicit collection group. For example:\n\n```\n   HISTORY        \u003cHISTORY\u003e       {structure}\n      CREATED        \u003c_CHAR*24\u003e      '2009-DEC-14 05:39:05.539'\n      CURRENT_RECORD  \u003c_INTEGER\u003e     1\n      RECORDS(10)    \u003cHIST_REC\u003e      {array of structures}\n\n      Contents of RECORDS(1)\n         DATE           \u003c_CHAR*24\u003e      '2009-DEC-14 05:39:05.579\n```\nwill have to be done as\n\n```\n  /HISTORY\n    Attr:TYPE=HISTORY\n      /CREATED\n      /CURRENT_RECORD\n      /RECORDS\n        Attr:TYPE=HIST_REC\n        Attr:ISARRAY=1\n        HDSCELL(1)\n           Attr:TYPE=HIST_REC\n           /DATE\n        HDSCELL(2)\n           Attr:TYPE=HIST_REC\n           /DATE\n```\n\nWe name the cell with its coordinates included to make it easy for\ndatCell to select the correct group simply by converting the\ncoordinates to (a,b). It also means that hdsTrace can easily be\nmodified to handle the presence of the group in the hierarchy.\nIf the name looks like /HISTORY/RECORDS/HDSCELL(5) hdsTrace\ncan easily convert that to the syntactically correct\n.HISTORY.RECORDS(5) by removing all occurrences of \"/HDSCELL\".\n\nNote also that in the final system HDSCELL is replaced with a longer\nstring that we know is longer that DAT__SZNAM (so that we can not\nend up with the name by mistake or fluke through the HDS layer).\n\nSlices of an array structure break this, as does vectorization (which\nis used by HDSTRACE a lot).\n\n### Type conversion\n\nHDS can do type conversion from/to strings for numeric and logical\ntypes. HDF5 can not do that in a H5Dwrite call so the type conversion\nhas to happen in datPut before that. The parameter system relies on this\nas parameters are always supplied as strings but the correct values\nare stored in the HDS parameter file.\n\n## HDS Groups\n\n`hdsLink` and `hdsFlush` are implemented above HDF5. A hash table and\nvector of locators is used to keep track of group membership. Uses\nuthash.\n\n## hdsClose\n\n`hdsClose` is documented to be obsolete but it is still called throughout\nADAM. Currently just calls `datAnnul`.\n\n## Primary locators?\n\nDoes HDF5 have the concept of a secondary locator?\n\nHDS defaults to top-level locators being \"primary\" and all other locators being\n\"secondary\". HDS is allowed to close the file if all primary locators are annulled.\nA secondary locator can be promoted to a primary locator using `datPrmry`. In\nHDF5 it seems that all locators are primary.\n\nIn Starlink software only one routine in NDF every sets a primary locator\nto a secondary locator (`ndf1_htop.f`), all others are setting to primary.\n\n## hdsLock/hdsFree\n\n`hdsLock` requires access to the underlying file descriptor associated with\nthe file. This is hard to obtain and no Starlink software calls the routine.\n`hdsFree` will not complain as it is only meant to do something if `hdsLock`\nhas been called.\n\n\n## TO DO\n\n`datRef` and `datMsg` currently do not report the full path to a file,\nunlike the HDS library. This is presumably an error in `hdsTrace()`\nimplementation which may have to determine the directory.  (and\nfurthermore we may have to store the directory at open time in case a\n`chdir` happens).\n\n`datParen` does not currently work out when it hits the top of the\ngroup hierarchy.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstarlink%2Fhds-v5","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstarlink%2Fhds-v5","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstarlink%2Fhds-v5/lists"}