{"id":21432585,"url":"https://github.com/ecmwf/pyfdb","last_synced_at":"2025-07-14T12:34:00.446Z","repository":{"id":62580306,"uuid":"412400975","full_name":"ecmwf/pyfdb","owner":"ecmwf","description":null,"archived":false,"fork":false,"pushed_at":"2024-11-18T16:07:19.000Z","size":6216,"stargazers_count":3,"open_issues_count":10,"forks_count":7,"subscribers_count":18,"default_branch":"master","last_synced_at":"2024-11-18T17:45:39.571Z","etag":null,"topics":["fdb","mars"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ecmwf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-01T09:13:21.000Z","updated_at":"2024-11-08T09:37:42.000Z","dependencies_parsed_at":"2024-01-13T07:47:43.349Z","dependency_job_id":"cd09c8c6-bb89-4386-84d8-5dfdfb2e1e20","html_url":"https://github.com/ecmwf/pyfdb","commit_stats":{"total_commits":42,"total_committers":7,"mean_commits":6.0,"dds":"0.40476190476190477","last_synced_commit":"dd7ccbc09b743ab52a310b71842dfe50f2dc3e15"},"previous_names":["ecmwf-projects/pyfdb"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ecmwf%2Fpyfdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ecmwf%2Fpyfdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ecmwf%2Fpyfdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ecmwf%2Fpyfdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ecmwf","download_url":"https://codeload.github.com/ecmwf/pyfdb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225233419,"owners_count":17441969,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fdb","mars"],"created_at":"2024-11-22T23:19:11.179Z","updated_at":"2025-07-14T12:34:00.426Z","avatar_url":"https://github.com/ecmwf.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pyfdb\n\n[![Static Badge](https://github.com/ecmwf/codex/raw/refs/heads/main/Project%20Maturity/emerging_badge.svg)](https://github.com/ecmwf/codex/raw/refs/heads/main/Project%20Maturity)\n[![codecov](https://codecov.io/gh/ecmwf/pyfdb/branch/develop/graph/badge.svg)](https://codecov.io/gh/ecmwf/pyfdb/tree/develop)\n\nThis repository contains the source code and documentation of pyfdb, a python FDB client which interfaces the FDB5 library in the system.\n\n## 1. Installation\n\nInstall pyfdb with python3 (\u003e= 3.6) and pip as follows:\n```bash\npython3 -m pip install --upgrade git+https://github.com/ecmwf/pyfdb.git@master\n```\n\nOr from PyPi (not yet available):\n```bash\npython3 -m pip install pyfdb\n```\n\nOr with the Conda package manager:\n```bash\nconda create -n client python=3.7\nconda activate client\npython -m pip install --upgrade git+https://github.com/ecmwf/pyfdb.git@master\n```\n\nSince eccodes version 2.37.0, the eccodes binaries are shipped with the python binary, which can\nlead to some issues, especially if the locally installed stack of the software is executed. To\novercome this, export the following environment variable (this will use the locally installed version\nof eccodes):\n\n```bash\nexport ECCODES_PYTHON_USE_FINDLIBS=1\n```\n\nOther options can be found on the corresponding eccodes page of pypi:\n[Eccodes - Pypi](https://pypi.org/project/eccodes/)\n\n## 2. Example\n\nAn example of archival, listing and retrieval via pyfdb is shown next. For the example to work, FDB5 must be installed in the system, as well as the shutil, eccodes-python and pyfdb python packages. The GRIB files involved can be found under the `tests/unit/` folder in the pyfdb Git repository (https://github.com/ecmwf/pyfdb).\n\n### Initialising FDB\n```python\nimport pyfdb\nimport shutil\n\nfdb = pyfdb.FDB()\n```\n\nA config and userconfig can also be passed directly to the initialization function:\n```python\nconfig = dict(\n    type=\"local\",\n    engine=\"toc\",\n    schema=\"/path/to/fdb_schema\",\n    spaces=[\n        dict(\n            handler=\"Default\",\n            roots=[\n                {\"path\": \"/path/to/root\"},\n            ],\n        )\n    ],\n)\n\nfdb = pyfdb.FDB(config = config, userconfig = {})\n# Now use fdb.list, fdb.archive etc\n```\nThe module level functions `pyfdb.list, pyfdb.archive` etc use the default `pyfdb.FDB()` initialization with the default config search path.\n\n### Archive\n```python\nkey = {\n    'domain': 'g',\n    'stream': 'oper',\n    'levtype': 'pl',\n    'levelist': '300',\n    'date': '20191110',\n    'time': '0000',\n    'step': '0',\n    'param': '138',\n    'class': 'rd',\n    'type': 'an',\n    'expver': 'xxxx'\n}\n\nfilename = 'x138-300.grib'\nfdb.archive(open(filename, \"rb\").read(), key)\n\nkey['levelist'] = '400'\nfilename = 'x138-400.grib'\nfdb.archive(open(filename, \"rb\").read())\n\nkey['expver'] = 'xxxy'\nfilename = 'y138-400.grib'\nfdb.archive(open(filename, \"rb\").read())\nfdb.flush()\n```\n\n### List\n\n#### direct function, request as dictionary\n```python\nrequest = {\n    'class': 'rd',\n    'expver': 'xxxx',\n    'stream': 'oper',\n    'date': '20191110',\n    'time': '0000',\n    'domain': 'g',\n    'type': 'an',\n    'levtype': 'pl',\n    'step': 0,\n    'levelist': [300, '500'],\n    'param': ['138', 155, 't']\n}\n\nfor el in pyfdb.list(request, True, True):\n    print(el['keys'])\n# {'class': 'rd', 'date': '20191110', 'domain': 'g', 'expver': 'xxxx', 'stream': 'oper', 'time': '0000', 'levtype': 'pl', 'type': 'an', 'levelist': '300', 'param': '138', 'step': '0'}\n```\n\n#### direct function, updated dictionary\n```python\nrequest['levelist'] = ['100', '200', '300', '400', '500', '700', '850', '1000']\nrequest['param'] = '138'\n\nfor el in pyfdb.list(request, True, True):\n    print(el['keys'])\n# {'class': 'rd', 'date': '20191110', 'domain': 'g', 'expver': 'xxxx', 'stream': 'oper', 'time': '0000', 'levtype': 'pl', 'type': 'an', 'levelist': '300', 'param': '138', 'step': '0'}\n# {'class': 'rd', 'date': '20191110', 'domain': 'g', 'expver': 'xxxx', 'stream': 'oper', 'time': '0000', 'levtype': 'pl', 'type': 'an', 'levelist': '400', 'param': '138', 'step': '0'}\n```\n\n#### Using the fdb object with the request as a dictionary\nAs an alternative, use the created FDB instance and start queries from there\n```python\nrequest['levelist'] = ['400', '500', '700', '850', '1000']\nfor el in fdb.list(request, True, True):\n    print(el)\n# {'class': 'rd', 'date': '20191110', 'domain': 'g', 'expver': 'xxxx', 'stream': 'oper', 'time': '0000', 'levtype': 'pl', 'type': 'an', 'levelist': '400', 'param': '138', 'step': '0'}\n```\n\n### Retrieve\n\n#### To a file\n```python\nimport tempfile\nimport os\n\ndir = tempfile.gettempdir()\n\nrequest = {\n    'domain': 'g',\n    'stream': 'oper',\n    'levtype': 'pl',\n    'step': '0',\n    'expver': 'xxxx',\n    'date': '20191110',\n    'class': 'rd',\n    'levelist': '300',\n    'param': '138',\n    'time': '0000',\n    'type': 'an'\n}\n\nfilename = os.path.join(dir, 'x138-300bis.grib')\nwith open(filename, 'wb') as o, fdb.retrieve(request) as i:\n    shutil.copyfileobj(i, o)\n\nrequest['levelist'] = '400'\nfilename = os.path.join(dir, 'x138-400bis.grib')\nwith open(filename, 'wb') as o, fdb.retrieve(request) as i:\n    shutil.copyfileobj(i, o)\n\nrequest['expver'] = 'xxxy'\nfilename = os.path.join(dir, 'y138-400bis.grib')\nwith open(filename, 'wb') as o, pyfdb.retrieve(request) as i:\n    shutil.copyfileobj(i, o)\n```\n\n#### Read into memory\n```python\ndatareader = pyfdb.retrieve(request)\n\n# reading a small chunk\nchunk = datareader.read(10)\n\nprint(chunk)\n# bytearray(b'GRIB2\\x0e\\x0e\\x01\\x00\\x00')\n\nprint('tell()', datareader.tell())\n# tell() 10\n\n# go back (partially) - seek(2)\ndatareader.seek(2)\nprint('tell()', datareader.tell())\n# tell() 2\n\n# reading a larger chunk\nchunk = datareader.read(40)\nprint(chunk)\n# bytearray(b'IB2\\x0e\\x0e\\x01\\x00\\x004\\x80b\\x96\\xff\\x80\\x8ad\\x01\\x90\\x13\\x0b\\n\\x00\\x00\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x15\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00')\n\n# go back - seek(0)\ndatareader.seek(0)\n```\n\n#### Decode GRIB\n```python\nfrom eccodes import StreamReader\nreader = StreamReader(datareader)\ngrib = next(reader)\ngrib.dump()\n# [...redacted...]\n```\n\n## 3. Development\n\n### Pre-Commit Hooks\n\nPre-commit hooks are supplied in `.pre-commit-config.yaml` to lint and format the code before committing. To activate this:\n```bash\npip install pre-commit\npre-commit install # Install the hooks so that they run before `git commit`\n```\nAt the moment this runs isort, black and flake8, if any of these encounter errors they can't autofix then the commit will be blocked. To install all needed dependencies, run:\n```bash\npip install \".[dev]\"\n```\n\n### Run Unit Tests\n\nTo run the unit tests, make sure that the `pytest` module is installed first. To install it, you can simply run\n\n```sh\npip install \".[test]\"\n```\n, which installs the optional testing dependencies needed for pytest, as well as liniting. Simply run\n\n```sh\npython -m pytest\n```\nto test against a source build of fdb5 use:\n```\nFDB_HOME=/path/to/build/fdb5 python -m pytest\n```\n\n### Run Unit Tests across multiple python versions with Tox\n\nTox is a useful tool to quickly run pytest across multiple python versions by managing a set of python environments for you. A tox.ini file is provided that targets python3.8 - 3.12. Note that this will also install older versions of libraries like numpy which helps to catch incompatibilities with older versions of those libraries too.\n\nTo run tox, [install it](https://tox.wiki/), modify the `FDB5_HOME = ../build` line in `tox.ini` to point to a build of fdb5, this will be reused for all the tests. If your fdb5 is built as part of a bundle and `FDB5_HOME` points to the bundle build root, you may need to copy `build/fdb5/etc/fdb` to `build/etc/fdb` because by default fdb looks for a schema in `build/etc/fdb`.\n\nThen run\n```sh\ntox\n```\nThe first run will take a while for it to install all the environments but after that it's very fast.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fecmwf%2Fpyfdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fecmwf%2Fpyfdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fecmwf%2Fpyfdb/lists"}