{"id":19310320,"url":"https://github.com/cedadev/nlds","last_synced_at":"2026-03-10T18:09:50.007Z","repository":{"id":39600573,"uuid":"385645492","full_name":"cedadev/nlds","owner":"cedadev","description":"Near Line Data Store","archived":false,"fork":false,"pushed_at":"2026-02-26T20:29:23.000Z","size":17764,"stargazers_count":15,"open_issues_count":16,"forks_count":0,"subscribers_count":6,"default_branch":"main","last_synced_at":"2026-02-26T20:33:31.461Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cedadev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-07-13T15:08:23.000Z","updated_at":"2026-01-22T14:40:54.000Z","dependencies_parsed_at":"2026-02-16T18:04:07.220Z","dependency_job_id":null,"html_url":"https://github.com/cedadev/nlds","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"purl":"pkg:github/cedadev/nlds","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cedadev%2Fnlds","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cedadev%2Fnlds/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cedadev%2Fnlds/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cedadev%2Fnlds/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cedadev","download_url":"https://codeload.github.com/cedadev/nlds/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cedadev%2Fnlds/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30346619,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T15:55:29.454Z","status":"ssl_error","status_checked_at":"2026-03-10T15:54:58.440Z","response_time":106,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T00:23:38.190Z","updated_at":"2026-03-10T18:09:49.978Z","avatar_url":"https://github.com/cedadev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"CEDA Near-Line Data Store\n=========================\n\n[![Testing](https://github.com/cedadev/nlds/actions/workflows/ci.yml/badge.svg)](https://github.com/cedadev/nlds/actions/workflows/ci.yml)\n[![Docs](https://github.com/cedadev/nlds/actions/workflows/pages/pages-build-deployment/badge.svg)](https://github.com/cedadev/nlds/actions/workflows/pages/pages-build-deployment)\n[![PEP8](https://img.shields.io/badge/code%20style-pep8-orange.svg)](https://www.python.org/dev/peps/pep-0008/)\n[![Coverage](https://cedadev.github.io/nlds/coverage.svg)](https://cedadev.github.io/nlds/coverage/htmlcov/)\n\nThis is the server code for the CEDA Near-Line Data Store (NLDS), consisting of \nan HTTP API and a cluster of rabbit consumer microservices. The \n[NLDS client](https://github.com/cedadev/nlds-client) is required to communicate \nwith the API, either via the command line interface or python client library.\n\nThe NLDS is a unified storage solution, allowing easy use of disk, s3 object \nstorage, and tape from a single interface. It utilises object storage as a cache\nfor the tape backend allowing for low-latency backup \n\nThe NLDS server is built upon [FastAPI](https://fastapi.tiangolo.com) for the \nAPI, [RabbitMQ](https://www.rabbitmq.com/) for the message broker, \n[minio](https://min.io/) for the s3 client, \n[SQLAlchemy](https://www.sqlalchemy.org/) for the database client and \n[xrootd](https://xrootd.slac.stanford.edu/) for the tape interactions.\n\nDocumentation can be found [here](https://cedadev.github.io/nlds/index.html).\n\nInstallation\n------------\n\nIf installing locally we strongly recommend the use of a virtual environment to \nmanage the dependencies.\n\n1.  Create a Python virtual environment:\n   \n    ```\n    python3 -m venv nlds-venv\n    ```\n\n2.  Activate the nlds-venv:\n\n    ```\n    source ~/nlds-venv/bin/activate\n    ```\n\n3.  You could either install the nlds package with editing capability from a \n    locally cloned copy of this repo (note the inclusion of the editable flag \n    `-e`), e.g.\n\n    ```\n    pip install -e ~/Coding/nlds\n    ```\n\n    or install this repo directly from github:\n\n    ```\n    pip install git+https://github.com/cedadev/nlds.git\n    ```\n\n4.  (Optional) There are several more requirements/dependencies defined:\n    *   `requirements-dev.txt` - contains development-specific (i.e. not \n    production appropriate) dependencies. Currently this consists of a psycopg2 \n    binary python package for interacting with PostgeSQL from a local NLDS \n    instance. \n    *   `requirements-deployment.txt` - contains deployment-specific \n    dependencies, excluding `XRootD`. Currently this consists of the psycopg2 \n    package but built from source instead of a precompiled binary. \n    *   `requirements-tape.txt` - contains tape-specific dependencies, notably \n    `XRootD`. \n    *   `tests/requirements.txt` - contains the dependencies for the test suite. \n    *   `docs/requirements.txt` - contains the dependencies required for \n    building the documentation with sphinx.\n\nServer Config\n-------------\n\nTo interface with the JASMIN accounts portal, for the OAuth2 authentication, a \n`.server_config` file has to be created. This contains infrastructure \ninformation and so is not included in the GitHub repository. See the \n[relevant documentation](https://cedadev.github.io/nlds/server-config/server-config.html) \nand [examples](https://cedadev.github.io/nlds/server-config/examples.html) for \nmore information.\n\nA Jinja-2 template for the `.server_config` file can also be found in the \n`templates/` directory.\n\nRunning the Server\n------------------\n\n1.  The NLDS API requires something to serve the API, usually uvicorn in a local \n    development environment:\n\n    ```\n    uvicorn nlds.main:nlds --reload\n    ```\n\n    This will create a local NLDS API server at `http://127.0.0.1:8000/`. \n    FastAPI displays automatically generated documentation for the REST-API, to \n    browse this go to http://127.0.0.1:8000/docs/\n\n2.  To run the microservices, you have three options:\n    1.  In individual terminals, after activating the virtual env, (e.g. \n        `source ~/nlds-venv/bin/activate`), start each of the microservice \n        consumers:\n        ```\n        nlds_q\n        index_q\n        catalog_q  \n        transfer_put_q   \n        transfer_get_q\n        logging_q\n        monitor_q\n        archive_put_q\n        archive_get_q\n        ```\n        This will send the output of each consumer to its own terminal (as well \n        as whatever is configured in the logger).\n\n    2.  Alternatively, you can use the scripts in the `test_run/` directory: \n        `start_test_run.py` to start and `stop_test_run.py` to stop. \n        This will start a [screen](https://www.gnu.org/software/screen/manual/screen.html) \n        session with all 8 processors (+ api server) in, sending each output to \n        a log in the `./nlds_log/` directory.\n        This method is good for getting a whole NLDS infrastructure up and running\n        quickly, but is not so great for debugging.\n\n    3.  Also in the `test_run/` directory is a script called `nlds-up` which will\n    activate the virtual environment and run one of the microservices listed above.\n    It requires the enviroment variable `NLDS_VENV` to be set to point to the \n    virtual environemnt created above (e.g. `~/nlds-venv/` - however, user paths \n    have to be expanded so, for the Mac, this would be \n    `/Users/\u003cyour_username\u003e/nlds-venv`).\n    It is recommended to set the environment variable in your shell profile.  For **bash** this is `~/.bash_profile`, and the environment can be set using the \n    line: \n    ```\n    export NLDS_VENV=\"/Users/\u003cyour_username\u003e/python-venvs/nlds-venv\"\n    ```\n    Running `nlds-up` then takes the name of one of the microservices above: e.g. \n    `nlds-up catalog_q`\n\n\nTests\n-----\n\nThe NLDS uses pytest for its unit test suite. Once `test/requirements.txt` have \nbeen installed, you can run the tests with \n```\npytest\n```\nin the root directory. Pytest is also used for integration testing in the \nseparate [nlds-test repo](https://github.com/cedadev/nlds-test). \n\nThe `pytest` test-coverage report can (hopefully) be found [here](https://cedadev.github.io/nlds/coverage/htmlcov/).\n\n\nLicense\n-------\n\nThe NLDS is available on a BSD 2-Clause License, see the [license](./LICENSE.txt) \nfor more info.\n\n\n\nAcknowledgements\n================\n\nNLDS was developed at the Centre for Environmental Data Analysis and supported \nthrough the ESiWACE2 project. The project ESiWACE2 has received funding from the \nEuropean Union's Horizon 2020 research and innovation programme under grant \nagreement No 823988.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcedadev%2Fnlds","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcedadev%2Fnlds","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcedadev%2Fnlds/lists"}