{"id":19863607,"url":"https://github.com/sandialabs/generic-data-display","last_synced_at":"2026-06-09T08:32:34.396Z","repository":{"id":142395538,"uuid":"587467539","full_name":"sandialabs/generic-data-display","owner":"sandialabs","description":null,"archived":false,"fork":false,"pushed_at":"2023-01-10T20:52:26.000Z","size":109,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-01T00:25:51.506Z","etag":null,"topics":["scr-2840","snl-visualization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sandialabs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-10T20:24:21.000Z","updated_at":"2024-08-02T03:11:39.000Z","dependencies_parsed_at":null,"dependency_job_id":"1a41ec26-823c-4609-abf4-1ccb6e0330b2","html_url":"https://github.com/sandialabs/generic-data-display","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sandialabs/generic-data-display","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sandialabs%2Fgeneric-data-display","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sandialabs%2Fgeneric-data-display/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sandialabs%2Fgeneric-data-display/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sandialabs%2Fgeneric-data-display/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sandialabs","download_url":"https://codeload.github.com/sandialabs/generic-data-display/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sandialabs%2Fgeneric-data-display/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34098931,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-09T02:00:06.510Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["scr-2840","snl-visualization"],"created_at":"2024-11-12T15:15:24.510Z","updated_at":"2026-06-09T08:32:34.375Z","avatar_url":"https://github.com/sandialabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"GD2: Generic Data Display\n=========================\nGD2 is a real time data visualization application that can be used to display any type of formatted user defined input data.\nGD2 consists of two components: a python backend that handles receipt and pre-processing of live user input data, and a javascript webserver that handles the display of data pre-processed by the backend.\nThe python backend is composed of a web-socket server connection component, a configuration parsing component, and multiple data processing components that are invoked through a user created JSON configuration file that adheres to the DSL GD2 defines.\nThe javascript frontend is built upon NASA's OpenMCT framework, and manages multiple backend connections using websockets.\nThe primary focus of GD2 is to make it incredibly easy to spin up a visual data display for real-time data: simply create a json config file to describe the input connections and fields you wish to display and pass that file as an argument to the python backend application.\n\n![example](assets/gd2_display.png)\n\n# Component Overview\nGD2 is made up of two main directories:\n- [generic_data_display](generic_data_display), which holds python code for any backend/utility processes\n- [frontend](frontend), which holds javascript code for the frontend server application\n\n## Python Backend\nGD2 comes with mutiple python processes to support processing, storing, and simulating data flow from input to the frontend.\n- `gd2_pipeline`: The processing component of GD2. Configured with a json file that describes the types of connections to make, format of the input data, and how to preprocess it for display purposes.\n- `gd2_data_store`: The database component of GD2. Used by the frontend to display historical data.\n- `gd2_data_sim`: The simulation component of GD2. Generates data for testing and for trying out features.\n- `gd2_validate`: A simple process that can be used to validate pipeline and data_store config files according to the json_schema.\n\n### Pipeline Features\nGD2 supports multiple input connection types, input data formats, and preprocessor functions for manipulating/formatting data.\nThe data formats and functions currently supported are listed below.\nFor more information on how to write a configuration to utilize the below features please read [the GD2 pipeline README.md](generic_data_display/pipeline/README.md)\n\n### Data Store Features\nGD2 stores data from the pipeline using a MongoDB database.\nFor information on how to configure the data store, please read [the GD2 data_store README.md](generic_data_display/data_store/README.md)\n\n### Data Sim Features\nFor testing and trial purposes, the GD2 data simulator can be spun up to create multiple data testing endpoints and is used to showcase the various GD2 capablities.\nThere are also examples on how to create GD2 pipeline and GD2 data store configuration files in the resources directory within this module.\nFor information on how to utilize the data simulator please read [the GD2 data_sim README.md](generic_data_display/data_sim/README.md)\n\n### Validate Features\nValidation is a simple script available when installing the gd2 python module.\nSimply run `gd2_validate` to see the options available for validating configuration files before usage in other GD2 backend processes.\n\n## Javascript Frontend\nGD2 expands on OpenMCT, a javascript framework developed at NASA used to display real-time telemetry data from their rovers and satellite missions.\nIt implements a generic interface for connecting to a proxied websocket data stream, and a display dictionary http endpoint.\nTogether, this allows the GD2 frontend to display any type of json data streamed through the GD2 pipeline.\nTesting deployments are configured via webpack dev server, and a docker container exists with an nginx configuration for deploying the frontend server application.\nFor more information on the frontend component, please read [the GD2 frontend README.md](frontend/README.md)\n\n# Installation Requirements\n## Python Backend\nTo run the python backend you will need at least `python3.7` and install the requirements found [here](requirements.txt).\nWe recommend using python virtual environments when installing GD2 directly.\nYou can create a python virtual environment, activate the environment, install GD2 and its requirements with the following commands:\n```\npython -m venv ~/path/to/gd2_venv\nsource ~/path/to/gd2_venv/Scripts/activate\npip install -r requirements.txt\npip install -e setup.py\n```\n\n## Javascript Frontend\n[See README in frontend for instructions](frontend/README.md)\n\n\n## Docker\nThis repository contains Dockerfiles for running the [python backend](Dockerfile.pipeline),\nthe [python data simulator](Dockerfile.data_sim), the [javascript frontend](Dockerfile.frontend),\nthe [historical data store](Dockerfile.data_store), and the [historical sidecar server](Dockerfile.sidecar).\nThe docker images and their dependencies can be built via [build.sh](build.sh).\n\n\n## Additional Third Party Software (TPS) Requirements\n### Kaitai\nGD2 uses the Kaitai parser to generate python structures used to parse binary data.\nTo use Kaitai, you must install the kaitai-struct-compiler (found here: https://kaitai.io/)\n\n### Protocol Buffer\nGD2 uses the protocol buffer compiler to generate python structures used to parse binary protobuf data.\nTo use Protocol Buffers you must install the protoc python compiler.\n\n# How to Run\n## Docker\nAn example Docker Compose script is [included](docker-compose.yml).\nBefore spinning up the docker containers, you will need to follow these instructions:\n- ensure `no_proxy=[no_proxy_info]` is set\n- ensure `http_proxy={proxy_url}` is set\n- ensure `https_proxy={proxy_url}` is set\n- ensure you have a docker container runtime installed and running\n- ensure that you have `docker-compose` installed\n- create a personal access token on gitlab in user/Settings/Access Tokens with read_registry and write_registry access selected (save this token)\n\nOnce the above steps are followed, to spin up the docker swarm, simply run `docker-compose up`.\nWith the default compose configuration, this will startup the GD2 frontend, backend, and data simulator, accessed at [http://localhost:8080](http://localhost:8080).\n\n### Modifying the configuration files\nUsing docker volumes, it is possible to change the configuation file utilized for a given application in the `docker-compose.yml` file.\nThis involves creating a `volumes` attribute for a given service and modifying the path to the `/opt/gd2/conf` directory in that mount.\nThe following example shows what to set for the `pipeline` service to modify the configuration file used (which is named `pipeline.json`) by pointing to the files in a local `gd2-conf` directory.\n```\n  pipeline:\n    image: ${CI_REGISTRY_IMAGE}/pipeline:latest\n    volumes:\n      - ./gd2-conf:/opt/gd2/conf\n    deploy:\n      mode: global\n      restart_policy:\n        condition: on-failure\n    depends_on:\n      - sidecar\n      - data_sim\n```\n\n## Startup Backend\nWith GD2 installed, you can run the following python command to spin up the processing pipeline application and frontend server:\n```\ngd2_pipeline --config-file path/to/config/file --http-port 8221\n```\nFrom inside the frontend directory.\n```\nnpm build \u0026\u0026 npm run start\n```\nYou can then connect to the frontend webpack server at `http://localhost:8080`\n\n## Startup Data Store\nWith GD2 installed, you can run the following python command and node command to spin up the data store pipeline application and sidecar server:\n```\ngd2_data_store --config-file path/to/config/file\n```\nFrom inside the frontend/sidecar_server directory.\n```\nnpm build \u0026\u0026 npm run start\n```\nIf you are planning on making changes to the side car server want to have the server automatically reload you can use the following command in the frontend/sidecar_server directory.\n```\nnodemon\n```\nYou can then connect to the frontend webpack server at `http://localhost:8080`\n\n## Run an Example Scenario\nFirst, startup the test data server:\n```\ngd2_data_sim --config-file generic_data_display/data_sim/resources/data_sim_configs/sim_config_tcp_kaitai_image.json --log-level debug\n```\nNext, startup the Generic Data Display backend using this config file:\n```\ngd2_pipeline --config-file generic_data_display/data_sim/resources/pipeline_configs/backend_config_tcp_kaitai_image.json --log-level debug\n```\nNext, startup the historical data store using the following config file:\n```\ngd2_data_store --config-file generic_data_display/data_sim/resources/data_store_configs/data_store_config_basic.json --log-level debug\n```\nThen, from inside the frontend directory\n```\nnpm build \u0026\u0026 npm run start\n```\nLastly, from inside the frontend/sidecar_server directory run\n```\nnpm build \u0026\u0026 npm run start\n```\nYou can then connect to the frontend webpack server at `http://localhost:8080`\nFor more info on how to run the examples see the [Examples README.md](generic_data_display/data_sim/README.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsandialabs%2Fgeneric-data-display","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsandialabs%2Fgeneric-data-display","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsandialabs%2Fgeneric-data-display/lists"}