{"id":22753987,"url":"https://github.com/slaclab/mps_database","last_synced_at":"2025-04-14T15:34:33.624Z","repository":{"id":38898558,"uuid":"85114186","full_name":"slaclab/mps_database","owner":"slaclab","description":null,"archived":false,"fork":false,"pushed_at":"2023-07-05T22:47:14.000Z","size":4252,"stargazers_count":2,"open_issues_count":1,"forks_count":7,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-28T04:27:31.524Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/slaclab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-03-15T19:44:20.000Z","updated_at":"2024-07-26T21:28:48.000Z","dependencies_parsed_at":"2023-02-14T10:45:21.336Z","dependency_job_id":null,"html_url":"https://github.com/slaclab/mps_database","commit_stats":null,"previous_names":[],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slaclab%2Fmps_database","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slaclab%2Fmps_database/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slaclab%2Fmps_database/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slaclab%2Fmps_database/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/slaclab","download_url":"https://codeload.github.com/slaclab/mps_database/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248906627,"owners_count":21181194,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-11T06:15:07.559Z","updated_at":"2025-04-14T15:34:33.586Z","avatar_url":"https://github.com/slaclab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LCLS-II MPS Database\n\nThis package contains `SQLAlchemy` python classes that provide access to the MPS `sqlite3` database. The classes are listed under the models subdirectory\n\n## Initialization\n\nIn order to use mps_database, you must first have access to conda, the virtual environment and package manager. \n\nIf you are already able to sucessfully run conda via the command line, then you can skip the \"Initializing Conda\" steps. This can be tested by running `conda` and comparing the following output:\n```\n$ conda\nusage: conda [-h] [-V] command ...\n\nconda is a tool for managing and deploying applications, environments and packages.\n...\n```\n\n### Initializing Conda\n\nRefer to documentation about using conda at SLAC.\n\n### Creating the Conda Environment\n\nOn `lcls-dev3`, after cloning the repo (`$ git clone git@github.com:slaclab/mps_database.git`) please navigate to the top of the mps_database directory, where environment.yml is located.\n\n\nThen, run the following commands to create the proper python-based virtual environment:\n\n```\n$ conda env create -f environment.yml\n\n$ conda activate mps-environment\n```\nThis creates a conda environment based off of the template file environment.yml. This .yml file includes a list of all packages and modules with their associated versions that will be installed within your conda environment. \n\nYou should now be within the created conda environment with a prompt such as:\n\n`(mps-environment) jsmith@lcls-dev3 mps_database]$`\n\n### Initializing the Project\nNow, we must initialize the mps_database module itself. In the top directory where the `setup.py` file is located, run:\n```\n$ pip install -e .\n```\nand look for the output:\n```\nInstalling collected packages: mps-database\n  Running setup.py develop for mps-database\nSuccessfully installed mps-database\n```\nNote: these steps only need to be completed once. After being initialized for the first time, refer to \"Standard Operation\" to utilize the environment.\n\n### Installing Packages\n\nShould you need to install any further packages to this environment, you can do so by running `conda install packagename` with the optional source location parameter. Sample package installations include:\n```\n(mps-environment) $ conda install packagename\n(mps-environment) $ conda install -c conda-forge packagename\n```\n### Deleting the Environment\n\nIf you ever need to delete the virtual environment, you can do so by running:\n\n`conda env remove --name mps-environment`\n\nHowever, you will have to create a new copy of the environment from the environment.yml file all over again. You should never need to delete this environment. \n\n## Operation\n### Environment Setup\n\nIn order to use this module, you must be within the conda virtual environment. \n\n\nTo activate the created environment, run:\n\n`$ conda activate mps-environment`\n\nTo turn off this environment and return to your standard environment and paths, run:\n\n`(mps-environment) $ conda deactivate`\n## Scripts\n\nAll the following operations are invoked by the `mps_database/tools/export_all.py` script when creating new databases. \n\n### Generate Configuration database\n\nThis script creates the file sqlite file `mps_config_imported.db`, the configuration database, that is used by other scripts to generate EPICS database and panels. It also creates the file `mps_config_imported_runtime.db`, the runtime database, used by the [MPS manager](https://github.com/slaclab/mps_manager).\n\n```\n$ ./import/import-csv.py\n```\n\n### Generate All files\n\nThis export the sqlite configuration database to all other files needed to run the system.  It generates central node configuration YAML files, pydm display files, epics DB files for link nodes and central nodes.  In the near future, it will also generate documentation and alarms.  Necessary arguments:\n\n--db \u003cdatabase_file\u003e\n--template \u003cpath to templates\u003e\n--destination \u003cpath to output\u003e\n\n```\n$ ./tools/export_all.py --db \u003csqlite db file\u003e --dest \u003cdestination path\u003e --template \u003cpath to templates, located in mps_database\u003e\n```\nThe link node databases will be generated in \u003cdestination path\u003e with the following structure:\n\n```\n\u003cdestination_path\u003e/link_node_db/app_db/\u003cCPU_NAME\u003e/\u003cCRATE_ID\u003e/\u003cSLOT_NUMBER\u003e\n```\n\nEach directory will contain the following EPICS database and configuration files for each application:\n- `mps.db`: EPICS database file\n- `mps.env`: EPICS environmental variables\n- `config.yaml`: Firmware configuration file\n\nExample of the result obtained for the EIC configuration:\n\n```\n.\n└── app_db\n    └── cpu-gunb0-mp01\n        └── 0001\n            ├── 02\n            │   ├── config.yaml\n            │   ├── mps.db\n            │   └── mps.env\n            ├── 03\n            │   ├── config.yaml\n            │   ├── mps.db\n            │   └── mps.env\n            ├── 06\n            │   ├── config.yaml\n            │   ├── mps.db\n            │   └── mps.env\n            └── 07\n                ├── config.yaml\n                ├── mps.db\n                └── mps.env\n```\n\nThe central node is a collection of 3 central nodes:\ncn1 = slot2 of B005-510 and handles LN groups 0-11\ncn2 = slot3 of B005-510 and handles LN groups 12-23\ncn3 = slot2 of L2KA00-XX and handles LN groups 0-8\n\nThe script generates files like:\n\n\u003cdestination path\u003e/central_node_db/cn#/\u003cdatabase_file\u003e\n\nThe central node IOC will load the correct databases based on it's configuration\n\nThere are three central node configuration yaml files also created, one for each CN.  They are output to \n\u003cdestination path\u003e/\u003cdb_name\u003e-cn#-\u003cdb_version\u003e.yaml.  They must be loaded into the proper CN via caput:\ncaput SIOC:SYS0:MP0#:CONFIG_LOAD \"\u003cproper_yaml_file\u003e\nIn the future, this will be automated, but for now is by hand.\n\n## Import CSV data\n\nThe script [import/import-csv.py](import/import-csv.py) generates a database using csv exported files from the `MPSInputList.xlsx` file (available from sharepoint). The generated database name is `mps_config_imported.db`.\n\n###### currently depricated ######\n## Tests\n\nThe [central_node_test.py](tools/central_node_test.py) script can be used to simulate firmware input data to the central node software. It sends update messages to the central node for each device state (e.g. valve on/valve off) and gets back a mitigation message, which is compared to the expected mitigation as described in the database.\n\nThe central node software must be compiled without the option `-DFW_ENABLE`, which enables the software to receive updates and send mitigation to the test script instead of exchanging messages with the central node firmware (via CPSW). The central node software can run as a standalone server (see [central_node_engine](https://github.com/slaclab/central_node_engine) package under `src/test/central_node_engine_server`). The central node IOC may also be used for testing, but it must be linked against the non-FW_ENABLED central_node_engine lib.\n\nOptions for the script are:\n```\n$ ./tools/central_node_test.py -h\nusage: central_node_test.py [-h] [--host hostname] [--port [size]]\n                            [--debug [debug]] [--device [device]] [--analog]\n                            [--report] [--delay]\n                            db\n\nSend link node update to central_node_engine server\n\npositional arguments:\n  db                 database file name (e.g. mps_gun.db)\n\noptional arguments:\n  -h, --help         show this help message and exit\n  --host hostname    Central node hostname\n  --port [size]      server port (default=4356)\n  --debug [debug]    set debug level output (default level 0)\n  --device [device]  device id (default - test all digital devices)\n  --analog           analog device\n  --report           generate pdf report\n  --delay            add 1 second delay between tests\n```\n\nExample of output:\n```\n$ ./tools/central_node_test.py mps_gun_config.db --host cpu-li00-mp01\nYAG01B PASSED\nGun Temperature PASSED\nWaveguide Temperature PASSED\nBUN1B Buncher Temperature PASSED\nSOL01B Temp PASSED\nSOL02B Temp PASSED\nVVR01 PASSED\nVVR02 PASSED\n```\n\nExample of verbose output, specifying a single device for testing:\n```\n$ ./tools/central_node_test.py mps_gun_config.db --host cpu-li00-mp01 --device 1 --debug 1\n+------------------------------------------------------------+\nDevice YAG01B\n+------------------------------------------------------------+\nDevice value    : 00\nInputs          : OUT_LMTSW=0 IN_LMTSW=0\nMitigation      : MS=0 AOM=0\nRecvd Mitigation: MS=0 AOM=0\n+------------------------------------------------------------+\nDevice value    : 01\nInputs          : OUT_LMTSW=1 IN_LMTSW=0\nMitigation      : MS=2 AOM=1\nRecvd Mitigation: MS=2 AOM=1\n+------------------------------------------------------------+\nDevice value    : 02\nInputs          : OUT_LMTSW=0 IN_LMTSW=1\nMitigation      : MS=2 AOM=2\nRecvd Mitigation: MS=2 AOM=2\n+------------------------------------------------------------+\nDevice value    : 03\nInputs          : OUT_LMTSW=1 IN_LMTSW=1\nMitigation      : MS=0 AOM=0\nRecvd Mitigation: MS=0 AOM=0\n+------------------------------------------------------------+\nYAG01B PASSED\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fslaclab%2Fmps_database","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fslaclab%2Fmps_database","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fslaclab%2Fmps_database/lists"}