{"id":25895429,"url":"https://github.com/diamondlightsource/ispyb-database","last_synced_at":"2025-10-23T20:04:14.307Z","repository":{"id":41096614,"uuid":"130190627","full_name":"DiamondLightSource/ispyb-database","owner":"DiamondLightSource","description":"Database schema and documentation for the Diamond flavour of the ISPyB database","archived":false,"fork":false,"pushed_at":"2025-07-01T09:52:28.000Z","size":4582,"stargazers_count":5,"open_issues_count":11,"forks_count":13,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-07-01T10:44:53.029Z","etag":null,"topics":["database-schema","ispyb","lims","mariadb","sql","sql-files","stored-procedures"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DiamondLightSource.png","metadata":{"files":{"readme":"README.md","changelog":"HISTORY.rst","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-04-19T09:19:32.000Z","updated_at":"2025-07-01T09:51:39.000Z","dependencies_parsed_at":"2023-10-18T18:32:04.947Z","dependency_job_id":"9f134b86-0efc-4c8c-949e-f4dd1310adb4","html_url":"https://github.com/DiamondLightSource/ispyb-database","commit_stats":{"total_commits":879,"total_committers":18,"mean_commits":"48.833333333333336","dds":"0.32764505119453924","last_synced_commit":"ef4a3ff2f981fe7548ed36691f1a7e38ce7053dd"},"previous_names":[],"tags_count":74,"template":false,"template_full_name":null,"purl":"pkg:github/DiamondLightSource/ispyb-database","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiamondLightSource%2Fispyb-database","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiamondLightSource%2Fispyb-database/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiamondLightSource%2Fispyb-database/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiamondLightSource%2Fispyb-database/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DiamondLightSource","download_url":"https://codeload.github.com/DiamondLightSource/ispyb-database/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiamondLightSource%2Fispyb-database/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264640004,"owners_count":23642323,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database-schema","ispyb","lims","mariadb","sql","sql-files","stored-procedures"],"created_at":"2025-03-02T22:31:19.948Z","updated_at":"2025-10-23T20:04:09.261Z","avatar_url":"https://github.com/DiamondLightSource.png","language":"Shell","readme":"[![Build Status](https://github.com/DiamondLightSource/ispyb-database/actions/workflows/build.yml/badge.svg)](https://github.com/DiamondLightSource/ispyb-database/actions/workflows/build.yml)\n\n# ispyb-database\n\nThis repository provides everything needed to create the Diamond flavour of the ISPyB database schema. As the schema evolves the update scripts will also be published here, so other users of the schema should be able to stay up-to-date.\n\n## Requirements\n\n* Server and a client packages of MariaDB 10.6 or later.\n* A Unix-like OS with `bash` shell.\n* The pandoc package (only required to build documentation at the end of the `build.sh` script.)\n* If binary logging is enabled in the DB system, then execute this before importing the test schema: `SET GLOBAL log_bin_trust_function_creators=ON;`\n\n## Installation\n\n1. Install MariaDB. The Wiki has ideas on how to install in a particular environment, including OS packages or containers via the provided Dockerfile. If you use the container approach with the provided Dockerfile, then steps 2 and 3 below are done for you automatically.\n2. Copy `.my.example.cnf` to `.my.cnf` and then edit that file to set the `user` and `password`, e.g. `user = root` and correspond `password`. Optionally, you can also set e.g. `host` and `port`.\n3. In a test environment run the `build.sh` file. This creates the database schema and applies the grants as described in the \"Schema\" and \"Grants\" sections below.\n\n### Schema\n\nTip: Execute `./build.sh` to create a development `ispyb_build` database and import all the schema and grants files into it.\n\nAlternatively, do it manually:\n\nRun this on the command-line to create a database and import the schema stored in the SQL files:\n\n```bash\nmariadb -e \"CREATE DATABASE ispyb\"\nmariadb ispyb \u003c schemas/ispyb/tables.sql\nmariadb ispyb \u003c schemas/ispyb/lookups.sql\nmariadb ispyb \u003c schemas/ispyb/data.sql\nmariadb ispyb \u003c schemas/ispyb/routines.sql\n```\n\nNote that the `data.sql` file contains test data, so is only useful in a development environment.\n\n### Grants\n\nThen apply the grants:\n\n```bash\nmariadb ispyb \u003c grants/ispyb_acquisition.sql\nmariadb ispyb \u003c grants/ispyb_import.sql\nmariadb ispyb \u003c grants/ispyb_processing.sql\nmariadb ispyb \u003c grants/ispyb_ro_nopii.sql\nmariadb ispyb \u003c grants/ispyb_scripts_processing.sql\nmariadb ispyb \u003c grants/ispyb_ssx_writer.sql\nmariadb ispyb \u003c grants/ispyb_touchscreen.sql\nmariadb ispyb \u003c grants/ispyb_web.sql\nmariadb ispyb \u003c grants/ispyb_web_verify_tests.sql\n```\n\nNote that the grants files are based on roles, so to actually use these grants, you also need to create database users and grant the roles to them. This is described in the header section of the grant files.\n\n### Miscellaneous Notes\n\nNote that SynchWeb currently assumes [`sql_mode`](https://mariadb.com/kb/en/library/sql-mode/) is not set. I.e. it assumes that you have a line like the below in the MariaDB .cnf file:\n\n```\nsql_mode=''\n```\n\nIn a development environment it might be useful to log all SQL errors. In MariaDB, you can install the [SQL Error Log Plugin](https://mariadb.com/kb/en/library/sql-error-log-plugin/) to get these logged to a file `sql_errors.log` inside your datadir. Run this from the mariadb command-line:\n\n```\nINSTALL SONAME 'sql_errlog';\n```\nYou can verify that it's installed and activated with:\n\n```\nSHOW PLUGINS SONAME WHERE Name = 'SQL_ERROR_LOG';\n```\n\n## Updating\n\nIn order to update a production database, please follow this procedure:\n\n1. For all *.sql files in `schemas/ispyb/updates` that have not already been run, read any comments inside the files to decide if/when you should run them. Run a file e.g. like this:\n```bash\nmariadb ispyb \u003c schemas/ispyb/updates/2019_03_29_BLSession_archived.sql\n```\n2. If `schemas/ispyb/routines.sql` has been updated since you installed it, you can simply re-run it. E.g.:\n```bash\nmariadb ispyb \u003c schemas/ispyb/routines.sql\n```\n3. If you ran the `routines.sql`, then re-apply the grants for the routines. E.g.:\n```bash\nmariadb ispyb \u003c grants/ispyb_acquisition.sql\nmariadb ispyb \u003c grants/ispyb_processing.sql\nmariadb ispyb \u003c grants/ispyb_web.sql\n```\n\n## Useful scripts\n\n* `build.sh`: Creates a development `ispyb_build` database and imports all the schema and grants files into it.\n* `release.sh`: Makes a release, see the `RELEASE.md` file for details.\n* `bin/backup.sh`: Makes a backup of the development database, producing four files: `tables.sql`, `routines.sql`, `lookups.sql`, `data.sql`.\n* `bin/missed_updates.sh`: Generates a list of files in the `schemas/ispyb/updates/` folder that haven't been applied.\n* `bin/export_session.sh`: Exports a given session's database rows to SQL files. These can then easily be imported into an empty ispyb database. This can be used e.g. to create a single-session ISPyB/SynchWeb instance if combined with all the relevant data files and processing files for the session, and an instance of SynchWeb.\n\n## Documentation\n\n* Please refer to the [```Wiki```](https://github.com/DiamondLightSource/ispyb-database/wiki) for database diagrams, stored procedure how-to, MariaDB installation and more  \n* A complete [```list of tables and columns```](https://alfred.diamond.ac.uk/documentation/ispyb-database/list_of_tables_and_columns.html) for more details about the tables and columns\n* A complete [```list of stored procedures```](https://alfred.diamond.ac.uk/documentation/ispyb-database/list_of_procs.html) for more details about the stored procedures\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiamondlightsource%2Fispyb-database","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdiamondlightsource%2Fispyb-database","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiamondlightsource%2Fispyb-database/lists"}