{"id":20640696,"url":"https://github.com/rhpvorderman/cromwell-mysql-starter","last_synced_at":"2026-02-06T19:30:56.022Z","repository":{"id":125428361,"uuid":"462364726","full_name":"rhpvorderman/cromwell-mysql-starter","owner":"rhpvorderman","description":"Scripts and information on starting a per user cromwell mysql database","archived":false,"fork":false,"pushed_at":"2022-03-02T10:41:26.000Z","size":4,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-03T14:56:32.481Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rhpvorderman.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-22T15:44:24.000Z","updated_at":"2022-02-22T15:44:24.000Z","dependencies_parsed_at":"2023-08-11T12:31:07.404Z","dependency_job_id":null,"html_url":"https://github.com/rhpvorderman/cromwell-mysql-starter","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rhpvorderman/cromwell-mysql-starter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rhpvorderman%2Fcromwell-mysql-starter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rhpvorderman%2Fcromwell-mysql-starter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rhpvorderman%2Fcromwell-mysql-starter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rhpvorderman%2Fcromwell-mysql-starter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rhpvorderman","download_url":"https://codeload.github.com/rhpvorderman/cromwell-mysql-starter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rhpvorderman%2Fcromwell-mysql-starter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29174046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-06T19:28:14.811Z","status":"ssl_error","status_checked_at":"2026-02-06T19:28:13.420Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T15:30:44.523Z","updated_at":"2026-02-06T19:30:56.001Z","avatar_url":"https://github.com/rhpvorderman.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"Creating a per-user MySQL database for Cromwell on HPC\n=======================================================\n\nUse case\n--------\n\nOn a compute cluster that uses a resource scheduler such as SLURM, multiple\nusers can login to a login node and submit jobs to the cluster. The SLURM\nresource scheduler does some job accouting based on the user's ID and the\nuser's department\n\nA Cromwell server does not work well in this use case, because it runs under\na single user. This means the job accounting does not work properly.\n\nCromwell's `run` command is used in these cases, but without a backing database\nthere is no possibility for call-caching, which is vital for production\nworkflows.\n\nThere are several options to alleviate this problem:\n\n1. Use a MySQL database and share a common database between all users.\n    pros:\n        + Highly efficient database calls\n    cons:\n        + Shared database can get locked by any of the users.\n        + Security risk: confidential information under the GDPR such as\n          patient IDs can end up in the database, where everyone can read it.\n2. Use Cromwell's in-memory database with a persistence file. Documented `here \u003chttps://cromwell.readthedocs.io/en/stable/Configuring/#database\u003e`_:\n    pros:\n        + Allows per-project databases to be set up easily.\n        + Everyone can share the same configuration as the database can be\n          set up in the cromwell executions directory.\n        + Permissions handled by filesystem permissions. No security risks.\n    cons:\n        + More memory usage required by cromwell.\n        + The database consumes enormous amounts of disk space. (Multiple\n          gigabytes).\n        + Cromwell's interaction with the database is slow and on very large\n          projects timeouts may occur when reading in the database persistence\n          file.\n3. Create a separate Cromwell server for every user that asks for it.\n    pros:\n        + Solves the problem with user permissions, filesystem permissions and\n          per-user job accounting.\n    cons:\n        + Enormous maintenance burden.\n\nCurrently we use option number 2 on our cluster. Unfortunately it does not\nscale towards thousands of jobs.\n\nOption 4: Allow creation of a per user MySQL database using command line scripts\n--------------------------------------------------------------------------------\n\nMySQL has a server-based architecture. Setting up a server is usuallly not\npossible without root permissions, however there are some features that enable\nsetting up MySQL for non-root users.\n\n+ Mysql is available trough the conda-forge channel and can be installed with\n  miniconda.\n+ Disabling of networking altogether\n+ Communicating via file socket\n\nThis way a per user database can be set up by the users themselves without\nintervention.\n\nstep 1: Installing mysql\n++++++++++++++++++++++++++++++++++++++++++++++++++++\nSet up miniconda with the conda-forge channel. Then:\n\n    conda create -y -n mysql mysql\n    conda activate mysql\n\nstep 2: Setting up a default configuration\n++++++++++++++++++++++++++++++++++++++++++++++++++++\n`mysqld` config values can be overwritten on the command line or can be set\nin a personal config file: `~/.my.cnf`.\n\nSet the following values::\n\n    [mysqld]\n    datadir=/path/to/mysqldb\n    skip-networking=TRUE\n    socket=/path/to/socket\n\nstep 3: Set up the cromwell configuration\n++++++++++++++++++++++++++++++++++++++++++\n\n.. code-block::\n\n    database {\n      profile = \"slick.jdbc.MySQLProfile$\"\n      db {\n        driver = \"com.mysql.cj.jdbc.Driver\"\n        url = \"jdbc:mysql:file:///path/to/socket/cromwell?rewriteBatchedStatements=true\"\n        user = \"user\"\n        password = \"pass\"\n        connectionTimeout = 5000\n      }\n    }\n\n\nstep 4: Start\n+++++++++++++++++++++++++++++++++++++\n+ On first start: ``mysqld --initialize``\n+ Start the database ``mysqld -D``\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frhpvorderman%2Fcromwell-mysql-starter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frhpvorderman%2Fcromwell-mysql-starter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frhpvorderman%2Fcromwell-mysql-starter/lists"}