{"id":46320546,"url":"https://github.com/sanger/unified_warehouse","last_synced_at":"2026-06-10T10:00:43.727Z","repository":{"id":21109205,"uuid":"24409681","full_name":"sanger/unified_warehouse","owner":"sanger","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-04T12:59:00.000Z","size":2072,"stargazers_count":2,"open_issues_count":21,"forks_count":8,"subscribers_count":5,"default_branch":"develop","last_synced_at":"2026-06-04T14:20:42.767Z","etag":null,"topics":["ruby"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sanger.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2014-09-24T09:53:04.000Z","updated_at":"2026-06-04T12:58:48.000Z","dependencies_parsed_at":"2023-12-08T11:29:19.614Z","dependency_job_id":"24b64f96-3a6b-45ff-a855-cc5f262b65b0","html_url":"https://github.com/sanger/unified_warehouse","commit_stats":null,"previous_names":[],"tags_count":510,"template":false,"template_full_name":null,"purl":"pkg:github/sanger/unified_warehouse","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sanger%2Funified_warehouse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sanger%2Funified_warehouse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sanger%2Funified_warehouse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sanger%2Funified_warehouse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sanger","download_url":"https://codeload.github.com/sanger/unified_warehouse/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sanger%2Funified_warehouse/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34146870,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-10T02:00:07.152Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ruby"],"created_at":"2026-03-04T15:03:49.973Z","updated_at":"2026-06-10T10:00:43.713Z","avatar_url":"https://github.com/sanger.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Unified Warehouse Builder\n\n[![Build Status](https://travis-ci.org/sanger/unified_warehouse.svg?branch=develop)](https://travis-ci.org/sanger/unified_warehouse)\n\nA denormalised warehouse for multiple LIMS. This project consumes messages from a message broker (RabbitMQ) and persists them in MLWH.\n\nPopulating a table in the warehouse is done asynchronously. This project is a Railtie application that facilitate asynchronous database population through a message queue. This application listens to a message queue, consumes the message, converts the message into a database model, and persists in the corresponding table. \n\nTake the example below.\n\n```json\n{\n    \"lims\": \"traction\",\n    \"aliquot\": {\n        \"id_lims\": \"LIMS123456\",\n        \"lims_uuid\": \"550e8400-e29b-41d4-a716-446655440000\",\n        \"aliquot_type\": \"DNA\",\n        \"source_type\": \"Blood\",\n        \"source_barcode\": \"SRC123456\",\n        \"sample_name\": \"SampleA\",\n        \"used_by_type\": \"Research\",\n        \"used_by_barcode\": \"USR123456\",\n        \"volume\": 50,\n        \"concentration\": 200,\n        \"last_updated\": \"2024-07-09T10:15:30Z\",\n        \"recorded_at\": \"2024-07-09T09:00:00Z\",\n        \"created_at\": \"2024-07-08T08:00:00Z\",\n        \"insert_size\": 350\n    }\n}\n```\n\nIf the message above is consumed by `unified_warehouse`, it will create a new record in `aliquot` table with the values set to the given attributes. If an exception occur, it will dead-letter the message. For message serialization logic, please follow [`lib/payload.rb`](https://github.com/sanger/unified_warehouse/blob/beea89418cc1987c426e7967b7c13ff9218b51fb/lib/payload.rb#L31-L33).\n\n## Usage (Development)\n\n### Requirement\n\n1. MySQL (currently 8.4) is required and usually installed with homebrew:\n\n       brew install mysql@8.4\n       brew link mysql@8.4 --force\n\n### Installation\n\n1. Clone the git repository\n2. Install the relevant ruby from `.ruby-version`  \n    Rbenv will read .ruby-version automatically   \n   `rbenv install`\n\n3. Run the setup process:  \n    3.1. Set up your MySQL root password inside the file config/database.yml  \n    3.2. Run `bin/setup ` \n\n**NB**: If getting an error while installing the `mysql2` gem, try:\n\n    bundle config build.mysql2 --with-opt-dir=$(brew --prefix openssl)\n    \nif that doesn't work, try:\n\n       gem install mysql2 -- \\\n        --with-mysql-lib=/Users/your_user/homebrew/Cellar/mysql/your_version/lib \\\n        --with-mysql-dir=/Users/your_user/homebrew/Cellar/mysql/your_version  \\\n        --with-mysql-config=/Users/your_user/homebrew/Cellar/mysql/your_version/bin/mysql_config \\\n        --with-mysql-include=/Users/your_user/homebrew/Cellar/mysql/your_version/include\n\nand try runnning `bundle install` again.\n\n#### Database preparation\n\nBefore you can use the system in any capacity, you must first prepare the database.\nThis should be handled by `bin/setup` above, but if not:\n\n    bundle exec rake db:setup\n\n#### (Optional) Create the views\n\nThis project provides the view `cherrypicked_samples` that links data with\nthe event warehouse. To create the view you need to run the command:\n\n    bundle exec rake db:views:schema:load\n\n### Running tests\n\nEnsure the test suite is running and passing:\n\n    bundle exec rspec\n\n### Integration tests\n\n#### Setup\n\n1. Initialize the integration tests setup for events warehouse (please check the\n   Integration Tests setup section at \u003chttps://github.com/sanger/event_warehouse/#integration-tests-setup\u003e)\n\n2. Reset the database\n\n       bundle exec rake db:reset\n\n3. Create the dependent views\n\n       bundle exec rake db:views:schema:load\n\nThese actions can also be performed automatically if you run the Docker container of the service\nand pass the environment variables:\n\n    RAILS_ENV=\"test\"\n    INTEGRATION_TEST_SETUP=\"true\"\n\n#### Running the integration tests\n\n1. Run the integration tests:\n\n        bundle exec rspec --tag integration\n\n### Execution\n\nExecute the worker to pick up messages in the queue and process them into the\ndatabase:\n\n        bundle exec warren consumer start\n\nThe consumer will run in the foreground, logging to the console. You can stop it with Ctrl-C.\n\nFor more warren actions, either use `bundle exec warren help` or see the\n[warren documentation](https://rubydoc.info/gems/sanger_warren)\n\n#### worker_count\n\nThe number of worker threads can be configured for the consumer in\n`warren_consumers.yml`. This setting is applied to the channel and affects how\nmessages from the subscription on the queue are processed. Setting this value\nto one uses a single thread and, therefore, a single writer to the database. If\nonly a few tables are written sequentially, a single worker has the advantage\nof avoiding lock contention. If not configured, the default value is 3.\n\n### Preparing to run locally with Traction Service\n\nRabbitMQ is essential for this process, so if you haven't already, install it using:\n\n    brew install rabbitmq\n    brew services start rabbitmq\n\nYou can now view the instance running at [http://localhost:15672/](http://localhost:15672/).\n\nYou may wish to start the warren consumers with:\n\n    bundle exec warren consumer start --path='config/warren_traction_service_dev.yml'\n\nThis will adjust the configuration options to be compatible with those suggested in the traction setup.\n\nAlso see [managing custom configs](#manage-custom-configs)\n\n### Manage Custom Configs\n\nIt is possible to run the consumers with a custom configuration, eg.\n\n    bundle exec warren consumer start --path='config/my_customized_config.local.yml'\n\nThe `.gitignore` file will automatically prevent these configurations from being committed.\n\n#### Troubleshooting\n\nIf you receive an error about a missing output file under `tmp/pid/` it may be that you need to create this directory manually.\nOnce the directory above has been inserted at the root of the repository, the error should go away.\n\n#### How To Section\n\nCOG-UK Ids - These ids are given to positive samples imported through the Lighthouse-UI. This process should automatically record those Ids in the sample table, and also into the lighthouse_sample table.\nTo migrate COG-UK Ids into the lighthouse_sample table manually via SQL, see [this Confluence page](https://ssg-confluence.internal.sanger.ac.uk/display/PSD/How+to+migrate+Cog+UK+IDs+into+the+lighthouse_sample+table).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsanger%2Funified_warehouse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsanger%2Funified_warehouse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsanger%2Funified_warehouse/lists"}