{"id":21721350,"url":"https://github.com/informaticsmatters/fragmentor-k8s-orchestration","last_synced_at":"2026-05-07T05:35:49.515Z","repository":{"id":80605046,"uuid":"313257039","full_name":"InformaticsMatters/fragmentor-k8s-orchestration","owner":"InformaticsMatters","description":"Ansible playbooks for kubernetes deployment of the fragmentor","archived":false,"fork":false,"pushed_at":"2024-12-16T10:01:04.000Z","size":217,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-01-25T18:43:18.847Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jinja","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/InformaticsMatters.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-16T09:53:20.000Z","updated_at":"2024-12-16T10:01:08.000Z","dependencies_parsed_at":null,"dependency_job_id":"f2efdbe7-574d-404a-96d4-6af24ee779d3","html_url":"https://github.com/InformaticsMatters/fragmentor-k8s-orchestration","commit_stats":null,"previous_names":["informaticsmatters/fragmentor-k8s-orchestration"],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InformaticsMatters%2Ffragmentor-k8s-orchestration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InformaticsMatters%2Ffragmentor-k8s-orchestration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InformaticsMatters%2Ffragmentor-k8s-orchestration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InformaticsMatters%2Ffragmentor-k8s-orchestration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/InformaticsMatters","download_url":"https://codeload.github.com/InformaticsMatters/fragmentor-k8s-orchestration/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244693749,"owners_count":20494503,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-26T02:15:59.619Z","updated_at":"2026-05-07T05:35:44.492Z","avatar_url":"https://github.com/InformaticsMatters.png","language":"Jinja","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fragmentor-k8s-orchestration\n\n![lint](https://github.com/InformaticsMatters/fragmentor-k8s-orchestration/workflows/lint/badge.svg)\n\n![GitHub tag (latest SemVer)](https://img.shields.io/github/tag/informaticsmatters/fragmentor-k8s-orchestration)\n\nAnsible playbooks for the Kubernetes-based execution of [fragmentor]\n**Playbooks** (formally the `fragmentor-ansible` repository).\n\n---\n\n![orchestration](docs/images/fragmentor-ansible-topology.001.png)\n\nThis repository's `site-player` play launches a Kubernetes **Pod** (based on the [fragmentor]\nrepository's _player_ image). It uses **ConfigMaps** to _inject_ a Nextflow configuration\nfile, a copy of your chosen `KUBECONFIG` file, and a copy of the play's `parameters.yaml`.\nThe _player_ **Pod** is also attached to a **PersistentVolumeClaim** for data\nprocessing and the `/pgcopy` volume that is shared with your configured database.\n\nThe `fragmentor-player` image (built from the [fragmentor] repository) contains all the\nplaybooks and code from the [fragmentor] repository so that it can orchestrate any play\nauthored in that repository: `standardise`, `fragment`, `inchi`, `extract`, and `combine`.\n\n## Prerequisites\nBefore you attempt to execute any fragmentation plays...\n\n1.  You will need a Kubernetes cluster with a ReadWriteMany storage class\n    (i.e. NFS or, if using AWS, EFS)\n2.  You will need a Kubernetes namespace that contains a pre-deployed postgres\n    server, by default the namespace is expected to be called `fragmentor`.\n    You can use the Ansible playbook and role in our [postgresql-ansible]\n    GitHub repository to create the namespace and install a PostgreSQL\n    database (See the **Kubernetes namespace setup** section below).\n3.  The database needs to be using a shareable (EFS/NFS) volume\n    (`ReadWriteMany`) for use with **pgcopy** command data - and you'll need\n    the name of the PVC. This is expected to be mounted into the database at\n    `/pgcopy`.\n4.  You will need at least 10 spare cores available and 8Gi of RAM in the\n    cluster to satisfy the needs of the database and player Pods.\n5.  The database server needs a user (`fragmentor`) with SUPERUSER privilege\n    and a database (`fairmolecules`)\n6.  The database service is expected to be exposed by a **Service**\n    called `postgres`\n7.  Your cluster must contain nodes with the label\n    `informaticsmatters.com/purpose=core` as the database Pod will require\n    a node with this label during scheduling\n8.  Your cluster must contain nodes with the label\n    `informaticsmatters.com/purpose-fragmentor=yes`. The fragmentor player and Nextflow\n    containers will only run on nodes that contain this label.\n9.  Each fragmentor node must have at least 4 cores and 8Gi RAM.\n10. You will need an AWS/S3 bucket that holds your vendor molecule data.\n    This bucket will also be used for the delivery of the extracted\n    fragmentation/graph data.\n11. You will need a Kubernetes cluster whose version matches the fragmentor `player` image.\n    At the time of writing this was Kubernetes 1.31 but check the\n    fragmentor's `Dockerfile-player` where the kubectl that's installed\n    is defined in the build-time argument `KUBECTL_VERSION`.\n12. You will need your Kubernetes config file.\n13. You will need AWS credentials (that allow for bucket access).\n14. You will need to be able to run `kubectl` from the command-line\n    as the player expects to use it to obtain the cluster host and its IP.\n    So ensure that `KUBECONFIG` is set appropriately.\n\n## Namespace database setup\nYou can conveniently create the required namespace and database using our\n[postgresql-ansible] Ansible playbooks.\n\n\u003e   You will need to decide now how much disk space you need to\n    give your database. There are volumes for the main database\n    and ReadWriteMany volume for 'pgcopy' actions.\n\nStart from the project root of a clone of the repository: -\n\n    python -m venv venv\n\n    source venv/bin/activate\n    pip install --upgrade pip\n    pip install -r requirements.txt\n\n...and create the database and corresponding namespace using an Ansible\nYAML-based parameter file. Here's an example that should work for 'small'\nfragmentation exercises (a few thousand molecules) on a typical AWS\ncluster: -\n\n```yaml\n---\npg_namespace: fragmentor\npg_aux_user: fragmentor\npg_aux_user_password: bullfinch\npg_aux_database: fairmolecules\npg_copy_vol_size_g: 5\npg_copy_vol_storageclass: efs\npg_vol_size_g: 5\npg_vol_storageclass: gp2\npg_cpu_request: 4\npg_cpu_limit: 8\npg_mem_request: 2Gi\npg_mem_limit: 4Gi\n```\n\n\u003e   Your fragmentation stages may require additional postgres configuration\n    in order to function properly (which typically relates to working memory\n    and cores/processes). If this is so then you can provide an 'additional'\n    postgres configuration file and name this using the\n    playbook's `pg_extra_configuration_file` variable.\n\nYou will need to set a few Kubernetes variables...\n\n    export K8S_AUTH_HOST=https://example.com\n    export K8S_AUTH_API_KEY=1234\n    export K8S_AUTH_VERIFY_SSL=no\n\nThen run the playbook...\n\n    ansible-playbook site.yaml -e @parameters.yaml\n    [...]\n\n## The kubeconfig ConfigMap\nThe player is configured to map the cluster's `KUBECONFIG` file into the nextflow\nPods created by the various playbooks. To do this you must create a **ConfigMap** in\nthe `fragmentor` namespace created in the step above.\n\nThis is done using `Kustomize` and the convenient bash script that copies your\nchosen `KUBECONFIG` file into the project's `kustomization` directory\nprior to running the kustomize command.\n\nSet your KUBECONFIG environment variable and then run the command: -\n\n    export KUBECONFIG=~/k8s-config/config-fragmentor\n    ./deploy-kubeconfig.sh\n\nYou are shown the resulting **ConfigMap** YAML before being asked to continue.\n\n\u003e   If you encounter a warning about missing the\n    kubectl.kubernetes.io/last-applied-configuration you most likely had a **ConfigMap**\n    deployed already and you can probably safely ignore this, but check the content\n    of the corresponding **Namespace** object, just to be safe.\n    The warning is issued because it's probably the first copy of the resource\n    and there is no annotation kustomize can use to figure out what to patch and change\n    with the exiting object.\n\n## Using a custom work volume\nIf you need to setup a custom PVC (which has to be **ReadWriteMany**) you can set one\nup, and if it's called `work` the playbooks will use it rather than deploy their own.\nHere's an example PV and PVC for a custom NFS work volume: -\n\n```yaml\n---\nkind: PersistentVolume\napiVersion: v1\nmetadata:\n  name: fragmentor-work\nspec:\n  capacity:\n    storage: 4Ti\n  volumeMode: Filesystem\n  claimRef:\n    name: work\n    namespace: fragmentor\n  accessModes:\n  - ReadWriteMany\n  persistentVolumeReclaimPolicy: Retain\n  nfs:\n    path: /nfs/kubernetes-fragmentor/work\n    server: 130.246.213.182\n```\n\n```yaml\n---\nkind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\n  name: work\n  namespace: fragmentor\nspec:\n  storageClassName: \"\"\n  volumeName: fragmentor-work\n  accessModes:\n  - ReadWriteMany\n  resources:\n    requests:\n      storage: 4Ti\n```\n\n## Running a fragmentor play\nthe _player_ Pod (implemented in and released from the [fragmentor] repository)\nprovides a series of Ansible playbooks for each stage of fragmentation: -\n\n-   **standardise**\n-   **fragment**\n-   **inchi**\n-   **extract**\n-   **combine-simple**\n\nAll of these playbooks are available in the _player_ container in Kubernetes,\nlaunched by this repo's playbook. What this repository's playbook does is\nlaunch the _player_ container (in a pre-existing Kubernetes **Namespace**)\nas a **Job**, mapping Ansible playbook parameters into the container prior\nto its execution. It sets an environment in the _player_ that is used to\nselect the play that is run.\n\nTo run a play you must set a set of play-specific parameters in the local file\n`parameters.yaml`. Once you've defined these you can run the _player_.\n\nStart from a virtual environment: -\n\n    python -m venv venv\n\n    source venv/bin/activate\n    pip install --upgrade pip\n    pip install -r requirements.txt\n\nAs always, set a few key environment parameters: -\n\n    export K8S_AUTH_HOST=https://example.com\n    export K8S_AUTH_API_KEY=?????\n    export K8S_AUTH_VERIFY_SSL=no\n\n    export KUBECONFIG=~/.kube/config\n\nFor access to AWS S3 you will need to provide the following, with the\n`AWS_ENDPOINT_URL` required for non-AWS buckets: -\n\n    export AWS_ACCESS_KEY_ID=?????\n    export AWS_SECRET_ACCESS_KEY=?????\n    export AWS_DEFAULT_REGION=????\n    export AWS_ENDPOINT_URL=????\n\nYou _name_ the play to run using our playbook's `fp_play` variable.\nIn this example we're running the *database reset* play and setting\nthe storage class to `nfs`: -\n\n    ansible-playbook site-player.yaml \\\n        -e fp_play=db-server-configure_create-database \\\n        -e fp_work_volume_storageclass=nfs\n\nAfter displaying the contents of the parameter file, the `KUBECONFIG` file and the\nplayer container image details each play will ask you to confirm that you want to run\nthe play before anything destructive happens.\n\nAs individual plays can take a considerable time to run the `site-player`\nplaybook does not wait for the result - you need to\ninspect the _player_ Pod yourself to check on the its progress.\n\n\u003e   The player playbooks here will prevent you from running another _play_ while\n    one is running.\n\n## Playbook tags (experimental)\nSome playbooks introduce `tags`. For example the **fragment** playbook\nuses the `fragmentation` tag to cover the tasks that actually perform the\nfragmentation, and `load_edges` and `load_nodes` for the logic that loads that data\ninto the database. This is used to allow you to skip certain playbook logic\nshould you need to re-run a play.\n\nYou can *skip* tagged logic using the `fp_play_skip_tags` variable.\nSo, to skip fragmentation logic you could add the following to your parameter file: -\n\n```yaml\nfp_play_skip_tags: fragmentation\n```\n\nThe supplied tags are a comma-separated list of tags (without spaces), so this\nwill skip loading nodes and edges: -\n\n```yaml\nfp_play_skip_tags: load_nodes,load_edges\n```\n\nYou can display the tags supported by a playbook by adding the `--list-tags` option\nto the `fp_play_extra_args` variable in your parameter file: -\n\n```yaml\nfp_play_extra_args: --list-tags\n```\n\nDoing this will run the playbook to display the tags. The playbook will not run\nid you add this argument so you can use it to safely check a play's tags.\n\n## Exposed parameter file content\nPlease be aware that when you rub a playbook the parameter file you provide to control\nthe play (fragment, extract, etc.) will be exposed in the Ansible output log here and\ninjected into the _player_ container using a Kubernetes `ConfigMap`.\nConsequently, any 'sensitive' information that you put in a parameter file\nwill be visible in the local log and to anyone who has access the the Kubernetes\n**fragmentor** `Namespace` used to run the plays.\n\n## Cheat-sheet\nWith a `parameter.yaml` file like the following in your project directory\n(which expects to find xchem/dsip data in your bucket) you should be able to\nrun a sequence of plays.\n\n\u003e   The parameter filename is assumed - if you use a different\n    filename you will need to add `-e fp_parameter_file=myfile.yaml` to the\n    playbook examples below.\n\n\u003e   You will need to provide the name of the AWS S3 bucket that you're\n    using to store RAW vendor data and extracted results. Here we're using the\n    bucket `im-fragnet` but (as these names are global you wil need to replace\n    it with the name of your bucket.\n\n```yaml\n---\ndatabase_login_host: postgres\ndeployment: production\nbucket: im-fragnet\nrunpath: /work\nadd_backup: no\nvendor: xchem_dsip\nversion: v1\nextracts:\n- lib:\n    vendor: xchem_dsip\n    version: v1\n    regenerate_index: yes\nhardware:\n  production:\n    parallel_jobs: 360\n```\n\n-   **Reset fragmentation database**\n\n\u003e   This must be done once, and only once, prior to running any fragmentation\n    plays. It formats the database by adding key tables and seeds then\n    with key records.\n\n```\n    ansible-playbook site-player.yaml \\\n        -e fp_play=db-server-configure_create-database\n```\n\n-   **Standardise**\n\n```\n    ansible-playbook site-player.yaml -e fp_play=standardise\n```\n\n-   **Fragment**\n\n```\n    ansible-playbook site-player.yaml -e fp_play=fragment\n```\n\n-   **InChi**\n\n```\n    ansible-playbook site-player.yaml -e fp_play=inchi\n```\n\n-   **Extract** (a dataset to graph CSV files)\n\n```\n    ansible-playbook site-player.yaml -e fp_play=extract\n```\n\n-   **Combine (Simple)** (multiple datasets into graph CSV files)\n\nUsing a slightly modified parameter file (shown below) you can then combine\nfragmented datasets.\n\n```yaml\n---\ndeployment: production\nbucket: im-fragnet\nrunpath: /work\nadd_backup: no\ncombine:\n- lib:\n    path: extract/xchem_dsip/v1\n    data_source: s3\n    s3_bucket: im-fragnet\n    s3_access_key: ?????\n    s3_secret_key: ?????\n    s3_region: ????\n    s3_url: ????\n- lib:\n    path: extract/xchem_spot/v1\n    data_source: s3\n    s3_bucket: im-fragnet\n    s3_access_key: ?????\n    s3_secret_key: ?????\n    s3_region: ????\n    s3_url: ????\noutput_destination: s3\npath_out: xchem_combi_alan_20201117\ns3_out_access_key: ?????\ns3_out_secret_key: ?????\ns3_out_region: ?????\ns3_out_url: ?????\n```\n\n```\n    ansible-playbook site-player.yaml -e fp_play=combine-simple\n```\n\n## A convenient player query playbook\nIf you don't have visual access to the cluster you can run\nthe following playbook, which summarises the phase of the currently executing\nplay. It will tell you if the current play is still running.\n\n    ansible-playbook site-player_query.yaml\n\nIt finishes with a summary message like this when a play is still running: -\n\n```\nTASK [player : Display query message] *****************************************\nWednesday 18 November 2020  13:17:54 +0000 (0:00:00.049)       0:00:07.401 ****\nok: [localhost] =\u003e {\n    \"msg\": \"The 'standardise' play is running\"\n}\n```\n\n...and something like this when a play has finished successfully: -\n\n```\nTASK [player : Display query message] ************************************************************************************\nok: [localhost] =\u003e {\n    \"msg\": \"The 'db-server-configure_create-database' play has finished (SUCCESSFULLY)\"\n}\n```\n\n## Killing the player (not recommended)\nIf the player is failing, and you want to kill it, and the Job that\nlaunched it, you can run the kill-player playbook: -\n\n    ansible-playbook site-player_kill-player.yaml\n\n---\n\n[fragmentor]: https://github.com/InformaticsMatters/fragmentor\n[postgresql-ansible]: https://github.com/InformaticsMatters/postgresql-ansible\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finformaticsmatters%2Ffragmentor-k8s-orchestration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finformaticsmatters%2Ffragmentor-k8s-orchestration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finformaticsmatters%2Ffragmentor-k8s-orchestration/lists"}