{"id":17117838,"url":"https://github.com/anderspitman/iobio-absent-anders","last_synced_at":"2026-01-04T08:56:14.766Z","repository":{"id":81835849,"uuid":"277915660","full_name":"anderspitman/iobio-absent-anders","owner":"anderspitman","description":null,"archived":false,"fork":false,"pushed_at":"2023-08-10T20:48:49.000Z","size":23,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-15T07:54:26.751Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anderspitman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-07T20:29:15.000Z","updated_at":"2024-06-14T09:00:53.000Z","dependencies_parsed_at":null,"dependency_job_id":"3d3b6aec-277c-4d38-ab3f-76ee7c852323","html_url":"https://github.com/anderspitman/iobio-absent-anders","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anderspitman%2Fiobio-absent-anders","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anderspitman%2Fiobio-absent-anders/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anderspitman%2Fiobio-absent-anders/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anderspitman%2Fiobio-absent-anders/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anderspitman","download_url":"https://codeload.github.com/anderspitman/iobio-absent-anders/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245195926,"owners_count":20575936,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-14T17:52:47.799Z","updated_at":"2026-01-04T08:56:14.710Z","avatar_url":"https://github.com/anderspitman.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Backend\n\nThe backend deployment is relatively robust. The stealthcheck service should\nkeep everything running. I've given a rundown of all our services to aid in\ndebugging in case something unexpected happens.\n\n\n## General information\n\n\n### SSH access\n\nAll of the machines involved should be accessible by the `ubuntu` user via the\n`iobioServers.cer` private key. If you don't have a copy ask Yi or Anders.\n\n\n### stealthcheck\n\nMost of our services, both on AWS and CHPC, are controlled by stealthcheck.\n[stealthcheck][0] is a very simple monitoring service I built. It does health\nchecks on our other services, and sends an email alert and attempts to restart\nthem if they go down.\n\n\n## AWS\n\n### stealthcheck\n\nThere are 2 instances running, stealthcheck1.iobio.io and\nstealthcheck2.iobio.io. stealthcheck1 is the one that monitors all the other\nservices, including stealthcheck2. The only purpose of stealthcheck2 is to\nmonitor stealthcheck1.\n\nSee the private repo [here](https://github.com/iobio/iobio-stealthcheck) for\nmore information about stealthcheck, including how to set up our AWS\nstealthcheck deployment from scratch if either of the VMs becomes\nunrecoverable.\n\nHowever, if you run into problems with stealthcheck, it's recommended to just\nshut it down (can simply stop stealthcheck1 and stealthcheck2 from the AWS\nconsole) and running the services manually until Anders gets back to fix it.\nAll of the necessary commands are in the [checks.json file][1] for\nstealthcheck1 (look under the `fail_command` keys). You can SSH into the\nvarious machines and run the commands shown. You'll lose monitoring and email\nalerts (ie you'll need to keep an eye on things manually), but it should be\nsufficient to limp along for several days.\n\nThe only two that are actually important are fibridge and\n`services.backend.iobio.io` (which is currently just for phenolyzer). You can\npretty safely ignore the others. Information for each service is below.\n\n\n### fibridge\n\nfibridge is used for local file support in the iobio apps. It's probably not\nthe end of the world if it goes down for a couple days, though we'll likely get\na few emails from users.\n\nFirst thing to try is rebooting (NOT terminating) the instance from the AWS\nEC2 dashboard.\n\nIf you want to debug directly you'll need to ssh into\nubuntu@lf-proxy.iobio.io and figure out why the executable isn't running on\nport 9001.\n\nThere are a couple unique things that can go wrong with fibridge. The first\nis if the TLS cert expires. We're currently using a Let's Encrypt cert,\nso we have to manage it manually. You can see from the stealthcheck config\nwhere the cert is stored. In order to renew it, you need to run:\n\n```\nsudo certbot renew\n```\n\nBut that won't work as long as fibridge is running on port 80, so you\nneed to kill fibridge first. But stealthcheck will restart it. You could\nstop stealthcheck, but what I usually do is `sudo killall fibridge-rs`\nand then quickly run `sudo certbot renew` before fibridge comes back\nup.\n\n**NOTE:** If you're running fibridge manually (ie stealthcheck is down), it\ncan appear to be working when it isn't. One way to verify it's actually ok\nis by running bam.iobio with local files. If it doesn't work there's a good\nchance fibridge needs to be restarted. Checking once a day is probably\nsufficient.\n\n\n### Phenolyzer\n\nPhenolyzer runs on services.backend.iobio.io. This has actually been quite\ntemperamental lately, going down every few days. I haven't had time to track\ndown the issue. You'll need to keep an eye out for monitoring emails, or if\nyou're monitoring manually (ie stealthcheck is down), you can use the\nfollowing curl to quickly verify that it's running:\n\n```\ncurl --fail --max-time 5 https://services.backend.iobio.io/\n```\n\nIf that command fails, instructions for fixing phenolyzer are [here][2].\n\n\n### quarantest\n\nquarantest should be kept alive by stealthcheck. In any case it's a\nnon-critical service, so if it goes down just wait until Anders gets back.\n\n\n### gru\n\nThe first thing to do if you suspect something is wrong with the AWS gru\nbackend is perform an instance refresh from the AWS console.\n\nThis is accomplished by going to `EC2\u003eAuto Scaling\u003eAuto Scaling\nGroups\u003egru-backend\u003eInstance refresh\u003eStart instance refresh\u003eStart`. I usually\nchange the settings to 50% and 60 seconds but it probably doesn't matter too\nmuch.\n\nIt takes a number of minutes to complete. You can monitor the nodes under\n`EC2\u003eLoad Balancing\u003eTarget Groups\u003egru-target-group\u003eTargets` to track the\nrefresh progress.\n\nIf that doesn't solve the problem, you'll also need to make sure the DNS\n(route 53), TLS certificates, and load balancer are working.\n\nYou can also try to SSH into the individual VMs (they are named like gru-1.X.X)\nand see if you can find anything fishy going on. I would be very surprised if\nit got to this point.\n\n\n### Multialign\n\nTony is currently in charge of Multialign. She'll probably be the first to\nnotice if it breaks anyway. Eventually we'll get this managed by stealthcheck.\n\n\n## CHPC\n\nOur CHPC PE deployment is pretty simple. It just consists of gru instance and\nMosaic instance. These are both managed by\n[systemd][3] services. The files for these\nservices are `/etc/systemd/system/iobio-gru-backend.service` and\n`/etc/systemd/system/mosaic.service`. Inspecting those files will tell you\nexactly how the services are run.\n\nIf something goes wrong, Chase would be the best person to ask to take a\nlook. You'll need sudo permissions for systemd in order to do most debugging.\n`sudo -l` will tell you what commands you have access. Refer to the systemd docs\nfor how to use the commands `systemctl` and `journalctl`. If you don't have\naccess to the commands, you'll need to email `helpdesk@chpc.utah.edu`.\n\n\n## CDDRC\n\nWe have an instance of Mosaic running on the machine\n`cddrc.utah.edu` for the CDDRC project. It shares the gru backend with the main\nPE instance.\n\nThe systemd file is at `/etc/systemd/system/mosaic.service`. Refer to the\nCHPC instructions above for details.\n\n\n\n[0]: https://github.com/anderspitman/stealthcheck\n\n[1]: https://github.com/iobio/iobio-stealthcheck/blob/master/stealthcheck1.iobio.io/checks.json\n\n[2]: https://github.com/iobio/iobio-backend-services/blob/master/docs/fixing_phenolyzer.md\n\n[3]: https://en.wikipedia.org/wiki/Systemd\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanderspitman%2Fiobio-absent-anders","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanderspitman%2Fiobio-absent-anders","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanderspitman%2Fiobio-absent-anders/lists"}