{"id":22621425,"url":"https://github.com/cca/moodle-backups","last_synced_at":"2025-03-29T02:24:37.015Z","repository":{"id":38886460,"uuid":"248359096","full_name":"cca/moodle-backups","owner":"cca","description":"back up specific courses from a Moodle instance","archived":false,"fork":false,"pushed_at":"2024-08-15T15:55:17.000Z","size":122,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-02-03T12:35:28.246Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cca.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-18T22:43:21.000Z","updated_at":"2024-08-15T15:55:21.000Z","dependencies_parsed_at":"2025-02-03T12:31:35.276Z","dependency_job_id":"5c7c769a-b586-4d38-a154-644121234564","html_url":"https://github.com/cca/moodle-backups","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cca%2Fmoodle-backups","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cca%2Fmoodle-backups/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cca%2Fmoodle-backups/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cca%2Fmoodle-backups/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cca","download_url":"https://codeload.github.com/cca/moodle-backups/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246127811,"owners_count":20727794,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-08T23:08:50.937Z","updated_at":"2025-03-29T02:24:36.996Z","avatar_url":"https://github.com/cca.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Moodle Backups\n\nBackup old Moodle courses to a Google Storage Bucket with a slower storage class.\n\n**As of Fall 2024, we no longer back up courses to Google Storage.** We are using the objectfs plugin to store Moodle files directly in Google Storage with autoclass to move seldom-used files to a slower storage class. This eliminates the need for this backup process but backups from 2021SU and prior are still stored in GSB.\n\nBackups are stored initially in the \"coldline\" storage class, reduced to \"archive\" after 240 days, and then deleted after 730 days using GSB Lifecycle Rules. The goal of using GSB is to save money, make management easier (e.g. the lifecycle automation), and make backups programmatically accessible. We have [a spreadsheet index](https://docs.google.com/spreadsheets/d/1mxO2PbKk088R9e3rU_XwUpxV_HwzIKBiIrK1xPy3zfU/edit?usp=sharing) of backups in the Libraries' InST Shared Drive folder which is used to identify backups for retrieval.\n\n## Setup\n\nRequires [fish shell](https://fishshell.com/), [gsutil](https://cloud.google.com/storage/docs/gsutil_install), and [kubectl](https://kubernetes.io/docs/reference/kubectl/). Fish and kubectl are in [homebrew](https://brew.sh).\n\n```sh\n\u003e brew install fish kubectl\n\u003e # Or install using gcloud. Do not install kubectl with both\n\u003e glcoud install gsutil kubectl\n```\n\nWe also need access to the [Moodle Course Archive](https://console.cloud.google.com/storage/browser/moodle-course-archive;tab=objects?project=cca-web-0) storage bucket as well as all of the Moodle kubernetes clusters (a \"staging\" `kubectl` context for tests and a \"production\" `kubectl` context for actual backups).\n\n## Semester Backup Workflow\n\nAfter a semester concludes, run the [Backups Index Report](https://moodle.cca.edu/report/customsql/view.php?id=30) for it and append its results to the Backups Index in Drive.\n\nTwo years after a semester has concluded, we can backup \"used\" courses in GSB and delete them from Moodle:\n\n- consult the Backups Index to determine which courses to backup\n  - our criteria tends to factor in course usage, number of modules, and visibility\n  - start with the Metacourses category (see below)\n  - export a list of course ID numbers (with no header row) and save it as data/ids.csv\n- [copy Panopto course groups to internal ones](https://github.com/cca/panopto_course_group_copy)\n- use one of the included backup procedures below\n\nWe backup the Metacourses category first because we don't want to delete the composite sections before the parent metacourse, then the metacourse backup will lack enrollments and their associated student work.\n\n### In the cloud (recommended)\n\nSetup:\n\n- in GCP, go to IAM \u003e Service Accounts, find the `moodle-backups` service account (SA)\n- create a JSON key file for the SA (\"more\" Actions \u003e Manage keys)\n- transfer the key file to the pod (e.g. `kubectl cp key.json POD:/bitnami/moodledata/key.json`)\n- prepare the pod to use google cloud utilities (Google has [official installation documentation](https://cloud.google.com/storage/docs/gsutil_install) that didn't fully work for me):\n\n```sh\ninstall_packages apt-transport-https ca-certificates gnupg curl\necho \"deb https://packages.cloud.google.com/apt cloud-sdk main\" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list\ncurl https://packages.cloud.google.com/apt/doc/apt-key.gpg | tee /usr/share/keyrings/cloud.google.gpg\napt-key add /usr/share/keyrings/cloud.google.gpg # TODO apt-key is deprecated\ninstall_packages google-cloud-cli\ngcloud auth activate-service-account --key-file /bitnami/moodledata/key.json\n```\n\nTo take advantage of speedier composite uploads, we might need to edit the service account's .boto file (/root/.boto) to add the properties described in the \"gsutil composite objects \u0026 CRC Mod\" section below. The last time I installed `gcloud`, it came with a compiled CRC Mod ready for these composite uploads. The SA has Storage Object Admin permission on the course archive bucket to take advantage of composite uploads.\n\nOnce ready, edit the `SEMESTER` variable at the top and then run onpodproc.sh on the container. It backs up 5 courses (or `./onpodproc N` to backup N at a time) from the list of IDs in ids.csv and writes to a log file.\n\nIt's best to delete the service account key (`rm /bitnami/moodledata/key.json` and IAM \u003e Service Accounts \u003e moodle-backups \u003e Manage Keys) when backups are done and to create a new key the next time we backup courses.\n\n### Via laptop\n\nThis method is a little easier to manage but introduces an extra network transfer (pod -\u003e laptop -\u003e storage instead of pod -\u003e storage) and is _much_ slower when running over a consumer internet connection with mediocre upload speeds.\n\n- run the included proc.fish script to iterate over ids.csv, its steps are\n  - `./backup mk 1234` backup a course\n  - `./backup dl --all` download all the backup files to the data dir\n  - `./backup cp $SEMESTER data/backup_*` transfer files to GSB (note: avoid copying any test files in the data dir)\n  - `./backup rm --all` delete the courses \u0026 their backups on the pod\n\n### Backups in recycle bin\n\nEvery time a course is backed up, the .mbz backup file is stored in the Recycle Bin for a configurable amount of time (see [the settings](https://moodle.cca.edu/admin/settings.php?section=tool_recyclebin)). It may be necessary to pause in the middle of a large number of backups sometimes until the bin is emptied. We can monitor these backup files with the [Backups in the Recycle Bin](https://moodle.cca.edu/report/customsql/view.php?id=15) report and the size of the \"trashdir\" directory underneath Moodle's data directory.\n\n### Deleted courses recreated\n\nWhen we delete a course from Moodle, its enrollments still exist in the Moodle support database, and thus during the next enrollment sync they will be recreated. We don't want to delete all of the old term's enrollments out of the support db _before_ performing backups, because then users will be removed from the course and the backup that's created won't reflect the real enrollment. The best way to manage this is to piecemeal backup courses in chunks, removing their entries from the support db at the same time as you delete the courses in Moodle. Course categories are a convenient way of chunking courses and make it easier to delete things in bulk:\n\n```sh\n# say we've finished backing up course categories 100 - 105\n# categories are mostly numbered to match their alphabetical order but things like \"metacourses\" can be exceptions\n# on the Moodle container:\nfor cat in (seq 100 105); do moosh -n category-delete $cat; done\n```\n\n```sql\n-- in the support db:\nDELETE FROM enrollments WHERE category_id IN (100, 101, 102, 103, 104, 105)\n```\n\n## Retrieving backups\n\nWhen we need a backup, `./backup retrieve $QUERY` retrieves it from the archive.\n\n## Testing\n\n```sh\n\u003e # run all tests\n\u003e test/test\n\u003e # test a specific command (or set of commands)\n\u003e test/test create\n```\n\nTests run against the staging Moodle cluster. The ID for a particular course (3606) in our staging instance is hard-coded into one test but can be overridden with a `TEST_COURSE` enrivonment variable.\n\n## gsutil composite objects \u0026 CRC Mod\n\n`gsutil` prints this notice if we try to upload a large file:\n\n\u003e ==\u003e NOTE: You are uploading one or more large file(s), which would run significantly faster if you enable parallel composite uploads. This feature can be enabled by editing the  \"parallel_composite_upload_threshold\" value in your .boto configuration file. However, note that if you do this large files will be uploaded as `composite objects \u003chttps://cloud.google.com/storage/docs/composite-objects\u003e`_, which means that any user who downloads such objects will need to have a compiled crcmod installed (see \"gsutil help crcmod\"). This is because without a compiled crcmod, computing checksums on composite objects is so slow that gsutil disables downloads of composite objects.\n\nCheck if a compiled CRC mod is available with `gsutil version -l` (it'll say \"compiled crcmod: True\"). If not, run `pip3 install -U crcmod` to install it. If pip claims it's already installed, it might not be running in the same python environment gsutil is using. Look at `gcloud info` for the Python Location and then run `$PYTHON_LOCATION -m pip install -U crcmod` (see [this comment](https://github.com/GoogleCloudPlatform/gsutil/issues/1123#issuecomment-772588861)). Finally, edit the .boto configuration file in your user's home directory to [enable parallel composite uploads](https://cloud.google.com/storage/docs/uploads-downloads#parallel-composite-uploads):\n\n```ini\n[GSUtil]\nparallel_composite_upload_threshold = 100M\nparallel_composite_upload_component_size = 50M\n```\n\nAlternatively, set the `parallel_composite_upload_threshold` to `0` to disable this message and then the `gsutil` clients that download files added without the parallel composite upload won't need crcmod. In testing, files uploaded as composite objects are able to be downloaded via the Google Cloud Console, so they can still be accessed from machines without a compiled crcmod.\n\n## LICENSE\n\n[ECL Version 2.0](https://opensource.org/licenses/ECL-2.0)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcca%2Fmoodle-backups","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcca%2Fmoodle-backups","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcca%2Fmoodle-backups/lists"}