{"id":13858993,"url":"https://github.com/mc2-center/minidream-r-env","last_synced_at":"2026-01-22T03:19:26.290Z","repository":{"id":37979381,"uuid":"129964357","full_name":"mc2-center/minidream-r-env","owner":"mc2-center","description":"Resources for setting up and managing an RStudio environment for interactive mini-DREAM activities","archived":false,"fork":false,"pushed_at":"2023-06-23T23:40:21.000Z","size":4641,"stargazers_count":3,"open_issues_count":3,"forks_count":1,"subscribers_count":14,"default_branch":"develop","last_synced_at":"2024-10-07T15:02:31.199Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mc2-center.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-17T20:58:29.000Z","updated_at":"2023-04-20T20:34:57.000Z","dependencies_parsed_at":"2024-05-30T22:37:11.054Z","dependency_job_id":"2347b31d-e5b2-4790-8ea5-a81c8bfa06c4","html_url":"https://github.com/mc2-center/minidream-r-env","commit_stats":null,"previous_names":["sage-bionetworks/minidream-r-env"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mc2-center%2Fminidream-r-env","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mc2-center%2Fminidream-r-env/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mc2-center%2Fminidream-r-env/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mc2-center%2Fminidream-r-env/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mc2-center","download_url":"https://codeload.github.com/mc2-center/minidream-r-env/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225938750,"owners_count":17548544,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-05T03:02:28.826Z","updated_at":"2026-01-22T03:19:26.252Z","avatar_url":"https://github.com/mc2-center.png","language":"Shell","funding_links":[],"categories":["Shell"],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n  miniDREAM R-Environment\n\u003c/h1\u003e\n\n\u003ch3 align=\"center\"\u003e\n  Resources for setting up and managing an RStudio environment for interactive \n  miniDREAM activities\n\u003c/h3\u003e\n\u003cbr/\u003e\n\n## 🛠️ Setup\n\n### AWS\n\n1. Choose an Ubuntu-based instance, as Ubuntu readily comes with `apt-get` and\n   Python. We recommend researching Ubuntu's ideal version with its\n   [end of life plot](https://www.ubuntu.com/info/release-end-of-life). For\n   example: `v16.04` is ideal for the year 2018.\n\n2. AWS Security groups need to have a TCP rule to open port number 8787.\n\n### RStudio Server\n\n1. Install [Docker-CE](https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-using-the-repository)\n   onto the instance if it's not already available.\n\n2. Allow for [non-root users to manage Docker](https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user).\n\n3. Install `docker compose` by installing the [Compose plugin](https://docs.docker.com/compose/install/linux/#install-the-plugin-manually):\n\n   ```\n   DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}\n   mkdir -p $DOCKER_CONFIG/cli-plugins\n   curl -SL https://github.com/docker/compose/releases/download/v2.18.1/docker-compose-linux-x86_64 \\\n      -o $DOCKER_CONFIG/cli-plugins/docker-compose\n   chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose\n   ```\n\n   You can test the installation with `docker compose version`\n\n4. Clone the repository containing the miniDREAM course modules onto the\n   instance. For example:\n\n   ```shell\n   git clone --branch minidream2023 https://github.com/mc2-center/minidream-challenge.git\n   ```\n\n   When building the server, it will expect the directory of course\n   materials to be named `minidream-challenge`. If you are not using\n   [minidream-challenge](https://github.com/mc2-center/minidream-challenge.git)\n   as the source of course materials, edit `docker-compose.yml` so that the\n   correct volume is mounted. See [Shared Files](#shared-files) for more details.\n\n5. Clone **this** repository onto the instance:\n\n   ```shell\n   git clone https://github.com/mc2-center/minidream-r-env.git\n   ```\n\n6. Switch to the `minidream-r-env` directory and start the server:\n\n   ```shell\n   cd minidream-r-env/\n   docker compose up --build -d\n   ```\n\n   It will take around 15-20 minutes to build the server for the first time.\n   Once complete, you should see the following:\n\n   ```shell\n   ...\n   [+] Running 3/3\n   ✔ Network minidream-r-env_default  Created                            0.0s\n   ✔ Container rstudio                Started                            1.6s\n   ✔ Container proxy                  Started                            0.6s\n   ```\n\n**Congrats!** 🎉 The interactive RStudio environment is now available at\nhttp://minidream.synapse.org/.\n\nFrom this point forward, you can:\n\n- start the server with:\n\n  ```shell\n  docker compose up -d\n  ```\n\n- stop the server with:\n\n  ```shell\n  docker compose down\n  ```\n\n   \u003e **Note**: stopping the server will reset the RStudio contents, e.g.\n   \u003e modules will be removed, etc.\n\n### Packages/Libraries\n\nThe RStudio server will come pre-installed with the following packages/libraries:\n\n* BiocManager\n* bitops\n* caTools\n* clusterProfiler\n* DOSE\n* getPass\n* here\n* imager\n* org.Hs.eg.db\n* pathview\n* revealjs\n* survival\n* tidyverse\n* tsne\n\nTo install a new package to the server, do:\n\n```\ndocker compose exec -it rstudio R -e \"install.packages('\u003cpackage name\u003e')\"\n```\n\n### Shared Files\n\nBy default, the server is configured to bind-mount `minidream-challenge` as\n`/shared` in the `rstudio` container:\n\n**docker-compose.yml**\n\n```\n...\nvolumes:\n   - ../minidream-challenge:/shared\n```\n\nIf you list the files of `/shared` with `docker compose exec -it rstudio ls -l /shared`,\nthe files listed will match the `minidream-challenge` directory.\n\nWhen a volume is mounted, it will preserve the same permissions it has on the\nhost. For example, let's say `minidream-challenge` has read-write-execute\nenabled for everyone (user, group, other) on the host; when it gets mounted,\n`/shared` will also have read-write-execute enabled for everyone.\n\nTo ensure that only the admins are allowed editing rights to the course\nmaterials on RStudio, change the group ownership as well as their permissions:\n\n```\ndocker compose exec rstudio chgrp -R rstudio-admin /shared \\\n   \u0026\u0026 docker compose exec rstudio chmod g+rw /shared\n```\n\nCheck the group ownership of `/shared` (just in case) with `docker compose exec -it rstudio ls -l /shared`\n\n---\n\n## 👤 Users\n\nWhen the server is first built, two default users are added - one instructor\n(admin), one student:\n\n```csv\nadmin,changeme,rstudio-user;rstudio-admin\nstudent,changeme,rstudio-user\n```\n\nwhere:\n\n- the first field is the username, e.g. `admin`\n- the second field is the password, e.g. `changeme`\n- the third field is the user's group(s), e.g. `rstudio-user;rstudio-admin`\n\n### Configure the User List\n\n1. Assuming you are still in the `minidream-r-env` directory, remove\n   `rstudio/utils/users.csv` (which is currently a symbolic link of\n   `users.csv.template`):\n\n   ```\n   rm rstudio/utils/users.csv\n   ```\n\n2. Create a new `users.csv` within the `rstudio/utils/` directory. Each line\n   should contain 3 fields, delimited by a comma (no spaces!):\n\n   - username\n   - password\n   - group(s)\n\n   If the user is a student, their group should be `rstudio-user`. If the user\n   is an admin, their groups should be `rstudio-user;rstudio-admin`. See the\n   default users list above for an example.\n\n3. Stop the server and rebuild:\n\n   ```shell\n   docker compose down\n   docker compose up --build -d\n   ```\n\n   This time, the server should be ready in less than a minute (assuming no\n   changes have been made to the server Dockerfile). Once ready, go to\n   http://minidream.synapse.org/ and try:\n\n   - Logging in as a student\n   - Logging in as an admin\n   - Changing the password of a sample user with `passwd` (in the terminal)\n\n### Adding New Users\n\nIf the course is already in progress, you can add new users with the `add_users`\ntool.\n\n1. Create a new CSV file, following the same format as above.\n\n2. Copy the CSV into the `rstudio` container:\n\n   ```\n   docker cp \u003cnew user list\u003e rstudio:\u003cnew user list\u003e\n   ```\n\n   You can check that the file has been copied over with:\n\n   ```\n   docker compose exec -it rstudio ls\n   ```\n\n3. Add the new users to the server:\n\n   ```\n   docker compose exec rstudio /root/utils/add_users.sh \u003cnew user list\u003e\n   ```\n\n   You should now be able to log in as one of the newly added users.\n\n---\n\n## 📚 Modules\n\nWhen the server is first built, none of the modules from the `minidream-challenge`\ndirectory will be available on RStudio - each module will need to be shared\nwith the users. We recommend sharing one module at a time, following the same\npace as the miniDREAM course.\n\n### Broadcasting a Module\n\nLet's go through an exercise of broadcasting a module to RStudio, starting\nwith Module 0.\n\nFor a quick overview, list the available modules that can be broadcasted. For\nexample:\n\n```\n$ docker compose exec -it rstudio ls -l /shared/modules\ntotal 32\ndrwxrwxr-x 4 admin rstudio-admin 4096 May 25 23:09 module0\ndrwxrwxr-x 3 admin rstudio-admin 4096 May 23 17:07 module1\ndrwxrwxr-x 3 admin rstudio-admin 4096 May 23 17:07 module2\ndrwxrwxr-x 3 admin rstudio-admin 4096 May 23 17:07 module3\ndrwxrwxr-x 3 admin rstudio-admin 4096 May 23 17:07 module4\ndrwxrwxr-x 3 admin rstudio-admin 4096 May 23 17:07 module5\ndrwxrwxr-x 3 admin rstudio-admin 4096 May 23 17:07 module6\ndrwxrwxr-x 3 admin rstudio-admin 4096 May 23 17:07 welcome\n```\n\n1. Assuming you are in the `minidream-r-env` directory, run the `broadcast_module`\n   tool:\n\n   ```\n   docker compose exec rstudio root/utils/broadcast_module.sh \\\n      shared/modules/module0 \\\n      rstudio-user\n   ```\n\n   This will share the contents of `shared/modules/module0` to all users\n   belonging to the `rstudio-user` group (which should be all of the users\n   in RStudio). A `modules` directory should now be available in the Files\n   pane in RStudio.\n\n2. (One-time only) Some course materials are dependent on scripts and data\n   located in `/home/shared/*`, which does not exist when the server is\n   first built. Create a symbolic link in the `rstudio` container so that\n   this filepath exists:\n\n   ```\n   docker compose exec -it rstudio ln -s /shared /home/shared\n   ```\n\n**Congrats!** 🎉 You just launched your first module!\n\n### Updating a Module\n\n1. From the home directory, pull the changes:\n\n   ```\n   git pull minidream-challenge\n   ```\n\n   Alternatively, changes can be directly applied inside the directory on the\n   instance:\n\n   ```\n   vim minidream-challenge/modules/\u003cmodule name\u003e/\u003cnotebook\u003e\n   ```\n\n2. Copy the new module contents to the container. For example:\n\n   ```\n   docker cp \\\n      minidream-challenge/modules/module0/intro-to-RStudio.Rmd \\\n      fc9ac0f0f15f:/shared/modules/module0/.\n   ```\n\n3. Switch to the `minidream-r-env` directory and re-broadcast the module:\n\n   ```\n   docker compose exec rstudio root/utils/broadcast_module.sh \\\n      shared/modules/module0 \\\n      rstudio-user\n   ```\n\n---\n\n## 🔄 Scoring Harness\n\n\u003e **Note**: this section will assume you are using `minidream-challenge` as\n\u003e the source for course materials.\n\n### Setup\n\n1. In either the same instance or a new one, install the latest version of Miniconda:\n\n   ```\n   mkdir -p ~/miniconda3\n   wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh\n   bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 \\\n      \u0026\u0026 rm -rf ~/miniconda3/miniconda.sh\n   miniconda3/bin/conda init bash\n   ```\n\n2. Exit the instance (to apply the changes), then log back in.\n\n3. Assuming you are in the home directory, create a new environment from file:\n\n   ```\n   conda env create -f minidream-challenge/environment.yml\n   ```\n\n   This will create a virtual environment called `minidream` that will contain\n   all of the necessary Python and R libraries to run the scoring harness.\n\n4. Activate the virtual environment.\n\n   ```\n   conda activate minidream\n   ```\n\n5. In the home directory, create a new file called `.synapseConfig`, which\n   will contain a Synapse Personal Access Token (PAT) needed to run the\n   submission system:\n\n   ```\n   [authentication]\n   authtoken=YOUR-PERSONAL-ACCESS-TOKEN-HERE\n   ```\n\n   Credentials used should have editing rights to the miniDREAM challenge\n   site. [Go here to generate a new PAT](https://www.synapse.org/#!PersonalAccessTokens:).\n\n### Configure the Workflow\n\nWithin `minidream-challenge`, edit the following scripts:\n\n* `scoring_harness/challenge_eval.sh`\n\n   - Update the absolute path to the conda virtual environment on your machine\n\n* `scoring_harness/challenge_config.py`\n\n   - Update CHALLENGE_SYN_ID to the synID of the project for the challenge site.\n   - Update CHALLENGE_NAME\n   - Update the list of ADMIN_USER_IDS to contain the user IDs of \n   - Update the evaluation IDs in `evaluation queues`:\n\n      ```\n      evaluation_queues = [\n         {\n            'id': \u003cneed to be updated\u003e,\n            'scoring_func': score,\n         }\n      ]\n      ```\n\n* `R/submission_helpers`\n   - Update the folder synIDs where module submissions will be uploaded:\n\n      ```\n      submission_folder \u003c- switch(\n        module,\n        \"0\" = \"syn000\",\n        \"1\" = \"syn111\",\n        \"2\" = \"syn222\",\n        \"3\" = \"syn333\",\n        \"4\" = \"syn444\",\n        \"5\" = \"syn555\",\n        \"6\" = \"syn666\",\n        \"7\" = \"syn777\"\n      )\n      ```\n\n   - update the evaluation ID for the `synSubmit` function:\n\n      ```\n      submission \u003c- synSubmit(evaluation = \"\u003cneed to be updated\u003e\",\n                           entity = activity_submission)\n      ```\n\n### Conduct Dry-runs\n\n1. If it's not already active, activate the `minidream` virtual environment. \n\n2. Assuming you are in the `minidream-challenge` directory, test whether you\n   can run `challenge.py` without any errors:\n\n   ```\n   python scoring_harness/challenge.py -h\n   ```\n\n3. We also recommend testing out one of the commands as well, ideally, the\n   same one listed in `challenge_eval.sh`, e.g.\n\n   ```\n   python scoring_harness/challenge.py score --all\n   ```\n\n   If it's working properly (and assuming there are no pending submission to\n   be evaluated yet), you should get some logs in STDOUT like this:\n\n   ```\n   ===========================================================================\n   2023-06-03T07:12:24.821643\n\n\n   Scoring: 9615336 - 2023 miniDREAM Module Submissions\n   ------------------------------------------------------------\n\n\n   done:  2023-06-03T07:12:25.028427\n   =========================================================================== \n   ```\n\n### Automate the Process\n\nOnce you are sure the scoring harness is working as expected, set up a \n[Cron job](https://help.ubuntu.com/community/CronHowto#Starting_to_Use_Cron)\nso that it will run every minute of every day.\n\n1. Open up the crontab:\n\n   ```\n   crontab -e\n   ```\n\n2. Add the following task to the file:\n\n   ```\n   * * * * * /home/\u003cyour username\u003e/minidream-challenge/scoring_harness/challenge_eval.sh\n   ```\n\n3. Save and exit the file.  Assuming it's working properly, logs will be saved\n   into `scoring_harness/log/score.log`\n\n\u003e **Note**: if errors are noted in the log file, we recommend stopping the cron\n\u003e job so that you can address the errors in the interim. You can temporarily stop\n\u003e the job by commenting out the task with #.\n\n\n**Congrats!** 🎉 You just finished setting up the challenge infrastructure for miniDREAM!\n\n---\n\n## Helpful Resources\n\n- `rstudio-server --help`\n\n  For example: `rstudio-server suspend-all` will remove the message: \"ERROR\n  session hadabend\" from an R session console after each service 'reboot'.\n\n- https://hub.docker.com/u/rocker/\n- [Bruno's RStudio environment walkthrough](https://www.synapse.org/#!Synapse:syn29616137/wiki/617456)\n- [Milen's repo and documentation](https://github.com/milen-sage/minidream-r-env)\n- [Bruno's repo and documentation](https://github.com/Sage-Bionetworks/minidream-r-env/tree/bgrande/minidream-2021)\n\n## Miscellaneous\n\n1. Check out leader board\n   Check out the course page wiki. Check \"scoreboards\" section, and click on the relevant module. Then, click on \"Wiki tools\" widget -\u003e \"Edit Project Wiki\". You should be able to uncomment the line related to leaderboard.\n\n2. Clear submission\n   After testing out your own submission, you might want to clear out previous submission. To do that, please use the `challengeutils` package [here](https://github.com/Sage-Bionetworks/challengeutils)\n\nAfter installing `challengeutils` package, you could use `challengeutils delete-submission \u003csubmission id\u003e` for deleting a certain submission.\n\n3. Update submission helpers\n   Check out file `submission_helpers.R` under folder R. This file by default could only edited by root user. You could edit it by following the steps:\n\n- Interact with the docker container as a root user `docker exec -it rstudio bin/bash`\n- Go to `/shared/R` folder\n- Update `submission_helpders.R` there by using `nano`\n  _Note_: To save a file in nano, use `^O` (`ctrlO` on Mac)\n\n4. Grant sudo access\n   This could be done by using `sudo usermod -a -G sudo username`\n\n5. Add users to docker group (Run docker commands without sudo)\n   To be able to run docker without using `sudo`, we will have to add users to the docker group.\n\n6. Check existing users in docker group\n\n```\ngetent group docker\n```\n\n2. Add a new user to docker group\n\n```\nsudo usermod -a -G docker username\n```\n\n_Note_: make sure that you are using jumpcloud username\nYou should be able to see something like:\n\n```\ndocker:x:999:ubuntu,username\n```\n\n3. To activate the changes to the docker group without restarting docker daemon:\n\n```\nnewgrp docker\n```\n\n_Note_: If the user is still getting error after the above steps, we might want to change group ownership of the `/var/run/docker.sock file` by using: `sudo chown root:docker /var/run/docker.sock`.\nReference could be found [here](https://linuxhandbook.com/docker-permission-denied/) and [here](https://askubuntu.com/questions/477551/how-can-i-use-docker-without-sudo#:~:text=I%20still%20had%20to%20give%20the%20/var/run/docker.sock%20socket%20and%20/var/run/docker%20directory%20the%20proper%20permissions%20to%20make%20it%20work%3A)\n\n6. Update forum link in `minidream-challenge/scoring_harness/messages.py`\n   After students submit their work, they would get a message from the system. You could update `support_forum_url` and `challenge_instructions_url`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmc2-center%2Fminidream-r-env","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmc2-center%2Fminidream-r-env","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmc2-center%2Fminidream-r-env/lists"}