Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://hemberg-lab.github.io/scRNA.seq.datasets/

Collection of public scRNA-Seq datasets used by our group
https://hemberg-lab.github.io/scRNA.seq.datasets/

aws dataset docker jenkins mkdocs openstack s3-storage single-cell

Last synced: 23 days ago
JSON representation

Collection of public scRNA-Seq datasets used by our group

Host: GitHub
URL: https://hemberg-lab.github.io/scRNA.seq.datasets/
Owner: hemberg-lab
License: gpl-3.0
Created: 2017-01-27T14:48:35.000Z (over 7 years ago)
Default Branch: master
Last Pushed: 2021-05-04T16:02:10.000Z (about 3 years ago)
Last Synced: 2024-05-22T18:14:10.701Z (about 1 month ago)
Topics: aws, dataset, docker, jenkins, mkdocs, openstack, s3-storage, single-cell
Language: R
Homepage: https://hemberg-lab.github.io/scRNA.seq.datasets/
Size: 761 KB
Stars: 161
Watchers: 15
Forks: 59
Open Issues: 13
Metadata Files:
- Readme: README.md
- License: LICENSE

Lists

awesome_single_cell - scRNA.seq.datasets - Collection of public scRNA-Seq datasets used by [Hemberg Lab](http://www.sanger.ac.uk/science/groups/hemberg-group) (Web portals and apps / Web portals)

README

# Public scRNA-Seq Datasets

This repository is used to build [scater](http://bioconductor.org/packages/scater/) objects and reports (in continuous integration manner) for various publicly available scRNA-Seq datasets used by our group. This pipeline is implemented using docker containers and cloud computing. The resulting website is available [here](https://hemberg-lab.github.io/scRNA.seq.datasets/). Below are some notes on the pipeline setup.

## Website

The website is generated using [MkDocs](http://www.mkdocs.org/) generator. Links to S3 storage and data annotations are added manually. If you are creating a pull request and adding new data please add its annotations to one of the files in the `website` folder.

## S3 storage

To list files on the S3 storage please use this [link](https://scrnaseq-public-datasets.s3.amazonaws.com/index.html).

## Instance setup

### AWS

1. Launch Amazon Linux EC2 instance.
2. Using security groups add access to the instance on port 8080.
3. Connect to instance and [install Jenkins](http://sanketdangi.com/post/62715793234/install-configure-jenkins-on-amazon-linux).
4. Add permission for Jenkins to run Docker:
```
sudo usermod -aG docker jenkins
```

Hard reboot your instance after that. Now Jenkins can run docker images.

5. Install `s3cmd` utility to be able to upload data to the S3 storage:
```
sudo apt-get install s3cmd
```

6. In Jenkins Export S3 key ID, secret key and region as environmental variables. Use secret text option provided by Jenkins. Some details available [here](http://serverfault.com/questions/724730/unable-to-use-aws-cli-in-jenkins-due-to-unable-to-locate-credentials-error).

7. File listing can be setup on AWS S3 bucket using this [plugin](https://github.com/rufuspollock/s3-bucket-listing).

### OpenStack Cloud (Sanger)

1. Launch Ubuntu Trusty instance (`j1.large` flavour)
2. Add the instance to the `default`, `cloudforms_icmp_in`, `cloudforms_ssh_in` `cloudforms_web_in` security groups.
3. Create additional security group: `TCP` with port 8080 (this is needed for Jenkins) and add your instance to this group.
4. Associate a floating IP (FLOATING_IP) number with your instance.
5. Login to instance:
```
ssh -i ~/.ssh/your_key.pem ubuntu@FLOATING_IP
```
6. In the instance [install Jenkins](https://jenkins.io/doc/book/installing/#debian-ubuntu).

To setup Jenkins after installation go to http://FLOATING_IP:8080

7. In the instance [install docker](https://docs.docker.com/engine/installation/linux/ubuntu/).

8. Resolve docker network issues (Sanger OpenStack problem only):
```{bash}
sudo bash -c "echo '{ \"bip\": \"10.10.0.1/16\", \"mtu\": 1400 }' > /etc/docker/daemon.json"
```

9. Add permission for Jenkins to run Docker:
```
sudo usermod -aG docker jenkins
```

Hard reboot your instance after that. Now Jenkins can run docker images.

10. Install `s3cmd` utility to be able to upload data to the S3 storage:
```
sudo apt-get install s3cmd
```

11. In Jenkins Export S3 key ID, secret key and region as environmental variables. Use secret text option provided by Jenkins. Some details available [here](http://serverfault.com/questions/724730/unable-to-use-aws-cli-in-jenkins-due-to-unable-to-locate-credentials-error).

## Jenkins build

```
# build and deploy
sh deploy.sh $WORKSPACE
```

## AWS Calculator

To calculate how much you can spend on AWS one can us the [AWS Calculator](https://calculator.s3.amazonaws.com/index.html).