https://github.com/remarkablemark/c9.io-scraper
c9.io scraper
https://github.com/remarkablemark/c9.io-scraper
c9io chrome chromedriver cloud9 nodejs selenium webdriverjs
Last synced: 2 months ago
JSON representation
c9.io scraper
- Host: GitHub
- URL: https://github.com/remarkablemark/c9.io-scraper
- Owner: remarkablemark
- License: mit
- Created: 2019-12-26T05:56:02.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2021-03-28T07:10:50.000Z (about 4 years ago)
- Last Synced: 2025-02-14T08:45:54.422Z (4 months ago)
- Topics: c9io, chrome, chromedriver, cloud9, nodejs, selenium, webdriverjs
- Language: JavaScript
- Homepage:
- Size: 19.5 KB
- Stars: 2
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# c9.io scraper
Web scraper for archived [c9.io](https://c9.io/login) projects. Built with [WebDriverJS](https://github.com/SeleniumHQ/selenium/wiki/WebDriverJs) ([Selenium](https://www.selenium.dev/) for [Node.js](https://nodejs.org/)). Bootstrapped from [webdriverjs-recipes](https://github.com/remarkablemark/webdriverjs-recipes).
## Prerequisites
[Node.js](https://nodejs.org/en/download/) and [npm](https://www.npmjs.com/get-npm):
```sh
brew install node
```
[Chrome](https://www.google.com/chrome/) and [ChromeDriver](https://chromedriver.chromium.org/downloads):
```sh
brew cask install google-chrome
brew install chromedriver
```
## Install
Clone repository:
```sh
git clone https://github.com/remarkablemark/c9.io-scraper.git
cd c9.io-scraper
```
If you're using [nvm](https://github.com/nvm-sh/nvm), you can set the node version:
```sh
nvm use
```
Install dependencies:
```sh
npm install
```
## Environment Variables
Fill out `.env` before running the scraper:
```sh
USERNAME=user # required
PASSWORD=pass # required
DOWNLOADS_DIR=~/Downloads/ # optional
```
If `DOWNLOADS_DIR` isn't set, it defaults to `~/Downloads/`.
## Scripts
In the project directory, you can run:
### `npm start`
The script logs into your [c9.io](https://c9.io/login) account, prepares all workspaces for download, and then downloads them.
Downloaded workspaces are recorded in `tmp/downloaded_workspaces.txt`.
## License
[MIT](LICENSE)