https://github.com/anon-exploiter/ine-dl
Python script to download INE courses including labs, exercises, quizzes, slides, and, videos!
https://github.com/anon-exploiter/ine-dl
courses ine ine-dl security
Last synced: 7 days ago
JSON representation
Python script to download INE courses including labs, exercises, quizzes, slides, and, videos!
- Host: GitHub
- URL: https://github.com/anon-exploiter/ine-dl
- Owner: Anon-Exploiter
- License: mit
- Created: 2022-07-23T09:44:36.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2025-01-04T22:48:14.000Z (9 months ago)
- Last Synced: 2025-04-03T22:36:34.144Z (6 months ago)
- Topics: courses, ine, ine-dl, security
- Language: Python
- Homepage:
- Size: 110 KB
- Stars: 28
- Watchers: 1
- Forks: 13
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# INE-DL (INE course downloader)
[](https://GitHub.com/Anon-Exploiter/ine-dl.js/graphs/commit-activity)
[](https://www.python.org/downloads/)
[](https://twitter.com/syed__umar)
[![LinkedIn][linkedin-shield]][linkedin-url]
[contributors-shield]: https://img.shields.io/github/contributors/Anon-Exploiter/ine-dl.svg?style=flat-square
[contributors-url]: https://github.com/Anon-Exploiter/ine-dl/graphs/contributors
[issues-shield]: https://img.shields.io/github/issues/Anon-Exploiter/ine-dl?style=flat-square
[issues-url]: https://github.com/Anon-Exploiter/ine-dl/issues
[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=flat-square&logo=linkedin&colorB=555
[linkedin-url]: https://www.linkedin.com/in/syedumararfeen/
**Python script to download INE courses including labs, exercises, quizzes, slides, and, videos!**
### Requirements
- Python (3.6.* - 3.8.*)
- Python `pip3`
- Python module `requests_toolbelt`
- Python module `requests`
- Python module `loguru`
- External downloader `aria2`
### How does this script work? (if you want to understand the code)
The script was written based on the APIs of **iOS application** to prevent Google's Invisible captcha implementation hence you will see a header (`X-Ine-Mobile`) hard coded in the script with an static API key required for the IOS API calls to succeed (this is hard-coded in the iOS application binary and can easily be grepped).
**Initialization:**
1. The script starts with loading the credentials (`username/email` and `password`) from `config.json` file
2. It then proceeds to login into the INE account and loading the `JWT` token into Authorization header to use in next API calls
3. The script then does an API call to download the metadata of all the INE courses present and write into `all_courses_metadata.json`
4. It then checks for the subscriptions the user account has and filters the courses fetched and creates another file named `all_courses_with_access.json` with the data
**Downloading of videos:**
1. Does an API call to fetch the `video name`, `url`, and checks if subtitle URL is present and downloads all those!
2. Also *filters based on the resolution* and goes from 1080, 720 -> low
**Downloading of slides** (format looks like the [following](https://user-images.githubusercontent.com/18597330/179960035-7a00d727-ebc0-4744-be50-356f53ad03af.png)):
1. An initial request returns the slide metadata containing a link to `index.html` file (with cookies in response headers)
2. We store the cookies and then download `browsersupport.js` and `player.js` files (static can be reused with other slides)
3. INE uses a mix of `css`, `js`, `woff`, and, `pngs` but all have incrementable file names (format differs for text and binary files)
4. Wrote a `while loop` until the incremented file returns 404 and then stops the downloading of category: `CSS/JS/WOFF/PNG`
5. It also downloads any attachments added with the slides.
**Downloading of quizzes** (with right answers):
1. This was quite a pickle, looked into the quiz solving API call, first request returns a `JSON` containing the whole quiz content
2. Second request is a `PUT` request with right/wrong answers from user, in response to this, the JSON body now contains a new key `is_correct` containing the right answer
3. Wrote logic for posting the JSON body taken from the initial request, modified it to required standards, the server doesn't need options to be selected either
4. The `PUT` request then returns the right answers, *two files are made with no answers and correct answers*.
**Downloading of exercises**:
1. This was quite simple, an API call returns the contents in both `Markdown` and `HTML` format
2. For ease of cross-platform usage, I've only stored the `HTML` content with `_exercise_` in file name.
**Downloading of labs**:
1. You lose an edge here, in most of the cases, the labs are stored on INE cloud.
2. The script only stores the `HTML content` of it (if present)
### Features
- Resume capability for a course video
- Download subtitles for the videos (if present)
- Download all courses without any prompt (option: -a / --all)
- Downloading slides, labs, exercises, quizzes, and, videos
### Install modules
pip3 install -r requirements.txt
### Install aria2
sudo apt install aria2
### Running the container in Docker
```bash
docker build -t ine-dl .
docker run -d --name ine-dl -v {HOSTDIRECTORY}:/app/ine-dl/downloads ine-dl
```
Make sure you replace {HOSTDIRECTORY} in the command to map to an on device save folder
### Works on
- ~Windows 7/8/8.1~ (It should work fine in WSL but not in cmd/PS)
- Ubuntu 18.04 LTS
- Ubuntu 20.04 LTS
### Download ine-dl
You can download the latest version of ine-dl by cloning the GitHub repository.
git clone https://github.com/Anon-Exploiter/ine-dl --depth 1
### Help menu
```bash
┌──(umar_0x01@b0x)-[~/scripts/ine-dl]
└─$ python3 ine.py
██╗███╗ ██╗███████╗ ██████╗ ██╗
██║████╗ ██║██╔════╝ ██╔══██╗██║
██║██╔██╗ ██║█████╗█████╗██║ ██║██║
██║██║╚██╗██║██╔══╝╚════╝██║ ██║██║
██║██║ ╚████║███████╗ ██████╔╝███████╗
╚═╝╚═╝ ╚═══╝╚══════╝ ╚═════╝ ╚══════╝
Usage: python3 ine.py --all
Help:
-h, --help show this help message and exit
Basic arguments (one or more):
-l LOG, --log-output LOG
Logs output of the script (if required later)
-lct, --list-categories
List all categories
-lcc, --list-courses List all courses
-lcct LCCT, --list-categories-courses LCCT
List all courses of a specific category UUID from -lct
Necessary arguments:
-p PROCESSES, --processes PROCESSES
Number of parallel processes to launch (2 if nothing specified)
-c COURSE, --course COURSE
Download course based on provided UUID from -lcc
-ct CATEGORY, --category CATEGORY
Download whole category based on provided UUID from -lct
-a, --all Download all courses of all categories
```
### Arguments usage
***Running the Script (displays help menu with no args)***
python ine-dl.py
***Listing all the courses***
python ine-dl.py -lc
***Listing course categories***
python ine-dl.py -lct
***Listing all the courses of a specific category***
python ine-dl.py -lcct {category_id}
***Logging the script's output into a log file***
python ine-dl.py -l logfile.log
***Downloading all the INE course (your subscription has access to, with/without parallel processing)***
python ine-dl.py --all
python ine-dl.py --all -p 2
***Downloading a single course***
python ine-dl.py -c {course_id}
***Downloading all courses of specified category (with/without parallel processes)***
python ine-dl.py -ct {category_id}
python ine-dl.py -ct {category_id} -p 2
### To DOs
- [x] Fetch all the courses and write into a file
- [x] Fetch & Match the subscriptions and then put stuff into the course file
- [x] Implement downloading of video files (highest resolution and so-on)
- [x] Implement quiz downloading
- [x] Write the quizzes into an text file and then write it's correct results (json:is_correct) into another file!
- [x] Implement exercise downloading
- [x] Implement iframe downloading
- [x] Implement html, css, js, woff, img files downloading
- [x] Implement lab downloading
- [x] Check if description_html exists and if not, write the json object of the whole lab for user satisfaction
- [x] Downloading the files_uuids zip/pdf files
- [x] Write a json into the course directory containing whole course data
- [x] Implement all argparse arguments
### What it can't do
- At the moment, learning paths and bootcamps downloading is not supported
- You can, however, download the courses within those paths and bootcamps manually by specifying their UUID
- This can also be automated using a bash script to download a lot of UUIDs one by one or using parallel processing
- https://github.com/Anon-Exploiter/ine-dl/issues/8
### Screenshots
### Credits
- Inspired as always by downloaders of [r0oth3x49](https://github.com/r0oth3x49)
- Some ideas were taken from [Jayapraveen's downloader](https://github.com/Jayapraveen/INE-courses-downloader). Though his script is nice, it has a lot of bugs, I spent almost 2 holidays fixing those and then thought of writing my own.
### Note
Please use the script w.r.t the usage guidelines of INE. Do not exhaust their backend servers. Do not dump and share the courses publicly.
Please use this on your own risk, If your account is blocked by the usage of this script, I won't be responsible.