{"id":13501776,"url":"https://github.com/nikhilkumarsingh/content-downloader","last_synced_at":"2026-01-16T11:08:59.746Z","repository":{"id":46006518,"uuid":"91945490","full_name":"nikhilkumarsingh/content-downloader","owner":"nikhilkumarsingh","description":"Python package to download files on any topic in bulk.","archived":false,"fork":false,"pushed_at":"2019-10-03T05:35:27.000Z","size":1188,"stargazers_count":107,"open_issues_count":3,"forks_count":63,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-09-17T14:56:04.981Z","etag":null,"topics":["bulk-downloader","command-line-tool","content-downloader","ctdl","file-download","google","multithreading","parallel-download","python","requests","tqdm"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nikhilkumarsingh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-05-21T09:16:03.000Z","updated_at":"2025-08-27T12:12:49.000Z","dependencies_parsed_at":"2022-09-10T04:26:27.239Z","dependency_job_id":null,"html_url":"https://github.com/nikhilkumarsingh/content-downloader","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nikhilkumarsingh/content-downloader","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nikhilkumarsingh%2Fcontent-downloader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nikhilkumarsingh%2Fcontent-downloader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nikhilkumarsingh%2Fcontent-downloader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nikhilkumarsingh%2Fcontent-downloader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nikhilkumarsingh","download_url":"https://codeload.github.com/nikhilkumarsingh/content-downloader/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nikhilkumarsingh%2Fcontent-downloader/sbom","scorecard":{"id":687203,"data":{"date":"2025-08-11","repo":{"name":"github.com/nikhilkumarsingh/content-downloader","commit":"8b14af3a6eadcc43581e0425dc1d218208de12df"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":1.7,"checks":[{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE.txt:0","Info: FSF or OSI recognized license: MIT License: LICENSE.txt:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":0,"reason":"12 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-55x5-fj6c-h6m8","Warn: Project is vulnerable to: PYSEC-2014-9 / GHSA-57qw-cc2g-pv5p","Warn: Project is vulnerable to: PYSEC-2021-19 / GHSA-jq4v-f5q6-mjqq","Warn: Project is vulnerable to: GHSA-pgww-xf46-h92r","Warn: Project is vulnerable to: PYSEC-2022-230 / GHSA-wrxv-2j5q-m38w","Warn: Project is vulnerable to: PYSEC-2018-12 / GHSA-xp26-p53h-6h2p","Warn: Project is vulnerable to: GHSA-9hjg-9r4m-mvj7","Warn: Project is vulnerable to: GHSA-9wx4-h78v-vm56","Warn: Project is vulnerable to: PYSEC-2023-74 / GHSA-j8r2-6x86-q33q","Warn: Project is vulnerable to: PYSEC-2015-17 / GHSA-pg2w-x9wp-vw92","Warn: Project is vulnerable to: PYSEC-2018-28 / GHSA-x84v-xcm2-53pg","Warn: Project is vulnerable to: PYSEC-2017-74"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-22T01:14:18.501Z","repository_id":46006518,"created_at":"2025-08-22T01:14:18.501Z","updated_at":"2025-08-22T01:14:18.501Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28478115,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T06:30:42.265Z","status":"ssl_error","status_checked_at":"2026-01-16T06:30:16.248Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bulk-downloader","command-line-tool","content-downloader","ctdl","file-download","google","multithreading","parallel-download","python","requests","tqdm"],"created_at":"2024-07-31T22:01:49.976Z","updated_at":"2026-01-16T11:08:59.709Z","avatar_url":"https://github.com/nikhilkumarsingh.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"[![PyPI](https://img.shields.io/badge/PyPi-v1.5-f39f37.svg)](https://pypi.python.org/pypi/ctdl)\n[![license](https://img.shields.io/github/license/mashape/apistatus.svg?maxAge=2592000)](https://github.com/nikhilkumarsingh/content-downloader/blob/master/LICENSE.txt)\n\n# content-downloader\n\n**content-downloader** a.k.a **ctdl** is a python package with **command line utility** and **desktop GUI** to download files on any topic in bulk!\n\n![](https://media.giphy.com/media/3oKIPlt7APHqWuVl3q/giphy.gif)\n\n![](https://media.giphy.com/media/xUPGcIvGpH3KvEmlnG/giphy.gif)\n\n## Features\n\n- ctdl can be used as a command line utility as well as a desktop GUI.\n\n- ctdl fetches file links related to a search query from **Google Search**.\n\n- Files can be downloaded parallely using multithreading.\n\n- ctdl is Python 2 as well as Python 3 compatible.\n\n## Installation\n\n- To install content-downloader, simply,\n\n  ```\n  $ pip install ctdl\n  ```\n\n- There seem to be some issues with parallel progress bars in tqdm which have\n  been resolved in this [pull](https://github.com/tqdm/tqdm/pull/385). Until this pull is merged, please use my patch by running this command:\n\n  ```\n  $ pip install -U git+https://github.com/nikhilkumarsingh/tqdm\n  ```\n\n## Desktop GUI usage\n\nTo use **ctdl** desktop GUI, open terminal and run this command:\n\n```\n$ ctdl-gui\n```\n\n## Command line usage\n\n```\n$ ctdl [-h] [-f FILE_TYPE] [-l LIMIT] [-d DIRECTORY] [-p] [-a] [-t]\n       [-minfs MIN_FILE_SIZE] [-maxfs MAX_FILE_SIZE] [-nr]\n       [query]\n```\nOptional arguments are:\n\n- -f FILE_TYPE : set the file type. (can take values like ppt, pdf, xml, etc.)\n\n                 Default value: pdf\n\n- -l LIMIT : specify the number of files to download.\n\n             Default value: 10\n\n- -d DIRECTORY : specify the directory where files will be stored.\n\n                 Default: A directory with same name as the search query in the current directory.\n\n- -p : for parallel downloading.\n\n- -minfs MIN_FILE_SIZE : specify minimum file size to download in Kilobytes (KB).\n\n                 Default: 0\n\n- -maxfs MAX_FILE_SIZE : specify maximum file size to download in Kilobytes (KB).\n\n                 Default: -1 (represents no maximum file size)\n\n- -nr : prevent download redirects.\n\n                 Default: False\n\n## Examples\n\n- To get list of available filetypes:\n\n  ```\n  $ ctdl -a\n  ```\n\n- To get list of potential high threat filetypes:\n\n  ```\n  $ ctdl -t\n  ```\n\n- To download pdf files on topic 'python':\n\n  ```\n  $ ctdl python\n  ```\n  This is the default behaviour which will download 10 pdf files in a folder named 'python' in current directory.\n\n- To download 3 ppt files on 'health':\n\n  ```\n  $ ctdl -f ppt -l 3 health\n  ```\n\n- To explicitly specify download folder:\n\n  ```\n  $ ctdl -d /home/nikhil/Desktop/ml-pdfs machine-learning\n  ```\n\n- To download files parallely:\n  ```\n  $ ctdl -f pdf -p python\n  ```\n\n- To search for and download in parallel 10 files in PDF format containing\n  the text \"python\" and \"algorithm\", without allowing any url redirects,\n  and where the file size is between 10,000 KB (10 MB) and 100,000KB (100 MB),\n  where KB means Kilobytes, which has an equivalent value expressed in Megabytes:\n  ```\n  $ ctdl -f pdf -l 10 -minfs 10000 -maxfs 100000 -nr -p \"python algorithm\"\n  ```\n\n## Usage in Python files\n\n```python\nfrom ctdl import ctdl\n\nctdl.download_content(\nfile_type = 'ppt',\nlimit = 5,\ndirectory = '/home/nikhil/Desktop/ml-pdfs',\nquery = 'machine learning using python')\n```\n\n## TODO\n\n- [X] Prompt user before downloading potentially threatful files\n\n- [X] Create ctdl GUI\n\n- [ ] Implement unit testing\n\n- [ ] Use DuckDuckgo API as an option\n\n## Want to contribute?\n\n- Clone the repository\n\n  ```\n  $ git clone http://github.com/nikhilkumarsingh/content-downloader\n  ```\n\n- Install dependencies\n  ```\n  $ pip install -r requirements.txt\n  ```\n\n  **Note:** There seem to be some issues with current version of tqdm. If you do not get\n  expected progress bar behaviour, try this patch:\n\n  ```\n  $ pip uninstall tqdm\n  $ pip install git+https://github.com/nikhilkumarsingh/tqdm\n  ```\n\n- In ctdl/ctdl.py, remove the `.` prefix from `.downloader` and `.utils` for\n  the following imports, so it changes from:\n  ```python\n  from .downloader import download_series, download_parallel\n  from .utils import FILE_EXTENSIONS, THREAT_EXTENSIONS\n  ```\n  to:\n  ```python\n  from downloader import download_series, download_parallel\n  from utils import FILE_EXTENSIONS, THREAT_EXTENSIONS\n  ```\n\n- Run the python file directly `python ctdl/ctdl.py ___` (instead of with `ctdl ___`)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnikhilkumarsingh%2Fcontent-downloader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnikhilkumarsingh%2Fcontent-downloader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnikhilkumarsingh%2Fcontent-downloader/lists"}