{"id":13451146,"url":"https://github.com/KoalaBear84/OpenDirectoryDownloader","last_synced_at":"2025-03-23T18:31:49.336Z","repository":{"id":37947042,"uuid":"164577069","full_name":"KoalaBear84/OpenDirectoryDownloader","owner":"KoalaBear84","description":"Indexes open directories","archived":false,"fork":false,"pushed_at":"2025-03-14T16:41:29.000Z","size":1994,"stargazers_count":1199,"open_issues_count":14,"forks_count":96,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-03-23T02:46:58.438Z","etag":null,"topics":["directory","http","https","indexer","open","parser"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KoalaBear84.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"KoalaBear84","patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2019-01-08T06:15:13.000Z","updated_at":"2025-03-22T02:25:07.000Z","dependencies_parsed_at":"2023-02-18T04:46:04.999Z","dependency_job_id":"6fc6a829-4188-451d-ac03-023fedae1de1","html_url":"https://github.com/KoalaBear84/OpenDirectoryDownloader","commit_stats":{"total_commits":867,"total_committers":15,"mean_commits":57.8,"dds":0.08881199538638984,"last_synced_commit":"4218a0b001815e1ce0179f57aad5cd9aba80a3f2"},"previous_names":[],"tags_count":209,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KoalaBear84%2FOpenDirectoryDownloader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KoalaBear84%2FOpenDirectoryDownloader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KoalaBear84%2FOpenDirectoryDownloader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KoalaBear84%2FOpenDirectoryDownloader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KoalaBear84","download_url":"https://codeload.github.com/KoalaBear84/OpenDirectoryDownloader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245149406,"owners_count":20568901,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["directory","http","https","indexer","open","parser"],"created_at":"2024-07-31T07:00:48.938Z","updated_at":"2025-03-23T18:31:49.317Z","avatar_url":"https://github.com/KoalaBear84.png","language":"C#","funding_links":["https://github.com/sponsors/KoalaBear84"],"categories":["C# #","HarmonyOS"],"sub_categories":["Windows Manager"],"readme":"# Open Directory Downloader\n\nIndexes open directories listings in 130+ supported formats, including FTP(S), Google Drive, Directory Lister, HFS, Bhadoo, GoIndex, Go2Index (alternatives), Mediafire, GoFile, GitHub.\n\n![](assets/Screenshot01.png)\n\nWritten in C# with .NET (Core), which means it is **cross platform**!\n\nDownloading is not (yet) implemented, but is already possible when you use the resulting file into another tool (for most of the formats).\n\nDownloading with [wget](https://www.gnu.org/software/wget/):\n`wget -x -i theurlsfile.txt`\n\nDownloading with [aria2c](https://aria2.github.io/) with `--aria2-urls` (theurlsfile-aria2.txt):\n`aria2c -i theurlsfile-aria2.txt`\n\nDownloading with [aria2c](https://aria2.github.io/) with normal theurlsfile.txt (Does not support directory structure):\n`aria2c -i theurlsfile.txt`\n\nFor aria2 you sometimes needs `--disable-ipv6` if downloading fails.\n\nIf you have improvements, supply me with a pull request! If you have a format not yet supported, please let me know.\n\n## Releases / Binaries\n\nFor builds (64-bit) for Windows, Linux and Mac, or ARM/ARM64 builds for Pi:\n\nhttps://github.com/KoalaBear84/OpenDirectoryDownloader/releases\n\nWhen using the self-contained releases you don't need to install the .NET (Core) Runtime.\n\n## Prerequisites\n\nWhen you are NOT using the self-contained releases, you need to install the latest/current Runtime version of .NET 9:\n\nhttps://dotnet.microsoft.com/download/dotnet/9.0/runtime\n\n## Usage\n\nCommand line parameters:\n\n| Short | Long | Description |\n| ----- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `-u` | `--url` | Url to scan |\n| `-t` | `--threads` | Number of threads (default 5) |\n| `-o` | `--timeout` | Number of seconds for timeout |\n| `-w` | `--wait` | Number of seconds to wait between calls (when single threaded is too fast..) |\n| `-q` | `--quit` | Quit after scanning (No \"Press a key\") |\n| `-c` | `--clipboard` | Automatically copy the Reddits stats once the scan is done |\n| `-j` | `--json` | Save JSON file |\n| `-f` | `--no-urls` | Do not save URLs file |\n| `-f` | `--aria2-urls` | Save aria2 urls files (with directory support) |\n| `-r` | `--no-reddit` | Do not show Reddit stats markdown |\n| `-l` | `--upload-urls` | Uploads urls file |\n| `-e` | `--exact-file-sizes` | Exact file sizes (WARNING: Uses HEAD requests which takes more time and is heavier for server) |\n| | `--fast-scan` | Only use sizes from HTML, no HEAD requests, even if the approx. size cannot be extracted from the HTML |\n| `-s` | `--speedtest` | Does a speed test after indexing |\n| `-a` | `--user-agent` | Use custom default User Agent |\n| | `--username` | Username |\n| | `--password` | Password |\n| | `--github-token` | GitHub Token |\n| `-H` | `--header` | Supply a custom header to use for each HTTP request. Can be used multiple times for multiple headers. See below for more info. |\n| | `--output-file` | Output file to use for urls file |\n| | `--proxy-address` | Proxy address, like \"socks5://127.0.0.1:9050\" (needed for .onion) |\n| | `--proxy-username` | Proxy username |\n| | `--proxy-password` | Proxy password |\n| | `--no-browser` | Disallow starting Chromium browser (for Cloudflare) |\n\n### Example\n\n#### Windows\n\n`OpenDirectoryDownloader.exe --url \"https://myopendirectory.com\"`\n\n#### Linux\n\n`./OpenDirectoryDownloader --url \"https://myopendirectory.com\"`\n\nIf you want to learn more or contribute, see the following paragraphs!\n\n### Custom Headers\n\nHeaders need to be provided in the following format: \n```\n\u003cHeader Name\u003e: \u003cHeader Value\u003e\n```\n **This syntax is compatible with e.g. cURL, so that you can copy the headers from a cURL command and re-use them with OpenDirectoryDownloader**. \n\nThis means you can easily \"fake\" a browser request: \n\n1. On the page/site you want to index, open your browsers dev tools (`F12` or `CTRL` + `SHIFT` + `i`)\n2. Go to the `Network` tab\n3. Reload the page\n4. Right-click on the first request/item in the network tab and select `Copy \u003e Copy as cURL (bash)` (might be called differently, depending on your browser)\n5. The copied command ends with lots of headers (`-H '\u003csomething\u003e' -H '\u003csomething else\u003e'`). Copy only this part of the command and append it to your OpenDirectoryDownloader command, like so: `OpenDirectoryDownloader --url \"https://myopendirectory.com\" -H 'header-name-1: header-value-1' -H 'header-name-2: header-value-2' ...` \n You can of course also use other options with this or omit the `--url` option to use the prompt instead.\n\nSetting some options like `--username` or `--user-agent` might override some headers, as explicit options take precedence. Option order does **not** matter (this applies to OpenDirectoryDownloader in general). \n\n### Copying on Linux\n\nWhen you want to copy (`C` key or `-c` flag) the stats at the end on Linux you need to have xclip installed.\n\n### Linux distros\n\nOn some distros you need extra dependencies. For Alpine: https://docs.microsoft.com/en-us/dotnet/core/install/linux-alpine\n\nFor others see: https://docs.microsoft.com/en-us/dotnet/core/install/linux\n\n## TLS errors (Windows 10)\n\nIf you received errors like this, please apply the registry file \"Enable TLS 1.3.reg\" from this [site](https://www.itechtics.com/tls-1-3/).\n\n```\nSystem.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.\n ---\u003e System.Security.Authentication.AuthenticationException: Authentication failed because the remote party sent a TLS alert: 'ProtocolVersion'.\n ---\u003e System.ComponentModel.Win32Exception (0x80090326): The message received was unexpected or badly formatted.\n ```\n\n## Cloudflare\n\n*EXPERIMANTAL!! READ THIS FIRST!*\n\nIT WILL NOT ALWAYS WORK!\n\nThere is experimental support for Cloudflare. When it detects a Cloudflare issue it will download a Chromium browser, start it, in which the Cloudflare protection can be solved. Sometimes this is a captcha which the user (you) needs to solve. For each browser session you have 60 seconds to complete. After that the browser will be killed and you can retry on next request.\n\nCloudflare does somehow detect that it is not the normal Chromium/Chrome browser and therefore it sadly will not always work. A good tip is move your mouse as soon as possible in the browser.\n\nSometimes it fails and pops up a browser for every request, and also kills it almost immediately when Cloudflare sees that there is no problem with the session. If this happens, kill the indexer!\n\nIf anybody have more info how to get Cloudflare to work better, let me know!\n\n## GitHub\n\nBy default GitHub has a rate limit of 60 request per hour, which is enough for 20 repositories with less than 100.000 items. You can increase this limit to 5000 per hour by creating a (personal) token:\n\n1. Go to https://github.com/settings/tokens/new\n2. Add a name like \"OpenDirectoryDownloader\"\n3. You don't have to select any scopes!\n4. Click \"Generate token\"\n5. Start OpenDirectoryDownloader with --githubtoken \u003cTOKEN\u003e\n\n## Docker\n\nEvery release will automatically push an image to the Docker Hub:\n\nhttps://hub.docker.com/repository/docker/koalabear84/opendirectorydownloader\n\nRun it like:\n\n`docker run --rm -v c:/Scans:/app/Scans -it koalabear84/opendirectorydownloader --quit --speedtest`\n\nIt will save the URLs files onto C:\\\\Scans (windows), or replace with a custom folder on other OS-ses.\n\n\\* You can also run it without `-v c:/scans:/app/Scans` if you don't want to save the results on your host.\n\n## Google Colab / Jupyter Notebook\n\n1. Open https://colab.research.google.com/github/KoalaBear84/OpenDirectoryDownloader/blob/master/OpenDirectoryDownloader.ipynb\n2. Run step 1 to setup the environment and install the latest OpenDirectoryDownloader\n3. Fill in the Url\n4. Run step 2\n5. Wait until indexing is completed\n6. Urls file can be found in Scans folder (see Folder icon on the left sidebar)\n\n## Onion / Tor support\n\n1. Make sure the Tor is running on your machine\n2. Use the correct proxy address notation, default for Tor is: \"socks5://127.0.0.1:9050\"\n3. Start it with `--proxy-address` parameter\n\n`OpenDirectoryDownloader.exe --url \"http://*.onion/\" --proxy-address \"socks5://127.0.0.1:9050\"`\n\n## Getting the code\n\n### For Visual Studio (Windows)\n\n1. Install Visual Studio: https://visualstudio.microsoft.com/vs/community/\n\n* With workload: \".NET Core cross-platform development\"\n* With individual components: Code tools \u003e Git for Windows and Code tools \u003e GitHub extension for Visual Studio\n\n1. Be sure to install Git: https://git-scm.com/downloads\n2. Clone the repository by clicking \"Clone or download\" and click \"Open in Visual Studio\"\n\n### For Visual Studio Code\n\n1. Download Visual Studio Code: https://code.visualstudio.com/download\n2. Be sure to install Git: https://git-scm.com/downloads\n3. Clone the repository: https://code.visualstudio.com/docs/editor/versioncontrol#_cloning-a-repository\n4. More help: https://docs.microsoft.com/en-us/dotnet/core/tutorials/with-visual-studio-code\n\n## Building\n\n1. Install the newest .NET 9 SDK: https://dotnet.microsoft.com/download/dotnet/9.0\n2. `git clone https://github.com/KoalaBear84/OpenDirectoryDownloader`\n3. `cd OpenDirectoryDownloader/src`\n4. `dotnet build .`\n5. `cd OpenDirectoryDownloader/bin/Debug/net9.0`\n6. `./OpenDirectoryDownloader --url \"https://myopendirectory.com\"`\n\nFor Linux (Might not be needed since .NET 7): \nThen, if you need to package it into a binary, you can use [warp-packer](https://github.com/dgiagio/warp#quickstart-with-net-core)\n\nWhen you have cloned the code, you can also run it without the SDK. For that, download the [\"Runtime\"](https://dotnet.microsoft.com/download) and do \"`dotnet run .`\" instead of build.\n\n## Google Drive\n\nFor Google Drive scanning you need to get a Google Drive API credentials file, it's free!\n\nYou can use a many steps manual option, or the 6 steps 'Quickstart' workaround.\n\nManual/customized:\n\n1. Go to https://console.cloud.google.com/projectcreate\n2. Fill in Project Name, like \"opendirectorydownloader\" or so, leave Location unchanged\n3. Change Project ID (optional)\n4. Click \"CREATE\"\n5. Wait a couple of seconds until the project is created and open it (click \"VIEW\")\n6. On the APIs pane, click \"Go to APIs overview\"\n7. Click \"ENABLE APIS AND SERVICES\"\n8. Enter \"Drive\", select \"Google Drive API\"\n9. Click \"ENABLE\"\n10. Go to \"Credentials\" menu in the left menu bar\n11. Click \"CONFIGURE CONSENT SCREEN\"\n12. Choose \"External\", click \"CREATE\"\n13. Fill in something like \"opendirectorydownloader\" in the \"Application name\" box\n14. At the bottom click \"Save\"\n15. Go to \"Credentials\" menu in the left menu bar (again)\n16. Click \"CREATE CREDENTIALS\"\n17. Select \"OAuth client ID\"\n18. Select \"Desktop app\" as \"Application type\"\n19. Change the name (optional)\n20. Click \"Create\"\n21. Click \"OK\" in the \"OAuth client created\" dialog\n22. In the \"OAuth 2.0 Client IDs\" section click on the just create Desktop app line\n23. In the top bar, click \"DOWNLOAD JSON\"\n24. You will get a file like \"client\\_secret\\_xxxxxx.apps.googleusercontent.com.json\", rename it to \"OpenDirectoryDownloader.GoogleDrive.json\" and replace the one in the release\n\nWow, they really made a mess of this..\n\nAlternative method (easier):\n\nThis will 'abuse' a 'Quickstart' project.\n\n1. Go to https://developers.google.com/drive/api/v3/quickstart/python\n2. Click the \"Enabled the Drive API\"\n3. \"Desktop app\" will already be selected on the \"Configure your OAuth client\" dialog\n4. Click \"Create\"\n5. Click \"DOWNLOAD CLIENT CONFIGURATION\"\n6. You will get a file like \"credentials.json\", rename it to \"OpenDirectoryDownloader.GoogleDrive.json\" and replace the one in the release\n\nOn the first use, you will get a browser screen that you need to grant access for it, and because we haven't granted out OAuth consent screen (This app isn't verified), we get an extra warning. You can use the \"Advanced\" link, and use the \"Go to yourappname (unsafe)\" link.\n\n## Support\n\nIf you like OpenDirectoryDownloader, please consider supporting me!\n\n[:heart: Sponsor](https://github.com/sponsors/KoalaBear84)\n\n## Contact me\n\nReddit https://www.reddit.com/user/KoalaBear84\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKoalaBear84%2FOpenDirectoryDownloader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FKoalaBear84%2FOpenDirectoryDownloader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKoalaBear84%2FOpenDirectoryDownloader/lists"}