Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/tanja-4732/od-get

A Rust tool for recursively crawling & downloading data from open directories
https://github.com/tanja-4732/od-get

cli crawler open-directory open-directory-downloader rust

Last synced: about 2 months ago
JSON representation

A Rust tool for recursively crawling & downloading data from open directories

Host: GitHub
URL: https://github.com/tanja-4732/od-get
Owner: Tanja-4732
License: agpl-3.0
Created: 2021-01-03T15:34:39.000Z (about 4 years ago)
Default Branch: main
Last Pushed: 2021-05-09T12:03:23.000Z (over 3 years ago)
Last Synced: 2024-10-12T18:57:15.931Z (3 months ago)
Topics: cli, crawler, open-directory, open-directory-downloader, rust
Language: Rust
Homepage: https://crates.io/crates/od-get
Size: 184 KB
Stars: 1
Watchers: 2
Forks: 1
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE.md

Awesome Lists containing this project

README

        # od-get

[![dependency status](https://deps.rs/crate/od-get/0.3.1/status.svg)](https://deps.rs/crate/od-get/0.3.1)

A Rust tool for recursively crawling & downloading data from [open directories](https://www.vice.com/en/article/d35x57/what-are-open-directories)

- [x] Filtering (regex) support

  - [x] Exclude file patterns

  - [x] Include file patterns

  - [x] Exclude folder patterns

  - [x] Include folder patterns

- [x] Customizable output

  - [x] Target directory

  - [ ] Verbosity

  - [x] Metadata-JSON file generation

  - [ ] Log file/dynamic terminal output

- [ ] Customizable limits

  - [ ] recursion depth limit

  - [ ] file count limit

  - [ ] file count offset (skip `n` files)

- [x] Multi threaded (using `rayon`)

- [x] Resume on error (avoid re-downloading files)

- [ ] Multi-level recursion

- [x] Disable download (only crawl to JSON)

(work in progress, one layer of recursion works)

## Licence & Copyright

Copyright (c) 2021 Bernd-L. All rights reserved.

![AGPL v3: Free as in Freedom](https://www.gnu.org/graphics/agplv3-with-text-162x68.png)

od-get is free software: you can redistribute it and/or modify it under the terms of the [GNU Affero General Public License](/LICENSE.md) as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

od-get is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the [GNU Affero General Public License](/LICENSE.md) for more details.

You should have received a copy of the [GNU Affero General Public License](/LICENSE.md) along with od-get. If not, see .

This project (including its source code and its documentation) is released under the terms of the [GNU Affero General Public License](/LICENSE.md).