awesome-datahoarding
List of data-hoarding related tools
https://github.com/simon987/awesome-datahoarding
Last synced: 1 day ago
JSON representation
-
Download utilities
-
Download automation
-
Web Archiving
- webrecorder - fidelity, ISO-compliant web archives in a user-friendly interface, providing access to archived content, and sharing collections
- Browsertrix Crawler - based high-fidelity crawling system, designed to run a complex, customizable browser-based crawl in a single Docker container
- Collect
- wail - Click User Instigated Preservation
- wikiteam
- Heritrix - scale, archival-quality web crawler
- HTTrack
- ArchiveBox - hosted web archive. Takes browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
-
General
- annie - DL alternative written in Golang
- aria2 - protocol & multi-source command-line download utility
- you-get
- ytdl-sub
- yt-dlp - DLP that behaves better
- CrowLeer
- curl
- Horahora
- httpie
- news-crawl
- Plowshare - line tool to manage file-sharing site
- Rclone
- rsync
- Suck-It - threaded)
- wget - interactive download of files from the Web
- wget2 - threaded
- wpull - compatible web downloader and crawler
-
Application-specific
- Instagram Scraper - scraper is a command-line application written in Python that scrapes and downloads an instagram user's photos and videos. Use responsibly.
- RedditDownloader
- Scribd-Downloader
- PyInstaLive
- yt-mango
- snscrape
- RipMe
- Tube Archivist - Hosted Docker container for automated/scheduled YouTube downloads of channels, playlists, etc.
- tumblr-utils
- BBCSoundDownloader
- ChanThreadWatch - style boards and checks for updates until the thread dies
- comics-downloader - line tool to download comicsand manga in pdf/epub/cbz/cbr from supported sites
- floatplane_ripper
- gallery-dl
- Discord-Channel-Scraper
- dzi-dl
- FanFicFare
- FicSave
- flickr_download
- Google Images Download
- iiif-dl - line tile downloader/assembler for IIIF endpoints/manifests
- imgbrd-grabber
- instaloader
- InstaLooter - less Instagram pictures and videos downloader.
- Youtube-MA
- PyInstaLive
- yt-mango
-
-
Compression
-
Download automation
- KGB Archiver
- WinRAR
- 7-Zip
- peazip
- PIGZ - threaded gzip
-
-
Network
-
Download automation
-
-
File systems
-
Download automation
-
-
File conversion
-
Utility Scripts
-
Download automation
- rclone_dirsize
- flac2mp3_V0.py - threaded python script to convert all flac files to mp3 V0 while keeping the directory structure
- Backblaze B2 sync backup script
- Misc download scripts
- TheFrenchGhosty's Ultimate YouTube-DL Scripts Collection - dl scripts to aid in YouTube channel archival
- rm_empty_subdir - directories on Windows
- void-cat-uploader
- youtube-dl_soundcloud - dl to download soundcloud playlists
-
-
Content sharing
-
Data curation
-
Download automation
- baobab
- Kiwix
- fucking-weeb
- MediaElch
- MediaInfo
- picard
- Mp3tag - to-use tool to edit metadata of audio files (Windows/Mac)
- WizTree
- VisiPics
- VisiPics
- diskover
- browsemonkey
- Calibre
- DataCurator-Filetree
- grepWin
- phockup
- Kiwix
- WizTree
- VisiPics
- beets
- Everything
- FileBot
- Hydrus
- jdupes
- TeraCopy
- tree
- WinDirStat
- SyncToy
-
-
Long-term data archiving
-
Download automation
-
-
APIs & Online tools
-
Backup
-
Download automation
-
-
Hardware / Monitoring
-
Download automation
- CrystalDiskInfo
- GSmartControl - OS S.M.A.R.T. utility with an easy to understand graphical interface
- Hard Drive Sentinel - OS SSD and HDD monitoring and analysis software
- smartmontools
-
-
Data recovery
-
Local Media
-
Download automation
- whipper - DA ripper preferring accuracy over speed. Generates .flac, .cue, and .log by default and automatically fetches metadata from musicbrainz. EAC log plugin is available.
- MakeMKV - platform DVD ripper that supports recent blu ray DVDs. It's mostly open source, but the blu ray secret sauce is still hidden
- Handbrake
-
Programming Languages
Categories
Sub Categories
Keywords
python
18
downloader
10
crawler
5
warc
5
cli
4
web-archiving
4
torrent
4
instagram
4
epub
3
ebook
3
tumblr
3
backup
3
sonarr
3
rss
3
youtube
3
radarr
3
youtube-dl
3
scraper
3
linux
3
golang
3
go
3
music
3
yt-dlp
3
plex
2
sftp
2
archive
2
kodi
2
filesystem
2
rust
2
download
2
couchpotato
2
crawling
2
self-hosted
2
web-crawler
2
lidarr
2
fanfiction
2
windows
2
metadata
2
php
2
reactjs
2
wget
2
instagram-scraper
2
heritrix
2
reddit
2
java
2
help-wanted
2
archiver
2
audio
2
twitter
2
flickr
2