Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

awesome-digital-preservation

Awesome list dedicated to digital and data preservation tools, sources, services and so on.
https://github.com/ruarxive/awesome-digital-preservation

Last synced: 5 days ago
JSON representation

  • Web archiving

    • Crawlers

      • WPull - Wget-compatible web downloader and crawler.
      • Conifer - collect and revisit web pages
      • grab-site - The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
      • Heritrix3 - Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
      • WAIL - Web Archiving Integration Layer: One-Click User Instigated Preservation
      • Browsetrix Crawler - run a high-fidelity browser-based crawler in a single Docker container
      • Wget - a free software package for retrieving files using HTTP, HTTPS, FTP and FTPS, the most widely used Internet protocols.

  • Replay tools

    • Crawlers

      • Archive Web.page - A High-Fidelity Web Archiving Extension for Chrome and Chromium based browsers
      • Reply Web.page - Serverless Web Archive Replay directly in the browser
      • pywb - Core Python Web Archiving Toolkit for replay and recording of web archives
      • webrecorder-player - Webrecorder Player for Desktop (OSX/Windows/Linux). (Built with Electron + Webrecorder)
      • ipwb - InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS

    • Analysis and data processing

      • AUT - The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
      • AUT Notebooks - Various examples of notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archives Unleashed Toolkit.
      • WARCIO - Streaming WARC/ARC library for fast web archive IO
      • Metawarc - Metadata extractor from WARC files
      • WarcDB - WarcDB: Web crawl data as SQLite databases
      • ArchiveSpark - An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
      • CDX Toolkit - A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

    • Page pushers

      • ArchiveBox - Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more
      • Wayback - A self-hosted toolkit for archiving webpages to the Internet Archive, archive.today, IPFS, and local file systems
      • Archivenow - A Tool To Push Web Resources Into Web Archives
      • iagitup - A command line tool to archive a git repository from GitHub to the Internet Archive.

    • Online services

  • Social Networks

    • Twitter

      • twarc - A command line tool (and Python library) for archiving Twitter JSON

    • Instagram

      • instaloader - Download pictures (or videos) along with their captions and other metadata from Instagram.

    • Universal

      • sfm-ui - Social Feed Manager user interface application.
      • Media downloader - download Instagram Reels, Stories, Post, Stalk Instagram Profile, Facebook Public Videos, YouTube Videos and YouTube to MP3 converter, SoundCloud MP3 and Dailymotion videos. Made from Node JS Express JS, React JS and Rapid API.

  • Other digital objects

    • Online storage

      • ydiskarc - command-line tool to backup public resources from Yandex.disk (disk.yandex.ru / yadi.sk) filestorage service
      • filegetter - A command-line tool to collect files from public data sources using URL patterns and config files

    • Messengers and chats

      • tgarc - A command line tool for archiving Telegram JSON

    • Specific CMS

      • wparc - Wordpress API data and files archival command line tool
      • spcrawler - A command-line tool to backup Sharepoint public installations data from open API endpoint

    • Public Data API

      • apibackuper - Python library and cmd tool to backup API calls

  • Standards and specifications

    • Public Data API

      • WARC Specifications - collection of WARC related specifications and formats
      • The WARC Format 1.1 - The Web ARChive (WARC) archive format specifies a method for combining multiple digital resources into an aggregate archive file together with related information.
      • CDX File format - format of CDX files, that are list of files in WARC files
      • The WACZ Format 1.1.1 - Web Archive Collection Zipped. WACZ is a media type that allows web archive collections to be packaged and shared on the web as a discrete file.

    • Public Data API

      • Awesome data takeout - An Awesome Data Takeout list of services to take out your personal data from major online services and providers
      • Awesome Web Archiving - An Awesome List for getting started with web archiving

  • Organizations

  • Knowledge bases

    • Public Data API

  • Major digital archives

    • Public Data API

Programming Languages
Python 19 JavaScript 4 Scala 2 TypeScript 2 HTML 2 Go 1 Jupyter Notebook 1 Roff 1 Java 1
Categories
Replay tools 17 Web archiving 7 Other digital objects 6 Standards and specifications 4 Social Networks 4 Related lists 2 Organizations 2 Major digital archives 1 Knowledge bases 1
Sub Categories
Crawlers 12 Public Data API 11 Analysis and data processing 7 Page pushers 4 Online storage 2 Universal 2 Specific CMS 2 Messengers and chats 1 Twitter 1 Instagram 1 Online services 1
Keywords
web-archiving 15 warc 14 python 7 backup 4 web-archives 4 archiving 4 webrecorder 4 wayback 4 pywb 4 internet-archive 4 wayback-machine 3 wacz 3 spark 3 social-media 2 service-worker 2 ipfs 2 memento 2 instagram-stories 2 instagram-downloader 2 instagram 2 python3 2 osint 2 osint-python 2 webarchiving 2 cli 2 archive 2 self-hosted 2 awesome-list 2 chromium 2 archival 2 awesome 2 heritrix 2 crawling 2 crawler 2 archives 2 docker 2 database 1 crawl 1 warc-files 1 spider 1 metadata 1 pyspark-notebook 1 notebooks 1 juypter-notebook 1 sqlite 1 web-data 1 archivespark 1 spark-framework 1 webarchive 1 cdx 1