{"id":13440196,"url":"https://github.com/rockdaboot/wget2","last_synced_at":"2025-05-15T06:02:33.255Z","repository":{"id":40472505,"uuid":"43870860","full_name":"rockdaboot/wget2","owner":"rockdaboot","description":"The successor of GNU Wget. Contributions preferred at https://gitlab.com/gnuwget/wget2. But accepted here as well 😍","archived":false,"fork":false,"pushed_at":"2025-02-02T18:44:54.000Z","size":14095,"stargazers_count":612,"open_issues_count":44,"forks_count":79,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-04-07T01:05:52.854Z","etag":null,"topics":["hacktoberfest"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rockdaboot.png","metadata":{"files":{"readme":"README","changelog":"ChangeLog","contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-10-08T07:44:28.000Z","updated_at":"2025-03-27T00:41:02.000Z","dependencies_parsed_at":"2023-09-30T20:31:07.038Z","dependency_job_id":"33092cf5-6a5a-47a3-a671-8e24af7ccc45","html_url":"https://github.com/rockdaboot/wget2","commit_stats":{"total_commits":3396,"total_committers":56,"mean_commits":"60.642857142857146","dds":"0.21142520612485272","last_synced_commit":"ff735fce14216b2f2b4fc88214c6ebfcc94b15cc"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rockdaboot%2Fwget2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rockdaboot%2Fwget2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rockdaboot%2Fwget2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rockdaboot%2Fwget2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rockdaboot","download_url":"https://codeload.github.com/rockdaboot/wget2/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248837276,"owners_count":21169373,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest"],"created_at":"2024-07-31T03:01:20.600Z","updated_at":"2025-04-14T06:52:58.072Z","avatar_url":"https://github.com/rockdaboot.png","language":"C","funding_links":[],"categories":["C"],"sub_categories":[],"readme":"[![Build status](https://gitlab.com/gnuwget/wget2/badges/master/pipeline.svg)](https://gitlab.com/gnuwget/wget2/pipelines)\n[![Fuzzing Status](https://oss-fuzz-build-logs.storage.googleapis.com/badges/wget2.svg)](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened\u0026can=1\u0026q=proj:wget2)\n[![Coverage status](https://gnuwget.gitlab.io/wget2/fuzz-coverage.svg)](https://gnuwget.gitlab.io/wget2/fuzz-coverage)\n[![Coverage status](https://gnuwget.gitlab.io/wget2/coverage.svg)](https://gnuwget.gitlab.io/wget2/coverage)\n[![Coverity Scan Build Status](https://scan.coverity.com/projects/7852/badge.svg)](https://scan.coverity.com/projects/rockdaboot-wget2)\n\nSolaris OpenCSW [![Build Status Solaris amd64](https://buildfarm.opencsw.org/buildbot/png?builder=wget2-solaris10-amd64)](https://buildfarm.opencsw.org/buildbot/builders/wget2-solaris10-amd64)\n[![Build Status Solaris i386](https://buildfarm.opencsw.org/buildbot/png?builder=wget2-solaris10-i386)](https://buildfarm.opencsw.org/buildbot/builders/wget2-solaris10-i386)\n[![Build Status Solaris Sparc](https://buildfarm.opencsw.org/buildbot/png?builder=wget2-solaris10-sparc)](https://buildfarm.opencsw.org/buildbot/builders/wget2-solaris10-sparc)\n[![Build Status Solaris SparcV9](https://buildfarm.opencsw.org/buildbot/png?builder=wget2-solaris10-sparcv9)](https://buildfarm.opencsw.org/buildbot/builders/wget2-solaris10-sparcv9)\n\n# Table of Content\n- [GNU Wget2 - Introduction](#GNU-Wget2---Introduction)\n- [Features](#Features)\n- [Links](#Links)\n- [Build Requirements](#Build-Requirements)\n- [Downloading and building from tarball](#Downloading-and-building-from-tarball)\n- [Building from git](#Building-from-git)\n  - [Download project and prepare sources with](#Download-project-and-prepare-sources-with)\n  - [Build Wget2 with](#Build-Wget2-with)\n  - [In Haiku build Wget2 with](#In-Haiku-build-Wget2-with)\n  - [Test the functionality](#Test-the-functionality)\n  - [Install Wget2 and libwget](#Install-Wget2-and-libwget)\n  - [Build static wget2.exe on GNU/Linux for Windows](#build-static-wget2exe-on-gnulinux-for-windows)\n- [License](#License)\n\n# GNU Wget2 - Introduction\n\nGNU Wget2 is the successor of GNU Wget, a file and recursive website downloader.\n\nDesigned and written from scratch it wraps around libwget, that provides the basic\nfunctions needed by a web client.\n\nWget2 works multi-threaded and uses many features to allow fast operation.\n\nIn many cases Wget2 downloads much faster than Wget1.x due to HTTP2, HTTP compression,\nparallel connections and use of If-Modified-Since HTTP header.\n\nGNU Wget2 is licensed under GPLv3+.\n\nLibwget is licensed under LGPLv3+.\n\n\n# Features\n\nA non-exhaustive list of features\n\n- Support for HTTP/1.1 and HTTP/2.0 protocol\n- [brotli](https://github.com/google/brotli) decompression support (Accept-Encoding: br)\n- [zstandard](https://github.com/facebook/zstd) decompression support, RFC8478 (Accept-Encoding: zstd)\n- [lzip](https://www.nongnu.org/lzip/) decompression support, (Accept-Encoding: lzip)\n- HPKP - HTTP Public Key Pinning (RFC7469) with persistent database\n- TCP Fast Open for plain text *and* for HTTPS\n- TLS Session Resumption including persistent session data cache\n- TLS False Start (with GnuTLS \u003e= 3.5.0)\n- HTTP2 support via nghttp2 and GnuTLS ALPN including streaming/pipelining\n- OCSP stapling + OCSP server querying as a fallback (experimental, needs GnuTLS \u003e= 3.3.11)\n- Use [libpsl](https://github.com/rockdaboot/libpsl) for cookie domain checking (using Public Suffix List)\n- Support link conversion (-k/--convert-links and -K/--backup-converted)\n- Support for RFC 6266 compliant Content-Disposition\n- RFC 6797 HSTS (HTTP Strict Transport Security)\n- Support for bzip2 Content-Encoding / Accept-Encoding compression type\n- New Year 2014 gimmick: added support for XZ Content-Encoding / Accept-Encoding compression type\n- Character encoding of input files may be specified despite from local and remote encoding (--input-encoding)\n- Support scanning RSS 2.0 feeds from local files (`--force-rss -i \u003cfilename\u003e`)\n- Support scanning RSS 2.0 feeds.\n- Support scanning Atom 1.0 feeds from local files (`--force-atom -i \u003cfilename\u003e`)\n- Support scanning Atom 1.0 feeds.\n- Support scanning URLs from local Sitemap XML file (`--force-sitemap -i \u003cfilename\u003e`)\n- Support scanning sitemap files given in robots.txt (Sitemap XML, gzipped Sitemap XML, plain text) including\nsitemap index files.\n- Support arbitrary number of proxies for parallel downloads\n- Multithreaded download of single files (option --chunk-size)\n- Internationalized Domain Names in Applications (compile-selectable IDNA2008 or IDNA2003)\n- ICEcast / SHOUTcast support via library (see examples/getstream.c)\n- respect /robots.txt \"Robot Exclusion Standard\" and `\u003cMETA name=\"robots\" ...\u003e`\n- new option --secure-protocol=PFS to have TLS only plus forcing Perfect Forward Secrecy (PFS)\n- IDN support for international domains\n- autotools support\n- proxy support\n- cookies (session/non-session), detection of supercookies via Mozilla Public Suffix List\n  (use the new option `--cookie-suffixes \u003cfilename\u003e`, better: put it into ~/.wgetrc)\n- recursive download of websites with or without spanning hosts\n- download of single web pages / resources\n- zlib/gzip compressed HTTP/HTTPS downloads (gzip, deflate)\n- number of parallel download threads is adjustable\n- include directive for config files (wildcards allowed)\n- support for keep-alive connections\n- included CSS, HTML, XML parser needed for recursive downloads\n- gettext support\n- HTTPS via libgnutls (and basic WolfSSL support)\n- support for Metalink RFC 6249 (Metalink/HTTP: Mirrors and Hashes)\n- support for Metalink RFC 5854 (Metalink Download Description Format / .meta4 files)\n- support for Metalink 3\n- Metalink checksumming via libgnutls\n- DNS lookup cache\n- IPv4 and IPv6 support\n- built and tested on Linux, OSX, OpenBSD, FreeBSD, Solaris, Windows\n\n\n# Links\n\n[Online Docs](https://gnuwget.gitlab.io/wget2/reference/)\n\n[Mailing List](https://savannah.gnu.org/mail/?group=wget)\n\n[Bug Tracker](https://gitlab.com/gnuwget/wget2/issues)\n\n[Development](https://gitlab.com/gnuwget/wget2)\n\n[Code Coverage](https://gnuwget.gitlab.io/wget2/coverage/)\n\n[Fuzz Code Coverage](https://gnuwget.gitlab.io/wget2/fuzz-coverage/)\n\n\n# Build Requirements\n\nThe following packages are needed to build the software\n\n* autotools (autoconf, autogen, automake, autopoint, libtool)\n* python (recommended for faster bootstrap)\n* rsync\n* tar\n* makeinfo (part of texinfo)\n* pkg-config \u003e= 0.28 (recommended)\n* doxygen (for creating the documentation)\n* pandoc (for creating the wget2 man page)\n* gettext \u003e= 0.18.2\n* libiconv (needed for IRI and IDN support)\n* libz \u003e= 1.2.3 (the distribution may call the package zlib*, eg. zlib1g on Debian)\n* liblzma \u003e= 5.1.1alpha (optional, if you want HTTP lzma decompression)\n* libbz2 \u003e= 1.0.6 (optional, if you want HTTP bzip2 decompression)\n* libbrotlidec/libbrotli \u003e= 1.0.0 (optional, if you want HTTP brotli decompression)\n* libzstd \u003e= 1.3.0 (optional, if you want HTTP zstd decompression)\n* libgnutls (3.3, 3.5 or 3.6)\n* libidn2 \u003e= 0.14 (libidn \u003e= 1.25 if you don't have libidn2)\n* flex \u003e= 2.5.35\n* libpsl \u003e= 0.5.0\n* libnghttp2 \u003e= 1.3.0 (optional, if you want HTTP/2 support)\n* libmicrohttpd \u003e= 0.9.51 (optional, if you want to run the test suite)\n* lzip (optional, if you want to build distribution tarballs)\n* lcov (optional, for coverage reports)\n* libgpgme \u003e= 0.4.2 (optional, for automatic signature verification)\n* libpcre | libpcre2 (optional, for filtering by PCRE|PCRE2 regex)\n* libhsts (optional, to support HSTS preload lists)\n* libwolfssl (optional, to support WolfSSL instead of GnuTLS)\n\nThe versions are recommended, but older versions may also work.\n\n\n# Downloading and building from tarball\n\n\t\twget https://gnuwget.gitlab.io/wget2/wget2-latest.tar.gz\n\t\ttar xf wget2-latest.tar.gz\n\t\tcd wget2-*\n\t\t./configure\n\t\tmake\n\t\tmake check\n\t\tsudo make install\n\n\n# Building from git\n\n### Download project and prepare sources with\n\n\t\tgit clone https://gitlab.com/gnuwget/wget2.git\n\t\tcd wget2\n\t\t./bootstrap\n\t\t# on shell failure try 'bash ./bootstrap'\n\n### Build Wget2 with\n\n\t\t./configure\n\t\tmake\n\n### In Haiku build Wget2 with\n\n        setarch x86\n        ./configure --prefix=/boot/home/config/non-packaged\n        rm /boot/home/config/non-packaged/wget2 \u0026\u0026 mv /boot/home/config/non-packaged/wget2_noinstall /boot/home/config/non-packaged/wget2\n\n### Test the functionality\n\n\t\tmake check\n\n### Install Wget2 and libwget\n\n\t\tsudo make install (or su -c \"make install\")\n\n### Build static wget2.exe on GNU/Linux for Windows\n\n\t\t(cd contrib; docker build -t wget2/static -f Dockerfile.win32.static .)\n\t\tdocker run --rm -v $PWD:/tmp wget2/static cp /usr/local/wget2/src/wget2.exe /tmp\n\t\t# now you have `wget2.exe` with debug symbols in your current directory\n\t\t# optional: remove debug symbols with\n\t\tstrip wget2.exe\n\t\t# optional: pack executable\n\t\tupx wget2.exe\n\n# License\n\nCopyright (C) 2015-2024 Free Software Foundation, Inc.\n\nThis program is free software; you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation; either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU General Public License for more details.\n\nA copy of the GNU General Public License can be found in the file LICENSE\nin the top directory of the official source distribution. The license is\nalso available in several formats through the World Wide Web, or\nvia http://www.gnu.org/licenses/licenses.html#GPL .\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frockdaboot%2Fwget2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frockdaboot%2Fwget2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frockdaboot%2Fwget2/lists"}