{"id":21035444,"url":"https://github.com/archiveteam/wget-lua","last_synced_at":"2025-04-04T16:12:33.003Z","repository":{"id":2245073,"uuid":"207030888","full_name":"ArchiveTeam/wget-lua","owner":"ArchiveTeam","description":"Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.","archived":false,"fork":false,"pushed_at":"2024-12-31T12:11:28.000Z","size":29366,"stargazers_count":117,"open_issues_count":14,"forks_count":16,"subscribers_count":20,"default_branch":"v1.21.3-at","last_synced_at":"2025-03-28T15:04:50.204Z","etag":null,"topics":["archiveteam","archiving","crawl","crawler","crawlers","crawling","downloader","ftp","lua","scraper","scraping","spider","warc","webarchiving","wget","wget-lua","zstd"],"latest_commit_sha":null,"homepage":"https://www.archiveteam.org/","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"alard/wget-lua","license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ArchiveTeam.png","metadata":{"files":{"readme":"README","changelog":"ChangeLog","contributing":"CONTRIBUTING.md","funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-07T22:05:35.000Z","updated_at":"2025-03-11T06:17:38.000Z","dependencies_parsed_at":"2023-07-05T21:32:21.964Z","dependency_job_id":"3f033818-dbd9-4670-8af8-05f09bf04efb","html_url":"https://github.com/ArchiveTeam/wget-lua","commit_stats":{"total_commits":4359,"total_committers":175,"mean_commits":"24.908571428571427","dds":0.5570084881853636,"last_synced_commit":"c1fe6093eda544fc7a933f7646225bec1ff4bd8d"},"previous_names":[],"tags_count":34,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArchiveTeam%2Fwget-lua","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArchiveTeam%2Fwget-lua/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArchiveTeam%2Fwget-lua/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArchiveTeam%2Fwget-lua/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ArchiveTeam","download_url":"https://codeload.github.com/ArchiveTeam/wget-lua/tar.gz/refs/heads/v1.21.3-at","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247208142,"owners_count":20901570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archiveteam","archiving","crawl","crawler","crawlers","crawling","downloader","ftp","lua","scraper","scraping","spider","warc","webarchiving","wget","wget-lua","zstd"],"created_at":"2024-11-19T13:14:56.206Z","updated_at":"2025-04-04T16:12:30.703Z","avatar_url":"https://github.com/ArchiveTeam.png","language":"C","readme":"                                                          -*- text -*-\nGNU Wget\n========\n                  Current Web home: https://www.gnu.org/software/wget/\n\nGNU Wget is a free utility for non-interactive download of files from\nthe Web.  It supports HTTP, HTTPS, and FTP protocols, as well as\nretrieval through HTTP proxies.\n\nIt can follow links in HTML pages and create local versions of remote\nweb sites, fully recreating the directory structure of the original\nsite.  This is sometimes referred to as \"recursive downloading.\"\nWhile doing that, Wget respects the Robot Exclusion Standard\n(/robots.txt).  Wget can be instructed to convert the links in\ndownloaded HTML files to the local files for offline viewing.\n\nRecursive downloading also works with FTP, where Wget can retrieve a\nhierarchy of directories and files.\n\nWith both HTTP and FTP, Wget can check whether a remote file has\nchanged on the server since the previous run, and only download the\nnewer files.\n\nWget has been designed for robustness over slow or unstable network\nconnections; if a download fails due to a network problem, it will\nkeep retrying until the whole file has been retrieved.  If the server\nsupports regetting, it will instruct the server to continue the\ndownload from where it left off.\n\nIf you are behind a firewall that requires the use of a socks style\ngateway, you can get the socks library and compile wget with support\nfor socks.\n\nMost of the features are configurable, either through command-line\noptions, or via initialization file .wgetrc.  Wget allows you to\ninstall a global startup file (/usr/local/etc/wgetrc by default) for\nsite settings.\n\nWget works under almost all Unix variants in use today and, unlike\nmany of its historical predecessors, is written entirely in C, thus\nrequiring no additional software, such as Perl.  The external software\nit does work with, such as OpenSSL, is optional.  As Wget uses the GNU\nAutoconf, it is easily built on and ported to new Unix-like systems.\nThe installation procedure is described in the INSTALL file.\n\nAs with other GNU software, the latest version of Wget can be found at\nthe master GNU archive site ftp.gnu.org, and its mirrors.  Wget\nresides at \u003cftp://ftp.gnu.org/pub/gnu/wget/\u003e.\n\nPlease report bugs in Wget to \u003cbug-wget@gnu.org\u003e.\n\nSee the file `MAILING-LIST' for information about Wget mailing lists.\nWget's home page is at \u003chttps://www.gnu.org/software/wget/\u003e.\n\nIf you would like to contribute code for Wget, please read\nCONTRIBUTING.md.\n\nWget was originally written and mainained by Hrvoje Niksic.  Please see\nthe file AUTHORS for a list of major contributors, and the ChangeLogs\nfor a detailed listing of all contributions.\n\n\nCopyright (C) 1995-2022 Free Software Foundation, Inc.\n\nThis program is free software; you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation; either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program; if not, write to the Free Software\nFoundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301\nUSA.\n\nAdditional permission under GNU GPL version 3 section 7\n\nIf you modify this program, or any covered work, by linking or\ncombining it with the OpenSSL project's OpenSSL library (or a\nmodified version of that library), containing parts covered by the\nterms of the OpenSSL or SSLeay licenses, the Free Software Foundation\ngrants you additional permission to convey the resulting work.\nCorresponding Source for a non-source form of such a combination\nshall include the source code for the parts of OpenSSL used as well\nas that of the covered work.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farchiveteam%2Fwget-lua","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farchiveteam%2Fwget-lua","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farchiveteam%2Fwget-lua/lists"}