{"id":21035447,"url":"https://github.com/archiveteam/puush-grab","last_synced_at":"2025-05-15T14:31:18.086Z","repository":{"id":9477957,"uuid":"11364535","full_name":"ArchiveTeam/puush-grab","owner":"ArchiveTeam","description":null,"archived":false,"fork":false,"pushed_at":"2014-01-11T19:54:10.000Z","size":229,"stargazers_count":4,"open_issues_count":0,"forks_count":3,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-05-12T08:28:49.922Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ArchiveTeam.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-07-12T08:46:50.000Z","updated_at":"2025-04-04T00:38:06.000Z","dependencies_parsed_at":"2022-07-19T04:47:44.797Z","dependency_job_id":null,"html_url":"https://github.com/ArchiveTeam/puush-grab","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArchiveTeam%2Fpuush-grab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArchiveTeam%2Fpuush-grab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArchiveTeam%2Fpuush-grab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArchiveTeam%2Fpuush-grab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ArchiveTeam","download_url":"https://codeload.github.com/ArchiveTeam/puush-grab/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254358690,"owners_count":22057959,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T13:14:56.918Z","updated_at":"2025-05-15T14:31:17.755Z","avatar_url":"https://github.com/ArchiveTeam.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"puush-grab\n==========\n\nGrabbing Puu.sh files before they expire.\n\nPlease read the wiki at http://archiveteam.org/index.php?title=Puu.sh for the latest information.\n\nSetup instructions\n=========================\n\nBe sure to replace `YOURNICKHERE` with the nickname that you want to be shown as, on the tracker. You don't need to register it, just pick a nickname you like.\n\nIn most of the below cases, there will be a web interface running at http://localhost:8001/. If you don't know or care what this is, you can just ignore it—otherwise, it gives you a fancy view of what's going on.\n\n**If anything goes wrong while running the commands below, please scroll down to the bottom of this page. There's troubleshooting information there.**\n\nRunning without a warrior\n-------------------------\nTo run this outside the warrior, clone this repository, cd into its directory and run:\n\n    pip install seesaw\n    ./get-wget-lua.sh\n\nthen start downloading with:\n\n    run-pipeline pipeline.py --concurrent 2 YOURNICKHERE\n\nFor more options, run:\n\n    run-pipeline --help\n\nIf you don't have root access and/or your version of pip is very old, you can replace \"pip install seesaw\" with:\n\n    wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py ; python get-pip.py --user ; ~/.local/bin/pip install --user seesaw\n\nso that pip and seesaw are installed in your home, then run\n\n    ~/.local/bin/run-pipeline pipeline.py --concurrent 2 YOURNICKHERE\n\nDistribution-specific setup\n-------------------------\n### For Debian/Ubuntu:\n\n    adduser --system --group --shell /bin/bash archiveteam\n    apt-get install -y git-core libgnutls-dev lua5.1 liblua5.1-0 liblua5.1-0-dev screen python-dev python-pip bzip2 zlib1g-dev\n    pip install seesaw\n    su -c \"cd /home/archiveteam; git clone https://github.com/ArchiveTeam/puush-grab.git; cd puush-grab; ./get-wget-lua.sh\" archiveteam\n    screen su -c \"cd /home/archiveteam/{{REPO_NAME}}/; run-pipeline pipeline.py --concurrent 2 --address '127.0.0.1' YOURNICKHERE\" archiveteam\n    [... ctrl+A D to detach ...]\n\nWget-lua is also available on [ArchiveTeam's PPA](https://launchpad.net/~archiveteam/+archive/wget-lua) for Ubuntu.\n\n### For CentOS:\n\nEnsure that you have the CentOS equivalent of bzip2 installed as well. You might need the EPEL repository to be enabled.\n\n    yum -y install gnutls-devel lua-devel python-pip zlib-devel\n    pip install seesaw\n    [... pretty much the same as above ...]\n\n### For openSUSE:\n\n    zypper install liblua5_1 lua51 lua51-devel screen python-pip libgnutls-devel bzip2 python-devel gcc make\n    pip install seesaw\n    [... pretty much the same as above ...]\n\n### For OS X:\n\nYou need Homebrew. Ensure that you have the OS X equivalent of bzip2 installed as well.\n\n    brew install python lua gnutls\n    pip install seesaw\n    [... pretty much the same as above ...]\n\n**There is a known issue with some packaged versions of rsync. If you get errors during the upload stage, {{REPO_NAME}} will not work with your rsync version.**\n\nThis supposedly fixes it:\n\n    alias rsync=/usr/local/bin/rsync\n\n### For Arch Linux:\n\nEnsure that you have the Arch equivalent of bzip2 installed as well.\n\n1. Make sure you have `python-pip2` installed.\n2. Install [https://aur.archlinux.org/packages/wget-lua/](the wget-lua package from the AUR). \n3. Run `pip2 install seesaw`.\n4. Modify the run-pipeline script in seesaw to point at `#!/usr/bin/python2` instead of `#!/usr/bin/python`.\n5. `adduser --system --group --shell /bin/bash archiveteam`\n6. `screen su -c \"cd /home/archiveteam/puush-grab/; run-pipeline pipeline.py --concurrent 2 --address '127.0.0.1' YOURNICKHERE\" archiveteam`\n\n### For FreeBSD:\n\nHonestly, I have no idea. `./get-wget-lua.sh` supposedly doesn't work due to differences in the `tar` that ships with FreeBSD. Another problem is the apparent absence of Lua 5.1 development headers. If you figure this out, please do let us know on IRC (irc.efnet.org #archiveteam).\n\nTroubleshooting\n=========================\n\nBroken? These are some of the possible solutions:\n\n### wget-lua was not successfully built\n\nIf you get errors about `wget.pod` or something similar, the documentation failed to compile - wget-lua, however, compiled fine. Try this:\n\n    cd get-wget-lua.tmp\n    mv src/wget ../wget-lua\n    cd ..\n\nThe `get-wget-lua.tmp` name may be inaccurate. If you have a folder with a similar but different name, use that instead and please let us know on IRC what folder name you had!\n\nOptionally, if you know what you're doing, you may want to use wgetpod.patch.\n\n### Problem with gnutls or openssl during get-wget-lua\n\nPlease ensure that gnutls-dev(el) and openssl-dev(el) are installed.\n\n### ImportError: No module named seesaw\n\nIf you're sure that you followed the steps to install `seesaw`, permissions on your module directory may be set incorrectly. Try the following:\n\n    chmod o+rX -R /usr/local/lib/python2.7/dist-packages\n\n### Issues in the code\n\nIf you notice a bug and want to file a bug report, please use the GitHub issues tracker.\n\nAre you a developer? Help write code for us! Look at our [developer documentation](http://archiveteam.org/index.php?title=Dev) for details.\n\n### Other problems\n\nHave an issue not listed here? Join us on IRC and ask! We can be found at irc.efnet.org #pushharder.\n\nTechnical details for tracker admin and developers\n==================================================\n\nWhen \"item name\" is referred, they are the base 62 encoded ids of each item. \"Item number\" refers to the base 10 integer of the item id. The tracker is loaded up by item names (not integers).\n\nThe script uses two base 62 encoding alphabets. The legacy alphabet consists of `0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz`. Puush, however, uses an encoding of `0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ`. They both start from the base 10 integer of 0 and incrementing up.  Therefore, it is important to be careful of this distinction. \n\n(When transitioning from one alphabet and generating item names, be sure to use a value that is unambiguously lower than the current item number in *both* alphabets. Then, switch alphabets.)\n\nIf the item contains a comma like ``abcd,abcg``, it is an item name range using the legacy alphabet. It covers from ``abcd`` to ``abcg`` which is 4 items in the range. If the item contains a colon like `abcd:abcg`, it is an item name using the Puush alphabet.\n\n\nDecentralized Puush Grab Script\n-------------------------------\n\nThis script will randomly choose Puush items and download them. It does not use a tracker and does not upload files.\n\nThis script is useful if you cannot use the Warrior or the tracker is not available. Before running this script, be sure you are willing to invest time running and taking care of the downloads.\n\nThe script requires Python 2.7 and wget-lua and can be run with the command as shown:\n\n    python ./decentralized_puush_grab.py\n\nIt will create a temporary wget directory, a data directory, and a report directory.\n\nYou can provide the `--delay SECONDS` argument to control the minimum delay in seconds.\n\nTo stop, create a file called STOP in the same directory.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farchiveteam%2Fpuush-grab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farchiveteam%2Fpuush-grab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farchiveteam%2Fpuush-grab/lists"}