{"id":28474223,"url":"https://github.com/athlon1600/useful","last_synced_at":"2025-10-13T10:35:09.782Z","repository":{"id":41431147,"uuid":"230974552","full_name":"Athlon1600/useful","owner":"Athlon1600","description":":+1: Useful shell scripts in one place","archived":false,"fork":false,"pushed_at":"2024-12-20T20:28:39.000Z","size":288,"stargazers_count":10,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-08-21T14:16:44.695Z","etag":null,"topics":["certbot","ffmpeg","laravel","nginx","php","shell","squid","youtube-dl","yt-dlp"],"latest_commit_sha":null,"homepage":"https://www.proxynova.com/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Athlon1600.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-12-30T20:18:06.000Z","updated_at":"2024-12-20T20:28:44.000Z","dependencies_parsed_at":"2023-12-22T02:22:32.656Z","dependency_job_id":"9b933040-60d9-422f-a146-2664998a3a62","html_url":"https://github.com/Athlon1600/useful","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Athlon1600/useful","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Athlon1600%2Fuseful","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Athlon1600%2Fuseful/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Athlon1600%2Fuseful/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Athlon1600%2Fuseful/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Athlon1600","download_url":"https://codeload.github.com/Athlon1600/useful/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Athlon1600%2Fuseful/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279014641,"owners_count":26085555,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-13T02:00:06.723Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["certbot","ffmpeg","laravel","nginx","php","shell","squid","youtube-dl","yt-dlp"],"created_at":"2025-06-07T13:06:27.600Z","updated_at":"2025-10-13T10:35:09.777Z","avatar_url":"https://github.com/Athlon1600.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# :bulb: useful\n\nA list of useful commands and bash scripts for many popular stacks.\n\n## Install Docker / Docker Compose\n\n```shell\ncurl -sSL https://get.docker.com/ | sh\n```\n\n## Squid Http Proxy\n\nInstall anonymous proxy server with port 8080 fully open and accessible to everyone:\n\n```shell\nbash \u003c(wget -O - https://raw.githubusercontent.com/Athlon1600/useful/master/squid/install.sh)\n```\n\nGenerate custom Squid installation script:\n\n[![](https://i.imgur.com/kJjVkPJ.png)](https://www.proxynova.com/proxy-articles/install-squid-proxy-server)\n\n- https://www.proxynova.com/proxy-articles/install-squid-proxy-server\n\n---\n\n```bash\n## number of \"open files\" by process\n\nlsof | awk '{print $1}' | uniq -c | sort -n\n```\n\n\n## youtube-dl/yt-dlp\n\nDownload binaries:\n\n```shell\ncurl -L https://yt-dl.org/downloads/latest/youtube-dl -o /usr/local/bin/youtube-dl \u0026\u0026 chmod a+rx /usr/local/bin/youtube-dl\ncurl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_linux -o /usr/local/bin/yt-dlp \u0026\u0026 chmod a+rx /usr/local/bin/yt-dlp\n```\n\nCheck if YouTube is blocked:\n\u003e curl -I -X GET https://www.youtube.com/watch?v=S0DRch3YLh0\n\nDownload all URLs in file:\n\n```shell\nyt-dlp --batch-file \"batch.txt\" -o \"%(upload_date)s-%(title)s.%(ext)s\"\n```\n\nDownload entire channel slowly:\n\n```shell\nyoutube-dl -o \"%(upload_date)s %(title)s.%(ext)s\" -f best -ciw --sleep-interval 30 --max-sleep-interval 90 https://www.youtube.com/@AlfoMedia\nyt-dlp -o \"%(upload_date)s %(title)s.%(ext)s\" -f best -ciw --sleep-interval 30 --max-sleep-interval 90 https://www.youtube.com/@AlfoMedia\n```\n\n## Apache Bench\n\nDownload Windows binaries from - https://www.apachelounge.com/download/\n\n```shell\n# -n = number of requests\n# -c = Number of multiple requests to perform at a time\n# -k = enable KeepAlive which is off by default\n\nab -n 1000 -c 30 -k http://example.com\n```\n\n## Siege\n\n```shell\nsudo apt-get install siege -y\nsiege -c 500 -r 5000 https://example.com/\nsiege -c 600 --time=5M https://example.com/\n```\n\nAlso edit:\n```shell\nvim $HOME/.siege/siege.conf \n```\n\nand raise `limit` from default 255.\n\n## NGINX + PHP 7.3\n\n\u003e sudo add-apt-repository ppa:ondrej/php\n\nBasic:\n\n```shell\nsudo apt-get -y install nginx php7.3-fpm\n```\n\nComplete:\n```shell\nsudo apt-get -y install nginx php7.3-fpm php7.3-curl php7.3-mbstring php7.3-zip php7.3-dom php7.3-mysql\n```\n\nExtra:\n```shell\napt-get -y install php7.3-zip\napt-get -y install build-essential\n```\n\n```shell\nln -s /etc/nginx/sites-available/newsite.com /etc/nginx/sites-enabled/\n```\n\n## Caddy\n\nInstall Caddy\n\n```shell\ncurl -sS https://raw.githubusercontent.com/Athlon1600/chat/master/scripts/caddy.sh | sh\n```\n\nFrom static binary (Ubuntu)\n```shell\ncurl -sL https://github.com/caddyserver/caddy/releases/download/v2.8.4/caddy_2.8.4_linux_amd64.tar.gz | tar -xz \u0026\u0026 rm -f caddy_2.8.4_linux_amd64.tar.gz LICENSE README.md \u0026\u0026 chmod +x caddy \u0026\u0026 mv caddy /usr/local/bin/caddy\n```\n\nFrom caddyserver.com with rate limiting extension:\n```shell\ncurl -o caddy -sL \"https://caddyserver.com/api/download?os=linux\u0026arch=amd64\u0026p=github.com/RussellLuo/caddy-ext/ratelimit\" \u0026\u0026 chmod +x caddy \u0026\u0026 mv caddy /usr/local/bin/caddy\n```\n\nLaunch simple static file server in current directory with automatic HTTTPS:\n\n```shell\npkill caddy\nnohup caddy file-server --root ./ --browse --listen :50519 \u003e /dev/null 2\u003e\u00261 %\n```\n\nShow requests as they happen:\n```shell\ntail -F /var/log/caddy/access.log | jq -r '\"\\(.request.host)\\t\\(.request.uri)\"'\n```\n\nTop Request URIs:\n```shell\ntail -n 9000 /var/log/caddy/access.log | jq -r '.request.uri' | sort | uniq -c | sort -nr | head -n 30\n```\n\nTop Request URIs in the last hour:\n```shell\ncat /var/log/caddy/access.log | jq --argjson time_ago $(date -d '1 hour ago' +%s) 'select(.ts \u003e $time_ago)' | jq -r '.request.uri' | sort | uniq -c | sort -nr | head -n 30\n```\n\nTop Request URIs from some IP:\n```shell\ntail -n 9000 /var/log/caddy/access.log | jq 'select(.request.remote_ip == \"192.0.2.4\")' | jq -r '.request.uri' | sort | uniq -c | sort -nr | head -n 30\n```\n\nTop client IPs (useful during DDOS attack):\n```shell\ntail -n 9000 /var/log/caddy/access.log | jq -r '.request.remote_ip' | sort | uniq -c | sort -nr | head -n 30\n```\n\n## Composer\n\n```shell\ncurl -sS https://getcomposer.org/installer | sudo php -- --install-dir=/usr/local/bin --filename=composer\n```\n\n\n## Laravel\n\n```shell\nsudo chown -R www-data:www-data /var/www/unblockvideos.com/storage\nphp artisan key:generate\n```\n\n- https://laravel.com/docs/6.x/deployment#nginx\n- https://laravel.com/docs/7.x/deployment#nginx\n\n```shell\nwget -O deploy.sh https://raw.githubusercontent.com/Athlon1600/useful/master/deploy-laravel.sh\n```\n\n## Node.js\n\n```shell\ncurl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash - \u0026\u0026 sudo apt-get install -y nodejs\n```\n\n## Nginx + Express\n\nConfiguration to proxy all requests backend to node express server running on port 3000.\n\n```shell\nwget -O /etc/nginx/nginx.conf https://raw.githubusercontent.com/Athlon1600/useful/master/nginx/express.conf\n```\n\nRestart the server.\n\n## Let's Encrypt\n\n```shell\nsudo snap install --classic certbot\n\nsudo certbot --nginx --agree-tos --register-unsafely-without-email --redirect\nsudo certbot --nginx --agree-tos --register-unsafely-without-email --no-redirect\n```\n\n## ngxtop\n\n```shell\nsudo apt-get -y install build-essential python-pip\npip install ngxtop\n```\n\nUsage:\n\n```shell\nngxtop -n 30\ndocker logs --tail 1000 -f nginx | cat | ngxtop -n 30\n\n// Request count by IP:\nngxtop top remote_addr -n 30\ndocker logs --tail 1000 -f nginx | cat | ngxtop top remote_addr -n 30\n```\n\nBlock IP:\n```shell\niptables -A INPUT -s 195.201.192.154 -j DROP\n```\n\nAdd `block_ip IP_ADDRESS` command to your linux:\n```shell\necho 'block_ip() { iptables -A INPUT -s \"$1\" -j DROP; }' \u003e\u003e ~/.bashrc \u0026\u0026 source ~/.bashrc\n```\n\n## php + SSL\n\ncURL error 60: SSL certificate problem: unable to get local issuer certificate (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)\n\n```bash\nwget https://curl.haxx.se/ca/cacert.pem\nmv cacert.pem /etc/php/7.3/cacert.pem\n\necho \"curl.cainfo=\\\"/etc/php/7.3/cacert.pem\\\"\" \u003e\u003e /etc/php/7.3/cli/php.ini\n```\n\nmisc other:\n\u003e php -r \"var_dump(ini_get('curl.cainfo'));\"\n\n\u003e php -r \"var_dump(openssl_get_cert_locations());\" \n\n\n## Aria2 - download file in parts in parallel\n\n```bash\nsudo apt-get -y install snapd\nsudo snap install aria2c\n\n// extra\nsudo snap install core\nexport PATH=$PATH:/snap/bin\n\naria2c -x 16 -j 16 https://archive.org/download/YouTubeCrawlSurveyDataset2009-2010/ytdata.sql.xz\n```\n\n`urls.txt`:\n\n```text\nhttps://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png\n\tout=logo.png\n```\n\n```bash\naria2c -iurls.txt -x 16 -j 16\n```\n\n## Streamlink\n\n```bash\nstreamlink https://site.com/index.m3u8 best -o best.mp4 --http-header \"Referer=https://www.youtube.com\"\nstreamlink https://site.com/index.m3u8 best -o streamlink.mp4 --http-header \"Referer=https://www.youtube.com\" --retry-streams 5 --retry-max 9999 --stream-segment-attempts 9999 --hls-playlist-reload-attempts 9999 --dash-manifest-reload-attempts 9999 --stream-segment-attempts 9999\n```\n\n## Redis\n\n```shell\nsudo apt-get -y install redis-server\n```\n\n## Google Big Query\n\n```shell\nbash \u003c(wget -O - https://raw.githubusercontent.com/Athlon1600/useful/master/google/bigquery.sh)\n```\n\n## Java\n\n```shell\nsudo apt-get -y install default-jre\nwget https://raw.githubusercontent.com/Athlon1600/useful/master/java/run.sh\n```\n\n## AWS\n\n```shell\napt-get install -y unzip curl\ncurl -L \"https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip\" -o \"awscliv2.zip\" \u0026\u0026 unzip awscliv2.zip \u0026\u0026 sudo ./aws/install\n\n## authenticate\naws configure set aws_access_key_id ACCESS_KEY_HERE ; aws configure set aws_secret_access_key SECRET_KEY_GOES_HERE\n\n## copy all files from current directory to a directory on s3 bucket\naws s3 cp . s3://bucket/folder --recursive --storage-class ONEZONE_IA\n```\n\n## MySQL - Backup/Restore\n\nInstall client first:\n\n```shell\nsudo apt-get -y install mysql-client\n```\n\nLogical Backup:\n\n```bash\nmysqldump -u root -pPASSWORD --quick --max_allowed_packet=512M --verbose --databases db_name \u003e db_name.sql\nmysqldump -u root -pPASSWORD --quick --max_allowed_packet=512M --verbose --all-databases \u003e all_databases.sql\n\n## backup directly to AWS S3\nmysqldump -u root -p --single-transaction --databases db1 db2 | gzip -9 | aws s3 cp - s3://bucket_name/$(date +%Y-%m-%d-%s)/dump.sql.gz --storage-class ONEZONE_IA\n\n## backup from REMOTE server\nmysqldump --ssl-mode=DISABLED --column-statistics=0 --host mysql.server.com -u root -pPASSWORD database \u003e database.sql\n```\nPhysical Backup:\n\n```bash\nsudo service mysql stop\nsudo cp -R /var/lib/mysql/. /root/backups\ntar cfvz /root/db-$(date +%F).tar.gz /root/backups/*\nsudo service mysql start\n```\n\nMove `/var/lib/mysql` from one server to another\n\n```bash\nsudo service mysql stop\nrsync --archive --verbose --stats --progress --human-readable /var/lib/mysql/* root@server.com:/var/lib/mysql/\nsudo service mysql start\n```\n\nStore Backups to AWS S3:\n\n```bash\naws s3 cp db_name.sql s3://bucket_name/backups/$(date +%Y-%m-%d)/db_name.sql --storage-class ONEZONE_IA\naws s3 cp db_name.sql s3://bucket_name/backups/$(date +%Y-%m-%d-%s)/db_name.sql --storage-class ONEZONE_IA\n```\n\nMore links\n- https://mysqldump.guru/\n\n## Speedtest by Ookla\n\nOne liner install for Ubuntu:\n\n```shell\ncurl -sL https://install.speedtest.net/app/cli/ookla-speedtest-1.2.0-linux-x86_64.tgz | tar -xz \u0026\u0026 chmod +x speedtest \u0026\u0026 mv speedtest /usr/local/bin/speedtest\n```\n\n### Misc\n\n\u003e dmesg -T\n\nRecently modified log files.\n\n\u003e ls -lhrt /var/log\n\n\n## Git\n\nForce current folder force to existing repository.\n\n```bash\n\ngit init\ngit add .\ngit commit -m \"First commit\"\ngit remote add origin https://github.com/Athlon1600/proxy-checker.git\n\ngit remote -v\ngit remote show origin\n\ngit push -f origin master\n```\n\n## ffmpeg\n\nInstall `ffmpeg-amd64` from static binary, and move to `/usr/local/bin`:\n\n```shell\ncurl -L https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-amd64-static.tar.xz | tar -xJf - --strip-components=1 \u0026\u0026 rm -rf GPLv3.txt manpages model readme.txt \u0026\u0026 chmod +x ffmpeg ffprobe \u0026\u0026 mv ffmpeg /usr/local/bin/ffmpeg \u0026\u0026 mv ffprobe /usr/local/bin/ffprobe\n```\n\n- https://johnvansickle.com/ffmpeg/\n\nadd subtitles to existing video\n\n```bash\nffmpeg -i subtitles.srt subtitles.ass\nffmpeg -i video.mkv -vf ass=subtitles.ass output.mp4\n```\n\n![ffmpeg](https://user-images.githubusercontent.com/1063088/148630169-22b9ff02-6a1e-420a-b216-60f4832c5f8d.png)\n\nextract exactly 1 hour of video starting at 10 seconds\n\n```bash\nffmpeg -i \"input.mp4\" -ss 00:00:10.0 -t 01:00:10.0 -c copy output.mp4\nffmpeg -i \"input.mp4\" -ss 00:00:01.0 -t 00:00:10.0 -c:v libx264 -c:a aac -strict experimental -b:a 128k output.mp4\n```\n\nCombine image and audio into video:\n\n```bash\nffmpeg -loop 1 -f image2 -r 2 -i image.jpg -i audio.mp3 -shortest result.mp4\n```\n\ndownload playlist\n\n```bash\nffmpeg -i https://example.com/index.m3u8 -c copy -bsf:a aac_adtstoasc vod.mp4\n```\n\n- https://www.proxynova.com/articles/trim-videos-using-ffmpeg\n\n### Links\n\nhttps://github.com/digitalocean/do_user_scripts\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fathlon1600%2Fuseful","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fathlon1600%2Fuseful","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fathlon1600%2Fuseful/lists"}