{"id":13491883,"url":"https://github.com/C3n7ral051nt4g3ncy/cURL_for_OSINT","last_synced_at":"2025-03-28T08:33:34.381Z","repository":{"id":57954317,"uuid":"528185875","full_name":"C3n7ral051nt4g3ncy/cURL_for_OSINT","owner":"C3n7ral051nt4g3ncy","description":"cURL Tool Usage for OSINT (Open-Source Intelligence)","archived":false,"fork":false,"pushed_at":"2023-02-04T09:25:12.000Z","size":190,"stargazers_count":237,"open_issues_count":0,"forks_count":23,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-26T23:08:24.628Z","etag":null,"topics":["curl","grep","intelligence","intelligence-gathering","open-source","osint"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/C3n7ral051nt4g3ncy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":"C3n7ral051nt4g3ncy","patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2022-08-23T22:42:00.000Z","updated_at":"2025-03-18T14:48:34.000Z","dependencies_parsed_at":"2023-02-18T15:16:09.680Z","dependency_job_id":null,"html_url":"https://github.com/C3n7ral051nt4g3ncy/cURL_for_OSINT","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/C3n7ral051nt4g3ncy%2FcURL_for_OSINT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/C3n7ral051nt4g3ncy%2FcURL_for_OSINT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/C3n7ral051nt4g3ncy%2FcURL_for_OSINT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/C3n7ral051nt4g3ncy%2FcURL_for_OSINT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/C3n7ral051nt4g3ncy","download_url":"https://codeload.github.com/C3n7ral051nt4g3ncy/cURL_for_OSINT/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245997023,"owners_count":20707386,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["curl","grep","intelligence","intelligence-gathering","open-source","osint"],"created_at":"2024-07-31T19:01:01.113Z","updated_at":"2025-03-28T08:33:34.342Z","avatar_url":"https://github.com/C3n7ral051nt4g3ncy.png","language":null,"funding_links":["https://github.com/sponsors/C3n7ral051nt4g3ncy"],"categories":["Others"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e \u003cimg width=\"233\" height=\"199\" src=\"https://user-images.githubusercontent.com/104733166/186285439-d9463805-3354-429a-baf4-3e960826028f.gif\"\u003e\u003cp/\u003e\n\n\u003cbr\u003e\n\n# cURL for OSINT\n### cURL Tool usage (with Grep) for OSINT (Open-Source Intelligence)\n\n\u003cbr\u003e\n\n![curl-logo](https://user-images.githubusercontent.com/104733166/186279152-3ca718a8-966c-48ac-b1f8-061435f57af6.svg)\n\u003cbr\u003e\n\u003cbr\u003e\n\u003cbr\u003e\n\n# Background Information\ncURL is a command line tool created in 1998 by [Daniel Stenberg](https://twitter.com/bagder), a Swedish Developper/Programmer.\n\u003cbr\u003e\nDaniel Stenberg's website: https://daniel.haxx.se\n\u003cbr\u003e\nThe latest version of the Tool and the code is open source right here on GitHub: https://github.com/curl/curl\n\n\u003cbr\u003e\n\n# What exactly does cURL do?\nI am not a cURL expert and it's quite difficult giving an exact explanation so I will try and give the most basic explanation.\ncURL means `Client URL` and you already know from above that it's a command line tool, so it's basically a tool that permits/enables data transfer over many network protocols (HTTPS, HTTP, FTP, it also supports SSL Certificates). The tool communicates with a web server with a URL and tells the server the data to be sent or received. (yes you read that correctly, **the tool can do both** 🤓)\n\n\u003cbr\u003e\n\n# Difficulty\ncURL is fairly easy to use, and if you have never opened your Terminal before, now is the time to do so 🤩.\n\n\u003cbr\u003e\n\n# Installation\nI simply installed cURL with [Homebrew](https://formulae.brew.sh/formula/curl)\n\u003cbr\u003e\nInstall Command: \n```$ brew install curl```\n\u003cbr\u003e\n\nOnce you have installed cURL, you can also install Grep: (more info on `grep` later on)\n\u003cbr\u003e\nInstall Command: \n```$ brew install grep```\n\n\u003cbr\u003e\n\n# Commands\nNow that you have everything installed, we are good to go ✅, so let's have a look at some commands for OSINT (Open-Source Intelligence).\n\u003cbr\u003e\n\u003cbr\u003e\n\u003cbr\u003e\n\u003cbr\u003e\n\n\n\n# IP Address Information\n\n\u003cimg width=\"233\" height=\"133\" src=\"https://user-images.githubusercontent.com/104733166/186291544-9cd73528-9145-4ddc-ac42-a3d604085ad0.jpg\"\u003e\n\n### Finding your own IP\nCommands for finding the IP address of the device you are on: (type each command separately, **each command does the same thing**)\n\n ```\n $ curl api.ipify.org\n ```\n\n ```\n $ curl ipinfo.io/ip\n ```\n\n ```\n $ curl ifconfig.me\n ```\n\u003cbr\u003e\nCommands to get your IPV4 and IPV6 addresses:\n\n ```\n $ curl -4 icanhazip.com\n ```\n\n ```\n $ curl -6 icanhazip.com\n ```\n \n \u003cbr\u003e\n\n\n\n\nExample below, we can see my IP (ProtonVPN) 😈 :\n\n\u003cimg width=\"333\" src=\"https://user-images.githubusercontent.com/104733166/186289031-2dbec60b-de7a-4740-9452-98c7f8b40164.png\"\u003e\n\n\u003cbr\u003e\n\n### Get more detailed information on your IP:\n```\n$ curl ipinfo.io\n```\nExample below, we can see that there is a lot more information on my IP such as city, country, organization, timezone, GPS coordinates:\n\n\u003cimg width=\"433\" src=\"https://user-images.githubusercontent.com/104733166/186290638-42617392-581a-4639-9275-607bcab2481f.png\"\u003e\n\nIt also works well with this command (.json format)\n\n```\n$ curl ipinfo.io/json\n```\nBelow is another command to make sure we really get the `JSON` and not the homepage `html`\n\n\n```\n$ curl -H \"Accept: application/json\" ipinfo.io\n```\n\n`-H takes a single parameter of an extra header to include in the request and Accept adds the Accept request headers to the request`\n\n\u003cbr\u003e\n\u003cbr\u003e\n\nLet's do more tests using Google's IP address.\u003cbr\u003e\n\nBelow are the commands when you know the IP address of the target , and as an example below, you want to know the **city**.\n\n```\n$ curl ipinfo.io/8.8.8.8/city\n```\n\u003cbr\u003e\nResults:\n\u003cimg width=\"366\" src=\"https://user-images.githubusercontent.com/104733166/186506023-a37ef2d1-fae7-47e7-a732-bc5b999ef898.png\"\u003e\n\u003cbr\u003e\n\nGet the **Country**:\n```\n$ curl ipinfo.io/8.8.8.8/country\n```\n\u003cbr\u003e\n\nGet the **Organization**:\n```\n$ curl ipinfo.io/8.8.8.8/org\n```\n\n\u003cbr\u003e\n\u003cbr\u003e\n\u003cbr\u003e\n\n\n# Whois Commands\n\n\u003cimg width=\"133\" src=\"https://user-images.githubusercontent.com/104733166/186533300-4d97e784-00e1-4b0b-8f84-2f40abf6cb9f.png\"\u003e\n\nI particularly like this easy command to get Whois information, this is not my work, it's the work of [Sector035](https://sector035.nl) and was shared on an article called [dial cURL for Content on OSINTCuriou.us](https://osintcurio.us/2019/06/25/dial-curl-for-content/)\n\n```\n$ curl cli.fyi/inputdomainhere\n```\n\u003cbr\u003e\n\nLet's test with the website of the French Presidential Palace:\n\u003cbr\u003e\n```\n$ curl cli.fyi/elysee.fr\n```\nPerfect results: 👍\n\n\u003cimg width=\"633\" src=\"https://user-images.githubusercontent.com/104733166/186533013-2716e467-0e82-4fb9-87bc-003cd0a355d1.png\"\u003e\n\n\u003cbr\u003e\n\nI will add my personal touch to this command, with some added color, the color doesn't show as a big difference with these results, but wait until you see how it works further on below with messy results and how the `jq` command can really make a difference.\n\n\n\u003cbr\u003e\n\n```\n$ curl cli.fyi/elysee.fr | jq --color-output\n```\n\n\u003cbr\u003e\n\n\u003cimg width=\"333\" alt=\"Screen Shot 2022-08-25 at 00 20 53\" src=\"https://user-images.githubusercontent.com/104733166/186533988-ea8abfd2-9546-4138-b016-8f75761cfba8.png\"\u003e\n\n\u003cbr\u003e\n\nI really like this command below:\n\u003cbr\u003e\n\n```\ncurl -s http://ip-api.com/json/[input Domain or IP address] | jq -r .as\n```\n\nWith this API you can input either a `domain` or an `IP address`, the commands after the IP or domain are `jq` which we will talk about further on below, and `-r` which means range, you can use `-r` or `--range`,  and depending on the information you require about the domain, you can put: .org or .isp or .city or .country or .RegionName or .as. \n\n`-s` just means silent (hiding progress)\n\nLet's try this command:\n\n```\n$ curl -s http://ip-api.com/json/visitiran.ir | jq -r .city\n```\n\u003cbr\u003e\n\n\u003cimg width=\"533\" src=\"https://user-images.githubusercontent.com/104733166/186725342-ec8d06ff-765a-4c6e-a17b-7f4662377b57.png\"\u003e\nWe can see the city is: `Tehran` for the domain we tested on.\n\n\u003cbr\u003e\n\n\u003cbr\u003e\n\n\n# Whois History\n\nThere are many sites you can get a `Free API Key` from.\u003cbr\u003e\nOne site I like is [whoisfreaks](https://whoisfreaks.com), just head over and join and you will get your API Key with 100 API calls.\n\nThis is a command for historical whois information:\n\n\u003cbr\u003e\n\n```\n$ curl -s  https://api.whoisfreaks.com/v1.0/whois\\?apiKey\\=API_Key_here\\\u0026whois\\=historical\\\u0026domainName\\=cnn.com | jq --color-output\n```\n\nDon't forget that you can also use `grep` if you just want to see email addresses and not the full information.\n\n\u003cbr\u003e\n\n\u003cimg width=\"1233\"  src=\"https://user-images.githubusercontent.com/104733166/187009377-9a3fff5b-fd0d-4e4b-845b-22be2b175f8a.png\"\u003e\n\n\u003cbr\u003e\n\n\u003cbr\u003e\n\n# Getting an email from a Domain with cURL + Grep\n\n\u003cimg width=\"333\" src=\"https://user-images.githubusercontent.com/104733166/186535617-dba770e5-72c2-4e7b-9854-2b49a821cb9f.jpg\"\u003e\n\nGrep Website: https://www.gnu.org/software/grep/manual/grep.html\n\n### What is Grep?\ngrep prints lines that contain a match for one or more patterns | Regex (Regular Expressions) \u003cbr\u003e\n\nWhy is it called grep?: \n- g/re/p: **globally search for a regular expression and print matching lines**\n\n\u003cbr\u003e\n\nThis is not my work, this is a fantastic email scraper with a cURL command, grep + regex string. \u003cbr\u003e\nIt was shared by [@Spiderfoot](https://twitter.com/spiderfoot): \n\n```\n$ curl -s [YOUR DOMAIN HERE] | grep -E -o \"\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,6}\\b\"\n```\n\nLet's honor Spiderfoot by taking their domain as an example:\n\n```\n$ curl -s https://www.spiderfoot.net | grep -E -o \"\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,6}\\b\"\n```\n\u003cbr\u003e \n\n\u003cimg width=\"1109\" src=\"https://user-images.githubusercontent.com/104733166/186537612-08a532b2-3456-4dd9-bd9a-72ecebcfac69.png\"\u003e\n\n\u003cbr\u003e\n\nGreat result!!! I timed it, it took less than 2 seconds.\u003cbr\u003e \nIt's quite easy becoming a fan of using cURL and Grep and learning about regex (regular expressions) \n\n\u003cbr\u003e\n\nI enjoyed reading the Spiderfoot explanation below in regards to this cURL + grep command:\n\n\u003cbr\u003e\n\n\n\u003eThe curl command simply prints the HTTP response to https://www.spiderfoot.net, and the grep command contains a regular expression (regex) that finds and prints all emails within the source code. You can see that in this case, it found one, support@spiderfoot.net. A similar process could be followed to extract other information such as phone numbers, names, hostnames, etc.\n\n\u003cbr\u003e\n\nLet's take a minute to analyze another Regex string that works for discovering emails:\n\n\u003cbr\u003e\n\nHow an email is structured: support[1]@[2]spiderfoot[3].[4]com[5]\n\n\u003cbr\u003e\n\n`[a-zA-Z0-9._-]\\+@[a-zA-Z]\\+.[a-zA-Z]\\+`\n\n- [1] [a-zA-Z0-9._-] The text/username before the @ can be lowercase a-z, uppercase A-Z, numbers 0-9, a period, underscore or hyphen\n- [2] [\\+@] Obviously, this can only be an @ sign, `\\+` this just means we want to combine the pattern without breaking it\n- [3] [a-zA-Z] Same as number 1, we can see here that this regex string does not include numbers, you could have numbers in a domain name, I prefer the string of Spiderfoot.\n- [4] [\\+.] We know there is always a period/dot `.` before a `com` or `net` or `org` or `us` or `gov`, whatever 🤓\n- [5] [a-zA-Z] Lowercase or Uppercase letter for com or net/org/us/io/gov/app/it (as above).\n\n\u003cbr\u003e\n\nRemember also that the `-s` command that Spiderfoot used means silent, and you can do `-s` or `--silent` to implement it, it does not show the progress meter or error messages but will still output the data ✔️\n\n\u003cbr\u003e\n\n\u003cbr\u003e\n\n\n# Domain Commands\n\u003cimg width=\"133\" src=\"https://user-images.githubusercontent.com/104733166/186541301-e800e802-9980-4f38-9854-c639d8d148ff.png\"\u003e\n\n\u003cbr\u003e\n\n```\n$ curl https://www.domain.com\n```\nAbove, this is the most basic of commands for beginners and will show the full response of the web server hosting that domain, the information returned is long and messy but can be very useful.\n\n*Notice that I put `https://www.` in front of the domain name, if I don't, I won't get a correct response, the response will be that the document has moved permanently.\n\nYou can avoid typing `https://www.` and can avoid the response of a `permanent move` with this command:\n\n```\n$ curl -L github.com\n```\n`-L, or --location`= **Follow redirects**\n\n\u003cbr\u003e\n\nView Headers Response with the `--head` command:\n\n```\n$ curl --head https://github.com\n```\n\nor \n\n```\n$ curl -I https://github.com\n```\n\u003cbr\u003e\n\n## Finding Subdomains\nSearching for `Tracelabs.org` subdomains.\n\nI did this simple command by looking around at how `crt.sh` works:\n\n```\n$ curl -s https://crt.sh/?q=tracelabs.org\u0026output=json | jd --color-output\n```\n\n\nI added `\\\u0026output\\=json` at the end of the link, and as I mentionned previously, adding  `| jq --color-output` puts some nice color in the results and `jq` seems to act like `pprint` in python and sorts results, making them look much neater. \n\nI could also have added `-o TL_results.txt` if I just wanted the results in a .txt file, and the `-o` command lets us choose the name of the file and format of the file.\n\n\u003cbr\u003e\n\n\u003cimg width=\"1145\" src=\"https://user-images.githubusercontent.com/104733166/186715598-6ec5cb66-2bb4-4b0b-92e5-f5a439518a7a.png\"\u003e\n\n\u003cbr\u003e\n\nResults in .txt:\n\n\u003cimg width=\"102\" src=\"https://user-images.githubusercontent.com/104733166/186715803-5ce6c1ab-7e96-4ddd-83ef-f79b90180427.png\"\u003e\n\n\u003cbr\u003e\n\n\u003cbr\u003e\n\n\n\n# Downloading files during the investigation\n\nLet's take this Interpol PDF file as an example that I found with some Dorking: https://rm.coe.int/3148-3-2-eurojust-presentation-interpol-approach-ukim-v2/1680791601\n\n- -o will save your file as you name it, meaning you need to choose a name\n- -O will save the file as it was called on the server\n\n```\n$ curl -o interpol.pdf https://rm.coe.int/3148-3-2-eurojust-presentation-interpol-approach-ukim-v2/1680791601\n\n```\n```\n$ curl -O https://rm.coe.int/3148-3-2-eurojust-presentation-interpol-approach-ukim-v2/1680791601\n```\nFantastic and fast! here is my downloaded file as I wanted it with the name I chose:\n\n\u003cimg width=\"151\" src=\"https://user-images.githubusercontent.com/104733166/186547270-fe5164ba-4ff6-4a56-86a9-28a010a90883.png\"\u003e\n\n\u003cbr\u003e\n\n\nBelow is a command I really like 🥳, you can request a progress bar for large files that take a long time to download:\n\n\u003eUse the command `--progress-bar` or  `-#`, they both do the same thing.\n\n```\n$ curl --progress-bar -o interpol.pdf https://rm.coe.int/3148-3-2-eurojust-presentation-interpol-approach-ukim-v2/1680791601\n```\n\u003cbr\u003e\n\n\u003cimg width=\"1589\" src=\"https://user-images.githubusercontent.com/104733166/186547994-65d017ce-b5a4-48a8-a462-5ec4b38eb3ae.png\"\u003e\n\n\u003cbr\u003e\n\nYou can also download multiple files at the same time which is awesome:\n\n```\n$ curl -O url -O url -O url \n```\n\u003cbr\u003e\n\nWe downloaded the file using cURL, how about finding emails on it without even opening the file?\n\ncURL and grep are great working together, but for this one, OUT goes cURL and IN comes grep! 👊\n\n\u003cbr\u003e\n\n\u003cbr\u003e\n\nLet's use the below command to see if the interpol file has some emails on it:\n\n```\n$ grep -a -E -o \"\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,6}\\b\" interpol.pdf\n```\n\nI put the `-a` command to force grep to treat the binary file as text, if I don't put it there I keep getting `Binary File matches`, without an email being printed.\n\n- `-E`= extended regular expression (regex)\n- `-o`= only matching | Meaning in this case Printing email addresses only\n\n\u003eThe email was found in a matter of milliseconds, it didn't even reach 1 sec, the PDF is over 20 pages and the email is on the last page so it would have taken much longer to find it, this proves how useful cURL and grep are.\n\n\u003cbr\u003e\n\n\u003cimg width=\"697\" alt=\"Screen Shot 2022-08-25 at 03 36 10\" src=\"https://user-images.githubusercontent.com/104733166/186554307-2542ef70-198a-41af-bc14-8befbdb629c5.png\"\u003e\n\n\u003cbr\u003e\n\nHow about searching through domain source code for exact words or phrases? 🕵🏻‍♂️ \u003cbr\u003e\nA lot can be found within source code, things can be intentionally hidden there, and if you compete in CTF events, the below command can help you to find something quick within the source code. \u003cbr\u003e\n\n\u003eThis is a fantastic example of combining cURL and grep:\n\nI hid a quote from Mr.Robot (fsociety) in the source code, this is not visible on the website, and manually going through source code can be quite annoying and time consuming.\n\nLet's use the `grep -i` command. \n\n`grep -i` enables the command to be case insensitive, meaning grep will search for uppercase or lowercase strings that matches the word or phrase.\n\n\u003cbr\u003e\n\nCommand:\n\n```\n$ curl -L tacs-sys.com | grep -i illusion\n```\nBingo! \n\n\u003cimg width=\"633\" src=\"https://user-images.githubusercontent.com/104733166/186700428-34fe951d-7877-40e7-9f46-223ef1d13ce4.png\"\u003e\n\n\u003cbr\u003e\n\n\u003cbr\u003e\n\n# Google UA Analytics code finder\n\nI created this one in a simple way, I know that I have Google Analytics.\u003cbr\u003e\nI will need to use cURL and grep together, and these 2 make a deadly team. 💪\n\n1. I want a command that can return a Google UA code fast from any website I choose\n2. I want to track the progress within my terminal\n3. I don't want to have to type `https://www.` before the domain name\n4. I will need to use grep and to create a regex string: \"UA\\-[0-9]+\", I know that all Google Analytics codes start with UA, a hyphen after the UA, and it can only be numbers after the hyphen and never letters, so pretty easy right? \n5. -E : Treats pattern as an extended regular expression\n6. -o : Prints only matched parts\n\n```\n$ curl --progress-bar -L tacs-sys.com | grep -E -o \"UA\\-[0-9]+\"\n```\n\n*Keep in mind that this string I created would work fine too: `\"UA+-[0-9]+\"`\n\n\u003cbr\u003e\n\n\u003cimg width=\"1339\" src=\"https://user-images.githubusercontent.com/104733166/186732128-6193749c-2692-4731-b7f6-0907126bed25.png\"\u003e\n\u003cbr\u003e\n\nWith the UA analytics code found, you can then use a reverse analytics site to check if the UA code is linked to another site: https://dnslytics.com/reverse-analytics\n\n\u003cbr\u003e\n\n\u003cbr\u003e\n\n# Reddit\n\n\u003cimg width=\"66\" src=\"https://user-images.githubusercontent.com/104733166/186503090-6a749cca-0302-4908-a197-4042a7a8aed6.png\"\u003e\n\nA command we saw higher up for searching for IP addresses would also work nicely for `reddit conversations`, **just add** `.json` at the end of the Reddit conversation link.\n\n\u003cbr\u003e\n\nExample: https://www.reddit.com/r/OSINT/comments/wvnko3/how_to_extract_an_email_from_a_google_maps[.json]\n\n```\n$ curl -H \"Accept: application/json\" https://www.reddit.com/r/OSINT/comments/wvnko3/how_to_extract_an_email_from_a_google_maps.json\n```\n\n\u003cbr\u003e\n\n\u003cimg width=\"2199\" src=\"https://user-images.githubusercontent.com/104733166/186504907-8cb16ad3-0213-420e-b13a-5758cb9dfa50.png\"\u003e\n\n\u003cbr\u003e\n\nYou can probably see that the above results look a little messy, the below command will put some color into the results and will put the `json` in a tidy format, this is thanks to the `jq`  command which acts like `pprint` in Python.\n\n\u003cbr\u003e\n\n```\n$ curl -H \"Accept: application/json\" https://www.reddit.com/r/OSINT/comments/wvnko3/how_to_extract_an_email_from_a_google_maps.json | jq --color-output\n```\n\n\u003cbr\u003e\n\nAs you can see from the command above, if you add  `| jq --color-output`, you get tidy results and great color which makes it a lot easier for our eyes.\n\n\u003cbr\u003e\n\n\u003cimg width=\"633\" src=\"https://user-images.githubusercontent.com/104733166/186518706-57748f3c-6d73-46b8-9020-cbf2952515d4.png\"\u003e\n\n\u003cbr\u003e\n\nWe can see the `unix timestamp` on the photo, it's then quite easy to convert this into a date and time as there is many free online converters.\n\n\u003cbr\u003e\n\n\n## What if you want all the information in a file?\nMaybe you want all the information for your investigation in a file.\u003cbr\u003e\nSimply add the `-o` command, meaning `output`, and add the file name you choose.\n\n\u003cbr\u003e\n\nCommand Example for the reddit post to get the information output into a file: 📁\n\n\u003cbr\u003e\n\n```\n$ curl -H \"Accept: application/json\" https://www.reddit.com/r/OSINT/comments/wvnko3/how_to_extract_an_email_from_a_google_maps.json -o reddit.json\n```\n\n\u003cbr\u003e\n\nBelow is the result in `.json` format, you can request the output file in `.txt`, `.html`, and more formats, it's up to you.\n\n\u003cimg width=\"797\" src=\"https://user-images.githubusercontent.com/104733166/186527850-5a75a828-fc16-4771-9742-475ffe233c9e.png\"\u003e\n\n\u003cbr\u003e\n\n\u003cbr\u003e\n\n# Various other Commands\nI learned these commands from Substack.\u003cbr\u003e\n\nYou can download sed with this command:\n\n```\nbrew install gnu-sed\n```\n\u003cbr\u003e\n\nI found the 2003 CIA World Factbook in a .txt file.\u003cbr\u003e\n\nThis is one pretty awesome command that I will use below and by using cURL combined with sed we can do a lot of interesting stuff.\u003cbr\u003e\nSED stands for stream editor.\n\nWikipedia Def:\n\u003eSed helps in operations like selecting the text, substituting text, modifying an original file, adding lines to text, or deleting lines from the text.\n\n\u003cbr\u003e\n\nCommand that will print every word in the book on a new line: \u003cbr\u003e\n\n```\n$ curl -s https://www.gutenberg.org/files/27558/27558.txt | gsed -r 's/\\s+/\\n/g'\n```\nIf you are on mac and downloaded `gnu-sed`, you need to use the `gsed` command and not `sed`.\n\n\u003cbr\u003e\n\nTry this command also:\n\n```\n$ echo Tactical OSINT Analyst loves to spend time on GitHub |  gsed -r 's/\\s+/\\n/g'\n```\n\n\u003cbr\u003e\n\n\u003cimg width=\"633\" src=\"https://user-images.githubusercontent.com/104733166/186768753-2ff1cbc9-6858-4b1a-bdfb-e56f7c406d5b.png\"\u003e\n\n\u003cbr\u003e\n\nAs you can see, each word is on a new line thanks to SED. SED is basically acting like a white space detector, meaning every time there is white space in between a word, SED will treat that as a new line.\n\n\u003cbr\u003e\n\nLet's bring grep into the picture, let's request to have the word China on a new line every time it's in the book.\n\n\u003cbr\u003e\n\n```\ncurl -s https://www.gutenberg.org/files/27558/27558.txt | gsed -r 's/\\s+/\\n/g' | grep -i china\n```\n\n\u003cbr\u003e\n\n\u003cimg width=\"633\" src=\"https://user-images.githubusercontent.com/104733166/186770745-3154fd21-7f70-400e-b718-33315e3e8ab2.png\"\u003e\n\n\u003cbr\u003e\n\nThis brought back so many results.\u003cbr\u003e\nNow I would like to know how many times the word China was mentionned in the 2003 CIA World Factbook:\n\n\u003cbr\u003e\n\n```\n$ curl -s https://www.gutenberg.org/files/27558/27558.txt | gsed -r 's/\\s+/\\n/g' | grep -i china | wc -l\n```\n\u003cbr\u003e\n\nOn the above command, `| wc -l` was added, meaning we can get a word count returned. \u003cbr\u003e\nTotal times the word China was mentionned in the book is: **916**\n\n\u003cbr\u003e\n\n\u003cimg width=\"833\" src=\"https://user-images.githubusercontent.com/104733166/186771768-427fbbf0-5574-4b91-8992-935bf903667d.png\"\u003e\n\n\n\n\n\u003cbr\u003e\n\n\u003cbr\u003e\n\n\u003cbr\u003e\n\n# cURL commands list\n\nCheat sheet by [Daniel Stenberg (Haxx)](https://daniel.haxx.se/)\n\n\n![cURL Cheat Sheet](https://user-images.githubusercontent.com/104733166/186732535-1fcbf63c-a984-435b-9b52-5112ad45f611.png)\n\n\u003cbr\u003e\n\n\u003cbr\u003e\n\n\u003cbr\u003e\n\n\n# grep commands list\n\n\n| Command | Or                    | Interpretation                               |\n| --------|:---------------------:| --------------------------------------------:|\n| -E      | --extended-regexp     | Extended Regular Expressions (Regex).        |\n| --help  | --help                | Print a usage message                        |\n| -V      | --version             | Print the version number of grep             |\n| -e      | --regexp=patterns     | Use patterns for Matching                    |\n| -a      | --text                | Process a binary file as if it were text     |\n| -f      | --file=FILE           | Get patterns from FILE                       |\n| -o      | --only-matching       | Print only matched parts of matching lines   |\n| -r      | --recursive           | process files in directory, recursively.     |\n| -F      | --fixed-strings       | Interpret patterns as fixed strings.         |\n| -i      | --ignore-case         | Ignore case distinctions in patterns.        |\n| -w      | --word-regexp         | Lines containing matches for whole words.    |\n| -x      | --line-regexp         | Matches exacrlt the whole lines              |\n| -c      | --count               | Print count of selected lines per file       |\n| -s      | --no-messages         | Suppress error messages                      |\n| -q      | --silent              | Quiet, do not write anything to output       |\n| -G      | --basic-regexp        | patterns as basic regular expressions        |\n| -v      | --invert-match        | Invert sense of matching (non matching lines)|\n| -m      | --max-count=NUM       | Stop after the first num selected lines      |\n| -n      | --line-number         | Print line number with output lines          |\n| -H      | --with-filename       | Print file name with output lines            |\n| -L      | --files-without-match | Print names of files with no selected lines  |\n| -l      | --files-with-match    | Print names of files with selected lines     |\n\n\u003cbr\u003e\n\nMore information: https://www.gnu.org/software/grep/manual/grep.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FC3n7ral051nt4g3ncy%2FcURL_for_OSINT","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FC3n7ral051nt4g3ncy%2FcURL_for_OSINT","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FC3n7ral051nt4g3ncy%2FcURL_for_OSINT/lists"}