{"id":20268999,"url":"https://github.com/patatetom/rds4xways","last_synced_at":"2025-07-03T00:07:18.521Z","repository":{"id":41566013,"uuid":"119024492","full_name":"patatetom/rds4xways","owner":"patatetom","description":"Extract SHA1 from Reference Data Set (RDS) provided by the National Software Reference Library (NSRL) for X-Ways Forensics (or any other tool that uses SHA1).","archived":false,"fork":false,"pushed_at":"2024-10-23T08:25:15.000Z","size":65918,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-11T03:51:40.851Z","etag":null,"topics":["bash","cryptography","dfir","fingerprint","forensic","forensics","hash","nsrl","python","rds","sha1","x-ways"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/patatetom.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-01-26T08:13:11.000Z","updated_at":"2024-10-23T08:25:19.000Z","dependencies_parsed_at":"2022-09-21T04:04:22.560Z","dependency_job_id":"388e5dd6-985b-47f2-9675-ece8fbe8d021","html_url":"https://github.com/patatetom/rds4xways","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/patatetom/rds4xways","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patatetom%2Frds4xways","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patatetom%2Frds4xways/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patatetom%2Frds4xways/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patatetom%2Frds4xways/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/patatetom","download_url":"https://codeload.github.com/patatetom/rds4xways/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patatetom%2Frds4xways/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263234955,"owners_count":23434920,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bash","cryptography","dfir","fingerprint","forensic","forensics","hash","nsrl","python","rds","sha1","x-ways"],"created_at":"2024-11-14T12:22:28.320Z","updated_at":"2025-07-03T00:07:18.367Z","avatar_url":"https://github.com/patatetom.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# rds4xways\n\nExtraction of SHA1 sets from [Reference Data Set](https://www.nist.gov/itl/ssd/software-quality-group/national-software-reference-library-nsrl/about-nsrl) (RDS) provided by the [National Software Reference Library](https://www.nist.gov/itl/ssd/software-quality-group/national-software-reference-library-nsrl) (NSRL) for X-Ways Forensics _(or any other tool that uses SHA1)_.\n\n\n## Prerequisites\n\n- relatively recent Linux distribution\n- a few Gb of memory\n- at least 12 Gb of free disk space\n- `bash` to bind tools\n- `unzip` to extract to stdout\n- `python` to convert files format and preserve some space\n- `sed` to add/delete headers and more\n- `egrep`, `fgrep` and `grep` to match strings\n- `cut` to cut fields\n- `tee` to duplicate data stream\n- `pv` to monitor progress of work\n- `wc` to do some counts\n\nExcept `pv`, `tee` and `unzip`, all the above mentioned tools should be present in a Linux distribution.\nIf the use and thus the installation of `pv` and `tee` is optional, the installation of `unzip` is required.\n\n\n\n## Download and mount\n\nThe full modern RDS Version 2.77 of june 2022 is [downloaded](https://www.nist.gov/itl/ssd/software-quality-group/national-software-reference-library-nsrl/nsrl-download/current-rds) (4,2 Gb) and used.\n\nThe content of the iso image `RDS_modern.iso` is made accessible through `/media/` :\n\n```bash\nmount -o ro ./RDS_modern.iso /media/\n```\n\n\n\n## Disk space\n\nUncompaction of the archive `NSRLFile.txt.zip` would require 30 Gb of disk space for 222 113 225 records :\n\n```bash\nunzip -p /media/NSRLFile.txt.zip | pv | wc\n28,6GiO 0:11:44 [41,6MiB/s] [    \u003c=\u003e                                           ]\n222113225 223814314 30736964414\n```\n\n\u003e `wc` results can be piped and formated with `numfmt --grouping` (if installed) or with this `bash` function `bignumbers(){ printf \"%'d - \" $( cat ) | sed 's/ - $//'; }`.\n\nExtraction of strictly necessary data with reformatting will save some precious gigabytes.\n\nThe formatting is carried out by the Python script `csv2tsv` which removes all double quots and separates the fields by a tabulation, which will make it easier to process them, especially with `cut`.\n\n\n\n## Extract data\n\nThe file `NSRLFile.txt` is structured as follows :\n\n```bash\nunzip -p /media/NSRLFile.txt.zip | head -3\n\"SHA-1\",\"MD5\",\"CRC32\",\"FileName\",\"FileSize\",\"ProductCode\",\"OpSystemCode\",\"SpecialCode\"\n\"0000001FFEF4BE312BAB534ECA7AEAA3E4684D85\",\"344428FA4BA313712E4CA9B16D089AC4\",\"7516A25F\",\".text._ZNSt14overflow_errorC1ERKSs\",33,219181,\"362\",\"\"\n\"00000052A9EEEC6C8348CFB2AEA77BC1FBF8D239\",\"F46CA74CA3D89E9D3CF8D8E5CD77842D\",\"2F9CC135\",\"__DATA__mod_init_func\",772,218747,\"362\",\"\"\n```\n\nOnly fields `SHA-1` and `ProductCode` are extracted from it :\n\n```bash\nunzip -p /media/NSRLFile.txt.zip | pv | sed 1d | ./csv2tsv | cut -f 1,6 | sed 's/$/x/g' | sort -u | tee nsrl | wc\n28,6GiO 0:24:18 [20,1MiB/s] [          \u003c=\u003e                          ]\n210502817 421005634 10299328069\n```\n\n\u003e the `sort -u` command used above can quickly run out of space when the `/tmp/` folder is mounted in memory : use its `-T /somedir/` option or the `$TMPDIR` environment variable in this case.\n\n```bash\nhead -3 nsrl\n0000001ffef4be312bab534eca7aeaa3e4684d85\t219181x\n00000052a9eeec6c8348cfb2aea77bc1fbf8d239\t218747x\n00000079fd7aac9b2f9c988c50750e1f50b27eb5\t190718x\n```\n\n\u003e the final character `x` is introduced for the later use of `fgrep`.\n\n\n\n## Extract all SHA1\n\n```bash\n( echo SHA-1; cut -f 1 nsrl | sort -u ) | tee sha1 | wc\n46688293 46688293 1914219978\n```\n\n**According to X-Ways documentation** : *Now, important top tip follows : If you are creating your own hash file to import, perhaps from another forensic tool, and if you are using SHA-1, be sure to make sure your column heading in your source file is written exactly as \"SHA-1\" and not \"SHA1\" or \"SHA\" or \"SHA 1\".* ***It has to be \"SHA-1\", exactly, to be understood.***\n\n```bash\nhead -3 sha1\nSHA-1\n0000001ffef4be312bab534eca7aeaa3e4684d85\n00000052a9eeec6c8348cfb2aea77bc1fbf8d239\n```\n\nThe file `sha1` weighs 1,8 Gb for 46 688 292 records.\n\n\n\n## Extract images SHA1\n\n***This only way of doing so, based on the extension of the file name, will import SHA1 that are not necessarily those of images and leave out SHA1 of images that will not have been imported because there is no extension to the file name.***\n\nExtensions used for the main image formats are searched :\n\n```bash\nre='\\.(jpg|jpeg|jfif|jif|jp2|jpx|j2k|j2c|png|gif|bmp|svg|tif|tiff|psd|pcx|webp|psd|emf|wmf)$'\nunzip -p /media/NSRLFile.txt.zip | pv | ./csv2tsv | cut -f 1,4 | egrep $re | cut -f 1 | sort -u | tee image.sha1 | wc\n25,0GiO 0:13:06 [32,5MiO/s] [           \u003c=\u003e                         ]\n1826385 1826385 74881785\n```\n```bash\nsed -i '1i SHA-1' image.sha1\n```\n\nThe file `image.sha1` weighs 72 Mb for 1 826 385 records (~4%).\n\n\n\n## Extract Microsoft SHA1\n\nExtract manufacturer :\n\n```bash\n./csv2tsv \u003c /media/NSRLMfg.txt | grep microsoft | tee microsoft | wc\n3 9 68\n```\n```bash\ncat microsoft\n5804\tmicrosoft corporation\n608\tmicrosoft\n609\tmicrosoft game studios\n```\n\nExtract products :\n\n```bash\n./csv2tsv \u003c /media/NSRLProd.txt | cut -f 1,2,5 | grep -f \u003c( cut -f 1 microsoft | sed -e 's/^/\\t/g' -e 's/$/$/g' ) | tee microsoft.product | wc\n7399 61640 402337\n```\n```bash\ncat microsoft.product\n62\tthe compaq personal computer startup diskette\t608\n62\tthe compaq personal computer startup diskette\t608\n…\n281008\twindows 11 consumer editions april 2022\t608\n281009\twindows 11 business editions april 2022\t608\n```\n\nExtract SHA1 :\n\n```bash\n( echo SHA-1; fgrep -f \u003c( cut -f 1 microsoft.product | sed -e 's/^/\\t/g' -e 's/$/x/g' | sort -u ) nsrl | cut -f 1 | sort -u ) | tee microsoft.sha1 | wc\n10991183 10991183 450638468\n```\n\nThe file `microsoft.sha1` weighs 430 Mb for 10 991 182 records (~23%).\n\n\n\n## Extract «Windows» SHA1\n\nExtract operating systems :\n\n```bash\n./csv2tsv \u003c /media/NSRLOS.txt | cut -f 1,2,4 | grep -f \u003c( cut -f 1 microsoft | sed -e 's/^/\\t/g' -e 's/$/$/g' ) | tee microsoft.os | wc\n483 3014 16063\n```\n```bash\ncat microsoft.os\n1000\twindows nt 3\t608\n1001\twindows 8 sp1 x64\t608\n…\n994\twindows 2003 sp2 x32\t608\n995\twindows 2003 sp2 x64\t608\n```\n\nExtract products :\n\n```bash\n./csv2tsv \u003c /media/NSRLProd.txt | cut -f 1,2,4 | grep -f \u003c( cut -f 1 microsoft.os | sed -e 's/^/\\t/g' -e 's/$/$/g' ) | tee windows.product | wc\n34043 204661 1325704\n```\n```bash\ncat windows.product\n62\tthe compaq personal computer startup diskette\t56\n62\tthe compaq personal computer startup diskette\t56\n…\n281074\tfuck putin\t189\n281220\tvampire: the masquerade - bloodhunt\t189\n```\n\nExtract SHA1 :\n\n```bash\n( echo SHA-1; fgrep -f \u003c( cut -f 1 windows.product | sed -e 's/^/\\t/g' -e 's/$/x/g' | sort -u ) nsrl | cut -f 1 | sort -u ) | tee windows.sha1 | wc\n25029548 25029548 1026211433\n```\n\nThe file `windows.sha1` weighs 979 Mb for 25 029 547 records (~54%).\n\n\n\n## More SHA1 sets\n\nWith the same constraints as for images, the variable `re` can be modified to extract file names with the `.exe` extension :\n\n```bash\n# re='\\.(com|sys|dll|exe)$'\nre='\\.exe$'\nunzip -p /media/NSRLFile.txt.zip | ./csv2tsv | cut -f 1,4 | egrep $re | cut -f 1 | sort -u \u003e executable.sha1\nsed -i '1i SHA-1' executable.sha1\n```\n\nImages can be reduced to Microsoft or Windows with the use of `comm` :\n\n```bash\ncomm -1 -2 microsoft.sha1 image.sha1 \u003e image.microsoft.sha1\ncomm -1 -2 windows.sha1 image.sha1 \u003e image.windows.sha1\n```\n\n\n\n## See also\n\n- [Testing the National Software Reference Library](https://www.sciencedirect.com/science/article/pii/S1742287612000345)\n- [hashwiniso](hashwiniso.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpatatetom%2Frds4xways","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpatatetom%2Frds4xways","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpatatetom%2Frds4xways/lists"}