{"id":27097346,"url":"https://github.com/joeavanzato/differ","last_synced_at":"2025-04-06T10:39:26.382Z","repository":{"id":263097289,"uuid":"862647469","full_name":"joeavanzato/differ","owner":"joeavanzato","description":"An easy-to-use, cross-platform utility for capturing and diffing file system metadata snapshots.","archived":false,"fork":false,"pushed_at":"2024-10-08T17:36:52.000Z","size":54,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-11-16T06:47:27.752Z","etag":null,"topics":["analysis","changes","diff","diff-analysis","filesystem","snapshot"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joeavanzato.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-25T00:33:28.000Z","updated_at":"2024-10-09T17:43:25.000Z","dependencies_parsed_at":"2024-11-16T06:47:29.486Z","dependency_job_id":"819822cf-3daa-4f87-beba-56e645cd188e","html_url":"https://github.com/joeavanzato/differ","commit_stats":null,"previous_names":["joeavanzato/differ"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joeavanzato%2Fdiffer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joeavanzato%2Fdiffer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joeavanzato%2Fdiffer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joeavanzato%2Fdiffer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joeavanzato","download_url":"https://codeload.github.com/joeavanzato/differ/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247471282,"owners_count":20944153,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analysis","changes","diff","diff-analysis","filesystem","snapshot"],"created_at":"2025-04-06T10:39:25.885Z","updated_at":"2025-04-06T10:39:26.376Z","avatar_url":"https://github.com/joeavanzato.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# differ\n## File System Metadata Snapshots made Easy\n\n### What is it?\n\ndiffer is a purpose-built tool for generating and comparing ('diffing') metadata snapshots of logical drives for any necessary purpose - this may include tasks such as determining changes made by a specific piece of software, changes between patches, malware analysis/sandboxing, integrity checks, etc.\n\ndiffer works cross-platform between Linux and Windows (and probably Mac as well but I don't have a test machine for that).\n\n### Why?\ndiffer was created because I had a need to perform a configurable file system metadata snapshot and subsequent comparison and I could not identify a simple and flexible open-source tool for this task.\n\nExample Usecases Include:\n* Baselining the contents of a logical drive to identify all changes following a system/software change\n* Establishing a baseline for use in Incident Response processes and to identify changes in system files or created/deleted files following a breach\n* Identifying differences in pre- and post- metadata snapshots during dynamic malware analysis (files created, files modified, files deleted)\n* Quickly hashing files in any number of directories based on extension allow or block lists to identify any unwanted software\n* Feeding data into allow/block lists to further DFIR processes/investigations\n* Hunting for specific file-types across a system or specific directories\n\n### How to use?\n\ndiffer can be run both through command-line arguments or fed a configuration file - the easiest way to use it is to download the most recent build - this will include differ.exe and differ_config.json.\n\n\n### Configuration File\n\nTo launch differ using a configuration file, just tell it where to find it like below;\n```\ndiffer.exe -config \"configs\\full_system_snapshot.json\"\ndiffer.exe -config \"configs\\full_scan_common_malware_extensions.json\"\ndiffer.exe -config \"some\\\\path\\\\to\\\\config.json\"\n```\n\nThe full_system_snapshot configuration file is shown below - this configuration tells differ to recursively snapshot the metadata for all files starting at C:\\ with no restrictions on extensions and performing the SHA1 hash of each encountered file.  CSV export is disabled by default.\n\nOn a common personal system using a nearly-full 2 TB M.2 SSD, this type of scan will take approximately 15-30 minutes depending on CPU availability.  The type of disk drive and connection mechanism will greatly influence the speed of the snapshot due to the potential for increased read-times.  I would recommend only snapshotting required directories and extensions when possible.\n\n```json\n{\n    \"directories\": [\n        \"C:\\\\\"\n    ],\n    \"use_extension_allowlist\": false,\n    \"extension_allowlist\": [\n        \".exe\"\n    ],\n    \"use_extension_blocklist\": false,\n    \"extension_blocklist\": [\n        \".txt\"\n    ],\n    \"hash_enabled\": true,\n    \"hash_algorithm\": \"sha1\",\n    \"do_csv_export\": false\n}\n```\n\n* directories - specify a list of directories to walk recursively for snapshot generation\n* use_extension_allowlist - if true, will skip all files that do not possess an extension present in the allowlist\n* use_extension_blocklist - if true, will skip all files that have an extension present in the blocklist\n* hash_enabled - if true, will hash all included files\n* hash_algorithm - can be sha1/sha256/md5\n* do_csv_export - if true, will generate a CSV output in addition to parquet\n\nBy default, differ will store a *.parquet file in the current working directory that contains the UNIX timestamp and hostname of the snapshot, such as '1727226208164680600_DESKTOP-KH2I9H2_differ_snapshot'.\n\nEnabling CSV exports results in an immediately human-readable file being produced if the user doesn't want to convert the provided parquet to some other format - this is mainly done for storage purposes.\n\n\n### Command-Line Arguments\n\n```\n-config some_file.json : When specified, differ will ignore all other command-line arguments and rely solely on the data contained within the configuration file for execution.\n-directory \"C:\\\\\" : Tells differ the directory to use as the starting point for a recursive file-walk snapshot\n-csv : Tells differ to also produce CSV output in addition to the default Parquet\n-hash md5 / -hash sha1 / -hash sha256 : Tells differ to also compute the hash of all scanned files using one of the specified algorithms\n-compare file1,file2 : Tells differ to 'diff' the two provided files - differ will automatically attempt to determine which one is older/newer based on the file naming format\n```\n\n### Comparing Snapshots\nTo compare two separate snapshots, use the '-compare' argument as follows:\n```\ndiffer.exe -compare 1727205513801559400_DESKTOP-KH2I9H2_differ_snapshot.parquet,1727224094973553500_DESKTOP-KH2I9H2_differ_snapshot.parquet\n```\ndiffer will perform a few different checks when looking for changes:\n* Files with the same path, name and extension but that...\n  * Have different hashes (modification)\n  * Have different modification times (modification)\n  * Have different file sizes (modification)\n* Files that do not appear in the older snapshot but do appear in the newer one (creation)\n* Files that do not appear in the newer snapshot but do appear in the previous one (deletion)\n\nAll differences are written to a CSV output file (snapshot_diff.csv) in the current working directory.\n\nBe aware there are caveats here - if a file is moved between two directories, we will count that as both a deletion and creation since we are not doing 'hash-scanning' across the entire snapshot at this time.\n\n\n### Common Extension Lists\nFor convenience, a few configuration files are provided inside the configs directory for common use-cases.  They are detailed below;\n\n* full_system_snapshot_(win|linux).json\n  * Recursively snapshot an entire drive starting at C:\\ (or \\\\ on Linux) with no restrictions on extension and also performing SHA1 hash.\n* quick_common_malware_hashscan.json\n  * Contains common directories where malware often lives and an extension allow-list for the most common file types encountered during incidents.\n* full_scan_common_malware_extensions.json\n  * Same as above but will scan for common malware extensions across the entire logical drive starting at C:\\.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoeavanzato%2Fdiffer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoeavanzato%2Fdiffer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoeavanzato%2Fdiffer/lists"}