{"id":13698658,"url":"https://github.com/Yelp/osxcollector","last_synced_at":"2025-05-04T03:31:44.356Z","repository":{"id":19374659,"uuid":"22615096","full_name":"Yelp/osxcollector","owner":"Yelp","description":"A forensic evidence collection \u0026 analysis toolkit for OS X","archived":true,"fork":false,"pushed_at":"2019-06-19T15:40:49.000Z","size":788,"stargazers_count":1885,"open_issues_count":18,"forks_count":242,"subscribers_count":122,"default_branch":"master","last_synced_at":"2025-04-23T13:58:48.124Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://yelp.github.io/osxcollector","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Yelp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-08-04T18:25:29.000Z","updated_at":"2025-04-17T03:17:08.000Z","dependencies_parsed_at":"2022-08-25T22:41:11.673Z","dependency_job_id":null,"html_url":"https://github.com/Yelp/osxcollector","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yelp%2Fosxcollector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yelp%2Fosxcollector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yelp%2Fosxcollector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yelp%2Fosxcollector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Yelp","download_url":"https://codeload.github.com/Yelp/osxcollector/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252283631,"owners_count":21723512,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T19:00:51.574Z","updated_at":"2025-05-04T03:31:43.215Z","avatar_url":"https://github.com/Yelp.png","language":"Python","funding_links":[],"categories":["Challenges","Python","Python (1887)","Tools","Security","事件响应工具","Hunting","Incident Response tools"],"sub_categories":["OS X Forensics","macOS 10.15 Catalina Setup","事件证据搜集（取证）","Evidence collection"],"readme":"![osxcollector](https://raw.githubusercontent.com/Yelp/osxcollector/master/osx-github.png)\n\n[![Build Status](https://travis-ci.org/Yelp/osxcollector.svg)](https://travis-ci.org/Yelp/osxcollector)\n[![PyPI](https://img.shields.io/pypi/v/osxcollector.svg)](https://pypi.python.org/pypi/osxcollector)\n\n# OSXCollector Manual\nOSXCollector is a forensic evidence collection \u0026 analysis toolkit for OSX.\n\n#### Forensic Collection\nThe collection script runs on a potentially infected machine and outputs a JSON file that describes the target machine. OSXCollector gathers information from plists, SQLite databases and the local file system.\n\n#### Forensic Analysis\nArmed with the forensic collection, an analyst can answer the question like:\n* _Is this machine infected?_\n* _How'd that malware get there?_\n* _How can I prevent and detect further infection?_\n\nYelp automates the analysis of most OSXCollector runs converting its output into an easily readable and actionable summary of _just the suspicious stuff_. Check out [OSXCollector Output Filters project](https://github.com/Yelp/osxcollector_output_filters) to learn how to make the most of the automated OSXCollector output analysis.\n\n## Performing Collection\n[`osxcollector.py`](https://raw.githubusercontent.com/Yelp/osxcollector/master/osxcollector/osxcollector.py) is a single Python file that runs without any dependencies on a standard OSX machine. This makes it really easy to run collection on any machine - no fussing with brew, pip, config files, or environment variables. Just copy the single file onto the machine and run it:\n\n`sudo osxcollector.py` is all it takes.\n\n```shell\n$ sudo osxcollector.py\nWrote 35394 lines.\nOutput in osxcollect-2014_12_21-08_49_39.tar.gz\n```\n\nIf you have just cloned the GitHub repository, `osxcollector.py` is inside `osxcollector/` directory, so you need to run it as:\n\n```shell\n$ sudo osxcollector/osxcollector.py\n```\n\n**IMPORTANT:** please make sure that `python` command on your Mac OS X machine uses the default Python interpreter shipped with the system and is not overridden, e.g. by the Python version installed through brew. OSXCollector relies on a couple of native Python bindings for OS X libraries, which might be not available in other Python versions than the one originally installed on your system.\nAlternatively, you can run `osxcollector.py` explicitly specifying the Python version you would like to use:\n\n```shell\n$ sudo /usr/bin/python2.7 osxcollector/osxcollector.py\n```\n\nThe JSON output of the collector, along with some helpful files like system logs, has been bundled into a .tar.gz for hand-off to an analyst.\n\n`osxcollector.py` also has a lot of useful options to change how collection works:\n* `-i INCIDENT_PREFIX`/`--id=INCIDENT_PREFIX`:\n  Sets an identifier which is used as the prefix of the output file. The default value is `osxcollect`.\n  ```shell\n  $ sudo osxcollector.py -i IncontinentSealord\n  Wrote 35394 lines.\n  Output in IncontinentSealord-2014_12_21-08_49_39.tar.gz\n  ```\n  Get creative with incident names, it makes it easier to laugh through the pain.\n\n* `-p ROOTPATH`/`--path=ROOTPATH`:\n  Sets the path to the root of the filesystem to run collection on. The default value is `/`. This is great for running collection on the image of a disk.\n  ```shell\n  $ sudo osxcollector.py -p '/mnt/powned'\n  ```\n\n* `-s SECTION`/`--section=SECTION`:\n  Runs only a portion of the full collection. Can be specified more than once. The full list of sections and subsections is:\n  * `version`\n  * `system_info`\n  * `kext`\n  * `startup`\n    * `launch_agents`\n    * `scripting_additions`\n    * `startup_items`\n    * `login_items`\n  * `applications`\n    * `applications`\n    * `install_history`\n  * `quarantines`\n  * `downloads`\n    * `downloads`\n    * `email_downloads`\n    * `old_email_downloads`\n  * `chrome`\n    * `history`\n    * `archived_history`\n    * `cookies`\n    * `login_data`\n    * `top_sites`\n    * `web_data`\n    * `databases`\n    * `local_storage`\n    * `preferences`\n  * `firefox`\n    * `cookies`\n    * `downloads`\n    * `formhistory`\n    * `history`\n    * `signons`\n    * `permissions`\n    * `addons`\n    * `extension`\n    * `content_prefs`\n    * `health_report`\n    * `webapps_store`\n    * `json_files`\n  * `safari`\n    * `downloads`\n    * `history`\n    * `extensions`\n    * `databases`\n    * `localstorage`\n    * `extension_files`\n  * `accounts`\n    * `system_admins`\n    * `system_users`\n    * `social_accounts`\n    * `recent_items`\n  * `mail`\n  * `full_hash`\n\n  ```shell\n  $ sudo osxcollector.py -s 'startup' -s 'downloads'\n  ```\n\n* `-c`/`--collect-cookies`:\n  Collect cookies' value.\n  By default OSXCollector does not dump the value of a cookie, as it may contain sensitive information (e.g. session id).\n\n* `-l`/`--collect-local-storage`:\n  Collect the values stored in web browsers' local storage.\n  By default OSXCollector does not dump the values as they may contain sensitive information.\n\n* `-d`/`--debug`:\n  Enables verbose output and python breakpoints. If something is wrong with OSXCollector, try this.\n\n  ```shell\n  $ sudo osxcollector.py -d\n  ```\n\n## Details of Collection\nThe collector outputs a `.tar.gz` containing all the collected artifacts. The archive contains a JSON file with the majority of information.  Additionally, a set of useful logs from the target system logs are included.\n\n#### Common Keys\n\n##### Every Record\nEach line of the JSON file records 1 _piece of information_.  There are some common keys that appear in every JSON record:\n* `osxcollector_incident_id`: A unique ID shared by every record.\n* `osxcollector_section`: The _section_ or type of data this record holds.\n* `osxcollector_subsection`: The _subsection_ or more detailed descriptor of the type of data this record holds.\n\n##### File Records\nFor records representing files there are a bunch of useful keys:\n* `atime`: The file accessed time.\n* `ctime`: The file creation time.\n* `mtime`: The file modified time.\n* `file_path`: The absolute path to the file.\n* `md5`: MD5 hash of the file contents.\n* `sha1`: SHA1 hash of the file contents.\n* `sha2`: SHA2 hash of the file contents.\n\nFor records representing downloaded files:\n* `xattr-wherefrom`: A list containing the source and referrer URLs for the downloaded file.\n* `xattr-quarantines`: A string describing which application downloaded the file.\n\n##### SQLite Records\nFor records representing a row of a SQLite database:\n* `osxcollector_table_name`: The table name the row comes from.\n* `osxcollector_db_path`: The absolute path to the SQLite file.\n\nFor records that represent data associated with a specific user:\n* `osxcollector_username`: The name of the user\n\n#### Timestamps\nOSXCollector attempts to convert timestamps to human readable date/time strings in the format `YYYY-mm-dd hh:MM:ss`. It uses heuristics to automatically identify various timestamps:\n* seconds since epoch\n* milliseconds since epoch\n* seconds since 2001-01-01\n* seconds since 1601-01-01\n\n#### Sections\n##### `version` section\n\nThe current version of OSXCollector.\n\n##### `system_info` section\n\nCollects basic information about the system:\n\n - system name\n - node name\n - release\n - version\n - machine\n\n##### `kext` section\n\nCollects the Kernel extensions from:\n- `/System/Library/Extensions`\n- `/Library/Extensions`\n\n##### `startup` section\n\nCollects information about the\n[LaunchAgents](https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man5/launchd.plist.5.html),\nLaunchDaemons, ScriptingAdditions,\n[StartupItems](https://developer.apple.com/library/mac/documentation/macosx/conceptual/bpsystemstartup/chapters/StartupItems.html)\nand other login items from:\n\n - `/System/Library/LaunchAgents`\n - `/System/Library/LaunchDaemons`\n - `/Library/LaunchAgents`\n - `~/Library/LaunchAgents`\n - `/Library/LaunchDaemons`\n - `/System/Library/ScriptingAdditions`\n - `/Library/ScriptingAdditions`\n - `/System/Library/StartupItems`\n - `/Library/StartupItems`\n - `~/Library/Preferences/com.apple.loginitems.plist`\n\nMore information about the Max OS X startup can be found here:\nhttp://www.malicious-streams.com/article/Mac_OSX_Startup.pdf\n\n##### `applications` section\n\nHashes installed applications and gathers install history from:\n\n - `/Applications`\n - `~/Applications`\n - `/Library/Receipts/InstallHistory.plist`\n\n##### `quarantines` section\n\nQuarantines are basically the info necessary to show the 'Are you sure you wanna run this?' when a user is trying to open a file downloaded from the Internet.\nFor some more details, checkout the Apple Support explanation of Quarantines:\nhttp://support.apple.com/kb/HT3662\n\nThis section collects also information from XProtect hash-based malware check\nfor quarantines files. The plist is at:\n`/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/XProtect.plist`\n\nXProtect also add minimum versions for Internet Plugins. That plist is at:\n`/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/XProtect.meta.plist`\n\n##### `downloads` section\n\nHashes all users' downloaded files from:\n\n - `~/Downloads`\n - `~/Library/Mail Downloads`\n - `~/Library/Containers/com.apple.mail/Data/Library/Mail Downloads`\n\n##### `chrome` section\n\nCollects following information from Google Chrome web browser:\n\n - History\n - Archived History\n - Cookies\n - Extensions\n - Login Data\n - Top Sites\n - Web Data\n\nThis data is extracted from `~/Library/Application Support/Google/Chrome/Default`\n\n##### `firefox` section\n\nCollects information from the different SQLite databases in a Firefox profile:\n\n - Cookies\n - Downloads\n - Form History\n - History\n - Signons\n - Permissions\n - Addons\n - Extensions\n - Content Preferences\n - Health Report\n - Webapps Store\n\nThis information is extracted from `~/Library/Application Support/Firefox/Profiles`\n\nFor more details about Firefox profile folder see\nhttp://kb.mozillazine.org/Profile_folder_-_Firefox\n\n##### `safari` section\n\nCollects information from the different plists and SQLite databases in a Safari\nprofile:\n\n - Downloads\n - History\n - Extensions\n - Databases\n - Local Storage\n\n##### `accounts` section\nCollects information about users' accounts:\n\n - system admins: `/private/var/db/dslocal/nodes/Default/groups/admin.plist`\n - system users: `/private/var/db/dslocal/nodes/Default/users`\n - social accounts: `~/Library/Accounts/Accounts3.sqlite`\n - users' recent items: `~/Library/Preferences/com.apple.recentitems.plist`\n\n##### `mail` section\nHashes files in the mail app directories:\n\n - `~/Library/Mail`\n - `~/Library/Mail Downloads`\n\n##### `full_hash` section\nHashes all the files on disk. All of 'em. This does not run by default. It must be triggered with:\n```shell\n$ sudo osxcollector.py -s full_hash\n```\n\n## Basic Manual Analysis\nForensic analysis is a bit of art and a bit of science. Every analyst will see a bit of a different story when reading the output from OSXCollector. That's part of what makes analysis fun.\n\nGenerally, collection is performed on a target machine because something is hinky: anti-virus found a file it doesn't like, deep packet inspect observed a callout, endpoint monitoring noticed a new startup item. The details of this initial alert - a file path, a timestamp, a hash, a domain, an IP, etc. - that's enough to get going.\n\n#### Timestamps\nSimply greping a few minutes before and after a timestamp works great:\n\n```shell\n$ cat INCIDENT32.json | grep '2014-01-01 11:3[2-8]'\n```\n\n#### Browser History\nIt's in there. A tool like [jq](http://stedolan.github.io/jq/) can be very helpful to do some fancy output:\n\n```shell\n$ cat INCIDENT32.json | grep '2014-01-01 11:3[2-8]' | jq 'select(has(\"url\"))|.url'\n```\n\n#### A Single User\n```shell\n$ cat INCIDENT32.json | jq 'select(.osxcollector_username==\"ivanlei\")|.'\n```\n\n## Automated Analysis\nThe [OSXCollector Output Filters project](https://github.com/Yelp/osxcollector_output_filters) contains filters that process and transform the output of OSXCollector. The goal of filters is to make it easy to analyze OSXCollector output.\n\n#### Development Tips\nThe functionality of OSXCollector is stored in a single file: `osxcollector.py`. The collector should run on a naked install of OS X without any additional packages or dependencies.\n\nEnsure that all of the OSXCollector tests pass before editing the source code. You can run the tests using: `make test`\n\nAfter making changes to the source code, run `make test` again to verify that your changes did not break any of the tests.\n\n## License\nThis work is licensed under the GNU General Public License and a derivation of [https://github.com/jipegit/OSXAuditor](https://github.com/jipegit/OSXAuditor)\n\n## Blog post\n\n* [OSXCollector: Forensic Collection and Automated Analysis for OS X](http://engineeringblog.yelp.com/2015/01/osxcollector-forensic-collection-and-automated-analysis-for-os-x.html) by Ivan Leichtling\n\n## Presentations\n\n* [OSXCollector: Automated forensic evidence collection \u0026 analysis for OS X](https://www.youtube.com/watch?v=l-lhyPcSd6I) by Kuba Sendor @ BruCON 0x07\n* [Squashing Rotten Apples: Automated forensics \u0026 analysis for Mac OS X with OSXCollector](https://www.youtube.com/watch?v=XeeCO8moyeE) by Kuba Sendor @ BSides Manchester 2015\n* [OSXCollector](http://macbrained.org/recap-august15-yelp/) by Ivan Leichtling @ Macbrained's August Meet-Up\n* [OSXCollector - Automated Forensic Evidence Collection \u0026 Analysis for OS X](https://www.youtube.com/watch?v=Yqny1rMTfyY) by Ivan Leichtling @ OpenNSM\n* [OSXCollector - Automated Forensic Evidence Collection \u0026 Analysis for OS X](https://www.youtube.com/watch?v=DfANq2ncaKU) by Ivan Leichtling @ Duo Tech Talk ([blog post](https://www.duosecurity.com/blog/duo-tech-talk-osxcollector-automated-forensic-evidence-collection-and-analysis-for-os-x))\n\n## External Presentations\n\n* [OSX Archaeology: Becoming Indiana Jones with OSXCollector and Strata](https://www.youtube.com/watch?v=9wvhOoXl2Os) by Chris Henderson \u0026 Justin Larson @ SAINTCon 2015\n\n## Resources\nWant to learn more about OS X forensics?\n* [Sarah Edward's mac4n6.com](http://www.mac4n6.com/) - The best presentations on Mac forensics.\n\nA couple of other interesting tools:\n* [KnockKnock](https://github.com/synack/knockknock) - KnockKnock is a command line python script that displays persistent OS X binaries that are set to execute automatically at each boot.\n* [Grr](https://github.com/google/grr) - Google Rapid Response: remote live forensics for incident response\n* [osquery](https://github.com/facebook/osquery) - SQL powered operating system instrumentation, monitoring, and analytics\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FYelp%2Fosxcollector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FYelp%2Fosxcollector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FYelp%2Fosxcollector/lists"}