{"id":13625550,"url":"https://github.com/etsy/MIDAS","last_synced_at":"2025-04-16T06:32:53.640Z","repository":{"id":12321648,"uuid":"14958024","full_name":"etsy/MIDAS","owner":"etsy","description":"Mac Intrusion Detection Analysis System","archived":true,"fork":false,"pushed_at":"2015-09-23T15:45:08.000Z","size":366,"stargazers_count":831,"open_issues_count":1,"forks_count":160,"subscribers_count":124,"default_branch":"master","last_synced_at":"2024-08-01T22:05:28.651Z","etag":null,"topics":["non-sox"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/etsy.png","metadata":{"files":{"readme":"README.archived.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-12-05T15:51:26.000Z","updated_at":"2024-07-28T08:48:03.000Z","dependencies_parsed_at":"2022-09-10T16:33:01.464Z","dependency_job_id":null,"html_url":"https://github.com/etsy/MIDAS","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/etsy%2FMIDAS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/etsy%2FMIDAS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/etsy%2FMIDAS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/etsy%2FMIDAS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/etsy","download_url":"https://codeload.github.com/etsy/MIDAS/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223700579,"owners_count":17188352,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["non-sox"],"created_at":"2024-08-01T21:01:57.561Z","updated_at":"2024-11-08T14:31:49.600Z","avatar_url":"https://github.com/etsy.png","language":null,"readme":"MIDAS\n=====\n\nMIDAS is a framework for developing a Mac Intrusion Detection Analysis System,\nbased on work and collaborative discussions between the Etsy and\nFacebook security teams. This repository provides a modular framework and a\nnumber of helper utilities, as well as an example module for detecting\nmodifications to common OS X persistence mechanisms.\n\nThe MIDAS project is based off concepts presented in [Homebrew Defensive\nSecurity] (http://www.slideshare.net/mimeframe/ruxcon-2012-15195589) and\n[Attack-Driven Defense]\n(http://www.slideshare.net/zanelackey/attackdriven-defense), as well as\nlessons learned during the development of the Tripyarn and BigMac products.\n\nOur mutual goal in releasing this framework is to foster more discussion in\nthis area and provide organizations with a starting point in instrumenting\nOS X endpoints to detect common patterns of compromise and persistence.\n\nOverview\n---------\n\nThe `midas` subdirectory is where the core MIDAS code lives. The entry point is\n`launcher.py`. From there, each module in `midas/modules` is executed and the\nstdout of the module is written to a log file. When deploying MIDAS, this is\nthe code that's put on user's systems.\n\nThe `develop` subdirectory is where development resources (like a `.pylintrc`)\nlive.\n\nThe `templates` resource is where template and base resources live. These can be\nused as a starting point when adding modules.\n\nArchitecture\n------------\n\nMIDAS allows you to define a set of \"modules\" that implement\nhost-based checks, verifications, analysis, etc.\n\n### Launcher\n\nThe `launcher.py` file exists at the top level of the `midas` directory. It\ngathers some simple information about the host it's executing on (such as time,\nhostname, etc) and defines the ways that it should handle modules of certain\nlanguages. To add a supported language, create a new instance of\n`TyLanguage` in `launcher.py` and add it to the `SUPPORTED_LANGUAGES` list. If\nyou'd like to change the way a certain language is supported (perhaps you'd\nlike all python modules to be executed with a custom version of python), you\ncan change the attributes (such as `execution_string`) of the language.\n\nOnce key definitions are made, the launcher will iterate through all files\n(note that directories are explicitly skipped) in the `modules` subdirectory.\nFor each file in the directory, if a language entry is found that indicates\nhow to deal with that filetype, the file is executed and the stdout of the\nmodule are appended to a log file in the `log` subdirectory. Note that a\nmodule `modules/example.py` will generate a log file `logs/example.log`.\n\n### Module language\n\nModules can be written in any language, so long as a named tuple for that\nlanguage is defined in `midas/launcher.py`. These named tuples (which\nalready exist for python, ruby and shell) exist so that MIDAS knows how\nto handle certain filetypes when it sees them.\n\nAs long as your code can be executed and prints something to stdout, it can be\na module.\n\n\nComponents\n----------\n\n### Example module\n\nThe file `midas/modules/example.py` is an example MIDAS module created to\nillustrate what a MIDAS module might look like. This module performs\nanalysis of LaunchAgents and LaunchDaemons on the host and logs any\nmodifications that it identifies. The rest of the checks and\nverifications analyze the host firewall configurations and log any additions or\ndifferences that are identified. **This is not meant to be a complete intrusion\ndetection mechanism alone**, instead it is meant as a reference example of what\na MIDAS module may look like.\n\n### Helpers\n\nThere are several helper files in `midas/lib/helpers` that are generally\ngrouped by category. Functions in these helpers can be imported by modules to\nassist in general tasks. Some functionality exposed by helpers include:\n\n- list all weak SSH keys on a host\n- find all files in a given directory with given permissions\n- list all startup items\n- list all LaunchAgents, LaunchDaemons, etc.\n- list and hash all kernel extensions\n- return the SSID of the currently connected WiFi network\n- return the IP and MAC address of the current network gateway\n- return DNS configuration information\n- and much, much more\n\n### Config system\n\nThe config file, which can be found at `midas/lib/config.py` is a way to group\ntogether information that can be abstracted away from modules. Usually there\nare things like strings that should be checked in a certain module validation,\ndirectories to search during a given check, etc. By abstracting the data away\nfrom the individual module/code, it makes it easier for people who might not\nmaintain the code to contribute to it.\n\nSince the config dictionary is operated on via a static method, it does not\nneed to instantiate the Config object in order to use it. To add a new value\nto the config dictionary, simply add an entry in the class.\n\n### ORM and table definitions\n\nMIDAS relies on a local datastore to do some simple host-based analytics with\nthe data gathered by modules. For this reason, MIDAS comes with a simple,\ncustom ORM.\n\n#### Table definitions\n\nTable schemas are described via a simple dictionary, all of which can be found\nin `midas/lib/tables/`. The dictionary is then parsed and valid SQL is created\nfrom the dictionary. Each item in the dictionary represents a column. The\ncolumn definition should be a key-value pair where the key is a strings that\nrepresents the name of the column and the value is a dictionary that describes\nthe column. The column definition dictionary can have the following attributes:\n\n- default\n  - If this is set, it will be the default value of the column. This is most frequently set to `\"NULL\"`\n- nullable\n  - If this is set to `False`, then the `NOT NULL` attribute will be used when creating the table\n- attrs\n  - Use this to add additional SQL syntax that you want added to the table creation statement that isn't supported by tyORM\n- primary_key\n  - If this is set to `True`, then the column listed will be set to the primary key\n\nSee the following same table definition for reference:\n\n```python\ntest_table = {\n    \"data_field_name\" : {\n        \"type\" : \"text\",\n        \"nullable\" : False,\n    },\n    \"other_data_field\" : {\n        \"type\" : \"text\",\n        \"default\" : \"NULL\",\n    }\n}\n```\n\n#### Instantiating an ORM object\n\nWhen instantiating an ORM object, the class takes one parameter: the database\nfilename. If the file doesn't exist, the ORM will create it.\n\nSee the following example code for reference:\n\n```python\nfrom lib.ty_orm import TyORM\nORM = TyORM(\"midas_hids.db\")\n```\n\n#### Transparent primary key system\n\nAlthough it is possible to specify a primary key when creating a table, TyORM\ntransparently creates an auto-incrementing `id` column and sets it as a primary\nkey. Although SQLite allows you to specify several primary keys, this is not\nnecessary. The `id` column is always used as the `WHERE` clause identifier when\ndoing `UPDATE` and `DELETE` operations.\n\n#### Creating\n\nUse the `insert` method to insert data into a table. The `insert` method takes\ntwo arguments: the table name and the data that you'd like to insert. The table\nname should be a string that describes the name of the table. The data should\nbe a dictionary such that the keys describe the columns that the value should\nbe inserted into.\n\nSee the following example `insert` call for reference:\n\n```python\nORM = TyORM(\"midas_hids.db\")\ndata = {\n    \"data_field_name\" : \"foo\",\n    \"other_data_field\" : \"bar\",\n}\nORM.insert(\"test_table\", data)\n```\n\n#### Reading\n\nUse the `select` method to read data from the database. The `select` method takes one mandatory argument and three optional arguments.\n\nThe mandatory argument that all `select` method calls needs to have is the table name that you'd like to select from. The optional arguments are as follows:\n\n- columns\n  - An array of columns that you would like returned\n- where\n  - A string that describes the \"WHERE\" clause that you would like added to the SQL query\n  - Note that this can be a string but if you're supplying user input, this should be an array such that the first item in the array is the where clause with '?' place holders and the second item in the array is an array with the representative values.\n- limit\n  - An integer describing the LIMIT value that you would like added to the SQL query\n- order_by\n  - A string dictating which column to order results by\n\nSee the following example `select` calls for reference:\n\n```python\nORM = TyORM(\"midas_hids.db\")\n\n# this will return all table data\nORM.select(\"test_table\")\n\n# this will return only the \"data_field_name\" column of the\n# first five columns where the \"data_field_name\" column is \"foobar\", ordered by \"data_field_name\"\nORM.select(\"test_table\", [\"data_field_name\"], \"data_field_name = 'foobar'\", 5, \"data_field_name\")\n```\n\n#### Updating\n\nUse the `update` method to update data from the database. Simply `select` some\ndata and change the returned dictionary to reflect the data you want the field\nto be updated to and, via some \"hidden\" values, the ORM will take care of the\nrest.\n\nSee the following example `update` call for reference:\n\n```python\nORM = TyORM(\"midas_hids.db\")\ndata = ORM.select(\"test_table\", \"*\", \"data_field_name = 'foobar123'\", 1)\ndata['data_field_name'] = 'newname123'\nORM.update(data)\n```\n\n#### Deleting\n\nUse the `delete` method to delete data from the database. Simply `select` some\ndata and call the delete method and the ORM will take care of the rest.\n\nSee the following example `delete` call for reference:\n\n```python\nORM = TyORM(\"midas_hids.db\")\ndata = ORM.select(\"test_table\", \"*\", \"data_field_name = 'foobar123'\", 1)\nORM.delete(data)\n```\n\n#### Initializing tables and dynamic ALTERs\n\nOne of the strengths of tyORM is it's ability to dynamically ALTER a table if\nthe table's schema doesn't match the table  definition dictionary.\n\nSimply call the `initialize_table` table method before operating on the table.\nThe `initialize_table` method takes two arguments: the table name and the table\ndefinition. The `initialize_table` method will create the table if it doesn't\nexist and it will alter the table if any new columns have been added.\n\nSee the following example code for reference:\n\n```python\ntest_table_data = {\n    \"data_field_name\" : \"foo\",\n    \"other_data_field\" : \"bar\",\n}\n\nORM = TyORM(\"midas_hids.db\")\nORM.initialize_table(\"test_table\", test_table_data)\n\n# operate on the ORM here\n```\n\nDue to limitations of SQLite, this only support new columns that are added, not\ncolumns that are removed.\n\n### Host based analytics\n\nThe file `lib/data_science.py` is used for simple host based data aggregation.\nThe `DataScience` class is used to query the database and log any changes,\ngiven a new dataset. Using data_science is very simple. The class constructor\ntakes three arguments:\n\n- a TyORM object that is already instantiated with the database which is to be\n  operated on\n- a dataset\n- a table name that the dataset should be compared against\n\nThe dataset should be an array of dictionaries. Each item in the array should\nbe a dictionary where each key of the dictionary represents a column in the\ndatabase and each corresponding value represents a corresponding value. It's\nOK if some columns of the table are not in the dictionary, but the `name`\ncolumn should always be present. Although TyORM has it's own transparent\nprimary key system using the `id` column, for the sake of `data_science`, the\n`name` column should be present and it should be unique. The `data_science`\ncode will then select all of the data from the given table and compare it\nagainst the supplied dataset, printing out log lines illustrating all data that\nhas been added, removed and changed.\n\n### Decorators\n\nThe file `midas/lib/decorators.py` contains a bunch of decorators that can be\nused for a variety of things, but currently predominantly code execution\nfrequency.\n\n### Property List parsing\n\nThis is the utility that MIDAS uses to operate on property list files such\nas LaunchAgents and LaunchDaemons. This is mostly the\n[biplist](https://github.com/wooster/biplist) python module, however, you\nshould never actually call any of the biplist functions directly.\n\nThe `read_plist` function is what you should call when trying to read a plist.\nWhen you call `read_plist`, it first tries to use biplist's readPlist. That\ncode determines if the plist is a binary plist or an XML plist. If it is an XML\nplist, it just uses python's plistlib to read the plist. If it is a binary\nplist, it uses a native python implementation to parse it and return it's\ncontents. If that, for whatever reason, fails, the `read_plist` function will\nthen try to shell out to `plutil` to parse the plist.\n\nThe `get_plist_key` function takes a plist and a key as input. It returns the\nkey (if the key exists) or False if it does not. This is so that, when\noperating on property lists, you don't have to roll your own exception handling\non every access.\n\n`read_plist` and `get_plist_key` are the only two functions that should be\ncalled from this file.\n\nCustomization\n-------------\n\nA MIDAS deployment in an organization typically follows these steps:\n\n- Create a private fork of MIDAS\n- Add modules and helpers that implement instrumentation specific to the environment\n- Deploy the code (with updated modules) to OS X endpoints in the organization\n- Set a crontab/LaunchAgent that executes MIDAS at a set interval\n- Use syslog/a log aggregation mechanism to forward the logs to a central logging\n  host\n- Analyze the collected data and create alerts for anomalies\n\nContributors\n------------\n\n+ __Mike Arpaia__ ([@mikearpaia](https://twitter.com/mikearpaia))\n+ __Chris Biettchert__ ([@chrisbiettchert](https://twitter.com/chrisbiettchert))\n+ __Ben Hughes__ ([@benjammingh](https://twitter.com/benjammingh))\n+ __Zane Lackey__ ([@zanelackey](https://twitter.com/zanelackey))\n+ __mimeframe__ ([@mimeframe](https://twitter.com/mimeframe))\n","funding_links":[],"categories":["Others","macOS Security","Security"],"sub_categories":["macOS 10.15 Catalina Setup"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fetsy%2FMIDAS","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fetsy%2FMIDAS","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fetsy%2FMIDAS/lists"}