{"id":13384317,"url":"https://github.com/NathanEpstein/Dora","last_synced_at":"2025-03-13T10:31:19.334Z","repository":{"id":83448253,"uuid":"51867244","full_name":"NathanEpstein/Dora","owner":"NathanEpstein","description":"Tools for exploratory data analysis in Python","archived":false,"fork":false,"pushed_at":"2024-01-18T03:14:44.000Z","size":138,"stargazers_count":644,"open_issues_count":0,"forks_count":73,"subscribers_count":43,"default_branch":"master","last_synced_at":"2024-10-26T05:39:28.476Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NathanEpstein.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2016-02-16T20:27:38.000Z","updated_at":"2024-09-27T14:10:18.000Z","dependencies_parsed_at":"2024-01-06T13:09:26.350Z","dependency_job_id":"23aeb3c1-bfef-40dc-93e9-21ea4fa02ebe","html_url":"https://github.com/NathanEpstein/Dora","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NathanEpstein%2FDora","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NathanEpstein%2FDora/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NathanEpstein%2FDora/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NathanEpstein%2FDora/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NathanEpstein","download_url":"https://codeload.github.com/NathanEpstein/Dora/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243386044,"owners_count":20282679,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T11:00:44.558Z","updated_at":"2025-03-13T10:31:19.038Z","avatar_url":"https://github.com/NathanEpstein.png","language":"Python","readme":"# Dora\nExploratory data analysis toolkit for Python.\n\n## Contents\n- [Summary](#summary)\n- [Setup](#setup)\n- [Usage](#use)\n  - [Reading Data \u0026 Configuration](#config)\n  - [Cleaning](#clean)\n  - [Feature Selection \u0026 Extraction](#feature)\n  - [Visualization](#visual)\n  - [Model Validation](#model)\n  - [Data Versioning](#version)\n- [Testing](#test)\n- [Contribute](#contribute)\n- [License](#license)\n\n\u003ca name=\"summary\"\u003e\u003c/a\u003e\n## Summary\n\nDora is a Python library designed to automate the painful parts of exploratory data analysis.\n\nThe library contains convenience functions for data cleaning, feature selection \u0026 extraction, visualization, partitioning data for model validation, and versioning transformations of data.\n\nThe library uses and is intended to be a helpful addition to common Python data analysis tools such as pandas, scikit-learn, and matplotlib.\n\n\u003ca name=\"setup\"\u003e\u003c/a\u003e\n## Setup\n\nTo ensure latest code, install this library from the Github repo. \n\n```\n\u003e\u003e\u003e from Dora import Dora\n```\n\n\u003ca name=\"use\"\u003e\u003c/a\u003e\n## Usage\n\n\u003ca name=\"config\" \u003e\u003c/a\u003e\n#### Reading Data \u0026 Configuration\n\n```python\n# without initial config\n\u003e\u003e\u003e dora = Dora()\n\u003e\u003e\u003e dora.configure(output = 'A', data = 'path/to/data.csv')\n\n# is the same as\n\u003e\u003e\u003e import pandas as pd\n\u003e\u003e\u003e dataframe = pd.read_csv('path/to/data.csv')\n\u003e\u003e\u003e dora = Dora(output = 'A', data = dataframe)\n\n\u003e\u003e\u003e dora.data\n   A   B  C      D  useless_feature\n0  1   2  0   left                1\n1  4 NaN  1  right                1\n2  7   8  2   left                1\n```\n\n\u003ca name=\"clean\" \u003e\u003c/a\u003e\n#### Cleaning\n\n```python\n# read data with missing and poorly scaled values\n\u003e\u003e\u003e import pandas as pd\n\u003e\u003e\u003e df = pd.DataFrame([\n...   [1, 2, 100],\n...   [2, None, 200],\n...   [1, 6, None]\n... ])\n\u003e\u003e\u003e dora = Dora(output = 0, data = df)\n\u003e\u003e\u003e dora.data\n   0   1    2\n0  1   2  100\n1  2 NaN  200\n2  1   6  NaN\n\n# impute the missing values (using the average of each column)\n\u003e\u003e\u003e dora.impute_missing_values()\n\u003e\u003e\u003e dora.data\n   0  1    2\n0  1  2  100\n1  2  4  200\n2  1  6  150\n\n# scale the values of the input variables (center to mean and scale to unit variance)\n\u003e\u003e\u003e dora.scale_input_values()\n\u003e\u003e\u003e dora.data\n   0         1         2\n0  1 -1.224745 -1.224745\n1  2  0.000000  1.224745\n2  1  1.224745  0.000000\n```\n\n\u003ca name=\"feature\" \u003e\u003c/a\u003e\n#### Feature Selection \u0026 Extraction\n\n```python\n# feature selection / removing a feature\n\u003e\u003e\u003e dora.data\n   A   B  C      D  useless_feature\n0  1   2  0   left                1\n1  4 NaN  1  right                1\n2  7   8  2   left                1\n\n\u003e\u003e\u003e dora.remove_feature('useless_feature')\n\u003e\u003e\u003e dora.data\n   A   B  C      D\n0  1   2  0   left\n1  4 NaN  1  right\n2  7   8  2   left\n\n# extract an ordinal feature through one-hot encoding\n\u003e\u003e\u003e dora.extract_ordinal_feature('D')\n\u003e\u003e\u003e dora.data\n   A   B  C  D=left  D=right\n0  1   2  0       1        0\n1  4 NaN  1       0        1\n2  7   8  2       1        0\n\n# extract a transformation of another feature\n\u003e\u003e\u003e dora.extract_feature('C', 'twoC', lambda x: x * 2)\n\u003e\u003e\u003e dora.data\n   A   B  C  D=left  D=right  twoC\n0  1   2  0       1        0     0\n1  4 NaN  1       0        1     2\n2  7   8  2       1        0     4\n```\n\n\u003ca name=\"visual\" \u003e\u003c/a\u003e\n#### Visualization\n\n```python\n# plot a single feature against the output variable\ndora.plot_feature('column-name')\n\n# render plots of each feature against the output variable\ndora.explore()\n```\n\n\u003ca name=\"model\" \u003e\u003c/a\u003e\n#### Model Validation\n\n```python\n# create random partition of training / validation data (~ 80/20 split)\ndora.set_training_and_validation()\n\n# train a model on the data\nX = dora.training_data[dora.input_columns()]\ny = dora.training_data[dora.output]\n\nsome_model.fit(X, y)\n\n# validate the model\nX = dora.validation_data[dora.input_columns()]\ny = dora.validation_data[dora.output]\n\nsome_model.score(X, y)\n```\n\n\u003ca name=\"version\" \u003e\u003c/a\u003e\n#### Data Versioning\n\n```python\n# save a version of your data\n\u003e\u003e\u003e dora.data\n   A   B  C      D  useless_feature\n0  1   2  0   left                1\n1  4 NaN  1  right                1\n2  7   8  2   left                1\n\u003e\u003e\u003e dora.snapshot('initial_data')\n\n# keep track of changes to data\n\u003e\u003e\u003e dora.remove_feature('useless_feature')\n\u003e\u003e\u003e dora.extract_ordinal_feature('D')\n\u003e\u003e\u003e dora.impute_missing_values()\n\u003e\u003e\u003e dora.scale_input_values()\n\u003e\u003e\u003e dora.data\n   A         B         C    D=left   D=right\n0  1 -1.224745 -1.224745  0.707107 -0.707107\n1  4  0.000000  0.000000 -1.414214  1.414214\n2  7  1.224745  1.224745  0.707107 -0.707107\n\n\u003e\u003e\u003e dora.logs\n[\"self.remove_feature('useless_feature')\", \"self.extract_ordinal_feature('D')\", 'self.impute_missing_values()', 'self.scale_input_values()']\n\n# use a previous version of the data\n\u003e\u003e\u003e dora.snapshot('transform1')\n\u003e\u003e\u003e dora.use_snapshot('initial_data')\n\u003e\u003e\u003e dora.data\n   A   B  C      D  useless_feature\n0  1   2  0   left                1\n1  4 NaN  1  right                1\n2  7   8  2   left                1\n\u003e\u003e\u003e dora.logs\n[]\n\n# switch back to your transformation\n\u003e\u003e\u003e dora.use_snapshot('transform1')\n\u003e\u003e\u003e dora.data\n   A         B         C    D=left   D=right\n0  1 -1.224745 -1.224745  0.707107 -0.707107\n1  4  0.000000  0.000000 -1.414214  1.414214\n2  7  1.224745  1.224745  0.707107 -0.707107\n\u003e\u003e\u003e dora.logs\n[\"self.remove_feature('useless_feature')\", \"self.extract_ordinal_feature('D')\", 'self.impute_missing_values()', 'self.scale_input_values()']\n```\n\n\u003ca name=\"test\" \u003e\u003c/a\u003e\n## Testing\n\nTo run the test suite, simply run `python3 spec.py` from the `Dora` directory.\n\n\u003ca name=\"contribute\" \u003e\u003c/a\u003e\n## Contribute\n\nPull requests welcome! Feature requests / bugs will be addressed through issues on this repository. While not every feature request will necessarily be handled by me, maintaining a record for interested contributors is useful.\n\nAdditionally, feel free to submit pull requests which add features or address bugs yourself.\n\n\n\u003ca name=\"license\" \u003e\u003c/a\u003e\n## License\n\n**The MIT License (MIT)**\n\n\u003e Copyright (c) 2016 Nathan Epstein\n\u003e\n\u003e Permission is hereby granted, free of charge, to any person obtaining a copy\n\u003e of this software and associated documentation files (the \"Software\"), to deal\n\u003e in the Software without restriction, including without limitation the rights\n\u003e to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n\u003e copies of the Software, and to permit persons to whom the Software is\n\u003e furnished to do so, subject to the following conditions:\n\u003e\n\u003e The above copyright notice and this permission notice shall be included in\n\u003e all copies or substantial portions of the Software.\n\u003e\n\u003e THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n\u003e IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n\u003e FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n\u003e AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n\u003e LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n\u003e OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\n\u003e THE SOFTWARE.\n","funding_links":[],"categories":["Python","Exploration","🐍 Python"],"sub_categories":["Useful Python Tools for Data Analysis"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNathanEpstein%2FDora","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FNathanEpstein%2FDora","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNathanEpstein%2FDora/lists"}