{"id":18322204,"url":"https://github.com/dataideaorg/dataidea-analysis-package","last_synced_at":"2025-04-09T14:42:40.265Z","repository":{"id":234313898,"uuid":"788638903","full_name":"dataideaorg/dataidea-analysis-package","owner":"dataideaorg","description":"This package helps simplify data analysis work for DATAIDEA students","archived":false,"fork":false,"pushed_at":"2024-04-21T19:54:50.000Z","size":7789,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"major","last_synced_at":"2025-03-21T20:47:33.274Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dataideaorg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-18T19:55:11.000Z","updated_at":"2024-04-21T19:54:53.000Z","dependencies_parsed_at":"2024-04-21T21:53:45.682Z","dependency_job_id":null,"html_url":"https://github.com/dataideaorg/dataidea-analysis-package","commit_stats":null,"previous_names":["dataidea/dataidea-analysis-package","dataideaorg/dataidea-analysis-package"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataideaorg%2Fdataidea-analysis-package","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataideaorg%2Fdataidea-analysis-package/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataideaorg%2Fdataidea-analysis-package/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataideaorg%2Fdataidea-analysis-package/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dataideaorg","download_url":"https://codeload.github.com/dataideaorg/dataidea-analysis-package/tar.gz/refs/heads/major","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248055275,"owners_count":21040151,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T18:23:45.342Z","updated_at":"2025-04-09T14:42:40.247Z","avatar_url":"https://github.com/dataideaorg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## What is the `dataidea` package?\n\nThis is a package we are currently developing to help new and old data analysists (especially DATAIDEA students) walk around some repetitive and sometimes disturbing tasks that a data analyst does day to day\n\nThis library currently extends and depends on majorly numpy, pandas as sklearn and these, among a few others will be installed once you install dataidea\n\n## Installing `dataidea`\n\n- To install dataidea, you must have python installed on your machine\n- It's advised that you install it in a virtual environment\n- You can install `dataidea` using the command below\n\n```\npip install dataidea\n```\n\n## Learning `dataidea`\n\nThe best way to get started with dataidea (and data analysis) is to complete the free course.\n\nTo see what’s possible with dataidea, take a look at the Quick Start\n\nRead through the Tutorials to learn how to load datasets, train your own models on your own datasets. Use the navigation to look through the dataidea documentation. Every class, function, and method is documented here.\n\n## Quickstart\n\n```python\nfrom dataidea.tabular import *\n```\n\n`dataidea`'s applications all use the same basic steps and code:\n\n- Create appropriate DataLoaders\n- Create a Trainer\n- Call a fit method\n- Make predictions or view results.\n\nIn this quick start, we’ll show these steps for classification and regression. As you’ll see, the code in each case is extremely similar, despite the very different models and data being used.\n\n## Loading datasets\n\n`dataidea` library makes loading the most common used dataset in the course easy, but also allows for loading personal dataset with one or 2 tweeks.\n\nIn the line of code below, we load the simple music dataset which inbuilt into dataidea for learning purposes\n\n```python\nmusic_data = loadDataset(name='music')\n```\n\nWe can see some values inside by using our usual `pandas` dataframe methods like `sample()`, `head()`, `tail()` etc\n\n```python\nmusic_data.sample(n=5)\n```\n\n\u003cdiv\u003e\n\u003ctable border=\"1\" class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003eage\u003c/th\u003e\n      \u003cth\u003egender\u003c/th\u003e\n      \u003cth\u003egenre\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003e3\u003c/th\u003e\n      \u003ctd\u003e26\u003c/td\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n      \u003ctd\u003eJazz\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e2\u003c/th\u003e\n      \u003ctd\u003e25\u003c/td\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n      \u003ctd\u003eHipHop\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e14\u003c/th\u003e\n      \u003ctd\u003e30\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n      \u003ctd\u003eAcoustic\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e1\u003c/th\u003e\n      \u003ctd\u003e23\u003c/td\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n      \u003ctd\u003eHipHop\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e19\u003c/th\u003e\n      \u003ctd\u003e35\u003c/td\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n      \u003ctd\u003eClassical\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\nWe can then create `TabularDataLoader` that allows for easy data manipulation like feature scaling, imputation and splitting for training etc, we use this to quickly prepare and load the data to a machine learning model\n\n```python\nmusic_data_loader = TabularDataLoader(data=music_data,\n                                numeric_features=['  age '],\n                                categorical_features=['gender'],\n                                outcome='genre'\n                               )\n```\n\nWe can (optionally) process the data, however this step is gonna be done for you once you decide to train a machine learning model\n\n```python\ntransformed_data, transformer = music_data_loader.transform()\n```\n\n```python\ntransformed_data[0].head()\n```\n\n\u003cdiv\u003e\n\u003ctable border=\"1\" class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003enum__  age\u003c/th\u003e\n      \u003cth\u003ecat__gender_0\u003c/th\u003e\n      \u003cth\u003ecat__gender_1\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003e0\u003c/th\u003e\n      \u003ctd\u003e1.637262\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n      \u003ctd\u003e1.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e1\u003c/th\u003e\n      \u003ctd\u003e0.204658\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n      \u003ctd\u003e1.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e2\u003c/th\u003e\n      \u003ctd\u003e-0.818631\u003c/td\u003e\n      \u003ctd\u003e1.0\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e3\u003c/th\u003e\n      \u003ctd\u003e-0.613973\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n      \u003ctd\u003e1.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e4\u003c/th\u003e\n      \u003ctd\u003e1.227947\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n      \u003ctd\u003e1.0\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\nNow we can fit a machine learning model, behind the scenes, the `Trainer` has some code to work with the `TabularDataLoader` to process your data quite thorouly, standardizing and imputing and the resulting model is actually a pipeline of these steps.\n\n```python\ntrainer = Trainer(data_loader=music_data_loader, model=RandomForestClassifier())\n```\n\nTo train our model, we just call the `train()` method on the trainer object\n\n```python\nmodel = trainer.train()\n```\n\nWe can obtain the accuracy real fast, this is obtained from a test set which is automatically picked from you data by the `TabularDataLoader`\n\n```python\naccuracy = trainer.evaluate()\n```\n\nSome times accuracy isn't the best measure for model performance, we can also use a classification report for classification problems\n\n```python\nclassification_report = trainer.report()\n```\n\n```python\nclassification_report\n```\n\n\u003cdiv\u003e\n\u003ctable border=\"1\" class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003eprecision\u003c/th\u003e\n      \u003cth\u003erecall\u003c/th\u003e\n      \u003cth\u003ef1-score\u003c/th\u003e\n      \u003cth\u003esupport\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003eAcoustic\u003c/th\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e0.000000\u003c/td\u003e\n      \u003ctd\u003e0.00\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eClassical\u003c/th\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e0.500000\u003c/td\u003e\n      \u003ctd\u003e0.666667\u003c/td\u003e\n      \u003ctd\u003e2.00\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eHipHop\u003c/th\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e2.00\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eaccuracy\u003c/th\u003e\n      \u003ctd\u003e0.750000\u003c/td\u003e\n      \u003ctd\u003e0.750000\u003c/td\u003e\n      \u003ctd\u003e0.750000\u003c/td\u003e\n      \u003ctd\u003e0.75\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003emacro avg\u003c/th\u003e\n      \u003ctd\u003e0.666667\u003c/td\u003e\n      \u003ctd\u003e0.833333\u003c/td\u003e\n      \u003ctd\u003e0.555556\u003c/td\u003e\n      \u003ctd\u003e4.00\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eweighted avg\u003c/th\u003e\n      \u003ctd\u003e1.000000\u003c/td\u003e\n      \u003ctd\u003e0.750000\u003c/td\u003e\n      \u003ctd\u003e0.833333\u003c/td\u003e\n      \u003ctd\u003e4.00\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\nIt's easy to save a model for future use, you can use the `save()` method on the traner\n\n```python\ntrainer.save(path='music_model.di')\n```\n\nNow (in future) we can load our saved model for prediction\n\n```python\nloaded_model = loadModel(filename='music_model.di')\n```\n\nNow let's make some predictions on some data\n\n```python\ndata_to_predict = pd.DataFrame(\n    data={\n        '  age ': [20, 35],\n        'gender': [1, 0]\n    })\n\ndata_to_predict\n```\n\n\u003cdiv\u003e\n\u003ctable border=\"1\" class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003eage\u003c/th\u003e\n      \u003cth\u003egender\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003e0\u003c/th\u003e\n      \u003ctd\u003e20\u003c/td\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e1\u003c/th\u003e\n      \u003ctd\u003e35\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n```python\npredicted = loaded_model.predict(X=data_to_predict)\n```\n\n```python\ndata_to_predict['predicted'] = predicted\n```\n\n```python\ndata_to_predict\n```\n\n\u003cdiv\u003e\n\u003ctable border=\"1\" class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003eage\u003c/th\u003e\n      \u003cth\u003egender\u003c/th\u003e\n      \u003cth\u003epredicted\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003e0\u003c/th\u003e\n      \u003ctd\u003e20\u003c/td\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n      \u003ctd\u003eHipHop\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e1\u003c/th\u003e\n      \u003ctd\u003e35\u003c/td\u003e\n      \u003ctd\u003e0\u003c/td\u003e\n      \u003ctd\u003eClassical\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n```python\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdataideaorg%2Fdataidea-analysis-package","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdataideaorg%2Fdataidea-analysis-package","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdataideaorg%2Fdataidea-analysis-package/lists"}