{"id":15063970,"url":"https://github.com/tslu1s/atlantic","last_synced_at":"2025-04-10T11:50:23.191Z","repository":{"id":62592946,"uuid":"534410108","full_name":"TsLu1s/Atlantic","owner":"TsLu1s","description":"Atlantic: Automated Data Preprocessing Framework for Supervised Machine Learning","archived":false,"fork":false,"pushed_at":"2024-06-17T21:17:35.000Z","size":1995,"stargazers_count":11,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-09-04T11:15:52.493Z","etag":null,"topics":["automation","automl","automl-pipeline","data-preprocessing","data-science","feature-selection","label-encoder","machine-learning","onehot-encoder","predictive-maintenance","predictive-modeling","preprocessing-pipeline","python","scikit-learn"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TsLu1s.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-08T22:11:36.000Z","updated_at":"2024-06-17T21:17:38.000Z","dependencies_parsed_at":"2022-11-03T23:05:43.235Z","dependency_job_id":"b563c673-3639-42a3-be36-86ace3a06d1a","html_url":"https://github.com/TsLu1s/Atlantic","commit_stats":{"total_commits":247,"total_committers":2,"mean_commits":123.5,"dds":"0.14574898785425106","last_synced_commit":"b2f4100fc5757162562c0409dc68cd205511bf42"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TsLu1s%2FAtlantic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TsLu1s%2FAtlantic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TsLu1s%2FAtlantic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TsLu1s%2FAtlantic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TsLu1s","download_url":"https://codeload.github.com/TsLu1s/Atlantic/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248215192,"owners_count":21066619,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","automl","automl-pipeline","data-preprocessing","data-science","feature-selection","label-encoder","machine-learning","onehot-encoder","predictive-maintenance","predictive-modeling","preprocessing-pipeline","python","scikit-learn"],"created_at":"2024-09-25T00:09:32.099Z","updated_at":"2025-04-10T11:50:23.174Z","avatar_url":"https://github.com/TsLu1s.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![LinkedIn][linkedin-shield]][linkedin-url]\n[![Contributors][contributors-shield]][contributors-url]\n[![Stargazers][stars-shield]][stars-url]\n[![MIT License][license-shield]][license-url]\n[![Downloads][downloads-shield]][downloads-url]\n[![Month Downloads][downloads-month-shield]][downloads-month-url]\n\n[contributors-shield]: https://img.shields.io/github/contributors/TsLu1s/Atlantic.svg?style=for-the-badge\u0026logo=github\u0026logoColor=white\n[contributors-url]: https://github.com/TsLu1s/Atlantic/graphs/contributors\n[stars-shield]: https://img.shields.io/github/stars/TsLu1s/Atlantic.svg?style=for-the-badge\u0026logo=github\u0026logoColor=white\n[stars-url]: https://github.com/TsLu1s/Atlantic/stargazers\n[license-shield]: https://img.shields.io/github/license/TsLu1s/Atlantic.svg?style=for-the-badge\u0026logo=opensource\u0026logoColor=white\n[license-url]: https://github.com/TsLu1s/Atlantic/blob/main/LICENSE\n[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=for-the-badge\u0026logo=linkedin\u0026colorB=555\n[linkedin-url]: https://www.linkedin.com/in/luísfssantos/\n[downloads-shield]: https://static.pepy.tech/personalized-badge/atlantic?period=total\u0026units=international_system\u0026left_color=grey\u0026right_color=blue\u0026left_text=Total%20Downloads\n[downloads-url]: https://pepy.tech/project/atlantic\n[downloads-month-shield]: https://static.pepy.tech/personalized-badge/atlantic?period=month\u0026units=international_system\u0026left_color=grey\u0026right_color=blue\u0026left_text=Month%20Downloads\n[downloads-month-url]: https://pepy.tech/project/atlantic\n\n\u003cbr\u003e\n\u003cp align=\"center\"\u003e\n  \u003ch2 align=\"center\"\u003e Atlantic - Automated Data Preprocessing Framework for Supervised Machine Learning\n  \u003cbr\u003e\n  \n## Framework Contextualization \u003ca name = \"ta\"\u003e\u003c/a\u003e\n\nThe `Atlantic` project constitutes an comprehensive and objective approach to simplify and automate data processing through the integration and objectively validated application of various preprocessing mechanisms, ranging from feature engineering, automated feature selection, multiple encoding versions and null imputation methods. The optimization methodology of this framework follows a evaluation structured in tree based models ensembles.\n\nThis project aims at providing the following application capabilities:\n\n* General applicability on tabular datasets: The developed preprocessing procedures are applicable on multiple domains associated with Supervised Machine Learning, regardless of the properties or specifications of the data.\n\n* Automated treatment of tabular data associated with predictive analysis: It implements a global and carefully validated data processing based on the characteristics of the data input columns.\n\n* Robustness and improvement of predictive results: The implementation of the `atlantic` automated data preprocessing pipeline aims at improving predictive performance directly associated with the processing methods implemented based on the data properties.  \n   \n#### Main Development Tools \u003ca name = \"pre1\"\u003e\u003c/a\u003e\n\nMajor frameworks used to built this project: \n   \n* [H2O.ai](https://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html)\n* [Scikit-learn](https://scikit-learn.org/stable/)\n* [XGBoost](https://xgboost.readthedocs.io/en/stable/)\n* [Optuna](https://optuna.org/)\n* [Pandas](https://pandas.pydata.org/)\n\n    \n## Framework Architecture \u003ca name = \"ta\"\u003e\u003c/a\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://i.ibb.co/C9dWJmk/ATL-Architecture-Final.png\" align=\"center\" width=\"700\" height=\"680\" /\u003e\n\u003c/p\u003e    \n\n## Where to get it \u003ca name = \"ta\"\u003e\u003c/a\u003e\n\nBinary installer for the latest released version is available at the Python Package Index [(PyPI)](https://pypi.org/project/atlantic/).  \n\n## Installation  \n\nTo install this package from Pypi repository run the following command:\n\n```\npip install atlantic\n```\n\n# Usage Examples\n    \n## 1. Atlantic - Automated Data Preprocessing Pipeline\n\nIn order to be able to apply the automated preprocessing `atlantic` pipeline you need first to import the package. \nThe following needed step is to load a dataset, split it and define your to be predicted target column name into the variable `target`.\nYou can customize the `fit_processing` method by altering the following running pipeline parameters:\n* split_ratio: Division ratio (Train\\Validation) in which the preprocessing methods will be evaluated within the loaded Dataset.\n* relevance: Minimal value of the total sum of relative feature importance percentage selected in the `H2O AutoML feature selection` step.\n* h2o_fs_models: Quantity of models generated for competition in step `H2O AutoML feature selection` to evaluate the relative importance of each feature (only leaderboard model is selected for evaluation).\n* encoding_fs: You can choose if you want to enconde categorical features in order to reduce loading time in `H2O AutoML feature selection` step.\n* vif_ratio: This value defines the minimal `threshold` for Variance Inflation Factor filtering (default value=10).\n\nOnce the data fitting process is complete, you can automaticaly optimize preprocessing transformations on all future dataframes with the same properties using the `data_processing` method.\n    \n```py\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom atlantic.pipeline import Atlantic\nimport warnings\nwarnings.filterwarnings(\"ignore\", category=Warning) # -\u003e For a clean console\n    \ndata = pd.read_csv('csv_directory_path', encoding='latin', delimiter=',') # Dataframe Loading Example\n# data[\"Target Column\"] = data[\"Target Column\"].astype(str) # -\u003e If Classification Task\n\ntrain,test = train_test_split(data, train_size = 0.8)\ntest,future_data = train_test_split(test, train_size = 0.6)\n\n# Resetting Index is Required\ntrain = train.reset_index(drop=True)\ntest = test.reset_index(drop=True)\nfuture_data = future_data.reset_index(drop=True)\n\nfuture_data.drop(columns=[\"Target_Column\"], inplace=True) # Drop Target\n\n### Fit Data Processing\n\natl = Atlantic(X = train,                # X:pd.DataFrame, target:str=\"Target_Column\"\n               target = \"Target Column\")    \n\natl.fit_processing(split_ratio = 0.75,   # split_ratio:float=0.75, relevance:float=0.99 [0.5,1]\n                   relevance = 0.99,     # h2o_fs_models:int [1,100], encoding_fs:bool=True\\False\n                   h2o_fs_models = 7,    # vif_ratio:float=10.0 [3,30]\n                   encoding_fs = True,\n                   vif_ratio = 10.0)\n\n### Transform Data Processing\n\ntrain = atl.data_processing(X = train)\ntest = atl.data_processing(X = test)\nfuture_data = atl.data_processing(X = future_data)\n\n### Export Atlantic Preprocessing Metadata\n\nimport dill as pickle\nwith open('fit_atl.pkl', 'wb') as output:\n    pickle.dump(atl, output)\n    \n```  \n\n## 2. Atlantic - Preprocessing Data\n    \n### 2.1 Encoding Versions\n \nThere are multiple preprocessing methods available to direct use. This package provides upgrated encoding `LabelEncoder`, `OneHotEncoder` and `InverseFrequency` ([IDF](https://pypi.org/project/cane/) based) methods with an automatic multicolumn application. \n \n```py\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split \nfrom atlantic.processing.encoders import AutoLabelEncoder, AutoIFrequencyEncoder, AutoOneHotEncoder\n\ntrain,test = train_test_split(data, train_size=0.8)\ntrain,test = train.reset_index(drop=True), test.reset_index(drop=True) # Required\n\ntarget = \"Target_Column\" # -\u003e target feature name\n    \ncat_cols = [col for col in data.select_dtypes(include=['object']).columns if col != target]\n\n### Encoders\n## Create Label Encoder\nencoder = AutoLabelEncoder()\n## Create InverseFrequency Encoder\nencoder = AutoIFrequencyEncoder()\n## Create One-hot Encoder\nencoder = AutoOneHotEncoder()\n\n## Fit\nencoder.fit(train[cat_cols])\n\n# Transform the DataFrame using Label\\IF\\One-hot Encoding\ntrain = encoder.transform(X = train)\ntest = encoder.transform(X = test)\n\n# Perform an inverse transform to convert it back the original categorical columns values\ntrain = encoder.inverse_transform(X = train)\ntest = encoder.inverse_transform(X = test)\n            \n```    \n   \n### 2.2 Feature Selection and Null Imputation Methods\n\nAtlantic provides automated feature selection methods (H2O AutoML and VIF-based) and null imputation techniques (Simple, KNN, and Iterative). Check out the \u003ca href=\"https://github.com/TsLu1s/atlantic/edit/main/examples/custom_preprocessing.py\" style=\"text-decoration:none;\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Custom%20Preprocessing-blue?style=for-the-badge\u0026logo=readme\u0026logoColor=white\" alt=\"Custom Preprocessing\"\u003e\n\u003c/a\u003e for detailed implementations of all preprocessing methods integrated in `Atlantic`.\n\n## Citation\n\nFeel free to cite Atlantic as following:\n\n```\n\n@article{SANTOS2023100532,\n  author = {Luis Santos and Luis Ferreira}\n  title = {Atlantic - Automated data preprocessing framework for supervised machine learning},\n  journal = {Software Impacts},\n  volume = {17},\n  year = {2023},\n  issn = {2665-9638},\n  doi = {http://dx.doi.org/10.1016/j.simpa.2023.100532},\n  url = {https://www.sciencedirect.com/science/article/pii/S2665963823000696}\n}\n\n```\n    \n## License\n\nDistributed under the MIT License. See [LICENSE](https://github.com/TsLu1s/Atlantic/blob/main/LICENSE) for more information.\n\n## Contact \n \n[Luis Santos - LinkedIn](https://www.linkedin.com/in/lu%C3%ADsfssantos/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftslu1s%2Fatlantic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftslu1s%2Fatlantic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftslu1s%2Fatlantic/lists"}