{"id":14970702,"url":"https://github.com/sk8997/pipeline-optimizer","last_synced_at":"2025-10-26T13:31:10.971Z","repository":{"id":153276654,"uuid":"627256593","full_name":"sk8997/pipeline-optimizer","owner":"sk8997","description":"Preprocessing infrastructure that simplifies and automates the machine learning pipeline","archived":false,"fork":false,"pushed_at":"2023-04-18T22:41:44.000Z","size":38,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-09-28T13:23:11.166Z","etag":null,"topics":["data-science","machine-learning","pandas","pipeline","preprocessing","sklearn","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sk8997.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-04-13T05:13:30.000Z","updated_at":"2023-04-19T19:09:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"1ff81d09-1107-46a4-a84a-d2798f4c0894","html_url":"https://github.com/sk8997/pipeline-optimizer","commit_stats":{"total_commits":54,"total_committers":2,"mean_commits":27.0,"dds":"0.40740740740740744","last_synced_commit":"a346eab5d23ad1259cc7e81038f79131d11ace82"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sk8997%2Fpipeline-optimizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sk8997%2Fpipeline-optimizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sk8997%2Fpipeline-optimizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sk8997%2Fpipeline-optimizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sk8997","download_url":"https://codeload.github.com/sk8997/pipeline-optimizer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219862887,"owners_count":16555951,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","machine-learning","pandas","pipeline","preprocessing","sklearn","transformers"],"created_at":"2024-09-24T13:44:00.726Z","updated_at":"2025-10-26T13:31:10.671Z","avatar_url":"https://github.com/sk8997.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\n\n![pipe-logo](https://user-images.githubusercontent.com/84877088/232358047-f8545063-5053-4a9e-a24e-e9c266283f5d.png)\n\n\n\n\n\u003cdiv align=\"center\"\u003e\n\u003ca href=\"https://github.com/sk8997/pipeline-optimizer/actions/workflows/tests.yml\"\u003e\n  \u003cimg src=\"https://github.com/sk8997/pipeline-optimizer/actions/workflows/tests.yml/badge.svg\" alt=\"Tests\" /\u003e\n\u003c/a\u003e\n\u003ca href=\"https://codecov.io/gh/sk8997/pipeline-optimizer\" \u003e \n \u003cimg src=\"https://codecov.io/gh/sk8997/pipeline-optimizer/branch/main/graph/badge.svg?token=BCWYCTXZPA\"/\u003e \n \u003c/a\u003e\n\u003c/div\u003e\n\nPipeline Optimizer is a Python library that aims to simplify and automate the machine learning pipeline, from preprocessing and testing to deployment. By providing a reusable infrastructure, the library allows you to manage custom preprocessing functions and reuse them effortlessly during the deployment of your project. This is particularly useful when dealing with a large number of custom functions.\n\nThe library currently features a single class called `SequentialTransformer` which allows you to add custom preprocessing functions using a simple decorator. This class also integrates with scikit-learn's `TransformerMixin`, making it compatible with the widely-used scikit-learn library.\n\n# Installation\n\n```bash\npip install pipeline_optimizer\n```\n\n# SequentialTransformer\n\n`SequentialTransformer` is a class that stores a list of preprocessing steps and applies them sequentially to input data. You can easily add a custom preprocessing function to its memory using the `@add_step` decorator. The class also provides methods to transform the input data, save the transformer to disk, and load it for later use.\n\n\nHere's a quick demonstration of how to use the `SequentialTransformer` class:\n\n# Step 1: Import necessary libraries\n\n```python \n\nimport pandas as pd\nfrom pipeline_optimizer import SequentialTransformer, add_step\n\n```\n\n\n# Step 2: Load your dataset\n\n```python\n\ndata = pd.DataFrame({\n    \"A\": [1, 2, 3, 4, 5],\n    \"B\": [5, 4, 3, 2, 1],\n    \"C\": [10, 20, 30, 40, 50]\n})\n\nlabels = pd.Series([0, 1, 0, 1, 1])\n\n```\n\n# Step 3: Define preprocessing functions and add them to the pipeline\n\n```python\n\npipe = SequentialTransformer()\n\n@add_step(pipe)\ndef drop_column(df: pd.DataFrame, col: str = \"B\") -\u003e pd.DataFrame:\n    return df.drop(columns=[col])\n\n@add_step(pipe)\ndef multiply(df: pd.DataFrame, col: str = \"A\", multiplier: float = 2) -\u003e pd.DataFrame:\n    df[col] = df[col] * multiplier\n    return df\n\n```\n\n# Step 4: Transform the input data\nAfter applying the preprocessing functions, the SequentialTransformer will drop column \"B\" and multiply column \"A\" by 2.\n\n```python\n\ntransformed_data = pipe.transform(data)\nprint(transformed_data)\n\n```\n\nOutput:\n\n```\n\n    A   C\n0   2  10\n1   4  20\n2   6  30\n3   8  40\n4  10  50\n\n```\n\n# \n\nStep 5: Save the transformer object\n\n```python\npipe.save(\"transformer.pkl\")\n```\n\n# Step 6: Load the saved transformer and apply it to deployment data\nYou can load the saved transformer using the pickle module and apply it to new deployment data to preprocess it.\n\n```python \n\nimport pickle\n\n# Load the saved transformer\nwith open(\"transformer.pkl\", \"rb\") as f:\n    loaded_pipe = pickle.load(f)\n\n# Deployment data\ndeployment_data = pd.DataFrame({\n    \"A\": [6],\n    \"B\": [3],\n    \"C\": [60]\n})\n\n# Transform the deployment data using the loaded transformer\ntransformed_deployment_data = loaded_pipe.transform(deployment_data)\nprint(transformed_deployment_data)\n\n```\n\nOutput:\n\n```\n   A   C\n0  12  60\n\n```\n\n# Integration with scikit-learn Pipeline\n\nA noteworthy feature of the `SequentialTransformer` is that it can be seamlessly integrated with scikit-learn's `Pipeline` class. This further simplifies the preprocessing and deployment processes, enabling you to create an end-to-end machine learning pipeline that combines custom preprocessing steps with scikit-learn estimators.\n\nBy incorporating the `SequentialTransformer` into an sklearn `Pipeline`, you can benefit from the full range of features provided by scikit-learn, such as cross-validation, grid search, and model evaluation.\n\nHere's a quick example of how to integrate initialized `SequentialTransformer` with an sklearn `Pipeline`:\n\n```python\n\npipe = SequentialTransformer()\n\n@add_step(pipe)\ndef drop_column(df: pd.DataFrame, col: str = \"B\") -\u003e pd.DataFrame:\n    return df.drop(columns=[col])\n\n@add_step(pipe)\ndef multiply(df: pd.DataFrame, col: str = \"A\", multiplier: float = 2) -\u003e pd.DataFrame:\n    df[col] = df[col] * multiplier\n    return df\n\n# Create an sklearn pipeline with the custom SequentialTransformer and a Linear Discriminant Analysis\npipeline = Pipeline([\n    (\"preprocessor\", pipe),  # Ensure the SequentialTransformer has been initialized and steps have been added\n    (\"lda\", LinearDiscriminantAnalysis())\n])\n\n# Fit the pipeline \npipeline.fit_transform(X, y)\n\n\n```\n\n\n# Comparison with scikit-learn\n\nWhen working with custom preprocessing functions using the scikit-learn library, you would typically define a custom class that inherits from `TransformerMixin` and implement `fit` and `transform` methods for each function. This can be time-consuming and may lead to code duplication.\n\nAlternatively, you can use scikit-learn's `FunctionTransformer` to create transformers from user-defined functions. However, using `FunctionTransformer` can become unwieldy when you have many preprocessing functions, as you need to create an instance of `FunctionTransformer` for each function and manage them individually.\n\nHere's an example of how you would use `FunctionTransformer` to accomplish the same preprocessing steps as in the previous example:\n\n```python\n\nfrom sklearn.preprocessing import FunctionTransformer\n\n# Define the preprocessing functions\ndef drop_column(df: pd.DataFrame, col: str) -\u003e pd.DataFrame:\n    return df.drop(columns=[col])\n\ndef multiply(df: pd.DataFrame, col: str, multiplier: float) -\u003e pd.DataFrame:\n    df[col] = df[col] * multiplier\n    return df\n\n# Create FunctionTransformer instances for each function\ndrop_column_transformer = FunctionTransformer(drop_column, kw_args={\"col\": \"B\"})\nmultiply_transformer = FunctionTransformer(multiply, kw_args={\"col\": \"A\", \"multiplier\": 2})\n\n# Apply the preprocessing functions to the toy dataset\ndata_dropped = drop_column_transformer.transform(data)\ndata_transformed = multiply_transformer.transform(data_dropped)\n\n```\n\nAs you can see, using `FunctionTransformer` requires creating separate instances for each preprocessing function and managing them individually. This approach can become cumbersome when dealing with a large number of custom functions. In contrast, the `SequentialTransformer` class in the Pipeline Optimizer library provides a more streamlined and efficient way to manage and apply multiple preprocessing functions.\n\nWith the Pipeline Optimizer library, you can easily define preprocessing functions and add them to the `SequentialTransformer` pipeline using the `@add_step` decorator. This approach is more concise and allows you to reuse your preprocessing functions across different projects effortlessly.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsk8997%2Fpipeline-optimizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsk8997%2Fpipeline-optimizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsk8997%2Fpipeline-optimizer/lists"}