{"id":15176651,"url":"https://github.com/kengz/feature_transform","last_synced_at":"2026-02-15T07:33:07.350Z","repository":{"id":40687528,"uuid":"503953805","full_name":"kengz/feature_transform","owner":"kengz","description":"Build Scikit ColumnTransformers by specifying configs.","archived":false,"fork":false,"pushed_at":"2025-01-12T18:23:20.000Z","size":626,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-08T07:46:43.670Z","etag":null,"topics":["auto-ml","automated-feature-preprocessor","columntransformer","data-preprocessing","feature-engineerig","machine-learning","scikit-learn"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kengz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-06-15T23:46:26.000Z","updated_at":"2025-01-12T18:22:56.000Z","dependencies_parsed_at":"2025-02-22T18:43:23.347Z","dependency_job_id":"86fb69d1-e696-4ab5-bbe0-08fbecaa5255","html_url":"https://github.com/kengz/feature_transform","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/kengz/feature_transform","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kengz%2Ffeature_transform","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kengz%2Ffeature_transform/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kengz%2Ffeature_transform/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kengz%2Ffeature_transform/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kengz","download_url":"https://codeload.github.com/kengz/feature_transform/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kengz%2Ffeature_transform/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29472880,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-15T06:58:05.414Z","status":"ssl_error","status_checked_at":"2026-02-15T06:58:05.085Z","response_time":118,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["auto-ml","automated-feature-preprocessor","columntransformer","data-preprocessing","feature-engineerig","machine-learning","scikit-learn"],"created_at":"2024-09-27T13:23:10.740Z","updated_at":"2026-02-15T07:33:07.322Z","avatar_url":"https://github.com/kengz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Feature Transform\n\n![Test](https://github.com/github/docs/actions/workflows/test.yml/badge.svg)\n\nBuild [Scikit ColumnTransformers](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html) by specifying configs.\n\n\u003e See also [TorchArc](https://github.com/kengz/torcharc) to build PyTorch models by specifying architectures.\n\n## Installation\n\n```bash\npip install feature_transform\n```\n\n## Usage\n\n1. specify column transformers in a YAML spec file, e.g. at `spec_filepath = \"./example/spec/basic.yaml\"`\n2. `import feature_transform as ft`.\n   1. (optional) if you have custom sklearn estimator/preprocessor, e.g. `Dummy`, register it with `ft.register_class(Dummy)`\n3. build with: `col_tfm = ft.build(spec_filepath)`\n\nThe returned object is a sklearn `ColumnTransformer` ready for normal use.\n\nSee more examples below, then see how it works at the end.\n\n---\n\n### Example: build ColumnTransformer from spec file\n\n```python\nfrom pathlib import Path\n\nimport joblib\nimport yaml\nfrom sklearn import datasets\n\nimport feature_transform as ft\n\nfilepath = Path(\".\") / \"feature_transform\" / \"example\" / \"spec\" / \"basic.yaml\"\n\n# The following are equivalent:\n\n# 1. build from YAML spec file\ncol_tfm = ft.build(filepath)\n\n# 2. build from dictionary\nwith filepath.open(\"r\") as f:\n    spec_dict = yaml.safe_load(f)\ncol_tfm = ft.build(spec_dict)\n\n# 3. use the underlying Pydantic validator to build the col_tfm\nspec = ft.Spec(**spec_dict)\ncol_tfm = spec.build()\n```\n\n---\n\nNext, load demo data for examples below:\n\n```python\n# ================================================\n# Load demo data\n\nx_df, y_sr = datasets.load_wine(return_X_y=True, as_frame=True)\n\nx_df.columns\n# Index(['alcohol', 'malic_acid', 'ash', 'alcalinity_of_ash', 'magnesium',\n#        'total_phenols', 'flavanoids', 'nonflavanoid_phenols',\n#        'proanthocyanins', 'color_intensity', 'hue',\n#        'od280/od315_of_diluted_wines', 'proline'],\n#       dtype='object')\n```\n\n---\n\n### Example: basic\n\nSpec file: [feature_transform/example/spec/basic.yaml](feature_transform/example/spec/basic.yaml)\n\n```yaml\ntransformers:\n  - transformer:\n      preprocessing.StandardScaler:\n    columns: [alcohol, total_phenols]\n  - transformer:\n      preprocessing.RobustScaler:\n    columns: [ash]\n```\n\n```python\ncol_tfm = ft.build(ft.SPEC_DIR / \"basic.yaml\")\n\nfeat_xs = col_tfm.fit_transform(x_df)\nfeat_xs\n# array([[ 1.51861254,  0.80899739,  0.20143885],\n#        ...,\n\n# save for later use\njoblib.dump(col_tfm, \"col_tfm.joblib\")\n\n# ... later, e.g. during batch inference\nloaded_col_tfm = joblib.load(\"col_tfm.joblib\")\nfeat_xs = loaded_col_tfm.transform(x_df)\n```\n\nColumnTransformer `col_tfm`:\n\n![](images/basic.png)\n\n---\n\n### Example: basic with pandas/polars dataframe\n\nSpec file: [feature_transform/example/spec/basic.yaml](feature_transform/example/spec/basic.yaml)\n\n```yaml\ntransformers:\n  - transformer:\n      preprocessing.StandardScaler:\n    columns: [alcohol, total_phenols]\n  - transformer:\n      preprocessing.RobustScaler:\n    columns: [ash]\n```\n\n```python\ncol_tfm = ft.build(ft.SPEC_DIR / \"basic.yaml\")\n# to use with dataframe, set output to \"pandas\" or \"polars\"\ncol_tfm.set_output(transform=\"pandas\")\n\nfeat_x_df = col_tfm.fit_transform(x_df)\nfeat_x_df\n# \tstandardscaler__alcohol\tstandardscaler__total_phenols\trobustscaler__ash\n# 0\t1.518613\t0.808997\t0.201439\n# 1\t0.246290\t0.568648\t-0.633094\n# ...\n\nfeat_x_df.describe()\n# \tstandardscaler__alcohol\tstandardscaler__total_phenols\trobustscaler__ash\n# count\t1.780000e+02\t178.000000\t178.000000\n# mean\t-8.382808e-16\t0.000000\t0.018754\n# std\t1.002821e+00\t1.002821\t0.789479\n# ...\n\n# save for later use\njoblib.dump(col_tfm, \"col_tfm.joblib\")\n\n# ... later, e.g. during batch inference\nloaded_col_tfm = joblib.load(\"col_tfm.joblib\")\nfeat_x_df = loaded_col_tfm.transform(x_df)\n```\n\nColumnTransformer `col_tfm`:\n\n![](images/basic.png)\n\n---\n\n### Example: specify name; use int columns\n\nSpec file: [feature_transform/example/spec/name-intcol.yaml](feature_transform/example/spec/name-intcol.yaml)\n\n```yaml\ntransformers:\n  - name: std\n    transformer:\n      preprocessing.StandardScaler:\n    columns: [0, 5]\n  - name: robust\n    transformer:\n      preprocessing.RobustScaler:\n    columns: [2]\n```\n\n```python\ncol_tfm = ft.build(ft.SPEC_DIR / \"name-intcol.yaml\")\n\nfeat_xs = col_tfm.fit_transform(x_df)\n# array([[ 1.51861254,  0.80899739,  0.20143885],\n#        ...,\n```\n\nColumnTransformer `col_tfm`:\n\n![](images/name-intcol.png)\n\n---\n\n### Example: pipeline\n\nSpec file: [feature_transform/example/spec/pipeline.yaml](feature_transform/example/spec/pipeline.yaml)\n\n```yaml\ntransformers:\n  - transformer:\n      preprocessing.StandardScaler:\n    columns: [alcohol, total_phenols]\n  - transformer:\n      Pipeline:\n        - impute.SimpleImputer:\n            strategy: constant\n        - preprocessing.RobustScaler:\n    columns: [ash]\n```\n\n```python\ncol_tfm = ft.build(ft.SPEC_DIR / \"pipeline.yaml\")\n\nfeat_xs = col_tfm.fit_transform(x_df)\nfeat_xs\n# array([[ 1.51861254,  0.80899739,  0.20143885],\n#        ...,\n```\n\nColumnTransformer `col_tfm`:\n\n![](images/pipeline.png)\n\n---\n\n### Example: ColumnTransformer settings\n\nSpec file: [feature_transform/example/spec/settings.yaml](feature_transform/example/spec/settings.yaml)\n\n```yaml\ntransformers:\n  - transformer:\n      preprocessing.StandardScaler:\n    columns: [alcohol, total_phenols]\n  - transformer:\n      preprocessing.RobustScaler:\n    columns: [ash]\n# use all processors\nn_jobs: -1\n# for more kwargs see https://scikit-learn.org/stable/modules/generated/sklearn.compose.make_column_transformer.html\n```\n\n```python\ncol_tfm = ft.build(ft.SPEC_DIR / \"settings.yaml\")\n\nfeat_xs = col_tfm.fit_transform(x_df)\nfeat_xs\n# array([[ 1.51861254,  0.80899739,  0.20143885],\n#        ...,\n```\n\nColumnTransformer `col_tfm`:\n\n![](images/settings.png)\n\n---\n\n### Example: full X, y feature transform with save/load\n\nSpec file (x): [feature_transform/example/spec/wine/x.yaml](feature_transform/example/spec/wine/x.yaml)\n\n```yaml\ntransformers:\n  - transformer:\n      preprocessing.StandardScaler:\n    columns: [alcohol, total_phenols, flavanoids, nonflavanoid_phenols, od280/od315_of_diluted_wines]\n  - transformer:\n      preprocessing.RobustScaler:\n    columns: [ash, alcalinity_of_ash, proanthocyanins, hue]\n  - transformer:\n      preprocessing.PowerTransformer:\n    columns: [malic_acid, magnesium, color_intensity, proline]\nn_jobs: -1\n```\n\nSpec file (y): [feature_transform/example/spec/wine/y.yaml](feature_transform/example/spec/wine/y.yaml)\n\n```yaml\ntransformers:\n  - transformer:\n      preprocessing.OneHotEncoder:\n        sparse_output: False\n    columns: [target]\n```\n\n```python\nimport joblib\nfrom sklearn import datasets\n\nimport feature_transform as ft\n\nx_df, y_sr = datasets.load_wine(return_X_y=True, as_frame=True)\ny_df = y_sr.to_frame()  # ColumnTransformer takes only dataframe/matrix as input\n\nx_col_tfm = ft.build(ft.SPEC_DIR / \"wine\" / \"x.yaml\")\ny_col_tfm = ft.build(ft.SPEC_DIR / \"wine\" / \"y.yaml\")\n\n# fit-transform\nfeat_xs = x_col_tfm.fit_transform(x_df)\nfeat_xs\n# array([[ 1.51861254,  0.80899739,  1.03481896, ...,  1.69074868,\n#          0.45145022,  1.06254129],\n#        ...,\n\nfeat_ys = y_col_tfm.fit_transform(y_df)\nfeat_ys\n# array([[1., 0., 0.],\n#        ...,\n\n# save for later use\njoblib.dump(x_col_tfm, \"x_col_tfm.joblib\")\njoblib.dump(y_col_tfm, \"y_col_tfm.joblib\")\n\n\n# ... later, e.g. during batch inference\nloaded_x_col_tfm = joblib.load(\"x_col_tfm.joblib\")\nfeat_xs = loaded_x_col_tfm.transform(x_df)\nfeat_xs\n# array([[ 1.51861254,  0.80899739,  1.03481896, ...,  1.69074868,\n#          0.45145022,  1.06254129],\n#        ...,\n```\n\nColumnTransformer `x_col_tfm`:\n\n![](images/wine-x.png)\n\nColumnTransformer `y_col_tfm`:\n\n![](images/wine-y.png)\n\n---\n\n### Example: use helper to suggest spec\n\nMost of the time, data preprocessing steps can be determined with rules-of-thumb; `ft.suggest` does exactly that (see [feature_transform/helper.py](feature_transform/helper.py) for details). This produces `spec_dict` that can be used directly with `ft.build` or for further editing.\n\n```python\nx_df, y_sr = datasets.load_wine(return_X_y=True, as_frame=True)\n\n# suggest spec_dict - use directly or save to yaml for further editing\nspec_dict = ft.suggest(x_df)\ncol_tfm = ft.build(spec_dict)\n\n# fit-transform\nfeat_xs = col_tfm.fit_transform(x_df)\nfeat_xs\n# array([[ 0.8973384 ,  0.20143885, -0.90697674, ...,  0.80804954,\n#         -0.43546273,  1.69074868],\n#         ...,\n```\n\nColumnTransformer `col_tfm`:\n\n![](images/suggest.png)\n\n---\n\n### Example: more\n\nSee more examples:\n\n- demo notebook from above [feature_transform/example/notebook/demo.py](feature_transform/example/notebook/demo.py)\n- spec files [feature_transform/example/spec/](feature_transform/example/spec/)\n- unit tests [test/validator/test_spec.py](test/validator/test_spec.py)\n\n## How does it work\n\nFeature Transform simply builds sklearn ColumnTransformer and its estimators/pipelines with 1-1 mapping from a spec file:\n\n1. Spec is defined via Pydantic [feature_transform/validator/](feature_transform/validator/). This defines:\n   - `spec`: the `Estimator, Pipeline, ColumnTransformer`\n2. If spec specifies:\n   1. `transformers=list[(name, transformer, columns)]`, then use [`ColumnTransformer`](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html)\n   1. `transformers=list[(transformer, columns)]`, then use [`make_column_transformer`](https://scikit-learn.org/stable/modules/generated/sklearn.compose.make_column_transformer.html) with auto-generated names\n\nSee more in the pydantic spec definition:\n\n- [feature_transform/validator/spec.py](feature_transform/validator/spec.py): the spec used by feature_transform\n\n### Guiding principles\n\nThe design of Feature Transform is guided as follows:\n\n1. simple: the module spec is straightforward:\n   1. it is simply sklearn class name with kwargs.\n   1. it supports official `sklearn` estimators, `Pipeline`, and custom-defined modules registered via `ft.register_class`\n1. expressive: it can be used to build both simple and advanced `ColumnTransformer` easily\n1. portable: it returns `ColumnTransformer` that can be used anywhere; it is not a framework.\n1. parametrizable: data-based feature transformation unlocks fast experimentation, e.g. by building logic for hyperparameter / data feature search\n\n## Development\n\n### Setup\n\n[Install uv](https://docs.astral.sh/uv/getting-started/installation/) for dependency management if you haven't already. Then run:\n\n```bash\n# setup virtualenv\nuv sync\n```\n\n### Unit Tests\n\n```bash\nuv run pytest\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkengz%2Ffeature_transform","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkengz%2Ffeature_transform","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkengz%2Ffeature_transform/lists"}