{"id":13578225,"url":"https://github.com/jrzaurin/pytorch-widedeep","last_synced_at":"2025-05-14T00:09:17.351Z","repository":{"id":40617415,"uuid":"107763164","full_name":"jrzaurin/pytorch-widedeep","owner":"jrzaurin","description":"A flexible package for multimodal-deep-learning to combine tabular data with text and images using Wide and Deep models in Pytorch","archived":false,"fork":false,"pushed_at":"2025-02-27T12:56:11.000Z","size":104442,"stargazers_count":1344,"open_issues_count":6,"forks_count":194,"subscribers_count":23,"default_branch":"master","last_synced_at":"2025-05-03T04:33:36.639Z","etag":null,"topics":["deep-learning","images","model-hub","multimodal-deep-learning","python","pytorch","pytorch-cv","pytorch-nlp","pytorch-tabular-data","pytorch-transformers","tabular-data","text"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jrzaurin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.MD","funding":null,"license":"LICENSE_APACHE_v2_0","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-10-21T08:11:44.000Z","updated_at":"2025-05-02T08:47:59.000Z","dependencies_parsed_at":"2024-04-04T14:28:20.235Z","dependency_job_id":"feaf9d0a-dddd-48f5-bf0c-143aa4b849a1","html_url":"https://github.com/jrzaurin/pytorch-widedeep","commit_stats":{"total_commits":633,"total_committers":8,"mean_commits":79.125,"dds":"0.24644549763033174","last_synced_commit":"3026c1ff11bf02b55eb2b6f7e26aacde03e18323"},"previous_names":[],"tags_count":31,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jrzaurin%2Fpytorch-widedeep","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jrzaurin%2Fpytorch-widedeep/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jrzaurin%2Fpytorch-widedeep/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jrzaurin%2Fpytorch-widedeep/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jrzaurin","download_url":"https://codeload.github.com/jrzaurin/pytorch-widedeep/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254044117,"owners_count":22005078,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","images","model-hub","multimodal-deep-learning","python","pytorch","pytorch-cv","pytorch-nlp","pytorch-tabular-data","pytorch-transformers","tabular-data","text"],"created_at":"2024-08-01T15:01:28.560Z","updated_at":"2025-05-14T00:09:12.343Z","avatar_url":"https://github.com/jrzaurin.png","language":"Python","funding_links":[],"categories":["Python","Open-source Related 开源开发相关","Tools \u0026 Libraries"],"sub_categories":["Hybrid Learning"],"readme":"\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"300\" src=\"mkdocs/sources/docs/figures/widedeep_logo.png\"\u003e\n\u003c/p\u003e\n\n[![PyPI version](https://badge.fury.io/py/pytorch-widedeep.svg)](https://pypi.org/project/pytorch-widedeep/)\n[![Python 3.8 3.9 3.10 3.11](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11-blue.svg)](https://pypi.org/project/pytorch-widedeep/)\n[![Build Status](https://github.com/jrzaurin/pytorch-widedeep/actions/workflows/build.yml/badge.svg)](https://github.com/jrzaurin/pytorch-widedeep/actions)\n[![Documentation Status](https://readthedocs.org/projects/pytorch-widedeep/badge/?version=latest)](https://pytorch-widedeep.readthedocs.io/en/latest/?badge=latest)\n[![codecov](https://codecov.io/gh/jrzaurin/pytorch-widedeep/branch/master/graph/badge.svg)](https://codecov.io/gh/jrzaurin/pytorch-widedeep)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/jrzaurin/pytorch-widedeep/graphs/commit-activity)\n[![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/jrzaurin/pytorch-widedeep/issues)\n[![Slack](https://img.shields.io/badge/slack-chat-green.svg?logo=slack)](https://join.slack.com/t/pytorch-widedeep/shared_invite/zt-soss7stf-iXpVuLeKZz8lGTnxxtHtTw)\n[![DOI](https://joss.theoj.org/papers/10.21105/joss.05027/status.svg)](https://doi.org/10.21105/joss.05027)\n\n# pytorch-widedeep\n\nA flexible package for multimodal-deep-learning to combine tabular data with\ntext and images using Wide and Deep models in Pytorch\n\n**Documentation:** [https://pytorch-widedeep.readthedocs.io](https://pytorch-widedeep.readthedocs.io/en/latest/index.html)\n\n**Companion posts and tutorials:** [infinitoml](https://jrzaurin.github.io/infinitoml/)\n\n**Experiments and comparison with `LightGBM`**: [TabularDL vs LightGBM](https://github.com/jrzaurin/tabulardl-benchmark)\n\n**Slack**: if you want to contribute or just want to chat with us, join [slack](https://join.slack.com/t/pytorch-widedeep/shared_invite/zt-soss7stf-iXpVuLeKZz8lGTnxxtHtTw)\n\nThe content of this document is organized as follows:\n\n- [pytorch-widedeep](#pytorch-widedeep)\n    - [Introduction](#introduction)\n    - [Architectures](#architectures)\n    - [The ``deeptabular`` component](#the-deeptabular-component)\n    - [The ``rec`` module](#the-rec-module)\n    - [Text and Images](#text-and-images)\n    - [Installation](#installation)\n      - [Developer Install](#developer-install)\n    - [Quick start](#quick-start)\n    - [Testing](#testing)\n    - [How to Contribute](#how-to-contribute)\n    - [Acknowledgments](#acknowledgments)\n    - [License](#license)\n    - [Cite](#cite)\n      - [BibTex](#bibtex)\n      - [APA](#apa)\n\n### Introduction\n\n``pytorch-widedeep`` is based on Google's [Wide and Deep Algorithm](https://arxiv.org/abs/1606.07792),\nadjusted for multi-modal datasets.\n\nIn general terms, `pytorch-widedeep` is a package to use deep learning with\ntabular data. In particular, is intended to facilitate the combination of\ntext and images with corresponding tabular data using wide and deep models.\nWith that in mind there are a number of architectures that can be implemented\nwith the library. The main components of those architectures are shown in the\nFigure below:\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"750\" src=\"mkdocs/sources/docs/figures/widedeep_arch_new.png\"\u003e\n\u003c/p\u003e\n\n\nIn math terms, and following the notation in the\n[paper](https://arxiv.org/abs/1606.07792), the expression for the architecture\nwithout a ``deephead`` component can be formulated as:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"500\" src=\"mkdocs/sources/docs/figures/architecture_1_math.png\"\u003e\n\u003c/p\u003e\n\n\nWhere \u0026sigma; is the sigmoid function, *'W'* are the weight matrices applied to the wide model and to the final\nactivations of the deep models, *'a'* are these final activations,\n\u0026phi;(x) are the cross product transformations of the original features *'x'*, and\n, and *'b'* is the bias term.\nIn case you are wondering what are *\"cross product transformations\"*, here is\na quote taken directly from the paper: *\"For binary features, a cross-product\ntransformation (e.g., “AND(gender=female, language=en)”) is 1 if and only if\nthe constituent features (“gender=female” and “language=en”) are all 1, and 0\notherwise\".*\n\nIt is perfectly possible to use custom models (and not necessarily those in\nthe library) as long as the the custom models have an property called\n``output_dim`` with the size of the last layer of activations, so that\n``WideDeep`` can be constructed. Examples on how to use custom components can\nbe found in the Examples folder and the section below.\n\n### Architectures\n\nThe `pytorch-widedeep` library offers a number of different architectures. In\nthis section we will show some of them in their simplest form (i.e. with\ndefault param values in most cases) with their corresponding code snippets.\nNote that **all** the snippets below shoud run locally. For a more detailed\nexplanation of the different components and their parameters, please refer to\nthe documentation.\n\nFor the examples below we will be using a toy dataset generated as follows:\n\n```python\nimport os\nimport random\n\nimport numpy as np\nimport pandas as pd\nfrom PIL import Image\nfrom faker import Faker\n\n\ndef create_and_save_random_image(image_number, size=(32, 32)):\n\n    if not os.path.exists(\"images\"):\n        os.makedirs(\"images\")\n\n    array = np.random.randint(0, 256, (size[0], size[1], 3), dtype=np.uint8)\n\n    image = Image.fromarray(array)\n\n    image_name = f\"image_{image_number}.png\"\n    image.save(os.path.join(\"images\", image_name))\n\n    return image_name\n\n\nfake = Faker()\n\ncities = [\"New York\", \"Los Angeles\", \"Chicago\", \"Houston\"]\nnames = [\"Alice\", \"Bob\", \"Charlie\", \"David\", \"Eva\"]\n\ndata = {\n    \"city\": [random.choice(cities) for _ in range(100)],\n    \"name\": [random.choice(names) for _ in range(100)],\n    \"age\": [random.uniform(18, 70) for _ in range(100)],\n    \"height\": [random.uniform(150, 200) for _ in range(100)],\n    \"sentence\": [fake.sentence() for _ in range(100)],\n    \"other_sentence\": [fake.sentence() for _ in range(100)],\n    \"image_name\": [create_and_save_random_image(i) for i in range(100)],\n    \"target\": [random.choice([0, 1]) for _ in range(100)],\n}\n\ndf = pd.DataFrame(data)\n```\n\nThis will create a 100 rows dataframe and a dir in your local folder, called\n`images` with 100 random images (or images with just noise).\n\nPerhaps the simplest architecture would be just one component, `wide`,\n`deeptabular`, `deeptext` or `deepimage` on their own, which is also\npossible, but let's start the examples with a standard Wide and Deep\narchitecture. From there, how to build a model comprised only of one\ncomponent will be straightforward.\n\nNote that the examples shown below would be almost identical using any of the\nmodels available in the library. For example, `TabMlp` can be replaced by\n`TabResnet`, `TabNet`, `TabTransformer`, etc. Similarly, `BasicRNN` can be\nreplaced by `AttentiveRNN`, `StackedAttentiveRNN`, or `HFModel` with\ntheir corresponding parameters and preprocessor in the case of the Hugging\nFace models.\n\n**1. Wide and Tabular component (aka deeptabular)**\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"400\" src=\"mkdocs/sources/docs/figures/arch_1.png\"\u003e\n\u003c/p\u003e\n\n\n```python\nfrom pytorch_widedeep.preprocessing import TabPreprocessor, WidePreprocessor\nfrom pytorch_widedeep.models import Wide, TabMlp, WideDeep\nfrom pytorch_widedeep.training import Trainer\n\n# Wide\nwide_cols = [\"city\"]\ncrossed_cols = [(\"city\", \"name\")]\nwide_preprocessor = WidePreprocessor(wide_cols=wide_cols, crossed_cols=crossed_cols)\nX_wide = wide_preprocessor.fit_transform(df)\nwide = Wide(input_dim=np.unique(X_wide).shape[0])\n\n# Tabular\ntab_preprocessor = TabPreprocessor(\n    embed_cols=[\"city\", \"name\"], continuous_cols=[\"age\", \"height\"]\n)\nX_tab = tab_preprocessor.fit_transform(df)\ntab_mlp = TabMlp(\n    column_idx=tab_preprocessor.column_idx,\n    cat_embed_input=tab_preprocessor.cat_embed_input,\n    continuous_cols=tab_preprocessor.continuous_cols,\n    mlp_hidden_dims=[64, 32],\n)\n\n# WideDeep\nmodel = WideDeep(wide=wide, deeptabular=tab_mlp)\n\n# Train\ntrainer = Trainer(model, objective=\"binary\")\n\ntrainer.fit(\n    X_wide=X_wide,\n    X_tab=X_tab,\n    target=df[\"target\"].values,\n    n_epochs=1,\n    batch_size=32,\n)\n```\n\n**2. Tabular and Text data**\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"400\" src=\"mkdocs/sources/docs/figures/arch_2.png\"\u003e\n\u003c/p\u003e\n\n\n```python\nfrom pytorch_widedeep.preprocessing import TabPreprocessor, TextPreprocessor\nfrom pytorch_widedeep.models import TabMlp, BasicRNN, WideDeep\nfrom pytorch_widedeep.training import Trainer\n\n# Tabular\ntab_preprocessor = TabPreprocessor(\n    embed_cols=[\"city\", \"name\"], continuous_cols=[\"age\", \"height\"]\n)\nX_tab = tab_preprocessor.fit_transform(df)\ntab_mlp = TabMlp(\n    column_idx=tab_preprocessor.column_idx,\n    cat_embed_input=tab_preprocessor.cat_embed_input,\n    continuous_cols=tab_preprocessor.continuous_cols,\n    mlp_hidden_dims=[64, 32],\n)\n\n# Text\ntext_preprocessor = TextPreprocessor(\n    text_col=\"sentence\", maxlen=20, max_vocab=100, n_cpus=1\n)\nX_text = text_preprocessor.fit_transform(df)\nrnn = BasicRNN(\n    vocab_size=len(text_preprocessor.vocab.itos),\n    embed_dim=16,\n    hidden_dim=8,\n    n_layers=1,\n)\n\n# WideDeep\nmodel = WideDeep(deeptabular=tab_mlp, deeptext=rnn)\n\n# Train\ntrainer = Trainer(model, objective=\"binary\")\n\ntrainer.fit(\n    X_tab=X_tab,\n    X_text=X_text,\n    target=df[\"target\"].values,\n    n_epochs=1,\n    batch_size=32,\n)\n```\n\n**3. Tabular and text with a FC head on top via the `head_hidden_dims` param\n  in `WideDeep`**\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"400\" src=\"mkdocs/sources/docs/figures/arch_3.png\"\u003e\n\u003c/p\u003e\n\n```python\nfrom pytorch_widedeep.preprocessing import TabPreprocessor, TextPreprocessor\nfrom pytorch_widedeep.models import TabMlp, BasicRNN, WideDeep\nfrom pytorch_widedeep.training import Trainer\n\n# Tabular\ntab_preprocessor = TabPreprocessor(\n    embed_cols=[\"city\", \"name\"], continuous_cols=[\"age\", \"height\"]\n)\nX_tab = tab_preprocessor.fit_transform(df)\ntab_mlp = TabMlp(\n    column_idx=tab_preprocessor.column_idx,\n    cat_embed_input=tab_preprocessor.cat_embed_input,\n    continuous_cols=tab_preprocessor.continuous_cols,\n    mlp_hidden_dims=[64, 32],\n)\n\n# Text\ntext_preprocessor = TextPreprocessor(\n    text_col=\"sentence\", maxlen=20, max_vocab=100, n_cpus=1\n)\nX_text = text_preprocessor.fit_transform(df)\nrnn = BasicRNN(\n    vocab_size=len(text_preprocessor.vocab.itos),\n    embed_dim=16,\n    hidden_dim=8,\n    n_layers=1,\n)\n\n# WideDeep\nmodel = WideDeep(deeptabular=tab_mlp, deeptext=rnn, head_hidden_dims=[32, 16])\n\n# Train\ntrainer = Trainer(model, objective=\"binary\")\n\ntrainer.fit(\n    X_tab=X_tab,\n    X_text=X_text,\n    target=df[\"target\"].values,\n    n_epochs=1,\n    batch_size=32,\n)\n```\n\n**4. Tabular and multiple text columns that are passed directly to\n  `WideDeep`**\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"500\" src=\"mkdocs/sources/docs/figures/arch_4.png\"\u003e\n\u003c/p\u003e\n\n```python\nfrom pytorch_widedeep.preprocessing import TabPreprocessor, TextPreprocessor\nfrom pytorch_widedeep.models import TabMlp, BasicRNN, WideDeep\nfrom pytorch_widedeep.training import Trainer\n\n\n# Tabular\ntab_preprocessor = TabPreprocessor(\n    embed_cols=[\"city\", \"name\"], continuous_cols=[\"age\", \"height\"]\n)\nX_tab = tab_preprocessor.fit_transform(df)\ntab_mlp = TabMlp(\n    column_idx=tab_preprocessor.column_idx,\n    cat_embed_input=tab_preprocessor.cat_embed_input,\n    continuous_cols=tab_preprocessor.continuous_cols,\n    mlp_hidden_dims=[64, 32],\n)\n\n# Text\ntext_preprocessor_1 = TextPreprocessor(\n    text_col=\"sentence\", maxlen=20, max_vocab=100, n_cpus=1\n)\nX_text_1 = text_preprocessor_1.fit_transform(df)\ntext_preprocessor_2 = TextPreprocessor(\n    text_col=\"other_sentence\", maxlen=20, max_vocab=100, n_cpus=1\n)\nX_text_2 = text_preprocessor_2.fit_transform(df)\nrnn_1 = BasicRNN(\n    vocab_size=len(text_preprocessor_1.vocab.itos),\n    embed_dim=16,\n    hidden_dim=8,\n    n_layers=1,\n)\nrnn_2 = BasicRNN(\n    vocab_size=len(text_preprocessor_2.vocab.itos),\n    embed_dim=16,\n    hidden_dim=8,\n    n_layers=1,\n)\n\n# WideDeep\nmodel = WideDeep(deeptabular=tab_mlp, deeptext=[rnn_1, rnn_2])\n\n# Train\ntrainer = Trainer(model, objective=\"binary\")\n\ntrainer.fit(\n    X_tab=X_tab,\n    X_text=[X_text_1, X_text_2],\n    target=df[\"target\"].values,\n    n_epochs=1,\n    batch_size=32,\n)\n```\n\n**5. Tabular data and multiple text columns that are fused via a the library's\n  `ModelFuser` class**\n\n\u003cp align=\"center\"\u003e\n    \u003cimg width=\"500\" src=\"mkdocs/sources/docs/figures/arch_5.png\"\u003e\n\u003c/p\u003e\n\n```python\nfrom pytorch_widedeep.preprocessing import TabPreprocessor, TextPreprocessor\nfrom pytorch_widedeep.models import TabMlp, BasicRNN, WideDeep, ModelFuser\nfrom pytorch_widedeep import Trainer\n\n# Tabular\ntab_preprocessor = TabPreprocessor(\n    embed_cols=[\"city\", \"name\"], continuous_cols=[\"age\", \"height\"]\n)\nX_tab = tab_preprocessor.fit_transform(df)\ntab_mlp = TabMlp(\n    column_idx=tab_preprocessor.column_idx,\n    cat_embed_input=tab_preprocessor.cat_embed_input,\n    continuous_cols=tab_preprocessor.continuous_cols,\n    mlp_hidden_dims=[64, 32],\n)\n\n# Text\ntext_preprocessor_1 = TextPreprocessor(\n    text_col=\"sentence\", maxlen=20, max_vocab=100, n_cpus=1\n)\nX_text_1 = text_preprocessor_1.fit_transform(df)\ntext_preprocessor_2 = TextPreprocessor(\n    text_col=\"other_sentence\", maxlen=20, max_vocab=100, n_cpus=1\n)\nX_text_2 = text_preprocessor_2.fit_transform(df)\n\nrnn_1 = BasicRNN(\n    vocab_size=len(text_preprocessor_1.vocab.itos),\n    embed_dim=16,\n    hidden_dim=8,\n    n_layers=1,\n)\nrnn_2 = BasicRNN(\n    vocab_size=len(text_preprocessor_2.vocab.itos),\n    embed_dim=16,\n    hidden_dim=8,\n    n_layers=1,\n)\n\nmodels_fuser = ModelFuser(models=[rnn_1, rnn_2], fusion_method=\"mult\")\n\n# WideDeep\nmodel = WideDeep(deeptabular=tab_mlp, deeptext=models_fuser)\n\n# Train\ntrainer = Trainer(model, objective=\"binary\")\n\ntrainer.fit(\n    X_tab=X_tab,\n    X_text=[X_text_1, X_text_2],\n    target=df[\"target\"].values,\n    n_epochs=1,\n    batch_size=32,\n)\n```\n\n**6. Tabular and multiple text columns, with an image column. The text columns\n  are fused via the library's `ModelFuser` and then all fused via the\n  deephead paramenter in `WideDeep` which is a custom `ModelFuser` coded by\n  the user**\n\nThis is perhaps the less elegant solution as it involves a custom component by\nthe user and slicing the 'incoming' tensor. In the future, we will include a\n`TextAndImageModelFuser` to make this process more straightforward. Still, is not\nreally complicated and it is a good example of how to use custom components in\n`pytorch-widedeep`.\n\nNote that the only requirement for the custom component is that it has a\nproperty called `output_dim` that returns the size of the last layer of\nactivations. In other words, it does not need to inherit from\n`BaseWDModelComponent`. This base class simply checks the existence of such\nproperty and avoids some typing errors internally.\n\n\n\u003cp align=\"center\"\u003e\n    \u003cimg width=\"600\" src=\"mkdocs/sources/docs/figures/arch_6.png\"\u003e\n\u003c/p\u003e\n\n\n```python\nimport torch\n\nfrom pytorch_widedeep.preprocessing import TabPreprocessor, TextPreprocessor, ImagePreprocessor\nfrom pytorch_widedeep.models import TabMlp, BasicRNN, WideDeep, ModelFuser, Vision\nfrom pytorch_widedeep.models._base_wd_model_component import BaseWDModelComponent\nfrom pytorch_widedeep import Trainer\n\n# Tabular\ntab_preprocessor = TabPreprocessor(\n    embed_cols=[\"city\", \"name\"], continuous_cols=[\"age\", \"height\"]\n)\nX_tab = tab_preprocessor.fit_transform(df)\ntab_mlp = TabMlp(\n    column_idx=tab_preprocessor.column_idx,\n    cat_embed_input=tab_preprocessor.cat_embed_input,\n    continuous_cols=tab_preprocessor.continuous_cols,\n    mlp_hidden_dims=[16, 8],\n)\n\n# Text\ntext_preprocessor_1 = TextPreprocessor(\n    text_col=\"sentence\", maxlen=20, max_vocab=100, n_cpus=1\n)\nX_text_1 = text_preprocessor_1.fit_transform(df)\ntext_preprocessor_2 = TextPreprocessor(\n    text_col=\"other_sentence\", maxlen=20, max_vocab=100, n_cpus=1\n)\nX_text_2 = text_preprocessor_2.fit_transform(df)\nrnn_1 = BasicRNN(\n    vocab_size=len(text_preprocessor_1.vocab.itos),\n    embed_dim=16,\n    hidden_dim=8,\n    n_layers=1,\n)\nrnn_2 = BasicRNN(\n    vocab_size=len(text_preprocessor_2.vocab.itos),\n    embed_dim=16,\n    hidden_dim=8,\n    n_layers=1,\n)\nmodels_fuser = ModelFuser(\n    models=[rnn_1, rnn_2],\n    fusion_method=\"mult\",\n)\n\n# Image\nimage_preprocessor = ImagePreprocessor(img_col=\"image_name\", img_path=\"images\")\nX_img = image_preprocessor.fit_transform(df)\nvision = Vision(pretrained_model_setup=\"resnet18\", head_hidden_dims=[16, 8])\n\n# deephead (custom model fuser)\nclass MyModelFuser(BaseWDModelComponent):\n    \"\"\"\n    Simply a Linear + Relu sequence on top of the text + images followed by a\n    Linear -\u003e Relu -\u003e Linear for the concatenation of tabular slice of the\n    tensor and the output of the text and image sequential model\n    \"\"\"\n    def __init__(\n        self,\n        tab_incoming_dim: int,\n        text_incoming_dim: int,\n        image_incoming_dim: int,\n        output_units: int,\n    ):\n\n        super(MyModelFuser, self).__init__()\n\n        self.tab_incoming_dim = tab_incoming_dim\n        self.text_incoming_dim = text_incoming_dim\n        self.image_incoming_dim = image_incoming_dim\n        self.output_units = output_units\n        self.text_and_image_fuser = torch.nn.Sequential(\n            torch.nn.Linear(text_incoming_dim + image_incoming_dim, output_units),\n            torch.nn.ReLU(),\n        )\n        self.out = torch.nn.Sequential(\n            torch.nn.Linear(output_units + tab_incoming_dim, output_units * 4),\n            torch.nn.ReLU(),\n            torch.nn.Linear(output_units * 4, output_units),\n        )\n\n    def forward(self, X: torch.Tensor) -\u003e torch.Tensor:\n        tab_slice = slice(0, self.tab_incoming_dim)\n        text_slice = slice(\n            self.tab_incoming_dim, self.tab_incoming_dim + self.text_incoming_dim\n        )\n        image_slice = slice(\n            self.tab_incoming_dim + self.text_incoming_dim,\n            self.tab_incoming_dim + self.text_incoming_dim + self.image_incoming_dim,\n        )\n        X_tab = X[:, tab_slice]\n        X_text = X[:, text_slice]\n        X_img = X[:, image_slice]\n        X_text_and_image = self.text_and_image_fuser(torch.cat([X_text, X_img], dim=1))\n        return self.out(torch.cat([X_tab, X_text_and_image], dim=1))\n\n    @property\n    def output_dim(self):\n        return self.output_units\n\n\ndeephead = MyModelFuser(\n    tab_incoming_dim=tab_mlp.output_dim,\n    text_incoming_dim=models_fuser.output_dim,\n    image_incoming_dim=vision.output_dim,\n    output_units=8,\n)\n\n# WideDeep\nmodel = WideDeep(\n    deeptabular=tab_mlp,\n    deeptext=models_fuser,\n    deepimage=vision,\n    deephead=deephead,\n)\n\n# Train\ntrainer = Trainer(model, objective=\"binary\")\n\ntrainer.fit(\n    X_tab=X_tab,\n    X_text=[X_text_1, X_text_2],\n    X_img=X_img,\n    target=df[\"target\"].values,\n    n_epochs=1,\n    batch_size=32,\n)\n```\n\n**7. A two-tower model**\n\nThis is a popular model in the context of recommendation systems. Let's say we\nhave a tabular dataset formed my triples (user features, item features,\ntarget). We can create a two-tower model where the user and item features are\npassed through two separate models and then \"fused\" via a dot product.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"350\" src=\"mkdocs/sources/docs/figures/arch_7.png\"\u003e\n\u003c/p\u003e\n\n\n```python\nimport numpy as np\nimport pandas as pd\n\nfrom pytorch_widedeep import Trainer\nfrom pytorch_widedeep.preprocessing import TabPreprocessor\nfrom pytorch_widedeep.models import TabMlp, WideDeep, ModelFuser\n\n# Let's create the interaction dataset\n# user_features dataframe\nnp.random.seed(42)\nuser_ids = np.arange(1, 101)\nages = np.random.randint(18, 60, size=100)\ngenders = np.random.choice([\"male\", \"female\"], size=100)\nlocations = np.random.choice([\"city_a\", \"city_b\", \"city_c\", \"city_d\"], size=100)\nuser_features = pd.DataFrame(\n    {\"id\": user_ids, \"age\": ages, \"gender\": genders, \"location\": locations}\n)\n\n# item_features dataframe\nitem_ids = np.arange(1, 101)\nprices = np.random.uniform(10, 500, size=100).round(2)\ncolors = np.random.choice([\"red\", \"blue\", \"green\", \"black\"], size=100)\ncategories = np.random.choice([\"electronics\", \"clothing\", \"home\", \"toys\"], size=100)\n\nitem_features = pd.DataFrame(\n    {\"id\": item_ids, \"price\": prices, \"color\": colors, \"category\": categories}\n)\n\n# Interactions dataframe\ninteraction_user_ids = np.random.choice(user_ids, size=1000)\ninteraction_item_ids = np.random.choice(item_ids, size=1000)\npurchased = np.random.choice([0, 1], size=1000, p=[0.7, 0.3])\ninteractions = pd.DataFrame(\n    {\n        \"user_id\": interaction_user_ids,\n        \"item_id\": interaction_item_ids,\n        \"purchased\": purchased,\n    }\n)\nuser_item_purchased = interactions.merge(\n    user_features, left_on=\"user_id\", right_on=\"id\"\n).merge(item_features, left_on=\"item_id\", right_on=\"id\")\n\n# Users\ntab_preprocessor_user = TabPreprocessor(\n    cat_embed_cols=[\"gender\", \"location\"],\n    continuous_cols=[\"age\"],\n)\nX_user = tab_preprocessor_user.fit_transform(user_item_purchased)\ntab_mlp_user = TabMlp(\n    column_idx=tab_preprocessor_user.column_idx,\n    cat_embed_input=tab_preprocessor_user.cat_embed_input,\n    continuous_cols=[\"age\"],\n    mlp_hidden_dims=[16, 8],\n    mlp_dropout=[0.2, 0.2],\n)\n\n# Items\ntab_preprocessor_item = TabPreprocessor(\n    cat_embed_cols=[\"color\", \"category\"],\n    continuous_cols=[\"price\"],\n)\nX_item = tab_preprocessor_item.fit_transform(user_item_purchased)\ntab_mlp_item = TabMlp(\n    column_idx=tab_preprocessor_item.column_idx,\n    cat_embed_input=tab_preprocessor_item.cat_embed_input,\n    continuous_cols=[\"price\"],\n    mlp_hidden_dims=[16, 8],\n    mlp_dropout=[0.2, 0.2],\n)\n\ntwo_tower_model = ModelFuser([tab_mlp_user, tab_mlp_item], fusion_method=\"dot\")\n\nmodel = WideDeep(deeptabular=two_tower_model)\n\ntrainer = Trainer(model, objective=\"binary\")\n\ntrainer.fit(\n    X_tab=[X_user, X_item],\n    target=interactions.purchased.values,\n    n_epochs=1,\n    batch_size=32,\n)\n```\n\n**8. Tabular with a multi-target loss**\n\nThis one is \"a bonus\" to illustrate the use of multi-target losses, more than\nactually a different architecture.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"200\" src=\"mkdocs/sources/docs/figures/arch_8.png\"\u003e\n\u003c/p\u003e\n\n\n```python\nfrom pytorch_widedeep.preprocessing import TabPreprocessor, TextPreprocessor, ImagePreprocessor\nfrom pytorch_widedeep.models import TabMlp, BasicRNN, WideDeep, ModelFuser, Vision\nfrom pytorch_widedeep.losses_multitarget import MultiTargetClassificationLoss\nfrom pytorch_widedeep.models._base_wd_model_component import BaseWDModelComponent\nfrom pytorch_widedeep import Trainer\n\n# let's add a second target to the dataframe\ndf[\"target2\"] = [random.choice([0, 1]) for _ in range(100)]\n\n# Tabular\ntab_preprocessor = TabPreprocessor(\n    embed_cols=[\"city\", \"name\"], continuous_cols=[\"age\", \"height\"]\n)\nX_tab = tab_preprocessor.fit_transform(df)\ntab_mlp = TabMlp(\n    column_idx=tab_preprocessor.column_idx,\n    cat_embed_input=tab_preprocessor.cat_embed_input,\n    continuous_cols=tab_preprocessor.continuous_cols,\n    mlp_hidden_dims=[64, 32],\n)\n\n# 'pred_dim=2' because we have two binary targets. For other types of targets,\n#  please, see the documentation\nmodel = WideDeep(deeptabular=tab_mlp, pred_dim=2).\n\nloss = MultiTargetClassificationLoss(binary_config=[0, 1], reduction=\"mean\")\n\n# When a multi-target loss is used, 'custom_loss_function' must not be None.\n# See the docs\ntrainer = Trainer(model, objective=\"multitarget\", custom_loss_function=loss)\n\ntrainer.fit(\n    X_tab=X_tab,\n    target=df[[\"target\", \"target2\"]].values,\n    n_epochs=1,\n    batch_size=32,\n)\n```\n\n### The ``deeptabular`` component\n\nIt is important to emphasize again that **each individual component, `wide`,\n`deeptabular`, `deeptext` and `deepimage`, can be used independently** and in\nisolation. For example, one could use only `wide`, which is in simply a\nlinear model. In fact, one of the most interesting functionalities\nin``pytorch-widedeep`` would be the use of the ``deeptabular`` component on\nits own, i.e. what one might normally refer as Deep Learning for Tabular\nData. Currently, ``pytorch-widedeep`` offers the following different models\nfor that component:\n\n0. **Wide**: a simple linear model where the nonlinearities are captured via\ncross-product transformations, as explained before.\n1. **TabMlp**: a simple MLP that receives embeddings representing the\ncategorical features, concatenated with the continuous features, which can\nalso be embedded.\n2. **TabResnet**: similar to the previous model but the embeddings are\npassed through a series of ResNet blocks built with dense layers.\n3. **TabNet**: details on TabNet can be found in\n[TabNet: Attentive Interpretable Tabular Learning](https://arxiv.org/abs/1908.07442)\n\nTwo simpler attention based models that we call:\n\n4. **ContextAttentionMLP**: MLP with at attention mechanism \"on top\" that is based on\n    [Hierarchical Attention Networks for Document Classification](https://www.cs.cmu.edu/~./hovy/papers/16HLT-hierarchical-attention-networks.pd)\n5. **SelfAttentionMLP**: MLP with an attention mechanism that is a simplified\n    version of a transformer block that we refer as \"query-key self-attention\".\n\nThe ``Tabformer`` family, i.e. Transformers for Tabular data:\n\n6. **TabTransformer**: details on the TabTransformer can be found in\n[TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://arxiv.org/pdf/2012.06678.pdf).\n7. **SAINT**: Details on SAINT can be found in\n[SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://arxiv.org/abs/2106.01342).\n8. **FT-Transformer**: details on the FT-Transformer can be found in\n[Revisiting Deep Learning Models for Tabular Data](https://arxiv.org/abs/2106.11959).\n9. **TabFastFormer**: adaptation of the FastFormer for tabular data. Details\non the Fasformer can be found in\n[FastFormers: Highly Efficient Transformer Models for Natural Language Understanding](https://arxiv.org/abs/2010.13382)\n10. **TabPerceiver**: adaptation of the Perceiver for tabular data. Details on\nthe Perceiver can be found in\n[Perceiver: General Perception with Iterative Attention](https://arxiv.org/abs/2103.03206)\n\nAnd probabilistic DL models for tabular data based on\n[Weight Uncertainty in Neural Networks](https://arxiv.org/abs/1505.05424):\n\n11. **BayesianWide**: Probabilistic adaptation of the `Wide` model.\n12. **BayesianTabMlp**: Probabilistic adaptation of the `TabMlp` model\n\nNote that while there are scientific publications for the TabTransformer,\nSAINT and FT-Transformer, the TabFasfFormer and TabPerceiver are our own\nadaptation of those algorithms for tabular data.\n\nIn addition, Self-Supervised pre-training can be used for all `deeptabular`\nmodels, with the exception of the `TabPerceiver`. Self-Supervised\npre-training can be used via two methods or routines which we refer as:\nencoder-decoder method and constrastive-denoising method. Please, see the\ndocumentation and the examples for details on this functionality, and all\nother options in the library.\n\n### The ``rec`` module\n\nThis module was introduced as an extension to the existing components in the\nlibrary, addressing questions and issues related to recommendation systems.\nWhile still under active development, it currently includes a select number\nof powerful recommendation models.\n\nIt's worth noting that this library already supported the implementation of\nvarious recommendation algorithms using existing components. For example,\nmodels like Wide and Deep, Two-Tower, or Neural Collaborative Filtering could\nbe constructed using the library's core functionalities.\n\nThe recommendation algorithms in the `rec` module are:\n\n1. [AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://arxiv.org/abs/1810.11921)\n2. [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://arxiv.org/abs/1703.04247)\n3. (Deep) Field Aware Factorization Machine (FFM): a Deep Learning version of the algorithm presented in [Field-aware Factorization Machines in a Real-world Online Advertising System](https://arxiv.org/abs/1701.04099)\n4. [xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems](https://arxiv.org/pdf/1803.05170)\n5. [Deep Interest Network for Click-Through Rate Prediction](https://arxiv.org/abs/1706.06978)\n6. [Deep and Cross Network for Ad Click Predictions](https://arxiv.org/abs/1708.05123)\n7. [DCN V2: Improved Deep \u0026 Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://arxiv.org/abs/2008.13535)\n8. [Towards Deeper, Lighter and Interpretable Click-through Rate Prediction](https://arxiv.org/abs/2311.04635)\n9. A basic Transformer-based model for recommendation where the problem is faced as a sequence.\n\nSee the examples for details on how to use these models.\n\n### Text and Images\nFor the text component, `deeptext`, the library offers the following models:\n\n1. **BasicRNN**: a simple RNN 2. **AttentiveRNN**: a RNN with an attention\nmechanism based on the\n[Hierarchical Attention Networks for DocumentClassification](https://www.cs.cmu.edu/~./hovy/papers/16HLT-hierarchical-attention-networks.pd)\n3. **StackedAttentiveRNN**: a stack of AttentiveRNNs\n4. **HFModel**: a wrapper around Hugging Face Transfomer-based models. At the moment\nonly models from the families BERT, RoBERTa, DistilBERT, ALBERT and ELECTRA\nare supported. This is because this library is designed to address\nclassification and regression tasks and these are the most 'popular'\nencoder-only models, which have proved to be those that work best for these\ntasks. If there is demand for other models, they will be included in the\nfuture.\n\nFor the image component, `deepimage`, the library supports models from the\nfollowing families:\n'resnet', 'shufflenet', 'resnext', 'wide_resnet', 'regnet', 'densenet', 'mobilenetv3',\n 'mobilenetv2', 'mnasnet', 'efficientnet' and 'squeezenet'.  These are\n offered via `torchvision` and wrapped up in the `Vision` class.\n\n###  Installation\n\nInstall using pip:\n\n```bash\npip install pytorch-widedeep\n```\n\nOr install directly from github\n\n```bash\npip install git+https://github.com/jrzaurin/pytorch-widedeep.git\n```\n\n#### Developer Install\n\n```bash\n# Clone the repository\ngit clone https://github.com/jrzaurin/pytorch-widedeep\ncd pytorch-widedeep\n\n# Install in dev mode\npip install -e .\n```\n\n### Quick start\n\nHere is an end-to-end example of a binary classification with the [adult\ndataset]([adult](https://www.kaggle.com/wenruliu/adult-income-dataset))\nusing `Wide` and `DeepDense` and defaults settings.\n\nBuilding a wide (linear) and deep model with ``pytorch-widedeep``:\n\n```python\nimport numpy as np\nimport torch\nfrom sklearn.model_selection import train_test_split\n\nfrom pytorch_widedeep import Trainer\nfrom pytorch_widedeep.preprocessing import WidePreprocessor, TabPreprocessor\nfrom pytorch_widedeep.models import Wide, TabMlp, WideDeep\nfrom pytorch_widedeep.metrics import Accuracy\nfrom pytorch_widedeep.datasets import load_adult\n\n\ndf = load_adult(as_frame=True)\ndf[\"income_label\"] = (df[\"income\"].apply(lambda x: \"\u003e50K\" in x)).astype(int)\ndf.drop(\"income\", axis=1, inplace=True)\ndf_train, df_test = train_test_split(df, test_size=0.2, stratify=df.income_label)\n\n# Define the 'column set up'\nwide_cols = [\n    \"education\",\n    \"relationship\",\n    \"workclass\",\n    \"occupation\",\n    \"native-country\",\n    \"gender\",\n]\ncrossed_cols = [(\"education\", \"occupation\"), (\"native-country\", \"occupation\")]\n\ncat_embed_cols = [\n    \"workclass\",\n    \"education\",\n    \"marital-status\",\n    \"occupation\",\n    \"relationship\",\n    \"race\",\n    \"gender\",\n    \"capital-gain\",\n    \"capital-loss\",\n    \"native-country\",\n]\ncontinuous_cols = [\"age\", \"hours-per-week\"]\ntarget = \"income_label\"\ntarget = df_train[target].values\n\n# prepare the data\nwide_preprocessor = WidePreprocessor(wide_cols=wide_cols, crossed_cols=crossed_cols)\nX_wide = wide_preprocessor.fit_transform(df_train)\n\ntab_preprocessor = TabPreprocessor(\n    cat_embed_cols=cat_embed_cols, continuous_cols=continuous_cols  # type: ignore[arg-type]\n)\nX_tab = tab_preprocessor.fit_transform(df_train)\n\n# build the model\nwide = Wide(input_dim=np.unique(X_wide).shape[0], pred_dim=1)\ntab_mlp = TabMlp(\n    column_idx=tab_preprocessor.column_idx,\n    cat_embed_input=tab_preprocessor.cat_embed_input,\n    continuous_cols=continuous_cols,\n)\nmodel = WideDeep(wide=wide, deeptabular=tab_mlp)\n\n# train and validate\ntrainer = Trainer(model, objective=\"binary\", metrics=[Accuracy])\ntrainer.fit(\n    X_wide=X_wide,\n    X_tab=X_tab,\n    target=target,\n    n_epochs=5,\n    batch_size=256,\n)\n\n# predict on test\nX_wide_te = wide_preprocessor.transform(df_test)\nX_tab_te = tab_preprocessor.transform(df_test)\npreds = trainer.predict(X_wide=X_wide_te, X_tab=X_tab_te)\n\n# Save and load\n\n# Option 1: this will also save training history and lr history if the\n# LRHistory callback is used\ntrainer.save(path=\"model_weights\", save_state_dict=True)\n\n# Option 2: save as any other torch model\ntorch.save(model.state_dict(), \"model_weights/wd_model.pt\")\n\n# From here in advance, Option 1 or 2 are the same. I assume the user has\n# prepared the data and defined the new model components:\n# 1. Build the model\nmodel_new = WideDeep(wide=wide, deeptabular=tab_mlp)\nmodel_new.load_state_dict(torch.load(\"model_weights/wd_model.pt\"))\n\n# 2. Instantiate the trainer\ntrainer_new = Trainer(model_new, objective=\"binary\")\n\n# 3. Either start the fit or directly predict\npreds = trainer_new.predict(X_wide=X_wide, X_tab=X_tab, batch_size=32)\n```\n\nOf course, one can do **much more**. See the Examples folder, the\ndocumentation or the companion posts for a better understanding of the content\nof the package and its functionalities.\n\n### Testing\n\n```\npytest tests\n```\n\n### How to Contribute\n\nCheck [CONTRIBUTING](https://github.com/jrzaurin/pytorch-widedeep/blob/master/CONTRIBUTING.MD) page.\n\n### Acknowledgments\n\nThis library takes from a series of other libraries, so I think it is just\nfair to mention them here in the README (specific mentions are also included\nin the code).\n\nThe `Callbacks` and `Initializers` structure and code is inspired by the\n[`torchsample`](https://github.com/ncullen93/torchsample) library, which in\nitself partially inspired by [`Keras`](https://keras.io/).\n\nThe `TextProcessor` class in this library uses the\n[`fastai`](https://docs.fast.ai/text.transform.html#BaseTokenizer.tokenizer)'s\n`Tokenizer` and `Vocab`. The code at `utils.fastai_transforms` is a minor\nadaptation of their code so it functions within this library. To my experience\ntheir `Tokenizer` is the best in class.\n\nThe `ImageProcessor` class in this library uses code from the fantastic [Deep\nLearning for Computer\nVision](https://www.pyimagesearch.com/deep-learning-computer-vision-python-book/)\n(DL4CV) book by Adrian Rosebrock.\n\n### License\n\nThis work is dual-licensed under Apache 2.0 and MIT (or any later version).\nYou can choose between one of them if you use this work.\n\n`SPDX-License-Identifier: Apache-2.0 AND MIT`\n\n### Cite\n\n#### BibTex\n\n```\n@article{Zaurin_pytorch-widedeep_A_flexible_2023,\nauthor = {Zaurin, Javier Rodriguez and Mulinka, Pavol},\ndoi = {10.21105/joss.05027},\njournal = {Journal of Open Source Software},\nmonth = jun,\nnumber = {86},\npages = {5027},\ntitle = {{pytorch-widedeep: A flexible package for multimodal deep learning}},\nurl = {https://joss.theoj.org/papers/10.21105/joss.05027},\nvolume = {8},\nyear = {2023}\n}\n```\n\n#### APA\n\n```\nZaurin, J. R., \u0026 Mulinka, P. (2023). pytorch-widedeep: A flexible package for\nmultimodal deep learning. Journal of Open Source Software, 8(86), 5027.\nhttps://doi.org/10.21105/joss.05027\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjrzaurin%2Fpytorch-widedeep","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjrzaurin%2Fpytorch-widedeep","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjrzaurin%2Fpytorch-widedeep/lists"}