{"id":16461513,"url":"https://github.com/tom-mohr/clevertable","last_synced_at":"2026-05-06T13:15:51.987Z","repository":{"id":153859393,"uuid":"630900883","full_name":"tom-mohr/clevertable","owner":"tom-mohr","description":"Consistent, intelligent transformation of text-based tabular data into numerical data.","archived":false,"fork":false,"pushed_at":"2023-06-14T06:15:12.000Z","size":183,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-10T10:14:01.149Z","etag":null,"topics":["conversion","data-science","numpy","pandas","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tom-mohr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-21T12:21:41.000Z","updated_at":"2023-05-30T09:07:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"da314d42-1dd6-41ff-938b-3d0d85663881","html_url":"https://github.com/tom-mohr/clevertable","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tom-mohr%2Fclevertable","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tom-mohr%2Fclevertable/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tom-mohr%2Fclevertable/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tom-mohr%2Fclevertable/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tom-mohr","download_url":"https://codeload.github.com/tom-mohr/clevertable/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240988681,"owners_count":19889542,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["conversion","data-science","numpy","pandas","python"],"created_at":"2024-10-11T11:08:26.585Z","updated_at":"2026-05-06T13:15:46.930Z","avatar_url":"https://github.com/tom-mohr.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CleverTable\n![Pytest](https://github.com/tom-mohr/clevertable/actions/workflows/pytest.yml/badge.svg)\n\nConsistent, intelligent transformation of text-based tabular data into numerical data.\u003cbr\u003e\nMinimal configuration required.\n\nInstallation:\n\n```bash\npip install clevertable\n```\n\nExample:\n\n```python\nfrom clevertable import *\n\nprofile = ConversionProfile({\n    # optionally specify converters for specific columns:\n    \"Country\": OneHot(),\n    \"Diagnosis\": Binary(positive=\"cancer\", negative=\"benign\"),\n    \"Hospitalized\": None,  # ignore column\n}, pre_processing=None)\n\ndf = profile.fit_transform(\"datasets/survey.xlsx\")  # transformed pandas.DataFrame\n```\n\n# Why this Library?\n\n- CleverTable makes it really easy to convert text-based tabular data\n  (optionally mixed with numbers) into numerical data, e.g. a medical survey\n  into a Pandas DataFrame or a NumPy array.\n- If something is obvious, you should not need to specify it.\n  CleverTable will try to make choices for you if you don't make them.\n- You stay in control: All choices made by CleverTable can be modified and overridden.\n\nThis is how CleverTable works: (see below for a full [tutorial](#tutorial))\n\n1. You create a new `profile = ConversionProfile()`.\n   Here, you can optionally specify certain converters.\n2. You call `profile.fit(data)` on a sample data set, which creates a fixed conversion profile.\n    - CleverTable chooses the best converter for each column if you don't specify it.\n    - The converter (chosen by you or by CleverTable) adapts its internal state to fit the data.\n3. You call `profile.transform(data)` on the actual data set (which may be the same as for `fit()`),\n   which converts the data according to the fixed profile.\n\nHere are some examples on what you can do with CleverTable:\n\n- Chain multiple converters to achieve complex conversions:\n  ```python\n  profile[\"Column 7\"] = [\n      Split(),\n      ForEach(Strip()),\n      Flatten(),\n      Infer()  # Infer() -\u003e CleverTable will choose what to put here\n  ]\n  ```\n- Use the `Infer()` converter where you want CleverTable to figure out the best solution (see above).\n- Concise shorthand writings with Python syntax:\n  ```python\n  profile[\"Column 1\"] = [  # Python lists create pipelines\n    str.lower,             # functions /\n    lambda s: s.strip(),   # lambda expressions are allowed\n  ]\n  profile[\"Column 2\"] = {\"Hello\": 1, \"Bye\": 2}\n  profile[\"Column 3\"] = Float(), 1  # tries conversion to float, defaults to 1 on error\n  ```\n- Incremental configuration: If a column already has a correct converter, you can further process the column\n  by adding another converter.\n  This implicitly creates a pipeline.\n  ```python\n  profile[\"Column 5\"] += OneHot()\n  ```\n- After `fit()`, you can access the inferred state of the converters.\n  ```python\n  my_weather_conv = profile[\"Weather\"]            # e.g. OneHot()\n  my_weather_categories = my_weather_conv.values  # e.g. [\"sunny\", \"cloudy\", \"rainy\"]\n  ```\n- Send multiple columns into one converter:\n  ```python\n  profile[\"Column 1\", \"Column 2\"] = max\n  ```\n- Send nested columns into one converter:\n  ```python\n  profile[(\"Column A\", \"Column B\"), \"Column C\"] = [Parallel(max, floor), min]  # min(max(A, B), floor(C))\n  ```\n\n# Tutorial\n\nSuppose you want to convert the following table of survey results in a 2D numpy array of numbers:\n\n| Country | Age | Diagnosis | Hospitalized | Education level | Symptoms        |\n|---------|-----|-----------|--------------|-----------------|-----------------|\n| China   | 32  | benign    | no           | University      | cough, fever    |\n| France  | 45  | cancer    | yes          | PhD             | fever           |\n| Italy   | 19  | benign    | yes          | High School     | cough           |\n| Germany | 56  | cancer    | yes          | High School     | fever and cough |\n| Nigeria | 23  | benign    | no           | University      | cough           |\n| India   | 34  | benign    | yes          | University      | cough, fever    |\n| ...     | ... | ...       | ...          | ...             | ...             |\n\nFor example, you might want to convert the `Country` column into a column of integers,\nwith every integer representing a different country.\u003cbr\u003e\nHowever:\n\n- You don't really care which number represents which country.\n- But you want to make sure that the same country always gets the same number,\n  even if you add more data to the table later.\n- You also want to know which integer was chosen for which country.\n\nThat's what CleverTable is for:\n\n- First, you call `fit()` on a sample data set, which creates a fixed conversion profile.\n- Then, you call `transform()` on the actual data set, and it converts the data according\n  to the fixed profile.\n\nMoreover, CleverTable does many things automatically:\n\n- It chooses the best converter if you don't specify it.\n- And then, the converter also adapts its internal state to fit the data.\n\nLet's see how that works:\n\n```python\nfrom clevertable import *\n\ntable = \"datasets/survey.xlsx\"  # filename or pandas.DataFrame\n\nprofile = ConversionProfile()\nprofile.fit(table)  # chooses best converters and creates a fixed conversion profile\n```\n\n`print(profile)` will show the inferred conversion profile:\n\n```python\n{\n    \"Country\": Enumerate('china', 'france', 'germany', ...),  # lots of countries\n    \"Age\": Float(),\n    \"Diagnosis\": Binary(),\n    \"Hospitalized\": Binary(),\n    \"Education level\": OneHot('high school', 'phd', 'university'),\n    \"Symptoms\": ListAndOr(),\n}\n```\n\nWe can access the individual converters and their properties by indexing the profile with the column name:\n\n```python\ncountry_converter = profile[\"Country\"]  # Enumerate('china', 'france', 'germany', ...)\n\n# see which integer corresponds to which country:\ncountries_list = country_converter.values  # ('china', 'france', 'germany', ...)\n```\n\nYou can now use this profile to convert data:\n\n```python\n# transform the whole table:\ndf = profile.transform(table)  # pandas.DataFrame\narr = df.to_numpy()  # 2D numpy array\n\n# transform a single data point:\ndata_point = {\"Country\": \"Germany\"}\ntransformed = profile.transform_single(data_point)  # {'Country': 2}\n```\n\nThe nice thing is that you can now use the fixed profile\nto find out after conversion where the numerical values originated from:\n\n```python\n# find out which country corresponds to the number 2:\ncountry_id = 2\ncountry = profile[\"Country\"].values[country_id]  # 'germany'\n```\n\nYou may have noticed that all the strings appear in lowercase.\nThat is because the `ConversionProfile` pre-processes all strings to lowercase by default.\nYou can disable this behavior by passing `pre_processing=None` to the constructor\nor setting this property after construction:\n\n```python\nprofile.pre_processing = None  # disable pre-processing\nprofile.pre_processing = str.lower  # default behavior\nprofile.pre_processing = lambda s: s.strip().lower()\n```\n\nIt's okay to provide a pre-processing function that doesn't work for some entries\n(e.g. `str.lower` will fail for non-string entries),\nbecause CleverTable will catch errors and ignore them during pre-processing.\n\nYou may also have noticed that the `Education level` column was converted to `OneHot()`,\neven though it contains arbitrary words, just like the `Country` column.\nThat's because CleverTable detected that there are too many different values\nin the `Country` column for a `OneHot()` converter, so it chose the `Enumerate()` converter.\n\nBut you can always override this behavior by explicitly setting the conversion method\nbefore calling `fit()`:\n\n```python\nfrom clevertable import *\n\ntable = \"datasets/survey.xlsx\"\n\nprofile = ConversionProfile()\n\n# explicitly specify some converters:\nprofile[\"Country\"] = OneHot()\nprofile[\"Diagnosis\"] = Binary(positive=\"cancer\", negative=\"benign\")\n\nprofile.fit(table)\n```\n\nIn this example, we also made sure that the \"Diagnosis\" column\nis choosing the correct positive and negative values.\n\nYou can also achieve the same by passing a dictionary to the constructor:\n\n```python\nfrom clevertable import *\n\ntable = \"datasets/survey.xlsx\"\n\nprofile = ConversionProfile({\n    \"Country\": OneHot(),\n    \"Diagnosis\": Binary(positive=\"cancer\", negative=\"benign\"),\n}).fit(table)  # fit() returns self\n```\n\nTwo final notes:\n\n- You can ignore columns by setting their converter to `None` (which is shorthand for the `Ignore()` converter).\n- You can use `fit_transform()` to perform `fit()` and `transform()` with the same data in one call.\n\nThis leaves us with this very concise code:\n\n```python\nfrom clevertable import *\n\ndf = ConversionProfile({\n    \"Country\": OneHot(),\n    \"Diagnosis\": Binary(positive=\"cancer\", negative=\"benign\"),\n    \"Hospitalized\": None,\n}, pre_processing=None).fit_transform(\"datasets/survey.xlsx\")\n```\n\nWhich produces the following transformed table:\n\n| Country=China | Country=France | ... | Country=Zimbabwe | Age | Diagnosis | Education level=High School | Education level=PhD | Education level=University | Symptoms=cough | Symptoms=fever |\n|---------------|----------------|-----|------------------|-----|-----------|-----------------------------|---------------------|----------------------------|----------------|----------------|\n| 1             | 0              | ... | 0                | 32  | 0         | 0                           | 0                   | 1                          | 1              | 1              |\n| 0             | 1              | ... | 0                | 45  | 1         | 0                           | 1                   | 0                          | 0              | 1              |\n| 0             | 0              | ... | 0                | 19  | 0         | 1                           | 0                   | 0                          | 1              | 0              |\n| 0             | 0              | ... | 0                | 56  | 1         | 1                           | 0                   | 0                          | 1              | 1              |\n| 0             | 0              | ... | 0                | 23  | 0         | 0                           | 0                   | 1                          | 1              | 0              |\n| 0             | 0              | ... | 0                | 34  | 0         | 0                           | 0                   | 1                          | 1              | 1              |\n\n# CLI\n\n`pip install clevertable` also makes the command `clevertable` available\nin the command line.\nIt can convert files with tabular data.\nExecute `clevertable --help` to see what arguments can be passed to the tool:\n\n```text\nusage: clevertable [-h] [-i IGNORE [IGNORE ...]] src out\n\nConsistent and intelligent conversion of tabular data into numerical values.\n\npositional arguments:\n  src                   Path to input file.\n  out                   Path to output file.\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -i IGNORE [IGNORE ...], --ignore IGNORE [IGNORE ...]\n                        Column names to ignore.\n```\n\n# How to Contribute\n\nBasic workflow of contribution:\n\n- Fork the repository\n- Create a new branch\n- Make your changes\n- Create a pull request\n- Wait for the pull request to be accepted or rejected\n- If accepted, you can delete your branch\n- If rejected, make the requested changes and push them to your branch\n- Repeat until pull request is accepted\n\nWhat to contribute:\n\n- New converters (classes that inherit from `Converter`)\n- Improvements to converter inference (logic in `Infer()` converter)\n- Improvements to default preprocessing\n- Make more features available through the CLI\n- New tests\n- New documentation, tutorials, examples\n- New ideas, suggestions, bug reports → create an issue or contact me directly\n\n# Documentation\n\nThere are only two classes that:\n\n- `ConversionProfile`: A collection of converters.\n- `Converter`: Transforms columns of data into columns of data.\n\n## Converters\n\nHere's a quick overview of all converters:\n\n| Converters                            | Description                                                                         | Shorthand | Example Usage                                                   |\n|---------------------------------------|-------------------------------------------------------------------------------------|-----------|-----------------------------------------------------------------|\n| Basic:                                |                                                                                     |           |                                                                 |\n| [`Float()`](#float)                   | Convert numbers into floats.                                                        |           |                                                                 |\n| [`Enumerate()`](#enumerate)           |                                                                                     |           |                                                                 |\n| [`OneHot()`](#onehot)                 |                                                                                     |           |                                                                 |\n| [`Binary()`](#binary)                 | Convert to 0 and 1. Detects common \"positive\" and \"negative\" terms in strings.      |           |                                                                 |\n| [`List()`](#list)                     |                                                                                     |           |                                                                 |\n| [`ListAndOr()`](#listandor)           |                                                                                     |           |                                                                 |\n| [`Map()`](#map)                       |                                                                                     | dict      | {\u003cbr\u003e\u0026nbsp;\u0026nbsp;\"foo\": 1,\u003cbr\u003e\u0026nbsp;\u0026nbsp;\"bar\": -2,\u003cbr\u003e}       |\n| [`Const()`](#const)                   | Return a constant value.                                                            | *any*     | 42\u003cbr\u003e\"foo\"                                                     |\n| Text Processing:                      |                                                                                     |           |                                                                 |\n| [`Strip()`](#strip)                   |                                                                                     |           |                                                                 |\n| [`Split()`](#split)                   |                                                                                     |           |                                                                 |\n| Combining Converters:                 |                                                                                     |           |                                                                 |\n| [`Pipeline()`](#pipeline)             | Apply multiple converters in sequence.                                              | list      | [\u003cbr\u003e\u0026nbsp;\u0026nbsp;Split(),\u003cbr\u003e\u0026nbsp;\u0026nbsp;ForEach(Strip()),\u003cbr\u003e] |\n| [`Try()`](#try)                       | Try multiple converters and return the first one that succeeds.                     | tuple     | (Float(), Binary())                                             |\n| [`ForEach()`](#foreach)               | Apply the same converter to all items.                                              |           |                                                                 |\n| [`Parallel()`](#parallel)             | Apply different converters to the respective items.                                 |           |                                                                 |\n| Special:                              |                                                                                     |           |                                                                 |\n| [`Id()`](#id)                         |                                                                                     |           |                                                                 |\n| [`Ignore()`](#ignore)                 | Drop the column.                                                                    | None      | None                                                            |\n| [`Infer()`](#infer)                   |                                                                                     |           |                                                                 |\n| [`Label()`](#label)                   |                                                                                     |           |                                                                 |\n| Dimensionality:                       |                                                                                     |           |                                                                 |\n| [`Flatten()`](#flatten)               | Flatten a tuple of tuples into a single tuple. This is often needed after `ForEach()` or `Parallel()`. |           |                                                                 |\n| [`Transpose()`](#transpose)           |                                                                                     |           |                                                                 |\n| Arbitrary Functions:                  |                                                                                     |           |                                                                 |\n| [`Function()`](#function)             | Apply a user-defined function to the data.                                          | callable  | lambda x: x**2                                                  |\n| [`StrictFunction()`](#strictfunction) | Apply a user-defined function to the data. Less flexible than `Function()`.         |           |                                                                 |\n\n---\n\n### Float\n\nConverts a column of numbers into a column of numbers.\nIf invalid values are encountered (`NaN`, `inf`, `None`, etc.),\na warning is printed and the value is replaced with `np.nan`.\nThis can be circumvented by passing a value to the `default` argument:\n\n```python\n\"Temperature\": Float(default=37.0)\n```\n\nYou can also specify `\"mean\"`, `\"median\"`, or `\"mode\"` as the default value.\nThis will choose the default value based on the data in the specified column:\n\n```python\n\"Temperature\": Float(default=\"mean\")\n```\n\n| Temperature | ⇒ | Temperature |\n|-------------|---|-------------|\n| 37.5        |   | 37.5        |\n| 40.0        |   | 40.0        |\n| 38.5        |   | 38.5        |\n|             |   | 38.75       |\n| 39.0        |   | 39.0        |\n\nResults in:\n\n```python\n\"Temperature\": Float(default=38.75)\n```\n\n---\n\n### Enumerate\n\nThis is the extension of the [`Binary()`](#binary) conversion method\nto columns with more than two possible values.\nThe values are converted into integers starting at 0,\nresulting in a single column of integers.\n\nThe possible values can be passed to the constructor:\n\n```python\n\"Country\": Enumerate(\"france\", \"germany\", \"italy\")\n```\n\n| Country | ⇒ | Country |\n|---------|---|---------|\n| france  |   | 0       |\n| italy   |   | 2       |\n| germany |   | 1       |\n\nTheir index in the argument list is used as the numerical value.\nIf no values are specified, the values found in the provided\ndata are sorted in lexically ascending order.\n\n---\n\n### OneHot\n\nIf each entry contains one of multiple possible values.\nThe possible values can be specified via the `values` argument:\n\n```python\n\"Education Level\": OneHot(\"primary\", \"secondary\", \"tertiary\")\n```\n\n| Education Level | ⇒ | Education Level=primary | Education Level=secondary | Education Level=tertiary |\n|-----------------|---|-------------------------|---------------------------|--------------------------|\n| primary         |   | 1                       | 0                         | 0                        |\n| secondary       |   | 0                       | 1                         | 0                        |\n| tertiary        |   | 0                       | 0                         | 1                        |\n\nIf no values are specified, the possible values are inferred from the data.\n\n---\n\n### Binary\n\nSimilar to [`Enumerate()`](#enumerate), but with just two possible values,\nand with some extra intelligence for this purpose.\nFor example, it can detect words commonly used for positive and negative values:\n\n- Positive: `yes`, `true`, `positive`, `1`, `female`\n- Negative: `no`, `false`, `negative`, `0`, `male`, `none`\n\nExample:\n\n```python\n\"Hospitalized\": Binary()\n```\n\n| Hospitalized | ⇒ | Hospitalized |\n|--------------|---|--------------|\n| no           |   | 0            |\n| yes          |   | 1            |\n| false        |   | 0            |\n| true         |   | 1            |\n| none         |   | 0            |\n\nResults in:\n\n```python\n\"Hospitalized\": Binary(positive={\"yes\", \"true\"},\n                       negative={\"no\", \"false\", \"none\"})\n```\n\nYou can explicitly specify the values of the `positive` class and the `negative` class via the constructor:\n\n```python\n\"Hospitalized\": Binary(positive=\"yes\", negative=\"no\")\n```\n\n| Hospitalized | ⇒ | Hospitalized |\n|--------------|---|--------------|\n| yes          |   | 1            |\n| no           |   | 0            |\n| no           |   | 0            |\n| yes          |   | 1            |\n\nIf only one argument is specified (either `positive` or `negative`),\nall other values present in the data are treated as instances of the other class:\n\n```python\n\"Time served\": Binary(negative=\"none\")\n```\n\n| Time served | ⇒ | Time served |\n|-------------|---|-------------|\n| none        |   | 0           |\n| 1 year      |   | 1           |\n| 4 years     |   | 1           |\n| none        |   | 0           |\n\nIt's also possible to specify more than one value for the ``positive`` and ``negative`` classes.\nExample:\n\n```python\n\"Hospitalized\": Binary(positive={\"yes\", \"true\"}, negative={\"no\", \"false\"})\n```\n\n| Hospitalized | ⇒ | Hospitalized |\n|--------------|---|--------------|\n| yes          |   | 1            |\n| no           |   | 0            |\n| false        |   | 0            |\n| true         |   | 1            |\n\nIf no positive or negative values are specified, a set of strings commonly used\nto indicate positive / negative values is tested against the available data.\nFor instance, in the example above, the specified arguments would have been\ninferred automatically as positive and negative.\n\nIf this approach is not successful, the lexically smallest value is chosen as the `negative` argument and\nthe `positive` argument is left empty, causing all other values to be treated as positive:\n\n```python\n\"Fruits\": Binary()\n```\n\n| Fruits | ⇒ | Fruits |\n|--------|---|--------|\n| banana |   | 1      |\n| apple  |   | 0      |\n| kiwi   |   | 1      |\n| apple  |   | 0      |\n\nResults in:\n\n```python\n\"Fruits\": Binary(negative=\"apple\")\n```\n\n---\n\n### List\n\nConverts lists of values into multiple binary columns.\n\n```python\n\"Symptoms\": List()\n```\n\n| Symptoms               | ⇒ | Symptoms=cough | Symptoms=fever | Symptoms=headache |\n|------------------------|---|----------------|----------------|-------------------|\n| fever, cough, headache |   | 1              | 1              | 1                 |\n| headache, cough        |   | 1              | 0              | 1                 |\n\nThe default delimiter is a comma.\u003cbr\u003e\nYou can specify a custom delimiter via the `delimiter` argument:\n\n```python\n\"Symptoms\": List(delimiter=\";\")\n\"Symptoms\": List(delimiter=[\",\", \";\"])  # also accepts lists\n```\n\nThe passed strings are interpreted as regular expressions.\n\n### ListAndOr\n\n```python\n\"Symptoms\": ListAndOr()\n```\n\n| Symptoms                  | ⇒ | Symptoms=cough | Symptoms=fever | Symptoms=headache |\n|---------------------------|---|----------------|----------------|-------------------|\n| fever, cough and headache |   | 1              | 1              | 1                 |\n| headache or cough         |   | 1              | 0              | 1                 |\n\nThe default delimiters are comma, \"and\" and \"or\".\u003cbr\u003e\nThe passed strings are interpreted as regular expressions.\n\n### Map\n\n### Strip\n\n### Split\n\n### Pipeline\n\n### Try\n\n```python\nTry(converter1, converter2, ...)\n```\n\nReturns value of the first converter that does not raise an exception,\nor the original value if all converters raise an exception.\n`Try()` always only applies one converter and returns its output (if it didn't fail).\n\n```python\n\"Product\": Try(Float(), Infer())  # will infer the converter for the samples that cannot be converted to floats\n```\n\n| Product | ⇒ | Product |\n|---------|---|---------|\n| Kiwi    |   | 48      |\n| Apple   |   | 0       |\n| 712356  |   | 712356  |\n| 261382  |   | 261382  |\n| Banana  |   | 1       |\n| Kiwi    |   | 48      |\n| ...     |   | ...     |\n\nThis would result in the following profile after `fit()`:\n\n```python\n\"Product\": Try(Float(), Enumerate(\"Apple\", \"Banana\", ...))\n```\n\n### ForEach\n\nApply the same converter to all items.\n\n### Parallel\n\n```python\nParallel(converter1, converter2, ...)\n```\n\nApply different converters to the respective items.\nUsually used in a Pipeline after other converters that create outputs with multiple items (e.g. `Split()`).\nAlso, you usually want to use `Flatten()` after this, as each individual converter will return a tuple of items,\neven if it only contains one item.\nExample:\n\n```python\n\"Latitude;Longitude\": [\n    Split(\";\"),  # must always result in two items, because Parallel() has 2 converters\n    Parallel(Ignore(), Float()),  # ignore latitude, convert longitude to float -\u003e [[], [longitude]]\n    Flatten(),  # -\u003e [longitude]\n]\n```\n\n| Latitude;Longitude  | ⇒ | Longitude |\n|---------------------|---|-----------|\n| 52.520008;13.404954 |   | 13.404954 |\n| 48.137154;11.576124 |   | 11.576124 |\n\n### Const\n\n### Id\n\nIdentity.\nKeeps the input unchanged.\n\n### Ignore\n\nDrops the column.\n\n```python\n\"registration_timestamp\": None\n```\n\nThis is chosen if no appropriate conversion method could be found.\n\n### Infer\n\nTries to infer the conversion method from the column name.\nAfter `fit()`, this converter will be replaced with the inferred converter in the profile.\n\nThis is the default converter for columns where no converter is specified.\nThis converter can however also be used anywhere else explicitly.\nExamples:\n\n```python\n\"col1\": [\n    str.upper,\n    Infer()\n],\n\"col2\": Try(Float(), Infer()),  # will infer the converter for the samples that cannot be converted to floats\n```\n\n### Label\n\n### Flatten\n\n### Transpose\n\nCan transpose nested tuples, given that the nested tuples are of equal length.\n\nFor example, look at this elegant implementation of the [`List()`](#list) converter:\n\n```python\n\"Symptoms\": [\n    Split(r\"\\s*,\\s*\"),  # split at comma\n    ForEach(OneHot()),\n    Transpose(),\n    ForEach(max),\n    Flatten()\n]\n```\n\n`Transpose()` allows us to apply `max` to each column of the one-hot encodings\nacross all tuple elements.\n\n### Function\n\n```python\nFunction(transform, labels=None)\n```\n\nShorthand: Instead of `Function(transform, None)`, just write `transform`, where `transform` is some callable.\n\nCreates a custom converter from a custom ``transform()`` function\n(and optionally, a custom ``labels()`` function).\nThis is a handy way to create a converter that doesn't need ``fit()``.\n\nUnlike ``StrictFunction()``, this class can handle functions that don't accept or return tuples,\nwhich often allows for more concise code.\n\nThis is achieved during ``fit()`` as follows:\n\n1. If all incoming items are 1-element tuples, it sets a flag ``UNPACK_OUTPUT`` to always\n   unpack the element before passing them to the wrapped function during ``transform()``.\n2. If during ``fit()`` the wrapped function doesn't return tuples,\n   it tries to turn that output into a tuple:\n    - If the output is always a non-string iterable, it will simply set a\n      flag ``CONVERT_ITERABLE`` to always convert the iterable output into a tuple during ``transform()``.\n    - Otherwise, it sets a flag ``WRAP_OUTPUT`` to always wrap the output in a\n      1-element tuple during ``transform()``.\n\nA similar logic is applied to the labels.\nIf a custom labels function is given, the following procedure is followed during ``labels()``:\n\n1. If the incoming labels are a 1-element tuples, the single label is unpacked before\n   it is passed to the custom labels function.\n2. If the custom labels function returns something other than a tuple,\n   this class tries to convert it into a tuple:\n    - If the output is a non-string iterable, it is converted into a tuple.\n    - Otherwise, the output is wrapped in a 1-element tuple.\n\nIf no custom labels function is given, the output labels are generated based on the\noutput cardinality inferred during ``fit()`` and according to the following logic:\n\n- If the number of incoming labels is identical to the output cardinality,\n  the labels will be returned unchanged.\n- If there are multiple incoming labels but a single output label,\n  the output label is formed by joining the incoming labels with ``, ``.\n- If there is a single incoming label but multiple output labels,\n  the output labels are formed by adding suffixes ``_0``, ``_1``, etc.\n  to the single input label.\n\n|                    | **1 Output Label** | **M Output Labels**          |\n|--------------------|--------------------|------------------------------|\n| **1 Input Label**  | identical          | suffixes ``_0``, ``_1``, ... |\n| **N Input Labels** | join with ``, ``   | identical if M=N, else ERROR |\n\nA special case are functions returning output of varying cardinality during ``fit()``.\nIn this case, a single label is returned.\nIf only one input label is given, that single label is returned.\nIf multiple input labels are given, they are joined with ``_``.\n\nThe following example turns a text column into two columns containing the ascii code of the first and last letter.\n\n```python\n\"Name\": lambda x: (ord(x[0]), ord(x[-1]))\n```\n\n| Name  | ⇒ | Name_0 | Name_1 |\n|-------|---|--------|--------|\n| Alice |   | 97     | 101    |\n| Bob   |   | 98     | 98     |\n\n(Remember that by default, all text entries are converted to\nlowercase before further processing.)\n\nAs you can see, the number of columns is inferred directly from the return value of the conversion function.\nIf the function returns a tuple, the resulting column names are indexed.\n\nYou can also set the labels explicitly with a lambda function\nthat takes the input column name as an argument and returns output column names:\n\n```python\n\"Name\": Function(lambda x: (ord(x[0]), ord(x[-1])),\n                 labels=lambda s: (f\"ord(first letter of {s})\", f\"ord(last letter of {s})\")),\n```\n\n| Name  | ⇒ | ord(first letter of Name) | ord(last letter of Name) |\n|-------|---|---------------------------|--------------------------|\n| Alice |   | 97                        | 101                      |\n| Bob   |   | 98                        | 98                       |\n\nHowever, remember that you can always simply use [`Label()`](#label) to rename the columns after the conversion,\nif you don't need the output column names to depend on the input column names.\n\n```python\n\"Name\": [lambda x: (ord(x[0]), ord(x[-1])),\n         Labels(\"ord(first letter)\", \"ord(last letter)\")],\n```\n\n### StrictFunction\n\n```python\nStrictFunction(transform, labels=None)\n```\n\nWorks mostly like [`Function()`](#function), but simpler:\n``transform`` and ``labels`` must both accept and return tuples.\nInstead of something like this:\n\n```python\n\"Name\": str.lower,\n```\n\nyou have to write this:\n\n```python\n\"Name\": StrictFunction(lambda x: (str.lower(x[0]),))  # notice the comma, which makes it a 1-element tuple\n```\n\nThat is, you will still receive 1-element tuples as tuples to the function,\neven if all input elements during `fit()` are 1-element tuples.\nAlso, you must now explicitly return a tuple,\neven if it is just a 1-element tuple, as otherwise an error will be raised.\n\nSee [`Function()`](#function) for a convenient extension of this converter.\n\n---\n\n## Understanding Multi-Column Converters\n\nA converter returns two things:\n\n- `transform()`: the items of the transformed data\n- `labels()`: a label for each item\n\nBoth return values are tuples.\n\nFor top-level converters, this then creates the corresponding amount of columns.\nThis includes the case of\n\n- a 1-element tuple `(item,)`, which is the case for most converters.\n- an empty tuple `()`, in which the result is ignored.\n  (In fact, this is exactly how `Ignore()` is implemented.)\n\nThis means, however, that for top-level converters, `labels()` and `transform()`\nmust return the same number of items.\nThat is because `labels()` is used to create the output column names.\nIf `transform()` returns a different number of items, that will raise an error for top-level converters.\n\nHowever, for nested converters, `labels()` and `transform()` can return different numbers of items.\nFor example `Split.labels()` always returns only one item,\nbecause the number of items returned by `Split.transform()` varies from input to input.\nTherefore, `Split()` can't be used as a top-level converter\nand has to be used inside a `Pipeline` or similar devices,\nso that other converters can ensure that the final output is of constant size.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftom-mohr%2Fclevertable","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftom-mohr%2Fclevertable","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftom-mohr%2Fclevertable/lists"}