{"id":16419710,"url":"https://github.com/salaah01/pandas-data-cleaner","last_synced_at":"2025-08-11T16:12:47.692Z","repository":{"id":104266427,"uuid":"454170937","full_name":"Salaah01/pandas-data-cleaner","owner":"Salaah01","description":"A package to aid with data cleaning using pandas.","archived":false,"fork":false,"pushed_at":"2022-02-02T02:55:20.000Z","size":28,"stargazers_count":5,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-30T23:02:03.150Z","etag":null,"topics":["datacleaning","pandas","python"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/pandas-data-cleaner/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Salaah01.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-31T21:09:38.000Z","updated_at":"2022-03-19T18:22:43.000Z","dependencies_parsed_at":"2023-03-13T14:59:30.745Z","dependency_job_id":null,"html_url":"https://github.com/Salaah01/pandas-data-cleaner","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Salaah01/pandas-data-cleaner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Salaah01%2Fpandas-data-cleaner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Salaah01%2Fpandas-data-cleaner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Salaah01%2Fpandas-data-cleaner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Salaah01%2Fpandas-data-cleaner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Salaah01","download_url":"https://codeload.github.com/Salaah01/pandas-data-cleaner/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Salaah01%2Fpandas-data-cleaner/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269917325,"owners_count":24496069,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-11T02:00:10.019Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datacleaning","pandas","python"],"created_at":"2024-10-11T07:25:40.399Z","updated_at":"2025-08-11T16:12:47.645Z","avatar_url":"https://github.com/Salaah01.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pandas Data Cleaner\nThis package is a data cleaning tool for Pandas DataFrames and other objects with a similar structure.\n\nThe tool is designed to help clean data by providing a function onto which you can apply various cleaning methods.\n\nThe main cleaning function can be found in `pandas_data_cleaner.base.clean_data`.\n\nThe app also provides an abstract base class `pandas_data_cleaner.base.CleaningStrategy` which can be used to implement custom cleaning strategies.\n\n## Installation\nTo install the application, run the following command:\n```bash\npip install pandas-data-cleaner\n```\n\n## Cleaning Data\nIn order to clean data, you need:\n* Pandas DataFrame\n* List of strategies to apply\n* Any additional arguments that you may need to pass to the cleaning function.\n\nLet's suppose we have the following DataFrame:\n\n```python\nimport pandas as pd\n\ndataframe = pd.DataFrame({\n    \"id\": [1, 2, 1],\n    \"structure_value\": [\"a\", \"a\", \"a\"],\n    \"status\": [\"ENABLED\", \"ENABLED\", \"DISABLED\"],\n})\n```\nAs a table, this would look like this:\n\n| id  | structure_value | status   |\n| --- | --------------- | -------- |\n| 1   | a               | ENABLED  |\n| 2   | a               | ENABLED  |\n| 1   | a               | DISABLED |\n\nIn this data frame, we can see that there are two rows with the same id but different values for status.\n\nAs part of our cleaning exercise, we want to keep the latest row of data as this is the most up-to-date.\n\nLet's try to apply the RemoveDuplicates cleaning strategy to the data frame:\n\n```python\nimport pandas as pd\nfrom pandas_data_cleaner.base import clean_data\nfrom pandas_data_cleaner.strategies import RemoveDuplicates\n\ndataframe = pd.DataFrame({\n    \"id\": [1, 2, 1],\n    \"structure_value\": [\"a\", \"a\", \"a\"],\n    \"status\": [\"ENABLED\", \"ENABLED\", \"DISABLED\"],\n})\n\ndataframe = clean_data(dataframe, [RemoveDuplicates])\n```\n\nRunning this will result in the following error:\n\n```bash\npandas_data_cleaner.exceptions.MissingOptionsError: Missing kwargs:\nremove_duplicates_subset_fields\nremove_duplicates_keep\n```\n\nThis lets us that we need to provide additional arguments when calling the cleaning function, these are:\n* `remove_duplicates_subset_fields`\n* `remove_dupplicates_keep`\n\nTo find out more information about the additional arguments required, you can run:\n```python\nRemoveDuplicates.info()\n```\nThis will return some information on how the strategy works as well as additional information on the arguments that are required.\n\nFor the `RemoveDuplicates` cleaning strategy, `remove_duplicates_subset_fields` is the fields we should perform the duplicate removal on and `remove_duplicates_keep` indicates given some duplicates are, which row should we keep.\n\nIf we now tweak our earlier code:\n\n\n```python\nimport pandas as pd\nfrom pandas_data_cleaner.base import clean_data\nfrom pandas_data_cleaner.strategies import RemoveDuplicates\n\ndataframe = pd.DataFrame({\n    \"id\": [1, 2, 1],\n    \"structure_value\": [\"a\", \"a\", \"a\"],\n    \"status\": [\"ENABLED\", \"ENABLED\", \"DISABLED\"],\n})\n\ndataframe = clean_data(\n    dataframe,\n    [RemoveDuplicates],\n    remove_duplicates_subset_fields=[\"id\"],\n    remove_duplicates_keep=\"last\"\n)\n```\n\nWe will now get the following data frame:\n\n```python\npd.DataFrame({\n    \"id\": [2, 1],\n    \"structure_value\": [\"a\", \"a\"],\n    \"status\": [\"ENABLED\", \"DISABLED\"],\n})\n```\n\nAs a table:\n\n| id  | structure_value | status   |\n| --- | --------------- | -------- |\n| 2   | a               | ENABLED  |\n| 1   | a               | DISABLED |\n\nAs we had set `remove_duplicates_subset_fields=[\"id\"]`, it found that there were two rows with the same ID. As we set `remove_duplicates_keep=\"last\"`, it kept the last row only.\n\nIn our example, we used only one cleaning strategy, but we are free to use as many as we like, we simply need to add all the strategies to the list of cleaning strategies to apply.\n\n## Creating Custom Cleaning Strategies\nLet's suppose we intend to create a new cleaning strategy that removes certain columns.\n\nWe would create a new class inheriting from `base.CleaningStrategy`:\n\n```python\nfrom pandas_data_cleaner.base import CleaningStrategy\n\n\nclass RemoveColumns(CleaningStrategy):\n    pass\n```\n\nWhen using this strategy, we need to know which column names to remove. We will therefore decide that, when using this class in the `clean_data` method, we need to provide a `remove_columns` argument.\n\nTo do this, we simply create a class attribute called `required_options` and set it to `[\"remove_columns\"]`.\n\nWe also will add some documentation to allow the end-user to receive some useful information when they run `RemoveColumns.info()`.\n\nOur new strategy will now look like this:\n\n```python\nclass RemoveColumns(CleaningStrategy):\n    \"\"\"Removes columns from a dataframe.\n\n    Required options:\n        `remove_columns` - (_t.List[str]) A list of columns to remove.\n    \"\"\"\n\n    required_options = [\"remove_columns\"]\n```\n\nNow, we need to create our cleaning method.\nOnce the cleaning method has been added, the class will look like the following:\n\n```python\nclass RemoveColumns(CleaningStrategy):\n    \"\"\"Removes columns from a dataframe.\n\n    Required options:\n        `remove_columns` - (List[str]) A list of columns to remove.\n    \"\"\"\n\n    required_options = [\"remove_columns\"]\n\n    def clean(self):\n        \"\"\"Executes the cleaning task.\"\"\"\n        self.dataframe.drop(\n            self.remove_columns, axis=1, inplace=True\n        )\n```\n\nLet's discuss how this cleaning method works.\nFirstly, whenever a user would use this strategy may run the following:\n\n```python\nclean_data(dataframe, [RemoveColumns], remove_columns=[\"id\", \"status\"])\n```\n\n`clean_data` will instantiate each cleaning strategy, in this case, just `RemoveColumns` providing the data frame as a required initial parameter as well as passing any keyword arguments to the function.\n\nEach strategy would then set both the `dataframe` and each keyword argument to the self object.\n\nThis means that within the clean method, we would have access:\n* `self.dataframe`\n* `self.remove_columns`.\n\nIf the command the user ran was instead:\n```python\nclean_data(dataframe, [RemoveColumns], remove_columns=[\"id\", \"status\"], foo=\"bar\")\n```\n\nThen within the clean method would have access:\n* `self.dataframe`\n* `self.remove_columns`\n* `self.foo`\n\nBy adding `remove_columns` to the `required_options` list, once this class is instantiated, we will be able to access `self.remove_columns`.\n\nNow that we have built our cleaning strategy let's run it:\n\n```python\ndataframe = pd.DataFrame({\n    \"id\": [1, 2, 3],\n    \"col1\": [1, 2, 3],\n    \"col2\": [1, 2, 3],\n    \"col3\": [1, 2, 3],\n})\n\ndataframe = clean_data(\n    dataframe,\n    [RemoveColumns],\n    remove_columns=[\"col1\", \"col2\"]\n)\n\nprint(dataframe)\n\n\u003e\u003e\u003e pd.DataFrame({\n    \"id\": [1, 2, 3],\n    \"col3\": [1, 2, 3],\n})\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalaah01%2Fpandas-data-cleaner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsalaah01%2Fpandas-data-cleaner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalaah01%2Fpandas-data-cleaner/lists"}