{"id":18538106,"url":"https://github.com/bigmb/mb_pandas","last_synced_at":"2026-02-17T13:08:38.984Z","repository":{"id":65817585,"uuid":"568312794","full_name":"bigmb/mb_pandas","owner":"bigmb","description":"Personal addition to pandas data ETL for faster and better performance","archived":false,"fork":false,"pushed_at":"2024-12-16T09:48:20.000Z","size":94,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-15T02:11:22.899Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bigmb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-11-20T05:40:51.000Z","updated_at":"2024-12-16T09:48:24.000Z","dependencies_parsed_at":"2024-11-06T19:43:57.277Z","dependency_job_id":"307d258b-7d2f-4c64-931b-4d11ff8d7bfe","html_url":"https://github.com/bigmb/mb_pandas","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/bigmb/mb_pandas","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bigmb%2Fmb_pandas","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bigmb%2Fmb_pandas/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bigmb%2Fmb_pandas/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bigmb%2Fmb_pandas/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bigmb","download_url":"https://codeload.github.com/bigmb/mb_pandas/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bigmb%2Fmb_pandas/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29472879,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-15T06:58:05.414Z","status":"ssl_error","status_checked_at":"2026-02-15T06:58:05.085Z","response_time":118,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T19:42:20.430Z","updated_at":"2026-02-15T07:11:33.387Z","avatar_url":"https://github.com/bigmb.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mb_pandas\n\nA Python package providing enhanced pandas functionality with async support and optimized operations.\n\n## Features\n\n- **Asynchronous DataFrame Loading**: Load large CSV and Parquet files efficiently using async I/O\n- **Optimized DataFrame Merging**: Merge large DataFrames using chunking or Dask\n- **Data Type Conversions**: Convert between string representations and Python objects\n- **DataFrame Profiling**: Generate detailed profiling reports and comparisons\n- **Data Transformation**: Various utilities for DataFrame transformations\n\n## Installation\n\n```bash\npip install mb_pandas\n```\n\n## Dependencies\n\n- Python \u003e= 3.8\n- numpy\n- pandas\n- colorama\n\n\u003c!-- - tqdm (for progress bars)\n- pyarrow (for parquet support)\n- dask (for distributed computing)\n- opencv-python (for image support)\n- pandas-profiling (for profiling features)\n- mb_utils (for logging utilities) --\u003e\n\n## Modules\n\n### transform.py\n\nFunctions for DataFrame transformations and merging operations.\n\n```python\nfrom mb_pandas.transform import merge_chunk, merge_dask, check_null, remove_unnamed,rename_columns\n\n# Merge large DataFrames in chunks\nresult = merge_chunk(df1, df2, chunksize=10000)\n\n# Merge using Dask for distributed computing\nresult = merge_dask(df1, df2)\n\n# Check and handle null values\ndf = check_null('data.csv', fillna=True)\n\n# Remove unnamed columns\ndf = remove_unnamed(df)\n\n# Rename column\ndf = rename_columns(data,'labels2','labels')\n```\n\n### dfload.py\n\nAsynchronous DataFrame loading utilities.\n\n```python\nfrom mb_pandas.dfload import load_any_df\n\n# Load any supported file format\ndf = load_any_df('data.csv')\ndf = load_any_df('data.parquet')\n\n# Convert string columns to Python objects\ndf = load_any_df('data.csv', literal_ast_columns=['json_col'])\n```\n\n### aio.py\n\nAsynchronous I/O utilities.\n\n```python\nfrom mb_pandas.aio import read_text, srun\n\n# Read file asynchronously\ncontent = await read_text('file.txt', context_vars={'async': True})\n\n# Run async function synchronously\nresult = srun(async_function, *args)\n```\n\n### convert_data.py\n\nData type conversion utilities.\n\n```python\nfrom mb_pandas.convert_data import convert_string_to_list, convert_string_to_dict, convert_string_to_type\n\n# Convert string representations to lists\ndf = convert_string_to_list(df, 'list_column')\n\n# Convert string representations to dictionaries\ndf = convert_string_to_dict(df, 'dict_column')\n\n# Convert strings to specific types\ndf = convert_string_to_type(df, 'number_column', int)\n```\n\n### profiler.py\n\nDataFrame profiling and comparison utilities.\n\n```python\nfrom mb_pandas.profiler import create_profile, profile_compare\n\n# Generate profiling report\ncreate_profile(df, 'report.html', target=['target_column'])\n\n# Compare two DataFrames\nprofile_compare(df1, df2, 'comparison.html')\n```\n\n## Key Functions\n\n### merge_chunk(df1, df2, chunksize=10000)\nMerge two DataFrames in chunks to handle large datasets efficiently.\n\n### merge_dask(df1, df2)\nMerge two DataFrames using Dask for improved performance with large datasets.\n\n### load_any_df(file_path, show_progress=True)\nLoad DataFrames from various file formats with progress tracking.\n\n### convert_string_to_list(df, column)\nConvert string representations of lists in a DataFrame column to actual lists.\n\n### create_profile(df, profile_name='report.html')\nGenerate a detailed profiling report for a DataFrame.\n\n## Error Handling\n\nAll functions include comprehensive error handling with descriptive messages:\n\n```python\ntry:\n    df = load_any_df('data.csv')\nexcept ValueError as e:\n    print(f\"Error loading file: {e}\")\n```\n\n## Logging\n\nMost functions accept an optional logger parameter for operation tracking:\n\n```python\nimport logging\nlogger = logging.getLogger()\ndf = load_any_df('data.csv', logger=logger)\n```\n\n## Performance Tips\n\n1. Use `merge_chunk` for large DataFrame merges that fit in memory\n2. Use `merge_dask` for very large datasets that benefit from distributed computing\n3. Enable `show_progress=True` to monitor long-running operations\n4. Use `minimal=True` in profiling for large datasets\n5. Consider sampling large datasets before profiling\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbigmb%2Fmb_pandas","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbigmb%2Fmb_pandas","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbigmb%2Fmb_pandas/lists"}