{"id":32657257,"url":"https://github.com/hansalemaos/outguncsv","last_synced_at":"2026-05-09T05:04:27.896Z","repository":{"id":66561954,"uuid":"604006392","full_name":"hansalemaos/outguncsv","owner":"hansalemaos","description":"Finds the right CSV separator and excludes bad lines in corrupt CSV files","archived":false,"fork":false,"pushed_at":"2023-04-07T12:32:06.000Z","size":10,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-31T12:01:48.769Z","etag":null,"topics":["corrupt","csv","pandas","python","read"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/outguncsv","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hansalemaos.png","metadata":{"files":{"readme":"README.MD","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-02-20T06:08:09.000Z","updated_at":"2023-02-20T06:09:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"91be7ac1-ef56-4d1f-acdd-6532a8e32848","html_url":"https://github.com/hansalemaos/outguncsv","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hansalemaos/outguncsv","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hansalemaos%2Foutguncsv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hansalemaos%2Foutguncsv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hansalemaos%2Foutguncsv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hansalemaos%2Foutguncsv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hansalemaos","download_url":"https://codeload.github.com/hansalemaos/outguncsv/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hansalemaos%2Foutguncsv/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32807862,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-08T08:22:46.396Z","status":"online","status_checked_at":"2026-05-09T02:00:06.633Z","response_time":123,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["corrupt","csv","pandas","python","read"],"created_at":"2025-10-31T12:01:02.552Z","updated_at":"2026-05-09T05:04:27.860Z","avatar_url":"https://github.com/hansalemaos.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Finds the right CSV separator and excludes bad lines\n### pip install outguncsv\n#### Example:\n```python\n\n# You have probably seen this before, right?\nimport pandas as pd\npd.read_csv(r\"https://github.com/zdavatz/diprela/raw/main/csv/diprela.csv\")\nTraceback (most recent call last):\n  File \"C:\\Users\\Gamer\\anaconda3\\envs\\dfdir\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3398, in run_code\n    exec(code_obj, self.user_global_ns, self.user_ns)\n  File \"\u003cipython-input-8-c5703c9ae399\u003e\", line 1, in \u003ccell line: 1\u003e\n  ....\n    File \"pandas\\_libs\\parsers.pyx\", line 1973, in pandas._libs.parsers.raise_parser_error\npandas.errors.ParserError: Error tokenizing data. C error: Expected 1 fields in line 3, saw 10\n\n\n\npd.read_csv(r\"https://github.com/zdavatz/diprela/raw/main/csv/diprela.csv\",on_bad_lines='skip')\n# Better, but everything in one column, and we have lost about 600 rows. \n    Schweizerische Nährwertdatenbank komplett (bearbeitet durch Lesley Grünenfelder) (Stand: 11.02.2023) ;;;;;;;;;;;;;;;;;;;;;;;;;;;Schweizerische Nährwertdatenbank komplett (bearbeitet durch Lesley Grünenfelder) (Stand: 30.01.2023) ;;;;;;;;;;;;;;;;;;;;;\n0     ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;                                                                                                                                                                                                        \n1    Agar Agar;;Gelier- und Bindemittel ;Verschiede...                                                                                                                                                                                                        \n2    Agavensirup;Agavendicksaft ;Zucker und Süsssto...                                                                                                                                                                                                        \n3    Ahornsirup;;Zucker und Süssstoffe;Verschiedene...                                                                                                                                                                                                        \n4    Älplermagronen;zubereitet;salzige Gerichte ;Ge...                                                                                                                                                                                                        \n..                                                 ...                                                                                                                                                                                                        \n521  Zwieback ;;Brot und Backware ;Getreide und Get...                                                                                                                                                                                                        \n522  Zwieback;Vollkorn;Brot und Backware ;Getreide ...                                                                                                                                                                                                        \n523  Zwiebel;gedünstet (ohne Zugabe von Fett und Sa...                                                                                                                                                                                                        \n524  Zwiebel;geröstet (ohne Zugabe von Fett und Sal...                                                                                                                                                                                                        \n525  Zwiebel;roh;Gemüse frisch ;Gemüse ;pro 100g es...                                                                                                                                                                                                        \n[526 rows x 1 columns]\n\n\n\n# If you have problems reading a CSV file, use: \nfrom outguncsv import read_balky_csv_files\nalf3 = r\"https://github.com/zdavatz/diprela/raw/main/csv/diprela.csv\"\ndf3 = read_balky_csv_files(\n    csvfiles=alf3,\n    encoding=\"utf-8\",\n    sep=None,\n    regexremove=(),\n    filepathcolumn=\"file\",\n    on_bad_lines=\"warn\",\n)\n\n\ndf3\nOut[4]: \n                                                      0  ...                                               file\n0     Schweizerische Nährwertdatenbank komplett (bea...  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n1                                                   NaN  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n2                                                  Name  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n3                                             Agar Agar  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n4                                           Agavensirup  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n                                                 ...  ...                                                ...\n1213                                            Zwiebel  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n1214                                      Zwiebelkuchen  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n1215                                      Zwiebelkuchen  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n1216                                      Zwiebelkuchen  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n1217                                      Zwiebelkuchen  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n[1218 rows x 50 columns]\n\n\n```\n\n```python\nfrom outguncsv import read_balky_csv_files\nimport glob\nalf = glob.glob(\n    r\"C:\\Users\\Gamer\\Documents\\Downloads\\anyascii-master\\input\\tables\\*.tsv\"\n)\ndf = read_balky_csv_files(\n    csvfiles=alf,  # list or string (url/file path)\n    encoding=\"utf-8\",\n    sep=None,  # if None, it does its best to find the best separator\n    regexremove=(),  # remove lines, regex must be in binary: rb\"^\\s*#\\s+.*$\",\n    filepathcolumn=\"file\",  # a new colum will be created with the file path\n    on_bad_lines=\"skip\",  # use either skip or warn, it won't work with error\n    # for *args, **kwargs -\u003e https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html\n\t\n\t\n)\n\nalf = r\"https://raw.githubusercontent.com/pandas-dev/pandas/main/doc/data/titanic.csv\"\ndf2 = read_balky_csv_files(\n    csvfiles=alf,\n    encoding=\"utf-8\",\n    sep=None,\n    regexremove=(),\n    filepathcolumn=\"file\",\n    on_bad_lines=\"skip\",\n)\nalf3 = r\"https://github.com/zdavatz/diprela/raw/main/csv/diprela.csv\"\ndf3 = read_balky_csv_files(\n    csvfiles=alf3,\n    encoding=\"utf-8\",\n    sep=None,\n    regexremove=(),\n    filepathcolumn=\"file\",\n    on_bad_lines=\"warn\",\n)\nalf4 = \"https://github.com/curran/data/raw/gh-pages/migrants/events.csv\"\ndf4 = read_balky_csv_files(\n    csvfiles=alf4,\n    encoding=\"utf-8\",\n    sep=\",\",\n    regexremove=(),\n    filepathcolumn=\"file\",\n    on_bad_lines=\"skip\",\n)\n\n\ndf\nOut[3]: \n       0  1                                               file\n0      𞤢  a  C:\\Users\\Gamer\\Documents\\Downloads\\anyascii-ma...\n1      𞤣  d  C:\\Users\\Gamer\\Documents\\Downloads\\anyascii-ma...\n2      𞤤  l  C:\\Users\\Gamer\\Documents\\Downloads\\anyascii-ma...\n3      𞤥  m  C:\\Users\\Gamer\\Documents\\Downloads\\anyascii-ma...\n4      𞤦  b  C:\\Users\\Gamer\\Documents\\Downloads\\anyascii-ma...\n   .. ..                                                ...\n14439  𜾿  -  C:\\Users\\Gamer\\Documents\\Downloads\\anyascii-ma...\n14440  𜿀  -  C:\\Users\\Gamer\\Documents\\Downloads\\anyascii-ma...\n14441  𜿁  -  C:\\Users\\Gamer\\Documents\\Downloads\\anyascii-ma...\n14442  𜿂  -  C:\\Users\\Gamer\\Documents\\Downloads\\anyascii-ma...\n14443  𜿃  -  C:\\Users\\Gamer\\Documents\\Downloads\\anyascii-ma...\n[14444 rows x 3 columns]\ndf2\nOut[4]: \n       0  1  2  ...    10 11                                               file\n0      1  0  3  ...   NaN  S  https://raw.githubusercontent.com/pandas-dev/p...\n1      2  1  1  ...   C85  C  https://raw.githubusercontent.com/pandas-dev/p...\n2      3  1  3  ...   NaN  S  https://raw.githubusercontent.com/pandas-dev/p...\n3      4  1  1  ...  C123  S  https://raw.githubusercontent.com/pandas-dev/p...\n4      5  0  3  ...   NaN  S  https://raw.githubusercontent.com/pandas-dev/p...\n..   ... .. ..  ...   ... ..                                                ...\n886  887  0  2  ...   NaN  S  https://raw.githubusercontent.com/pandas-dev/p...\n887  888  1  1  ...   B42  S  https://raw.githubusercontent.com/pandas-dev/p...\n888  889  0  3  ...   NaN  S  https://raw.githubusercontent.com/pandas-dev/p...\n889  890  1  1  ...  C148  C  https://raw.githubusercontent.com/pandas-dev/p...\n890  891  0  3  ...   NaN  Q  https://raw.githubusercontent.com/pandas-dev/p...\n[891 rows x 13 columns]\ndf3\nOut[5]: \n                                                      0  ...                                               file\n0     Schweizerische Nährwertdatenbank komplett (bea...  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n1                                                   NaN  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n2                                                  Name  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n3                                             Agar Agar  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n4                                           Agavensirup  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n                                                 ...  ...                                                ...\n1213                                            Zwiebel  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n1214                                      Zwiebelkuchen  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n1215                                      Zwiebelkuchen  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n1216                                      Zwiebelkuchen  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n1217                                      Zwiebelkuchen  ...  https://github.com/zdavatz/diprela/raw/main/cs...\n[1218 rows x 50 columns]\ndf4\nOut[6]: \n           0  ...                                               file\n0        NaN  ...  https://github.com/curran/data/raw/gh-pages/mi...\n1    57234.0  ...  https://github.com/curran/data/raw/gh-pages/mi...\n2    56633.0  ...  https://github.com/curran/data/raw/gh-pages/mi...\n3    72740.0  ...  https://github.com/curran/data/raw/gh-pages/mi...\n4    55194.0  ...  https://github.com/curran/data/raw/gh-pages/mi...\n..       ...  ...                                                ...\n955  36496.0  ...  https://github.com/curran/data/raw/gh-pages/mi...\n956  36500.0  ...  https://github.com/curran/data/raw/gh-pages/mi...\n957  36499.0  ...  https://github.com/curran/data/raw/gh-pages/mi...\n958  36503.0  ...  https://github.com/curran/data/raw/gh-pages/mi...\n959  36502.0  ...  https://github.com/curran/data/raw/gh-pages/mi...\n[960 rows x 22 columns]\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhansalemaos%2Foutguncsv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhansalemaos%2Foutguncsv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhansalemaos%2Foutguncsv/lists"}