{"id":17017590,"url":"https://github.com/datahappy1/csv_to_parquet_converter","last_synced_at":"2025-06-27T21:37:01.576Z","repository":{"id":122695293,"uuid":"223762180","full_name":"datahappy1/csv_to_parquet_converter","owner":"datahappy1","description":"csv to parquet and vice versa file converter based on Pandas written in Python3","archived":false,"fork":false,"pushed_at":"2021-03-23T19:43:30.000Z","size":33,"stargazers_count":9,"open_issues_count":0,"forks_count":9,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-12T09:43:47.673Z","etag":null,"topics":["aws-s3","converter","csv","pandas","parquet","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datahappy1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-11-24T15:01:53.000Z","updated_at":"2022-11-08T01:03:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"4167e943-f943-47ea-ab8d-42a0d6aab7b5","html_url":"https://github.com/datahappy1/csv_to_parquet_converter","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/datahappy1/csv_to_parquet_converter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datahappy1%2Fcsv_to_parquet_converter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datahappy1%2Fcsv_to_parquet_converter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datahappy1%2Fcsv_to_parquet_converter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datahappy1%2Fcsv_to_parquet_converter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datahappy1","download_url":"https://codeload.github.com/datahappy1/csv_to_parquet_converter/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datahappy1%2Fcsv_to_parquet_converter/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262337154,"owners_count":23296031,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-s3","converter","csv","pandas","parquet","python3"],"created_at":"2024-10-14T06:37:01.961Z","updated_at":"2025-06-27T21:37:01.568Z","avatar_url":"https://github.com/datahappy1.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 class=\"code-line\" data-line-start=0 data-line-end=1 \u003e\u003ca id=\"csv_to_parquet_and_parquet_to_csv_converter_0\"\u003e\u003c/a\u003ecsv to parquet and parquet to csv converter\u003c/h1\u003e\n\u003ch2 class=\"code-line\" data-line-start=1 data-line-end=2 \u003e\u003ca id=\"10000ft_Overview_1\"\u003e\u003c/a\u003e10000ft. Overview\u003c/h2\u003e\nThis tool is able to convert .csv files to .parquet files used for columnar storage typically in the Hadoop ecosystem. It is also able to\nconvert .parquet files to .csv files. This is achieved thanks to the 4 built-in Pandas dataframe methods \u003ccode\u003eread_csv\u003c/code\u003e, \u003ccode\u003eread_parquet\u003c/code\u003e, \u003ccode\u003eto_csv\u003c/code\u003e and \u003ccode\u003eto_parquet\u003c/code\u003e.\nBecause Pandas uses \u003ccode\u003es3fs\u003c/code\u003e for AWS S3 integration, so you are free to choose whether the location of the source and/or converted target files is on your local machine or in AWS S3.\n\u003ch2 class=\"code-line\" data-line-start=3 data-line-end=4 \u003e\u003ca id=\"How_to_install_and_run_3\"\u003e\u003c/a\u003eHow to install and run\u003c/h2\u003e\n\u003col\u003e\n\u003cli class=\"has-line-data\" data-line-start=\"10\" data-line-end=\"11\"\u003e\n\u003cp class=\"has-line-data\" data-line-start=\"10\" data-line-end=\"11\"\u003esetup and activate a virtual environment\u003c/p\u003e\n\u003c/li\u003e\n\u003cli class=\"has-line-data\" data-line-start=\"4\" data-line-end=\"5\"\u003e\n\u003cp class=\"has-line-data\" data-line-start=\"4\" data-line-end=\"5\"\u003epip3 install -r requirements.txt\u003c/p\u003e\n\u003c/li\u003e\n\u003cli class=\"has-line-data\" data-line-start=\"5\" data-line-end=\"10\"\u003e\n\u003cp class=\"has-line-data\" data-line-start=\"5\" data-line-end=\"10\"\u003ein case you wish to use AWS S3 as a source file and/or a target file location for the conversion,\u003cbr\u003e\nset environment variables like:\u003cbr\u003e\n\u003ccode\u003eaws_access_key_id = \u0026lt;your AWS IAM access key id\u0026gt;\u003c/code\u003e\u003cbr\u003e\n\u003ccode\u003eaws_secret_access_key = \u0026lt;your AWS IAM secret access key value\u0026gt;\u003c/code\u003e\u003cbr\u003e\nPandas uses s3fs to integrate with AWS S3, please see \u003ca href=\"https://s3fs.readthedocs.io/en/latest/\"\u003ehttps://s3fs.readthedocs.io/en/latest/\u003c/a\u003e in case of any authentication issues.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli class=\"has-line-data\" data-line-start=\"11\" data-line-end=\"48\"\u003e\n\u003cp class=\"has-line-data\" data-line-start=\"11\" data-line-end=\"12\"\u003erun \u003ccode\u003epython __main__.py\u003c/code\u003e with the requiered arguments \u003ccode\u003e-sfp\u003c/code\u003e for source file path, \u003ccode\u003e-tfp\u003c/code\u003e for target file path, set like:\u003c/p\u003e\n\u003ch2 class=\"code-line\" data-line-start=12 data-line-end=13 \u003e\u003ca id=\"for_csv_to_parquet_conversion_12\"\u003e\u003c/a\u003efor csv to parquet conversion:\u003c/h2\u003e\n\u003ch4 class=\"code-line\" data-line-start=14 data-line-end=15 \u003e\u003ca id=\"local_csv_file_to_local_parquet_file_14\"\u003e\u003c/a\u003elocal csv file to local parquet file:\u003c/h4\u003e\n\u003cp class=\"has-line-data\" data-line-start=\"15\" data-line-end=\"17\"\u003e\u003ccode\u003e-sfp C:\\your local folder\\source file name.csv\u003c/code\u003e\u003cbr\u003e\n\u003ccode\u003e-tfp C:\\your local folder\\target file name.parquet\u003c/code\u003e\u003c/p\u003e\n\u003ch4 class=\"code-line\" data-line-start=18 data-line-end=19 \u003e\u003ca id=\"local_csv_file_to_s3_parquet_file_18\"\u003e\u003c/a\u003elocal csv file to s3 parquet file:\u003c/h4\u003e\n\u003cp class=\"has-line-data\" data-line-start=\"19\" data-line-end=\"21\"\u003e\u003ccode\u003e-sfp C:\\your local folder\\source file name.csv\u003c/code\u003e\u003cbr\u003e\n\u003ccode\u003e-tfp s3://your bucket name/your bucket \u0026quot;folder\u0026quot; prefix/target file name.parquet\u003c/code\u003e\u003c/p\u003e\n\u003ch4 class=\"code-line\" data-line-start=22 data-line-end=23 \u003e\u003ca id=\"s3_csv_file_to_local_parquet_file_22\"\u003e\u003c/a\u003es3 csv file to local parquet file:\u003c/h4\u003e\n\u003cp class=\"has-line-data\" data-line-start=\"23\" data-line-end=\"25\"\u003e\u003ccode\u003e-sfp s3://your bucket name/your bucket \u0026quot;folder\u0026quot; prefix/source.csv\u003c/code\u003e\u003cbr\u003e\n\u003ccode\u003e-tfp C:\\your local folder\\target.parquet\u003c/code\u003e\u003c/p\u003e\n\u003ch4 class=\"code-line\" data-line-start=26 data-line-end=27 \u003e\u003ca id=\"s3_csv_file_to_s3_parquet_file_26\"\u003e\u003c/a\u003es3 csv file to s3 parquet file:\u003c/h4\u003e\n\u003cp class=\"has-line-data\" data-line-start=\"27\" data-line-end=\"29\"\u003e\u003ccode\u003e-sfp s3://your bucket name/your bucket \u0026quot;folder\u0026quot; prefix/source file name.csv\u003c/code\u003e\u003cbr\u003e\n\u003ccode\u003e-tfp s3://your bucket name/your bucket \u0026quot;folder\u0026quot; prefix/target file name.parquet\u003c/code\u003e\u003c/p\u003e\n\u003ch2 class=\"code-line\" data-line-start=30 data-line-end=31 \u003e\u003ca id=\"for_parquet_to_csv_conversion_30\"\u003e\u003c/a\u003efor parquet to csv conversion:\u003c/h2\u003e\n\u003ch4 class=\"code-line\" data-line-start=32 data-line-end=33 \u003e\u003ca id=\"local_parquet_file_to_local_csv_file_32\"\u003e\u003c/a\u003elocal parquet file to local csv file:\u003c/h4\u003e\n\u003cp class=\"has-line-data\" data-line-start=\"33\" data-line-end=\"35\"\u003e\u003ccode\u003e-sfp C:\\your local folder\\source file name.parquet\u003c/code\u003e\u003cbr\u003e\n\u003ccode\u003e-tfp C:\\your local folder\\target file name.csv\u003c/code\u003e\u003c/p\u003e\n\u003ch4 class=\"code-line\" data-line-start=36 data-line-end=37 \u003e\u003ca id=\"local_parquet_file_to_s3_csv_file_36\"\u003e\u003c/a\u003elocal parquet file to s3 csv file:\u003c/h4\u003e\n\u003cp class=\"has-line-data\" data-line-start=\"37\" data-line-end=\"39\"\u003e\u003ccode\u003e-sfp C:\\your local folder\\source file name.parquet\u003c/code\u003e\u003cbr\u003e\n\u003ccode\u003e-tfp s3://your bucket name/your bucket \u0026quot;folder\u0026quot; prefix/target file name.csv\u003c/code\u003e\u003c/p\u003e\n\u003ch4 class=\"code-line\" data-line-start=40 data-line-end=41 \u003e\u003ca id=\"s3_parquet_file_to_local_csv_file_40\"\u003e\u003c/a\u003es3 parquet file to local csv file:\u003c/h4\u003e\n\u003cp class=\"has-line-data\" data-line-start=\"41\" data-line-end=\"43\"\u003e\u003ccode\u003e-sfp s3://your bucket name/your bucket \u0026quot;folder\u0026quot; prefix/source file name.parquet\u003c/code\u003e\u003cbr\u003e\n\u003ccode\u003e-tfp C:\\your local folder\\target.csv\u003c/code\u003e\u003c/p\u003e\n\u003ch4 class=\"code-line\" data-line-start=44 data-line-end=45 \u003e\u003ca id=\"s3_parquet_file_to_s3_csv_file_44\"\u003e\u003c/a\u003es3 parquet file to s3 csv file:\u003c/h4\u003e\n\u003cp class=\"has-line-data\" data-line-start=\"45\" data-line-end=\"47\"\u003e\u003ccode\u003e-sfp s3://your bucket name/your bucket \u0026quot;folder\u0026quot; prefix/source file name.parquet\u003c/code\u003e\u003cbr\u003e\n\u003ccode\u003e-tfp s3://your bucket name/your bucket \u0026quot;folder\u0026quot; prefix/target file name.csv\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli class=\"has-line-data\" data-line-start=\"48\" data-line-end=\"52\"\u003e\n\u003cp class=\"has-line-data\" data-line-start=\"48\" data-line-end=\"51\"\u003eyou can add these optional arguments:\u003cbr\u003e\n\u003ccode\u003e-cols\u003c/code\u003e argument is used to define a subset of columns from the source file, meaning that only the columns passed as a \u003ccode\u003elist\u003c/code\u003e to this argument will get loaded and converted, example: \u003ccode\u003e[\u0026quot;column_name_1\u0026quot;, \u0026quot;column_name_2\u0026quot;]\u003c/code\u003e\u003cbr\u003e\n\u003ccode\u003e-comp\u003c/code\u003e argument is used for overriding the default parquet compression type (\u003ccode\u003esnappy\u003c/code\u003e) in case of converting from a csv to parquet file, example: \u003ccode\u003egzip\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch2 class=\"code-line\" data-line-start=52 data-line-end=53 \u003e\u003ca id=\"How_to_verify_a_parquet_file_52\"\u003e\u003c/a\u003eHow to verify a parquet file:\u003c/h2\u003e\n\u003cul\u003e\u003cli\u003ehttp://parquet-viewer-online.com/\u003c/li\u003e\u003c/ul\u003e\n\u003ch2 class=\"code-line\" data-line-start=55 data-line-end=56 \u003e\u003ca id=\"Further_documentation_55\"\u003e\u003c/a\u003eFurther documentation\u003c/h2\u003e\n\u003cul\u003e\n  \u003cli\u003ehttps://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html\u003c/li\u003e\n  \u003cli\u003ehttps://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html\u003c/li\u003e\n  \u003cli\u003ehttps://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html\u003c/li\u003e\n  \u003cli\u003ehttps://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_parquet.html\u003c/li\u003e\n  \u003cli\u003ehttps://s3fs.readthedocs.io/en/latest/\u003c/li\u003e\n\u003c/ul\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatahappy1%2Fcsv_to_parquet_converter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatahappy1%2Fcsv_to_parquet_converter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatahappy1%2Fcsv_to_parquet_converter/lists"}