{"id":16021825,"url":"https://github.com/davidanthoff/csv-comparison","last_synced_at":"2025-04-05T03:43:50.510Z","repository":{"id":74502420,"uuid":"153319671","full_name":"davidanthoff/csv-comparison","owner":"davidanthoff","description":null,"archived":false,"fork":false,"pushed_at":"2020-07-04T17:28:15.000Z","size":143,"stargazers_count":13,"open_issues_count":1,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-02-10T12:12:46.424Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://www.queryverse.org/benchmarks/","language":"Julia","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davidanthoff.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-16T16:39:35.000Z","updated_at":"2023-10-13T12:30:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"29e0f13c-66ce-4a04-be49-d8536ac44fec","html_url":"https://github.com/davidanthoff/csv-comparison","commit_stats":{"total_commits":119,"total_committers":2,"mean_commits":59.5,"dds":0.008403361344537785,"last_synced_commit":"44e7c665e83d9832bb879c70d9068bbb5a870539"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidanthoff%2Fcsv-comparison","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidanthoff%2Fcsv-comparison/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidanthoff%2Fcsv-comparison/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidanthoff%2Fcsv-comparison/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davidanthoff","download_url":"https://codeload.github.com/davidanthoff/csv-comparison/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247284918,"owners_count":20913691,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-08T18:06:06.364Z","updated_at":"2025-04-05T03:43:50.491Z","avatar_url":"https://github.com/davidanthoff.png","language":"Julia","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Comparing various CSV reading packages\n\n## Overview\n\nA CSV reader benchmark suite. Results are regularly published on the [Queryverse benchmark](https://www.queryverse.org/benchmarks/) page.\n\n## Setup\n\nYou need Julia 1.1.0, Julia 0.6.4, Python 3.X and R installed.\n\nFor Julia 0.6.4, you need to install the CSV.jl and TextParse.jl package.\n\nFor Julia 1.1.0, activate the project in the root folder and instantiate it.\n\nFor Python, install pandas and pyarrow.\n\nFor R, install data.table and readr.\n\nOn Windows, run `deps/build.jl` to download `EmptyStandbyList.exe` executable.\n\n## Generating test data\n\nRun `src/writedata.jl` to generate the input data.\n\nWARNING: The `writedata.jl` script generates _a lot_ of data, currently about 150 GB. Make sure you have enough disc space!\n\nYou can configure the details of the write process by adding a file `src/local_writeconfig.jl`, and then adding any of the following lines to configure various aspects:\n\n```julia\n# Configure what row number cases you want to generate\nns = [100, 10_000, 1_000_000]\n\n# Configure what column number cases you want to generate\ncs = [20, 200]\n\n# Share of missing values in columns with missing data\nmissing_share = 0.5\n\n# Configure how many rows one very large test file should have\nlargefile_rown = 150_000_000\n\n# Configure what colun types you want to generate\nuniform_types = [\n    :float64,      # Float64 with all digits\n    :float64short, # Float64 with only 4 digist after the decimal point\n    :int64,        # Int64\n    :datetime,     # DateTime\n    :string,       # Random string\n    :stringcat,    # Randomly pick one out of 5 strings for each cell\n    :stringescaped # A string that includes an escaped quote char\n]\n```\n\n## Running the benchmarks\n\nRun the `src/main.jl` script to run the benchmarks. You need to run the script in elevated mode, so on Windows as an administrator, on Mac and Linux with `sudo`.\n\nYou can configure the details of the benchmarking process by adding a file `src/local_config.jl`, and then adding any of the following lines to configure various aspects:\n\n```julia\n# Configure which row cases you want to benchmark\nns = [100, 10_000, 1_000_000]\n\n# Configure which column cases you want to benchmark\ncs = [20, 200]\n\n# Configure how many samples you want to take per benchmarking case\nsamples = 5\n\n# Configure which packages to run\ntests_to_run = [\n    :textparse,       # TextParse.jl\n    :csvfiles,        # CSVFiles.jl\n    :textparse06,     # TextParse.jl on Julia 0.6\n    :csv,             # CSV.jl\n    :csv06,           # CSV.jl on Julia 0.6\n    # :csvreader,     # CSVReader.jl\n    # :tablereader,   # TableReader.jl\n    :pandas,          # Pandas.jl\n    :rfreads,         # R data.table single threaded\n    :rfreadp,         # R data.table parallel\n    :rreadr,          # R readr\n    :dataframes,      # DataFrames.jl\n    :pythonpandas,    # Python pandas\n    :pythonarrows,    # Python pyarrow single threaded\n    :pythonarrowp     # Python pyarrow parallel\n]\n\n# Path to Julia 0.6 binary\njl06bin = \"julia-0.6\"\n\n# Path to Julia 1.1 binary\njlbin = \"julia\"\n\n# Path to Rscript binary\nrbin = \"Rscript\"\n\n# Path to Python binary\npythonbin = \"python3\"\n```\n\n## Contributing\n\nIssues and PRs with contributions are most welcome!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidanthoff%2Fcsv-comparison","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidanthoff%2Fcsv-comparison","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidanthoff%2Fcsv-comparison/lists"}