{"id":20749125,"url":"https://github.com/g-research/parquetsharp","last_synced_at":"2025-05-14T23:02:19.432Z","repository":{"id":37735517,"uuid":"142328410","full_name":"G-Research/ParquetSharp","owner":"G-Research","description":"ParquetSharp is a .NET library for reading and writing Apache Parquet files.","archived":false,"fork":false,"pushed_at":"2025-04-28T08:01:09.000Z","size":1823,"stargazers_count":206,"open_issues_count":15,"forks_count":55,"subscribers_count":33,"default_branch":"master","last_synced_at":"2025-05-14T23:01:50.259Z","etag":null,"topics":["apache-arrow","apache-parquet","big-data","columnar-storage","csharp","dotnet","parquet"],"latest_commit_sha":null,"homepage":"https://g-research.github.io/ParquetSharp/","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/G-Research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-07-25T16:48:59.000Z","updated_at":"2025-05-07T19:17:44.000Z","dependencies_parsed_at":"2023-10-16T12:01:45.266Z","dependency_job_id":"d39f804e-7467-4896-83fa-e1424d66f5a8","html_url":"https://github.com/G-Research/ParquetSharp","commit_stats":{"total_commits":201,"total_committers":22,"mean_commits":9.136363636363637,"dds":0.5472636815920398,"last_synced_commit":"f5148e3c681e8c24fedc8dca5f742d9ae2456cea"},"previous_names":[],"tags_count":75,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/G-Research%2FParquetSharp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/G-Research%2FParquetSharp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/G-Research%2FParquetSharp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/G-Research%2FParquetSharp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/G-Research","download_url":"https://codeload.github.com/G-Research/ParquetSharp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254243355,"owners_count":22038045,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-arrow","apache-parquet","big-data","columnar-storage","csharp","dotnet","parquet"],"created_at":"2024-11-17T08:21:10.032Z","updated_at":"2025-05-14T23:02:19.397Z","avatar_url":"https://github.com/G-Research.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Main logo](images/logo/svg/ParquetSharp_SignatureLogo_RGB-Black.svg)\n\n## Introduction\n\n**ParquetSharp** is a cross-platform .NET library for reading and writing Apache [Parquet][1] files.\n\nParquetSharp is implemented in C# as a [PInvoke][2] wrapper around [Apache Parquet C++][3] to provide high performance and compatibility. Check out [ParquetSharp.DataFrame][4] if you need a convenient integration with the .NET [DataFrames][5].\n\nSupported platforms:\n\n| Chip  | Linux    | Windows  | macOS    |\n| :---- | :------: | :------: | :------: |\n| x64   | \u0026#x2714; | \u0026#x2714; | \u0026#x2714; |\n| arm64 | \u0026#x2714; |          | \u0026#x2714; |\n\n|                       | Status                                                                                                                                                                                                                         |\n| --------------------: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |\n| **Release Nuget**     | [![NuGet latest release](https://img.shields.io/nuget/v/ParquetSharp.svg)](https://www.nuget.org/packages/ParquetSharp)                                                                                                        |\n| **Pre-Release Nuget** | [![NuGet latest pre-release](https://img.shields.io/nuget/vpre/ParquetSharp.svg)](https://www.nuget.org/packages/ParquetSharp/absoluteLatest)                                                                                  |\n| **CI Build**          | [![CI Status](https://github.com/G-Research/ParquetSharp/actions/workflows/ci.yml/badge.svg?branch=master\u0026event=push)](https://github.com/G-Research/ParquetSharp/actions/workflows/ci.yml?query=branch%3Amaster+event%3Apush) |\n\n## Why use Parquet?\n\n**Apache Parquet** is an [open source][6], column-oriented data file format designed for efficient data storage and retrieval. It provides efficient data compression and encoding schemes with enhanced performance to handle complex data in bulk. Relative to CSV files, Parquet executes queries **34x faster** while taking up **87% less space**. [Source][7]\n\n[1]: https://parquet.apache.org/\n[2]: https://docs.microsoft.com/en-us/cpp/dotnet/how-to-call-native-dlls-from-managed-code-using-pinvoke\n[3]: https://github.com/apache/arrow\n[4]: https://github.com/G-Research/ParquetSharp.DataFrame\n[5]: https://docs.microsoft.com/en-us/dotnet/api/microsoft.data.analysis.dataframe\n[6]: https://github.com/apache/parquet-format\n[7]: https://towardsdatascience.com/demystifying-the-parquet-file-format-13adb0206705\n\n## Quickstart\n\nThe following examples show how to write and then read a Parquet file with three columns representing a timeseries of object-value pairs.\nThese use the low-level API, which is the recommended API for working with native .NET types and closely maps to the API of Apache Parquet C++.\nFor reading and writing data in the [Apache Arrow](https://arrow.apache.org/) format, an [Arrow-based API](https://g-research.github.io/ParquetSharp/guides/Arrow.html) is also provided.\n\n### 1. Initialize a new project\n\nFirst, let's create a new console application:\n\n```bash\ndotnet new console -n ParquetExample\ncd ParquetExample\n```\n\nIn your project directory, you'll find a `Program.cs` file that we'll use to write a Parquet file, and then read it back.\n\n### 2. Install ParquetSharp\n\nParquetSharp is available as a [NuGet package](https://www.nuget.org/packages/ParquetSharp/). You can install it using the following command:\n\n```bash\ndotnet add package ParquetSharp\n```\n\n### 3. Write a Parquet File\n\nThis example shows how to write a Parquet file with three columns: `Timestamp`, `ObjectId`, and `Value`.\n\nUpdate your `Program.cs` with the following code:\n\n```csharp\nusing System;\nusing ParquetSharp;\n\nclass Program\n{\n    static void Main()\n    {\n        var timestamps = new DateTime[] { DateTime.Now, DateTime.Now.AddMinutes(1) };\n        var objectIds = new int[] { 1, 2 };\n        var values = new float[] { 1.23f, 4.56f };\n\n        var columns = new Column[]\n        {\n            new Column\u003cDateTime\u003e(\"Timestamp\"),\n            new Column\u003cint\u003e(\"ObjectId\"),\n            new Column\u003cfloat\u003e(\"Value\")\n        };\n\n        using var file = new ParquetFileWriter(\"float_timeseries.parquet\", columns);\n        using var rowGroup = file.AppendRowGroup();\n\n        using (var timestampWriter = rowGroup.NextColumn().LogicalWriter\u003cDateTime\u003e())\n        {\n            timestampWriter.WriteBatch(timestamps);\n        }\n        using (var objectIdWriter = rowGroup.NextColumn().LogicalWriter\u003cint\u003e())\n        {\n            objectIdWriter.WriteBatch(objectIds);\n        }\n        using (var valueWriter = rowGroup.NextColumn().LogicalWriter\u003cfloat\u003e())\n        {\n            valueWriter.WriteBatch(values);\n        }\n\n        file.Close();\n        Console.WriteLine(\"Parquet file written successfully!\");\n    }\n}\n```\n\nYou can execute it with:\n\n```bash\ndotnet run\n```\n\n### 4. Read a Parquet File\n\nAfter writing the Parquet file, we can read it back by updating the `Program.cs` file with the following code:\n\n```csharp\nusing System;\nusing ParquetSharp;\n\nclass Program\n{\n    static void Main()\n    {\n        using var file = new ParquetFileReader(\"float_timeseries.parquet\");\n\n        for (int rowGroup = 0; rowGroup \u003c file.FileMetaData.NumRowGroups; ++rowGroup)\n        {\n            using var rowGroupReader = file.RowGroup(rowGroup);\n            var groupNumRows = checked((int)rowGroupReader.MetaData.NumRows);\n\n            var groupTimestamps = rowGroupReader.Column(0).LogicalReader\u003cDateTime\u003e().ReadAll(groupNumRows);\n            var groupObjectIds = rowGroupReader.Column(1).LogicalReader\u003cint\u003e().ReadAll(groupNumRows);\n            var groupValues = rowGroupReader.Column(2).LogicalReader\u003cfloat\u003e().ReadAll(groupNumRows);\n\n            Console.WriteLine(\"Read Parquet file:\");\n            for (int i = 0; i \u003c groupNumRows; ++i)\n            {\n                Console.WriteLine($\"Timestamp: {groupTimestamps[i]}, ObjectId: {groupObjectIds[i]}, Value: {groupValues[i]}\");\n            }\n        }\n\n        file.Close();\n    }\n}\n```\n\nOnce again, run the program with:\n\n```bash\ndotnet run\n```\n\nThis should give you an output similar to:\n```\nRead Parquet file:\nTimestamp: 2025-01-25 10:15:25 AM, ObjectId: 1, Value: 1.23\nTimestamp: 2025-01-25 10:16:25 AM, ObjectId: 2, Value: 4.56\n```\n\n## Documentation\n\nFor more detailed guides on how to use ParquetSharp and an API reference, please see [the documentation](https://g-research.github.io/ParquetSharp/).\n\n## Rationale\n\nWe desired a Parquet implementation with the following properties:\n- Cross platform (originally Windows and Linux - but now also macOS).\n- Callable from .NET Core.\n- Good performance.\n- Well maintained.\n- Close to official Parquet reference implementations.\n\nNot finding an existing solution meeting these requirements, we decided to implement a .NET wrapper around apache-parquet-cpp (now part of Apache Arrow) starting at version 1.4.0. The library tries to stick closely to the existing C++ API, although it does provide higher level APIs to facilitate its usage from .NET. The user should always be able to access the lower-level API.\n\n## Performance\n\nThe following benchmarks can be reproduced by running `ParquetSharp.Benchmark.csproj`. The relative performance of ParquetSharp 10.0.1 is compared to [Parquet.NET](https://github.com/aloneguid/parquet-dotnet) 4.6.2, an alternative open-source .NET library that is fully managed. The Decimal tests focus purely on handling the C# `decimal` type, while the TimeSeries tests benchmark three columns of the types `{int, DateTime, float}`. Results are from a Ryzen 5900X on Linux 6.2.7 using the dotnet 6.0.14 runtime.\n\nIf performance is a concern for you, we recommend benchmarking your own workloads and testing different encodings and compression methods. For example, disabling dictionary encoding for floating point columns can often significantly improve performance.\n\n|              | Decimal (Read) | Decimal (Write) | TimeSeries (Read) | TimeSeries (Write) |\n| -----------: | :------------: | :-------------: | :---------------: | :----------------: |\n| Parquet.NET  | 1.0x           | 1.0x            | 1.0x              | 1.0x               |\n| ParquetSharp | 4.0x Faster    | 3.0x Faster     | 2.8x Faster       | 1.5x Faster        |\n\n## Known Limitations\n\nBecause this library is a thin wrapper around the Parquet C++ library, misuse can cause native memory access violations.\n\nTypically this can arise when attempting to access an instance whose owner has been disposed. Because some objects and properties are exposed by Parquet C++ via regular pointers (instead of consistently using `std::shared_ptr`), dereferencing these after the owner class instance has been destructed will lead to an invalid pointer access.\n\nAs only 64-bit runtimes are available, ParquetSharp cannot be referenced by a 32-bit project.  For example, using the library from F# Interactive requires running `fsiAnyCpu.exe` rather than `fsi.exe`.\n\n## Building\n\n### Dev Container\n\nParquetSharp can be built and tested within a [dev container](https://containers.dev). This is a probably the easiest way to get started, as all the C++ dependencies are prebuilt into the container image.\n\n#### GitHub Codespaces\n\nIf you have a GitHub account, you can simply open ParquetSharp in a new GitHub Codespace by clicking on the green \"Code\" button at the top of this page.\n\nChoose the \"unspecified\" CMake kit when prompted and let the C++ configuration run. Once done, you can build the C++ code via the \"Build\" button in the status bar at the bottom.\n\nYou can then build the C# code by right-clicking the ParquetSharp solution in the Solution Explorer on the left and choosing \"Build\". The Test Explorer will then get populated with all the C# tests too.\n\n#### Visual Studio Code\n\nIf you want to work locally in [Visual Studio Code](https://code.visualstudio.com), all you need is to have [Docker](https://docs.docker.com/get-docker/) and the [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) installed.\n\nSimply open up your copy of ParquetSharp in VS Code and click \"Reopen in container\" when prompted. Once the project has been opened, you can follow the GitHub Codespaces instructions above.\n\n\u003cdetails\u003e\n\u003csummary\u003ePodman and SELinux workarounds\u003c/summary\u003e\nUsing the dev container on a Linux system with podman and SELinux requires some workarounds.\n\nYou'll need to edit `.devcontainer/devcontainer.json` and add the following lines:\n\n```json\n  \"remoteUser\": \"root\",\n  \"containerUser\": \"root\",\n  \"workspaceMount\": \"\",\n  \"runArgs\": [\"--volume=${localWorkspaceFolder}:/workspaces/${localWorkspaceFolderBasename}:Z\"],\n  \"containerEnv\": { \"VCPKG_DEFAULT_BINARY_CACHE\": \"/home/vscode/.cache/vcpkg/archives\" }\n```\n\nThis configures the container to run as the root user,\nbecause when you run podman as a non-root user your user id is mapped\nto root in the container, and files in the workspace folder will be owned by root.\n\nThe workspace mount command is also modified to add the `:Z` suffix,\nwhich tells podman to relabel the volume to allow access to it from within the container.\n\nFinally, setting the `VCPKG_DEFAULT_BINARY_CACHE` environment variable\nmakes the root user in the container use the vcpkg cache of the vscode user.\n\u003c/details\u003e\n\n#### CLI\n\nIf the CLI is how you roll, then you can install the [Dev Container CLI](https://github.com/devcontainers/cli) tool and issue the following command in the your copy of ParquetSharp to get up and running:\n\n```bash\ndevcontainer up\n```\n\nBuild the C++ code and run the C# tests with:\n\n```bash\ndevcontainer exec ./build_unix.sh\ndevcontainer exec dotnet test csharp.test\n```\n\n#### Code Formatting\n\nWhen formatting for the first time, you'll need to restore the formatter tool:\n\n```bash\ndotnet tool restore\n```\n\nThen, you can format any time with the following command which is also executed by the CI format checker:\n\n```bash\ndotnet jb cleanupcode \"csharp\" \"csharp.test\" \"csharp.benchmark\" --profile=\"Built-in: Reformat Code\" --settings=\"ParquetSharp.DotSettings\" --verbosity=WARN\n```\n\n### Native\n\nBuilding ParquetSharp natively requires the following dependencies:\n- A modern C++ compiler toolchain\n- .NET SDK 8.0\n- Apache Arrow (19.0.1)\n\nFor building Arrow (including Parquet) and its dependencies, we recommend using Microsoft's [vcpkg](https://vcpkg.io).\nThe build scripts will use an existing vcpkg installation if either of the `VCPKG_INSTALLATION_ROOT` or `VCPKG_ROOT` environment variables are defined, otherwise vcpkg will be downloaded into the build directory.\n\n#### Windows\n\nBuilding ParquetSharp on Windows requires Visual Studio 2022 (17.0 or higher).\n\nOpen a Visual Studio Developer PowerShell and run the following commands to build the C++ code and run the C# tests:\n\n```pwsh\nbuild_windows.ps1\ndotnet test csharp.test\n```\n\n`cmake` must be available in the PATH for the build script to succeed.\n\n#### Unix\n\nBuild the C++ code and run the C# tests with:\n\n```bash\n./build_unix.sh\ndotnet test csharp.test\n```\n\n### Known Issues\nAn issue that may occur when building ParquetSharp locally using `build_windows.ps1` is Visual Studio not being detected by CMake:\n```pwsh\nCMake Error at CMakeLists.txt:2 (project):   Generator\n\n  Visual Studio 17 2022\n\ncould not find any instance of Visual Studio.\n```\nThis is a known issue: [(1)](https://stackoverflow.com/questions/60068168/cmake-problem-could-not-find-any-instance-of-visual-studio) [(2)](https://stackoverflow.com/questions/59953960/cmake-and-vs-2017-could-not-find-any-instance-of-visual-studio). It can be solved by ensuring that all required Visual Studio Build Tools are properly installed and that the relevant version of Visual Studio is available, and finally rebooting the machine. Another potential solution is to reinstall Visual Studio with the required build tools.\n\nWhen building, you may come across the following problem with `Microsoft.Cpp.Default.props`: \n```pwsh\nerror MSB4019: The imported project \"C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\MSBuild\\Microsoft\\V\nC\\v170\\Microsoft.Cpp.Default.props\" was not found. Confirm that the expression in the Import declaration \"C:\\Program Fi\nles (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\MSBuild\\Microsoft\\VC\\v170\\\\Microsoft.Cpp.Default.props\" is correct, a\nnd that the file exists on disk.\n```\nTo resolve this, make sure that the \"Desktop development with C++\" option is selected when installing Visual Studio Build Tools. If installation is successful, the required directory and files should be present. \n\nAnother common issue is the following:\n```pwsh\nCMake Error at CMakeLists.txt:2 (project):\n  The CMAKE_C_COMPILER:\n\n    C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe\n\n  is not a full path to an existing compiler tool.\n\nCMake Error at CMakeLists.txt:2 (project):\n  The CMAKE_CXX_COMPILER:\n\n    C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe\n\n  is not a full path to an existing compiler tool.\n```\nThis is also related to installed Visual Studio modules. Make sure to install \"C++/CLI support for build tools\" from the list of optional components for Desktop development with C++ for the relevant version of Visual Studio.\n\nFor any other build issues, please [open a new discussion](https://github.com/G-Research/ParquetSharp/discussions).\n\n## Contributing\n\nWe welcome new contributors! We will happily receive PRs for bug fixes or small changes. If you're contemplating something larger please get in touch first by opening a GitHub Issue describing the problem and how you propose to solve it.\n\n## Security\n\nPlease see our [security policy](https://github.com/G-Research/ParquetSharp/blob/master/SECURITY.md) for details on reporting security vulnerabilities.\n\n## License\n\nCopyright 2018-2023 G-Research\n\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use these files except in compliance with the License.\nYou may obtain a copy of the License at\n\nhttp://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fg-research%2Fparquetsharp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fg-research%2Fparquetsharp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fg-research%2Fparquetsharp/lists"}