{"id":31822842,"url":"https://github.com/jagedn/nf-parquet-benchmark","last_synced_at":"2026-02-15T22:32:00.503Z","repository":{"id":313538115,"uuid":"1051771104","full_name":"jagedn/nf-parquet-benchmark","owner":"jagedn","description":null,"archived":false,"fork":false,"pushed_at":"2025-09-07T07:56:53.000Z","size":54973,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-11T14:17:53.890Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Nextflow","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jagedn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-06T17:28:20.000Z","updated_at":"2025-09-22T13:22:51.000Z","dependencies_parsed_at":"2025-09-06T19:37:07.045Z","dependency_job_id":null,"html_url":"https://github.com/jagedn/nf-parquet-benchmark","commit_stats":null,"previous_names":["jagedn/nf-parquet-benchmark"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jagedn/nf-parquet-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jagedn%2Fnf-parquet-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jagedn%2Fnf-parquet-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jagedn%2Fnf-parquet-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jagedn%2Fnf-parquet-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jagedn","download_url":"https://codeload.github.com/jagedn/nf-parquet-benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jagedn%2Fnf-parquet-benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29490882,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-15T19:29:10.908Z","status":"ssl_error","status_checked_at":"2026-02-15T19:29:10.419Z","response_time":118,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-11T14:17:49.677Z","updated_at":"2026-02-15T22:32:00.467Z","avatar_url":"https://github.com/jagedn.png","language":"Nextflow","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CSV vs. Parquet Nextflow Plugin Comparison\n\nThis project aims to compare the performance of Nextflow's CSV and Parquet plugins by running a series of data processing pipelines on two different datasets. The goal is to highlight the efficiency gains, particularly in terms of storage and processing time, that can be achieved by using the **Parquet format**.\n\n-----\n\n## Datasets\n\nThe project uses two example datasets to demonstrate the performance differences:\n\n* **`gene_rpkm_matrix_HBC_day0_day7`**: This is a smaller dataset used for a baseline comparison.\n* **`MRCE_5percentFDR`**: This is a larger, more realistic dataset. Its CSV version is huge (110 MB) and has to be compressed for efficient storage. The compressed CSV file is 26 MB, while the equivalent Parquet file is significantly smaller at just 17 MB. This dataset clearly illustrates the storage benefits of the Parquet format.\n\n-----\n\n## Project Structure\n\nThe project is structured with two main types of pipelines:\n\n1.  **Conversion Pipelines**: These pipelines, named **`convert`** and **`convertMrce`**, are used to convert the raw CSV data into the Parquet format. This is the initial step to prepare the data for the comparison.\n\n2.  **Search Pipelines**: These pipelines perform various search and data manipulation tasks on both the CSV and Parquet versions of the datasets. By running the same operations on both data formats, we can directly compare the execution times and resources consumed, demonstrating the performance advantages of Parquet.\n\n-----\n\n## Running the Project\n\nTo run this project, you'll need a working Nextflow installation and the necessary plugins for CSV and Parquet. Follow these steps:\n\n1.  **Clone the Repository:**\n\n    ```bash\n    git clone https://github.com/jagedn/nf-parquet-benchmark \n    cd nf-parquet-benchmark\n    cd data\n    unzip MR*.zip\n    cd .. \n    ```\n\n2.  **Run the Conversion Pipeline:** First, convert the CSV data to Parquet.\n\n    ```bash\n    nextflow run convert.nf\n    nextflow run convertMrce.nf\n    ```\n\n3.  **Run the Search Pipelines:** Now, run the search pipelines on both the CSV and Parquet data to perform the comparison.\n\n    ```bash\n    time nextflow run search_mcre_csv.nf\n    time nextflow run search_mcre_parquet.nf\n    time nextflow run search_mcre_projection.nf\n    ```\n\nAfter the runs complete, you can analyze the Nextflow reports to compare the execution times, CPU usage, and memory consumption for the different pipelines. The results should clearly show the improved performance when processing the Parquet files.\n\nFor example, these are the results in my laptop\n\n   ```bash\n    time nextflow run search_mcre_csv.nf\n   \n     N E X T F L O W   ~  version 25.07.0-edge\n    \n    Launching `search_mcre_csv.nf` [peaceful_ptolemy] DSL2 - revision: 4d70b72f7c\n    \n    3384\n    \n    real\t0m7,888s\n    user\t0m28,881s\n    sys\t0m1,126s\n    \n    time nextflow run search_mcre_parquet.nf\n\n     N E X T F L O W   ~  version 25.07.0-edge\n    \n    Launching `search_mcre_parquet.nf` [stupefied_feynman] DSL2 - revision: 4f9460e097\n    \n    3384\n    \n    real\t0m5,714s\n    user\t0m15,665s\n    sys\t0m0,802s\n\n    time nextflow run search_mcre_projection.nf\n    \n     N E X T F L O W   ~  version 25.07.0-edge\n    \n    Launching `search_mcre_projection.nf` [backstabbing_swirles] DSL2 - revision: 8704ffa7f3\n    \n    3384\n    \n    real\t0m5,655s\n    user\t0m12,780s\n    sys\t0m0,705s\n   ```\n\nAs you can see, CSV format took more than double of time to filter the data that optimized projection parquet \n\n(Readme generated at 99% by an AI because I'm a bit lazzy)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjagedn%2Fnf-parquet-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjagedn%2Fnf-parquet-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjagedn%2Fnf-parquet-benchmark/lists"}