{"id":34070370,"url":"https://github.com/mklarqvist/libflagstats","last_synced_at":"2026-04-01T22:09:27.941Z","repository":{"id":62580350,"uuid":"183177985","full_name":"mklarqvist/libflagstats","owner":"mklarqvist","description":"Efficient C functions to compute the summary statistics (flagstats) for sequencing read sets.","archived":false,"fork":false,"pushed_at":"2019-12-16T02:00:13.000Z","size":335,"stargazers_count":15,"open_issues_count":0,"forks_count":3,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-12-16T07:08:16.768Z","etag":null,"topics":["avx2","avx512","bioinformatics","genetics","popcnt","popcount","pospopcnt","simd","sse4"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mklarqvist.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-04-24T07:57:38.000Z","updated_at":"2022-04-11T12:45:57.000Z","dependencies_parsed_at":"2022-11-03T21:02:44.766Z","dependency_job_id":null,"html_url":"https://github.com/mklarqvist/libflagstats","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mklarqvist/libflagstats","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mklarqvist%2Flibflagstats","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mklarqvist%2Flibflagstats/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mklarqvist%2Flibflagstats/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mklarqvist%2Flibflagstats/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mklarqvist","download_url":"https://codeload.github.com/mklarqvist/libflagstats/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mklarqvist%2Flibflagstats/sbom","scorecard":{"id":652597,"data":{"date":"2025-08-11","repo":{"name":"github.com/mklarqvist/libflagstats","commit":"93f68238b662add883ead01b717ca176ec9d07f4"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.4,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":3,"reason":"Found 6/16 approved changesets -- score normalized to 3","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 22 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-21T13:51:21.638Z","repository_id":62580350,"created_at":"2025-08-21T13:51:21.638Z","updated_at":"2025-08-21T13:51:21.638Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31018566,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-27T03:51:26.850Z","status":"ssl_error","status_checked_at":"2026-03-27T03:51:09.693Z","response_time":164,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avx2","avx512","bioinformatics","genetics","popcnt","popcount","pospopcnt","simd","sse4"],"created_at":"2025-12-14T07:07:45.830Z","updated_at":"2026-03-27T04:23:41.251Z","avatar_url":"https://github.com/mklarqvist.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# libflagstats\n\n[![PyPI version](https://badge.fury.io/py/pyflagstats.svg)](https://badge.fury.io/py/pyflagstats)\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mklarqvist/libflagstats/blob/master/python/pyflagstats.ipynb)\n\n\nThese functions compute the summary statistics for the SAM FLAG field (flagstat)\nusing fast SIMD instructions. They are based on the\n[positional-popcount](https://github.com/mklarqvist/positional-popcount)\n(`pospopcnt`) subroutines implemented in\n[libalgebra](https://github.com/mklarqvist/libalgebra) and assumes that the data\nis available as contiguous streams (e.g. [column projection](https://en.wikipedia.org/wiki/Projection_(relational_algebra))). The heavily\nbranched and bit-dependent [SAMtools](https://github.com/samtools/samtools/) code can be rewritten using a mask-select\npropagate-carry approach that feeds into bit-transposed Harley-Seal-based carry-save adder\nnetworks.\n\nThe positional population count is described in the paper:\n* [Efficient Computation of Positional Population Counts Using SIMD Instructions](https://arxiv.org/abs/1911.02696) by Marcus D. R. Klarqvist and Wojciech Muła and Daniel Lemire\n\n## Speedup\n\nThis benchmark shows the speedup of the pospopcnt algorithms compared to\n[samtools](https://github.com/samtools/samtools) using a human HiSeqX readset\nwith 824,541,892 reads. See [Results](#results) for additional information. On\nthis readset, the pospopcnt-based functions are \u003e3802-fold faster compared to\nBAM and 402.6-fold faster compared to CRAM. Note that the BAM format does not\nhave column projection capabilities. Around 80% of the CPU time is spent\nretrieving data from disk for the `pospopcnt`-functions. In this example, we\ncompress data blocks of 512k records (1 MB) using LZ4. \n\n| Approach        | Time       | Speedup |\n|-----------------|------------|---------|\n| Samtools – BAM  | 30m 50.06s | 1       |\n| Samtools – CRAM | 4m 50.68s  | 6.36    |\n| libflagstats-LZ9       | 0.72s      | 2569.53 |\n| libflagstats-raw       | 0.48s      | 3802.90 |\n\nWe also directly compared the potential speed of the naive flagstat subroutine\nin samtools againt these functions if samtools would be rewritten with efficient\ncolumn projection and the compression method changed to LZ4. In this context,\nthese functions are still 6.58-fold faster.\n\n| Approach             | Time   | Speedup |\n|----------------------|--------|---------|\n| samtools-rewrite+LZ4 | 4.74 s | 1       |\n| libflagstats            | 0.72s  | 6.58    |\n\n### Usage\n\nFor Linux/Mac: Compile the test suite with: `make`. LZ4 and Zstd needs to be\ninstalled on the target machine or pass the `LZ4_PATH` and `ZSTD_PATH` flags to\nmake for non-standard locations.\n\nFirst, import 2-byte FLAG words from a SAM file:\n```bash\nsamtools view NA12878D_HiSeqX_R12_GRCh37.bam | cut -f 2 | ./utility \u003e NA12878D_HiSeqX_R12_GRCh37_flags.bin\n```\n\nCompress the binary file using 512k blocks:\n```bash\nfor i in {1..10}; do ./bench compress -i NA12878D_HiSeqX_R12_GRCh37_flags.bin -l -c $i; done\nfor i in {1..10}; do ./bench compress -i NA12878D_HiSeqX_R12_GRCh37_flags.bin -l -f -c $i; done\nfor i in {1..20}; do ./bench compress -i NA12878D_HiSeqX_R12_GRCh37_flags.bin -z -c $i; done\n```\n\nEvaluate the flagstat subroutines by first decompressing the file twice while\nclearing cache and then computing samtools and pospopcnt.\n```bash\nfor i in {1..10}; do ./bench decompress -i NA12878D_HiSeqX_R12_GRCh37_flags.bin_HC_c${i}.lz4; done\nfor i in {1..10}; do ./bench decompress -i NA12878D_HiSeqX_R12_GRCh37_flags.bin_fast_a${i}.lz4; done\nfor i in {1..20}; do ./bench decompress -i NA12878D_HiSeqX_R12_GRCh37_flags.bin_c${i}.zst; done\n```\n\n### History\n\nThese functions were developed for [pil](https://github.com/mklarqvist/pil).\n\n## Problem statement\n\nThe FLAG field in the [SAM interchange\nformat](https://github.com/samtools/hts-specs) is defined as the union of\n[1-hot](https://en.wikipedia.org/wiki/One-hot) encoded states for a given read.\nFor example, the following three states evaluating to true\n\n```\n00000001: read paired\n01000000: first in pair\n00001000: mate unmapped\n--------\n01001001: Decimal (73)\n```\n\nare stored in a packed 16-bit value (only the LSB is shown here). There are 12\nstates described in the SAM format:\n\n| One-hot           | Description                               |\n|-------------------|-------------------------------------------|\n| 00000000 00000001 | Read paired                               |\n| 00000000 00000010 | Read mapped in proper pair                |\n| 00000000 00000100 | Read unmapped                             |\n| 00000000 00001000 | Mate unmapped                             |\n| 00000000 00010000 | Read reverse strand                       |\n| 00000000 00100000 | Mate reverse strand                       |\n| 00000000 01000000 | First in pair                             |\n| 00000000 10000000 | Second in pair                            |\n| 00000001 00000000 | Not primary alignment                     |\n| 00000010 00000000 | Read fails platform/vendor quality checks |\n| 00000100 00000000 | Read is PCR or optical duplicate          |\n| 00001000 00000000 | Supplementary alignment                   |\n\nComputing FLAG statistics from readsets involves iteratively incrementing up to\n16 counters. The native properties of a column-oriented storage, specifically\ncolumn projection, already deliver good performance because of data locality\n(memory contiguity) and value typing. We want to maximize compute on large\narrays of values by exploiting vectorized instructions, if available.\n\n## Goals\n\n* Achieve high-performance on large arrays of values.\n* Support machines without SIMD (scalar).\n* Specialized algorithms for SSE4 up to AVX512.\n\n## Technical approach\n\n### Results\n### Datasets\nAligned data:\n* https://dnanexus-rnd.s3.amazonaws.com/NA12878-xten/mappings/NA12878D_HiSeqX_R1.bam\n\nUnaligned data:\n* https://dnanexus-rnd.s3.amazonaws.com/NA12878-xten/reads/NA12878D_HiSeqX_R1.fastq.gz\n* https://dnanexus-rnd.s3.amazonaws.com/NA12878-xten/reads/NA12878D_HiSeqX_R2.fastq.gz\n\n### Speed\n\n| Comp. Method | Decomp. | Sam-branchless | sam  | flagstat |\n|--------------|---------|----------------|------|----------|\n| LZ4-HC-c1    | 988     | 8924           | 4991 | 1107     |\n| LZ4-HC-c2    | 993     | 8848           | 5076 | 1132     |\n| LZ4-HC-c3    | 938     | 8686           | 4930 | 1049     |\n| LZ4-HC-c4    | 846     | 8803           | 4876 | 933      |\n| LZ4-HC-c5    | 824     | 8525           | 5117 | 974      |\n| LZ4-HC-c6    | 770     | 8536           | 4774 | 851      |\n| LZ4-HC-c7    | 680     | 8404           | 4748 | 837      |\n| LZ4-HC-c8    | 644     | 8453           | 4662 | 755      |\n| LZ4-HC-c9    | **580**     | 8434           | 4740 | **722**      |\n| LZ4-fast-c2  | 814     | 8658           | 4886 | 990      |\n| LZ4-fast-c3  | 837     | 8576           | 4840 | 941      |\n| LZ4-fast-c4  | 889     | 8627           | 4861 | 1026     |\n| LZ4-fast-c5  | 826     | 8590           | 4885 | 1037     |\n| LZ4-fast-c6  | 823     | 8629           | 5034 | 951      |\n| LZ4-fast-c7  | 837     | 8606           | 4999 | 985      |\n| LZ4-fast-c8  | 834     | 8604           | 4920 | 962      |\n| LZ4-fast-c9  | 853     | 8615           | 4944 | 951      |\n| LZ4-fast-c10 | 853     | 8648           | 5024 | 950      |\n| Zstd-c1      | 3435    | 11438          | 7798 | 3630     |\n| Zstd-c2      | 3577    | 11231          | 8110 | 3767     |\n| Zstd-c3      | 3403    | 11250          | 7922 | 3553     |\n| Zstd-c4      | 3562    | 11223          | 7949 | 3649     |\n| Zstd-c5      | 2919    | 10584          | 7263 | 2986     |\n| Zstd-c6      | 2964    | 10680          | 7545 | 3015     |\n| Zstd-c7      | 2681    | 10591          | 7067 | 2715     |\n| Zstd-c8      | 2641    | 10523          | 7103 | 2850     |\n| Zstd-c9      | 2352    | 10453          | 6669 | 2463     |\n| Zstd-c10     | 2309    | 10094          | 6756 | 2509     |\n| Zstd-c11     | 2344    | 10018          | 6430 | 2467     |\n| Zstd-c12     | 2116    | 9916           | 6242 | 2252     |\n| Zstd-c13     | 2107    | 9844           | 6183 | 2236     |\n| Zstd-c14     | 1955    | 9616           | 5969 | 2044     |\n| Zstd-c15     | 1716    | 9562           | 5807 | 1808     |\n| Zstd-c16     | 1286    | 9208           | 5592 | 1448     |\n| Zstd-c17     | 1278    | 8996           | 5592 | 1396     |\n| Zstd-c18     | 1192    | 8907           | 5294 | 1306     |\n| Zstd-c19     | 1181    | 8931           | 5362 | 1293     |\n| Zstd-c20     | 1175    | 8982           | 5369 | 1303     |\n\n```bash\n$ time samtools flagstat NA12878D_HiSeqX_R12_GRCh37.bam\n824541892 + 0 in total (QC-passed reads + QC-failed reads)\n0 + 0 secondary\n5393628 + 0 supplementary\n0 + 0 duplicates\n805383403 + 0 mapped (97.68% : N/A)\n819148264 + 0 paired in sequencing\n409574132 + 0 read1\n409574132 + 0 read2\n781085884 + 0 properly paired (95.35% : N/A)\n797950890 + 0 with itself and mate mapped\n2038885 + 0 singletons (0.25% : N/A)\n9537902 + 0 with mate mapped to a different chr\n4425946 + 0 with mate mapped to a different chr (mapQ\u003e=5)\n\nreal    30m50.059s\nuser    30m10.638s\nsys 0m38.440s\n```\n\n```bash\n$ time samtools flagstat NA12878D_HiSeqX_R12_GRCh37.cram\n824541892 + 0 in total (QC-passed reads + QC-failed reads)\n0 + 0 secondary\n5393628 + 0 supplementary\n0 + 0 duplicates\n805383403 + 0 mapped (97.68% : N/A)\n819148264 + 0 paired in sequencing\n409574132 + 0 read1\n409574132 + 0 read2\n781085884 + 0 properly paired (95.35% : N/A)\n797950890 + 0 with itself and mate mapped\n2038885 + 0 singletons (0.25% : N/A)\n9537902 + 0 with mate mapped to a different chr\n4425946 + 0 with mate mapped to a different chr (mapQ\u003e=5)\n\nreal    4m50.684s\nuser    3m37.394s\nsys 1m12.396s\n```\n\n```c\n#define flagstat_loop(s, c) do {                                \\\n        int w = (c \u0026 BAM_FQCFAIL)? 1 : 0;                       \\\n        ++(s)-\u003en_reads[w];                                      \\\n        if (c \u0026 BAM_FSECONDARY ) {                              \\\n            ++(s)-\u003en_secondary[w];                              \\\n        } else if (c \u0026 BAM_FSUPPLEMENTARY ) {                   \\\n            ++(s)-\u003en_supp[w];                                   \\\n        } else if (c \u0026 BAM_FPAIRED) {                           \\\n            ++(s)-\u003en_pair_all[w];                               \\\n            if ((c \u0026 BAM_FPROPER_PAIR) \u0026\u0026 !(c \u0026 BAM_FUNMAP) ) ++(s)-\u003en_pair_good[w]; \\\n            if (c \u0026 BAM_FREAD1) ++(s)-\u003en_read1[w];              \\\n            if (c \u0026 BAM_FREAD2) ++(s)-\u003en_read2[w];              \\\n            if ((c \u0026 BAM_FMUNMAP) \u0026\u0026 !(c \u0026 BAM_FUNMAP)) ++(s)-\u003en_sgltn[w]; \\\n            if (!(c \u0026 BAM_FUNMAP) \u0026\u0026 !(c \u0026 BAM_FMUNMAP)) {      \\\n                ++(s)-\u003en_pair_map[w];                           \\\n            }                                                   \\\n        }                                                       \\\n        if (!(c \u0026 BAM_FUNMAP)) ++(s)-\u003en_mapped[w];              \\\n        if (c \u0026 BAM_FDUP) ++(s)-\u003en_dup[w];                      \\\n} while (0)\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmklarqvist%2Flibflagstats","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmklarqvist%2Flibflagstats","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmklarqvist%2Flibflagstats/lists"}