{"id":16864744,"url":"https://github.com/38/d4-format","last_synced_at":"2025-05-15T13:07:55.748Z","repository":{"id":40782450,"uuid":"248287897","full_name":"38/d4-format","owner":"38","description":"The D4 Quantitative Data Format","archived":false,"fork":false,"pushed_at":"2025-04-25T02:19:35.000Z","size":2502,"stargazers_count":161,"open_issues_count":31,"forks_count":20,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-04-25T02:57:06.195Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/38.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-18T16:42:42.000Z","updated_at":"2025-04-25T02:19:39.000Z","dependencies_parsed_at":"2025-03-10T10:45:46.247Z","dependency_job_id":"b2372591-d4f1-416c-b309-bc39f66b9b86","html_url":"https://github.com/38/d4-format","commit_stats":{"total_commits":265,"total_committers":12,"mean_commits":"22.083333333333332","dds":0.1773584905660377,"last_synced_commit":"480e0a36a453fd3ec40b6768458f8892e6f36510"},"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/38%2Fd4-format","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/38%2Fd4-format/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/38%2Fd4-format/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/38%2Fd4-format/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/38","download_url":"https://codeload.github.com/38/d4-format/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254346624,"owners_count":22055808,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T14:43:43.324Z","updated_at":"2025-05-15T13:07:50.728Z","avatar_url":"https://github.com/38.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# `d4` - Dense Depth Data Dump\n\n## Synopsis\n\nThe Dense Depth Data Dump (D4) format and tool suite provide an alternative to BigWig for fast analysis and compact storage of quantitative genomics datasets (e.g., RNA-seq, ChIP-seq, WGS depths, etc.). It supports random access, multiple tracks (e.g., RNA-seq, ChiP-seq, etc. from the same sample), HTTP range requests, and statistics on arbitrary genome intervals. The D4tools software is built on a [Rust crate](https://docs.rs/d4/). We provide both a [C-API](https://github.com/38/d4-format/tree/master/d4binding/include) and a [Python API](https://github.com/38/d4-format/tree/master/pyd4/) with an [Jupyter notebook](https://github.com/38/d4-format/blob/master/pyd4/examples/D4%20Tutorial%20in%20Python.ipynb) providing examples of how to to read, query, and create single-track and multi-track D4 files.\n\nUsage examples are provided below. Also, check out the [slide deck](https://docs.google.com/presentation/d/1vqs6mnfiVryfMAxqDyJrZsX6HI39NbwWqvB7DUCLTgw) that describes the motivation, performance and toolkits for D4\n\n## Motivation\n\nModern DNA sequencing is used as a readout for diverse assays, with the count of aligned sequences, or \"read depth\", serving as the quantitative signal for many underlying cellular phenomena. Despite wide use and thousands of datasets, existing formats used for the storage and analysis of read depths are limited with respect to both size and speed. For example, it is faster to recalculate sequencing depth from an alignment file than it is to analyze the text output from that calculation. We sought to improve on existing formats such as BigWig and compressed BED files by creating the Dense Depth Data Dump (D4) format and tool suite. The D4 format is adaptive in that it profiles a random sample of aligned sequence depth from the input BAM or CRAM file to determine an optimal encoding that  minimizes file size, while also enabling fast data access. We show that D4 uses less disk space for both RNA-Seq and whole-genome sequencing and offers 3 to 440 fold speed improvements over existing formats for random access, aggregation and summarization for scalable downstream analyses that would be otherwise intractable.\n\n## Manuscript\n\nTo learn more, please read the publication: [https://www.nature.com/articles/s43588-021-00085-0](https://www.nature.com/articles/s43588-021-00085-0). Note We ran the experiments described in the manuscript on a server with following hardward and software  \n\n- Processor: Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz\n- RAM: 376GB\n- OS: CentOS 7.6.180 w/ Linux Kernel 3.0.10\n- Rust Version: 1.47.0-nightly\n\n## Basic Usage by Examples (each should take seconds)\n\n### Create a D4 file\n\nThe `d4tools create` subcommand is used to convert BAM,CRAM,BigWig and BedGraph file to D4 file.\n\n```text\nUSAGE:\n    create [FLAGS] [OPTIONS] \u003cinput-file\u003e [output-file]\n\nFLAGS:\n    -z, --deflate      Enable the deflate compression\n    -A, --dict-auto    Automatically determine the dictionary type by random sampling\n        --dump-dict    Do not profile the BAM file, only dump the dictionary\n    -h, --help         Prints help information\n    -S, --sparse       Sparse mode, this is same as '-zR0-1', which enable secondary table compression and disable\n                       primary table\n    -V, --version      Prints version information\n\nOPTIONS:\n        --deflate-level \u003clevel\u003e          Configure the deflate algorithm, default 5\n    -d, --dict-file \u003cdict_spec_file\u003e     Provide a file that defines the values of the dictionary\n    -R, --dict-range \u003cdict_spec\u003e         Dictionary specification, use \"a-b\" to specify the dictionary is encoding\n                                         values from A to B(exclusively)\n    -f, --filter \u003cregex\u003e                 A regex that matches the genome name should present in the output file\n    -g, --genome \u003cgenome_file\u003e           The genome description file (Used by BED inputs)\n    -q, --mapping-qual \u003cmapping-qual\u003e    The minimal mapping quality (Only valid with CRAM/BAM inputs)\n    -r, --ref \u003cfai_file_path\u003e            Reference genome file (Used by CRAM inputs)\n    -t, --threads \u003cnum_of_threads\u003e       Specify the number of threads D4 can use for encoding\n\nARGS:\n    \u003cinput-file\u003e     Path to the input file\n    \u003coutput-file\u003e    Path to the output file\n```\n\n- From CRAM/BAM file\n\n```bash\n  d4tools create -Azr hg19.fa.gz.fai hg002.cram hg002.d4\n```\n\n- From BigWig file\n\n```bash\n  d4tools create -z input.bw output.d4\n```\n\n- From a BedGraph file (extension must be \".bedgraph\")\n\n```bash\n  d4tools create -z -g hg19.genome input.bedgraph output.d4\n```\n\n### View a D4 File\n\n```text\nUSAGE:\n    view [FLAGS] \u003cinput-file\u003e [chr:start-end]...\n\nFLAGS:\n    -h, --help           Prints help information\n    -g, --show-genome    Show the genome file instead of the file content\n    -V, --version        Prints version information\n\nARGS:\n    \u003cinput-file\u003e          Path to the input file\n    \u003cchr:start-end\u003e...    Regions to be viewed\n```\n\n- Convert a d4 file to a bedgraph file\n\n```text\n$ d4tools view hg002.d4 | head -n 10\nchr1    0       9998    0\nchr1    9998    9999    6\nchr1    9999    10000   9\nchr1    10000   10001   37\nchr1    10001   10002   59\nchr1    10002   10003   78\nchr1    10003   10004   100\nchr1    10004   10005   116\nchr1    10005   10006   130\nchr1    10006   10007   135\n```\n\n- Print given regions\n\n```text\n$ d4tools view hg002.d4 1:1234560-1234580 X:1234560-1234580\n1       1234559 1234562 28\n1       1234562 1234565 29\n1       1234565 1234566 30\n1       1234566 1234572 31\n1       1234572 1234573 29\n1       1234573 1234576 28\n1       1234576 1234578 27\n1       1234578 1234579 26\nX       1234559 1234562 26\nX       1234562 1234563 25\nX       1234563 1234565 26\nX       1234565 1234574 25\nX       1234574 1234575 26\nX       1234575 1234576 25\nX       1234576 1234578 26\nX       1234578 1234579 25\n```\n\n- Print the genome layout\n\n```text\n$ d4tools view -g hg002.d4 | head -n 10\n1       249250621\n2       243199373\n3       198022430\n4       191154276\n5       180915260\n6       171115067\n7       159138663\n8       146364022\n9       141213431\n10      135534747\n```\n\n### Run stat on a D4 file\n\n```text\nUSAGE:\n    stat [OPTIONS] \u003cinput_d4_file\u003e\n\nFLAGS:\n    -h, --help       Prints help information\n    -V, --version    Prints version information\n\nOPTIONS:\n    -r, --region \u003cbed_file_path\u003e      A bed file that describes the region we want to run the stat\n    -s, --stat \u003cstat_type\u003e            The type of statistics we want to perform, by default average. You can specify\n                                      statistic methods: perc_cov, mean, median, hist, percentile=X% (If this is not speficied\n                                      d4tools will use mean by default)\n    -t, --threads \u003cnum_of_threads\u003e    Number of threads\n\nARGS:\n    \u003cinput_d4_file\u003e\n```\n\n- Mean cov for each Chrom\n\n```text\n$ d4tools stat hg002.d4\nchr1    0       249250621       27.075065016588262\nchr10   0       135534747       31.59483947684648\nchr11   0       135006516       25.970025943044114\nchr11_gl000202_random   0       40103   14.47213425429519\nchr12   0       133851895       25.80992053194316\nchr13   0       115169878       24.18613685602758\nchr14   0       107349540       24.25194093053403\nchr15   0       102531392       23.04176524785697\nchr16   0       90354753        28.106620932271266\nchr17   0       81195210        25.58382477242192\n...\n```\n\n- Median cov for each Chrom\n\n```text\n$ d4tools stat -s median hg002.d4 | head -n 10\n1       0       249250621       25\n10      0       135534747       26\n11      0       135006516       26\n12      0       133851895       26\n13      0       115169878       26\n14      0       107349540       25\n15      0       102531392       24\n16      0       90354753        24\n17      0       81195210        25\n18      0       78077248        26\n```\n\n- Top 5% for the given region defined in a bed file\n\n```text\n$ d4tools stat -s percentile=95 -r region.bed hg002.d4\n1       2000000 3000000 33\n2       0       150000000       38\n```\n\n- Percent of bases at or above coverage levels (perc_cov)\n```text\n$ d4tools stat -H -s perc_cov=1,2 -r data/input_10nt.multiple_ranges.bed data/input_10nt.d4 \n#Chr    Start   End     1x      2x\nchr     0       2       0.000   0.000\nchr     0       8       0.625   0.375\nchr     0       10      0.600   0.300\nchr     1       6       0.600   0.400\nchr     3       9       1.000   0.500\nchr     4       5       1.000   1.000\nchr     5       10      0.800   0.400\n```\n\n### Reading D4 File Served by static HTTP Server\n\nD4 now supports showing and run statistics for D4 files that is served on a HTTP server without downloading the file to local.\nFor printing the file content, simple use the following command:\n\n```\n$ d4tools show https://d4-format-testing.s3.us-west-1.amazonaws.com/hg002.d4 | head -n 10\n1       0       9998    0\n1       9998    9999    6\n1       9999    10000   10\n1       10000   10001   38\n1       10001   10002   55\n1       10002   10003   72\n1       10003   10004   93\n1       10004   10005   110\n1       10005   10006   126\n1       10006   10007   131\n```\n\nTo run statistics on a D4 file on network, we required the D4 file contains the data index to avoid full file accessing.\n\n- (On the server side) Prepare the D4 file that need to be accessed on web\n\n```bash\nd4tools index build --sum hg002.d4\n```\n\n- (On the client side) Run mean depth statistics on this file\n\n```\n$ d4tools stat https://d4-format-testing.s3.us-west-1.amazonaws.com/hg002.d4\n1       0       249250621       23.848327146193952\n2       0       243199373       25.02162749408075\n3       0       198022430       23.086504175309837\n4       0       191154276       23.18471121200553\n5       0       180915260       23.2536419094774\n6       0       171115067       24.515156108374722\n7       0       159138663       24.398102314080646\n8       0       146364022       26.425789139628865\n9       0       141213431       19.780247114029827\n10      0       135534747       25.475887087464\n\n....\n```\n\n## Build\n\n### Prerequisites\n\nTo build `d4`, Rust toolchain is required. To install Rust toolchain,\nplease run the following command and follow the prompt to complete the\nRust installation.\n\n```bash\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n```\n\n`gcc` or `clang` is required to build `htslib` embeded with the `d4` library.\nFor details, please check the htslib repository.\n\n### Build Steps\n\nNormally, the build step is quite easy. Just\n\n```bash\n# For Debug Build\ncargo build\n# For Release Build\ncargo build --release\n```\n\nAnd it will produce the `d4tools` binary which you can find at either\n`target/debug/d4tools` or `target/release/d4tools` depending on which build mode\nyou choose.\n\n#### Troubleshooting\n\n- Compiling error: asking for -fPIC or -fPIE option\n\nFor some environment, the Rust toolchain will ask compile the `-fPIC` or `-fPIE` to build the `d4tools` binary.\nIn this case, you should be able to use the following workaround:\n\n```bash\n# To build a debug build :\ncd d4tools \u0026\u0026 cargo rustc --bin d4tools -- -C relocation-model=static\n\n# To build a release build :\ncd d4tools \u0026\u0026 cargo rustc --bin d4tools --release -- -C relocation-model=static\n```\n\n### Installation (\u003c 2 minutes)\n\n- Install bioconda \n\nAssuming you have bioconda environment installed and configured, you can simply install d4tools and d4binding from bioconda repository\n\n```bash\nconda install d4tools\n```\n\n- Install from crates.io: Assuming you have Rust compiler toolchain, you can install it from crate.io as well.\n\n```\ncargo install d4tools\n```\n\n- Install from source code: The following steps allows you to install d4tools from source code. You can choose to install the d4tools binary by running\n\n```bash\ncargo install --path .\n```\n\n### Using D4 in C/C++\n\nD4 provides a C binding that allows the D4 library used in C and C++.\nHere's the steps to build D4 binding.\n\n1. Install or build the binding library\n\n- The easist way to install d4binding library is using bioconda. \n\n```\nconda install d4binding\n```\nThen the header file will be installed under `\u003cconda-dir\u003e/include`. And `libd4binding.so` or `libd4binding.dylib` will be installed under `\u003cconda-dir\u003e/lib`.\n\n- Alternatively, you can choose install from the source code as well:\n\n```bash\n# Build the D4 binding library, for debug build, remove \"--release\" argument\ncargo build --package=d4binding --release\n```\n\nAfter running this command, you should be able to find the library \"target/release/libd4binding.so\".\n\n2. Use D4 in C\n\nHere's a small example that prints all chromosome name and size defined in a D4 file.\n\n```c\n#include \u003cstdio.h\u003e\n#include \u003cd4.h\u003e\n\nint main(int argc, char** argv) \n{\n    d4_file_t* fp = d4_open(\"input.d4\", \"r\");\n\n    d4_file_metadata_t mt = {};\n    d4_file_load_metadata(fp, \u0026mt);\n\n    int i;\n    for(i = 0; i \u003c mt.chrom_count; i ++)\n        printf(\"# %s %d\\n\", mt.chrom_name[i], mt.chrom_size[i]);\n    \n    d4_close(fp);\n    return 0;\n}\n```\n\n3. Compile C++ code against D4 binding library\n\n```\ngcc print-chrom-info.c -o print-chrom-info -I d4binding/include -L target/release -ld4binding  \n```\n\nFor more examples, see `d4binding/examples/`\n\n### Sample Data\n\n- WGS  [https://home.chpc.utah.edu/~u0875014/hg002.cram](https://home.chpc.utah.edu/~u0875014/hg002.cram)\n- RNASeq [https://www.encodeproject.org/files/ENCFF164HRL/](https://www.encodeproject.org/files/ENCFF164HRL/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F38%2Fd4-format","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F38%2Fd4-format","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F38%2Fd4-format/lists"}