{"id":13457451,"url":"https://github.com/nservant/HiC-Pro","last_synced_at":"2025-03-24T14:32:13.895Z","repository":{"id":26632988,"uuid":"30088728","full_name":"nservant/HiC-Pro","owner":"nservant","description":"HiC-Pro: An optimized and flexible pipeline for Hi-C data processing","archived":false,"fork":false,"pushed_at":"2024-03-28T14:18:22.000Z","size":48288,"stargazers_count":386,"open_issues_count":161,"forks_count":182,"subscribers_count":18,"default_branch":"master","last_synced_at":"2024-10-29T02:31:03.182Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nservant.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2015-01-30T20:09:25.000Z","updated_at":"2024-10-28T03:51:17.000Z","dependencies_parsed_at":"2022-07-14T12:17:12.110Z","dependency_job_id":"c73be114-41ee-410f-a522-0c489674d46f","html_url":"https://github.com/nservant/HiC-Pro","commit_stats":{"total_commits":584,"total_committers":22,"mean_commits":"26.545454545454547","dds":0.535958904109589,"last_synced_commit":"57a0bf2cb1d7ebeb44b4f11e78058b504260c5c9"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nservant%2FHiC-Pro","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nservant%2FHiC-Pro/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nservant%2FHiC-Pro/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nservant%2FHiC-Pro/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nservant","download_url":"https://codeload.github.com/nservant/HiC-Pro/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245289608,"owners_count":20591101,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T09:00:17.918Z","updated_at":"2025-03-24T14:32:10.351Z","avatar_url":"https://github.com/nservant.png","language":"Python","funding_links":[],"categories":["Pipelines"],"sub_categories":[],"readme":"\n# HiC-Pro\n\n### An optimized and flexible pipeline for Hi-C data processing\n\n[![Build Status](https://travis-ci.com/nservant/HiC-Pro.svg?branch=devel_py3)](https://travis-ci.com/nservant/HiC-Pro)\n\n![Conda](https://img.shields.io/badge/Conda-build-brightgreen.svg)\n![Singularity](https://img.shields.io/badge/Singularity-build-brightgreen.svg) \n[![Docker](https://img.shields.io/badge/Docker-manual-yellow.svg)](https://hub.docker.com/repository/docker/nservant/hicpro)\n\n![MultiQC](https://img.shields.io/badge/MultiQC-1.8-blue.svg)\n[![Forum](https://img.shields.io/badge/Groups-%20join%20chat%20%E2%86%92-4fb99a.svg?style=flat-square)](https://groups.google.com/forum/#!forum/hic-pro)\n[![DOI](https://img.shields.io/badge/DOI-10.1186%2Fs13059--015--0831--x-lightgrey.svg?style=flat-square)](https://doi.org/10.1186/s13059-015-0831-x)\n\n----\n\nFind documentation and examples at [http://nservant.github.io/HiC-Pro/](http://nservant.github.io/HiC-Pro/)\n\nFor any question about HiC-Pro, please contact nicolas.servant@curie.fr or use the [HiC-Pro forum](https://groups.google.com/forum/#!forum/hic-pro)\n\n## What is HiC-Pro ?\n\nHiC-Pro was designed to process Hi-C data, from raw fastq files (paired-end Illumina data) to normalized contact maps. It supports the main Hi-C protocols, including digestion protocols as well as protocols that do not require restriction enzymes such as DNase Hi-C. In practice, HiC-Pro was successfully applied to many data-sets including dilution Hi-C, in situ Hi-C, DNase Hi-C, Micro-C, capture-C, capture Hi-C or HiChip data.  \nThe pipeline is flexible, scalable and optimized. It can operate either on a single laptop or on a computational cluster. HiC-Pro is sequential and each step of the workflow can be run independantly.  \nHiC-Pro includes a fast implementatation of the iterative correction method (see the [iced python package](https://github.com/hiclib/iced) for more information).\nFinally, HiC-Pro can use phasing data to build [allele-specific contact maps](doc/AS.md).\n\nIf you use HiC-Pro, please cite :\n\n*Servant N., Varoquaux N., Lajoie BR., Viara E., Chen CJ., Vert JP., Dekker J., Heard E., Barillot E.* HiC-Pro: An optimized and flexible pipeline for Hi-C processing. Genome Biology 2015, 16:259 [doi:10.1186/s13059-015-0831-x](https://doi.org/10.1186/s13059-015-0831-x)\n\n## Containers\n\n### Using HiC-Pro through `conda`\n\nIn order to ease the installation of HiC-Pro dependancies, we provide a `yml` file for conda with all required tools.\nIn order to build your conda environment, first install [miniconda](https://docs.conda.io/en/latest/miniconda.html) and use :\n\n```\nconda env create -f MY_INSTALL_PATH/HiC-Pro/environment.yml -p WHERE_TO_INSTALL_MY_ENV\nconda activate WHERE_TO_INSTALL_MY_ENV\n```\n\n### Using the HiC-Pro `Docker` image\n\nA docker image is automatically build and available on [Docker Hub](https://hub.docker.com/repository/docker/nservant/hicpro)\nTo pull a Docker image, simply use :\n\n```\ndocker pull nservant/hicpro:latest\n```\n\nNote that the `tag` may depend on the HiC-Pro version.\n\nYou can also build your own image from the root folder using\n\n```\ndocker build -t hicpro:3.1.0 .\n```\n\n### Using HiC-Pro through `Singularity`\n\nHiC-Pro provides a Singularity container to ease its installation process.\nA ready-to-use container is available [here](https://zerkalo.curie.fr/partage/HiC-Pro/singularity_images/hicpro_latest_ubuntu.img).\n\nIn order to build you own Singularity image;\n\n1- Install singularity\n\n- Linux : http://singularity.lbl.gov/install-linux\n- MAC : http://singularity.lbl.gov/install-mac\n- Windows : http://singularity.lbl.gov/install-windows\n\n2- Build the singularity HiC-Pro image using the 'Singularity' file available in the HiC-Pro root directory.\n\n```\nsudo singularity build hicpro_latest_ubuntu.img MY_INSTALL_PATH/HiC-Pro/envs/Singularity\n```\n\n3- Run HiC-pro\n\nYou can then either use HiC-Pro using the 'exec' command ;\n\n```\nsingularity exec hicpro_latest_ubuntu.img HiC-Pro -h\n```\n\nOr directly use HiC-Pro within the Singularity shell\n\n```\nsingularity shell hicpro_latest_ubuntu.img\nHiC-Pro -h\n```\n\n## How to install it ?\n\nThe HiC-Pro pipeline requires the following dependencies :\n\n- The [bowtie2](http://bowtie-bio.sourceforge.net/bowtie2/index.shtml) mapper\n- Python (\u003e3.7) with *pysam (\u003e=0.15.4)*, *bx-python(\u003e=0.8.8)*, *numpy(\u003e=1.18.1)*, and *scipy(\u003e=1.4.1)* libraries.  \n**Note that the current version no longer supports python 2**\n- R with the *RColorBrewer* and *ggplot2 (\u003e2.2.1)* packages\n- g++ compiler\n- samtools (\u003e1.9)\n- Unix sort (**which support -V option**) is required ! For Mac OS user, please install the GNU core utilities !\n\nNote that Bowtie \u003e2.2.2 is strongly recommanded for allele specific analysis.  \n\nTo install HiC-Pro, be sure to have the appropriate rights and run :\n\n```\ntar -zxvf HiC-Pro-master.tar.gz\ncd HiC-Pro-master\n## Edit config-install.txt file if necessary\nmake configure\nmake install\n```\n\nNote that if some of these dependencies are not installed (i.e. not detected in the $PATH), HiC-Pro will try to install them.  \nYou can also edit the *config-install.txt* file and manually defined the paths to dependencies.\n\n\n|               | SYSTEM CONFIGURATION                                                          |\n|---------------|-------------------------------------------------------------------------------|\n| PREFIX        | Path to installation folder                                                   |\n| BOWTIE2_PATH  | Full path the bowtie2 installation directory                                  |\n| SAMTOOLS_PATH | Full path to the samtools installation directory                              |\n| R_PATH        | Full path to the R installation directory                                     |\n| PYTHON_PATH   | Full path to the python installation directory                                |\n| CLUSTER_SYS   | Scheduler to use for cluster submission. Must be TORQUE, SGE, SLURM or LSF    |\n\n\n## Annotation Files\n\nIn order to process the raw data, HiC-Pro requires three annotation files. Note that the pipeline is provided with some Human and Mouse annotation files.  \n**Please be sure that the chromosome names are the same than the ones used in your bowtie indexes !**\n\n- **A BED file** of the restriction fragments after digestion. This file depends both of the restriction enzyme and the reference genome. See the [FAQ](doc/FAQ.md) and the [HiC-Pro utilities](doc/UTILS.md) for details about how to generate this file. A few annotation files are provided with the HiC-Pro sources as examples.\n\n```\n   chr1   0       16007   HIC_chr1_1    0   +\n   chr1   16007   24571   HIC_chr1_2    0   +\n   chr1   24571   27981   HIC_chr1_3    0   +\n   chr1   27981   30429   HIC_chr1_4    0   +\n   chr1   30429   32153   HIC_chr1_5    0   +\n   chr1   32153   32774   HIC_chr1_6    0   +\n   chr1   32774   37752   HIC_chr1_7    0   +\n   chr1   37752   38369   HIC_chr1_8    0   +\n   chr1   38369   38791   HIC_chr1_9    0   +\n   chr1   38791   39255   HIC_chr1_10   0   +\n   (...)\n```\n\n- **A table file** of chromosomes' size. This file can be easily find on the UCSC genome browser. Of note, pay attention to the contigs or scaffolds, and be aware that HiC-pro will generate a map per chromosomes pair. For model organisms such as Human or Mouse, which are well annotated, we usually recommand to remove all scaffolds.  \n\n```\n   chr1    249250621\n   chr2    243199373\n   chr3    198022430\n   chr4    191154276\n   chr5    180915260\n   chr6    171115067\n   chr7    159138663\n   chr8    146364022\n   chr9    141213431\n   chr10   135534747\n   (...)\n```\n\n- **The bowtie2 indexes**. See the [bowtie2 manual page](http://bowtie-bio.sourceforge.net/bowtie2/index.shtml) for details about how to create such indexes.\n\n\n## How to use it ?\n\nFirst have a look at the help message !\n\n```\n  HiC-Pro --help\n  usage : HiC-Pro -i INPUT -o OUTPUT -c CONFIG [-s ANALYSIS_STEP] [-p] [-h] [-v]\n  Use option -h|--help for more information\n\n  HiC-Pro 3.1.0\n  ---------------\n  OPTIONS\n\n   -i|--input INPUT : input data folder; Must contains a folder per sample with input files\n   -o|--output OUTPUT : output folder\n   -c|--conf CONFIG : configuration file for Hi-C processing\n   [-p|--parallel] : if specified run HiC-Pro on a cluster\n   [-s|--step ANALYSIS_STEP] : run only a subset of the HiC-Pro workflow; if not specified the complete workflow is run\n      mapping: perform reads alignment - require fast files\n      proc_hic: perform Hi-C filtering - require BAM files\n      quality_checks: run Hi-C quality control plots\n      merge_persample: merge multiple inputs and remove duplicates if specified - require .validPairs files\n      build_contact_maps: Build raw inter/intrachromosomal contact maps - require .allValidPairs files\n      ice_norm : run ICE normalization on contact maps - require .matrix files\n   [-h|--help]: help\n   [-v|--version]: version\n```\n\n- Copy and edit the configuration file *'config-hicpro.txt'* in your local folder. See the [manual](doc/MANUAL.md) for details about the configuration file\n\n- Put all input files in a rawdata folder. The input files have to be organized with **one folder per sample**, such as;\n\n```\n   + PATH_TO_MY_DATA\n     + sample1\n       ++ file1_R1.fastq.gz\n       ++ file1_R2.fastq.gz\n       ++ ...\n     + sample2\n       ++ file1_R1.fastq.gz\n       ++ file1_R2.fastq.gz\n     *...\n```\n\n- Run HiC-Pro on your laptop in standalone model\n\n```\n    MY_INSTALL_PATH/bin/HiC-Pro -i FULL_PATH_TO_DATA_FOLDER -o FULL_PATH_TO_OUTPUTS -c MY_LOCAL_CONFIG_FILE\n```\n\n  - Run HiC-Pro on a cluster (TORQUE/SGE/SLURM/LSF)\n\n```\n   MY_INSTALL_PATH/bin/HiC-Pro -i FULL_PATH_TO_DATA_FOLDER -o FULL_PATH_TO_OUTPUTS -c MY_LOCAL_CONFIG_FILE -p\n```\n\nIn the latter case, you will have the following message :\n\n```\n  Please run HiC-Pro in two steps :\n  1- The following command will launch the parallel workflow through 12 torque jobs:\n  qsub HiCPro_step1.sh\n  2- The second command will merge all outputs to generate the contact maps:\n  qsub HiCPro_step2.sh\n```\n\nExecute the displayed command from the output folder:\n\n```\n  qsub HiCPro_step1.sh\n```\n\nOnce executed succesfully (may take several hours), run the step using:\n\n```\n  qsub HiCPro_step2.sh\n```\n\n## Test Dataset\n\nThe test dataset and associated results are available [here](https://zerkalo.curie.fr/partage/HiC-Pro/).\nSmall fastq files (2M reads) extracted from the Dixon et al. 2012 paper are available for test.\n\n```\n ## Get the data. Will download a test_data folder and a configuration file\n wget https://zerkalo.curie.fr/partage/HiC-Pro/HiCPro_testdata.tar.gz \u0026\u0026 tar -zxvf HiCPro_testdata.tar.gz\n\n ## Edit the configuration file and set the path to Human bowtie2 indexes\n\n ## Run HiC-Pro\n time HICPRO_INSTALL_DIR/bin/HiC-Pro -c config_test_latest.txt -i test_data -o hicpro_latest_test\n\nRun HiC-Pro 3.1.0\n--------------------------------------------\nThu Mar 19, 12:18:10 (UTC+0100)\nBowtie2 alignment step1 ...\nLogs: logs/dixon_2M_2/mapping_step1.log\nLogs: logs/dixon_2M/mapping_step1.log\n\n--------------------------------------------\nThu Mar 19, 12:18:57 (UTC+0100)\nBowtie2 alignment step2 ...\nLogs: logs/dixon_2M_2/mapping_step2.log\nLogs: logs/dixon_2M/mapping_step2.log\n\n--------------------------------------------\nThu Mar 19, 12:19:08 (UTC+0100)\nCombine R1/R2 alignment files ...\nLogs: logs/dixon_2M_2/mapping_combine.log\nLogs: logs/dixon_2M/mapping_combine.log\n\n--------------------------------------------\nThu Mar 19, 12:19:13 (UTC+0100)\nMapping statistics for R1 and R2 tags ...\nLogs: logs/dixon_2M_2/mapping_stats.log\nLogs: logs/dixon_2M/mapping_stats.log\n\n--------------------------------------------\nThu Mar 19, 12:19:15 (UTC+0100)\nPairing of R1 and R2 tags ...\nLogs: logs/dixon_2M_2/mergeSAM.log\nLogs: logs/dixon_2M/mergeSAM.log\n\n--------------------------------------------\nThu Mar 19, 12:19:25 (UTC+0100)\nAssign alignments to restriction fragments ...\nLogs: logs/dixon_2M_2/mapped_2hic_fragments.log\nLogs: logs/dixon_2M/mapped_2hic_fragments.log\n\n--------------------------------------------\nThu Mar 19, 12:20:10 (UTC+0100)\nMerge chunks from the same sample ...\nLogs: logs/dixon_2M/merge_valid_interactions.log\nLogs: logs/dixon_2M_2/merge_valid_interactions.log\n\n--------------------------------------------\nThu Mar 19, 12:20:11 (UTC+0100)\nMerge stat files per sample ...\nLogs: logs/dixon_2M/merge_stats.log\nLogs: logs/dixon_2M_2/merge_stats.log\n\n--------------------------------------------\nThu Mar 19, 12:20:11 (UTC+0100)\nRun quality checks for all samples ...\nLogs: logs/dixon_2M/make_Rplots.log\nLogs: logs/dixon_2M_2/make_Rplots.log\n\n--------------------------------------------\nThu Mar 19, 12:20:22 (UTC+0100)\nGenerate binned matrix files ...\nLogs: logs/dixon_2M/build_raw_maps.log\nLogs: logs/dixon_2M_2/build_raw_maps.log\n\n--------------------------------------------\nThu Mar 19, 12:20:22 (UTC+0100)\nRun ICE Normalization ...\nLogs: logs/dixon_2M/ice_500000.log\nLogs: logs/dixon_2M/ice_1000000.log\nLogs: logs/dixon_2M_2/ice_500000.log\nLogs: logs/dixon_2M_2/ice_1000000.log\n\nreal\t2m15,736s\nuser\t4m3,277s\nsys\t0m24,423s\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnservant%2FHiC-Pro","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnservant%2FHiC-Pro","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnservant%2FHiC-Pro/lists"}