{"id":13755650,"url":"https://github.com/kundajelab/simdna","last_synced_at":"2025-10-21T19:22:59.880Z","repository":{"id":49990080,"uuid":"64431056","full_name":"kundajelab/simdna","owner":"kundajelab","description":"A python library for creating simulated regulatory DNA sequences","archived":false,"fork":false,"pushed_at":"2023-02-22T21:59:01.000Z","size":4030,"stargazers_count":38,"open_issues_count":2,"forks_count":21,"subscribers_count":29,"default_branch":"master","last_synced_at":"2024-11-06T22:49:07.516Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kundajelab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2016-07-28T22:03:26.000Z","updated_at":"2024-08-06T04:44:34.000Z","dependencies_parsed_at":"2024-01-17T15:04:29.163Z","dependency_job_id":"bcddc582-5ab3-47db-9969-a1736167bdf1","html_url":"https://github.com/kundajelab/simdna","commit_stats":{"total_commits":149,"total_committers":12,"mean_commits":"12.416666666666666","dds":"0.38926174496644295","last_synced_commit":"72e6a3be39c08b09d00ab07034de5bff8b8587ac"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kundajelab%2Fsimdna","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kundajelab%2Fsimdna/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kundajelab%2Fsimdna/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kundajelab%2Fsimdna/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kundajelab","download_url":"https://codeload.github.com/kundajelab/simdna/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224906622,"owners_count":17389901,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T11:00:17.791Z","updated_at":"2025-10-21T19:22:54.830Z","avatar_url":"https://github.com/kundajelab.png","language":"Python","funding_links":[],"categories":["Software packages"],"sub_categories":["Data wrangling"],"readme":"# [simdna: simulated datasets of DNA](https://kundajelab.github.io/simdna/)\n[![license](https://img.shields.io/github/license/mashape/apistatus.svg?maxAge=2592000)](https://github.com/kundajelab/simdna/blob/master/LICENSE)\n[![DOI](https://zenodo.org/badge/64431056.svg)](https://zenodo.org/badge/latestdoi/64431056)\n[![Build Status](https://travis-ci.org/kundajelab/simdna.svg?branch=master)](https://travis-ci.org/kundajelab/simdna)\n\n\n## Installation\n```\ngit clone https://github.com/kundajelab/simdna.git\ncd simdna\npython setup.py develop\n```\n\n## Overview\n\nThis is a tool for generating simulated regulatory sequence for use in experiments/analyses. \nThere are essentially two \"phases\" to design that SimDNA facilitates.\nFirst is generating realistic background sequences.\nSecond is embedding elements of interest in an existing sequence or set of secquences.\nThese phases are captured by the two core classes underlying the packages basic functioning: backgrounds are\n generated by a BackgroundGenerator, and then elements are embedded by calling Embedders on the generated sequences.\n \n \n### Backgrounds\n\nSimDNA afford several ways of generating background sequences to embed elements into.  The simplest of\n these is generating a set of completely randomized background sequences according to a set of probabilities \n for each individual nucleotide. See [Background Sequence](###Background Sequence) for more information on different\n  ways to generate a background.\n\n### Embedders\nAn Embedder often consist of an EmbeddableGenerator and a PositionGenerator. The EmbeddableGenerator produces the \nmotif/grammar instance to be embedded, and the PositionGenerator determines the placement of the motif/grammar. A single Embedder may insert multiple motifs into a sequence. The RepeatedEmbedder class (which itself is a kind of \nEmbedder) may be used to call any Embedder class multiple times, where the number of times to call the Embedder is \ngenerated by a QuantityGenerator will return a number of times to embed. This number can be fixed, or sampled from\n some distribution.\nSee [Embedders](###Embedders) for more information on different\n  ways to construct embedders.\n  \n### Pipeline\n\nAt a high level, SimDNA sequence generators are assembled in a modular fashion, where the high-level classes build upon the outputs of lower-level classes.\nThe most high-level class is the SequenceSetGenerator, which can be supplied to the printSequences function to generate a collection of sequences.\n - An example of a SequenceSetGenerator is GenerateSequenceNTimes, which takes two arguments: a SingleSequenceGenerator\n  (which generates individual sequences) and a number N that determines how many times to call the SingleSequenceGenerator\n - An example of a SingleSequenceGenerator is EmbedInABackground. The EmbedInABackground class takes two arguments: a\n BackgroundGenerator, and a list of Embedder objects. The backgroundGenerator generates the background sequence, and then the Embedder objects are called successively to insert patterns into the background sequence.\n - An example of an Embedder object is the SubstringEmbedder. A SubstringEmbedder consists of two parts: a\n SubstringGenerator and a PositionGenerator. The SubstringGenerator produces the DNA string to be embedded, and the PositionGenerator determines the position at which the DNA string will be embedded.\n - An example of a SubstringGenerator is a PwmSampler, which samples from a PWM (or, more accurately, a PFM a.k.a. a\n  Position Frequency Matrix). A PwmSampler is instantiated using a Pwm object that is defined by specifying the matrix of letter frequencies.\n\n\nA PwmSampler can be optionally wrapped in a ReverseComplementWrapper. The ReverseComplementWrapper is itself a type of SubstringGenerator that contains an inner SubstringGenerator. The ReverseComplementWrapper will call the inner SubstringGenerator and reverse-complement the resulting DNA string with 50% probability.\nSimilarly, the SubstringEmbedder can be wrapped in a RepeatedEmbedder. The RepeatedEmbedder is itself a type of Embedder that has two parts: an inner Embedder object and a QuantityGenerator. The RepeatedEmbedder will call the inner Embedder class a number of times that is determined by the QuantityGenerator.\nBy building up the sequence generator in this modular way, it becomes easy to mix-and-match functionality. For example, it is possible to define a new type of quantity generator that may act as a wrapper for another quantity generators (such as the ZeroInflator or MinMaxWrapper) while still having access to all the other simdna classes.\n\n\n### A simple example\n\nHere SimDNA is used to construct 1000 sequences, each 200 nucleotides long, with backgrounds sampled from a specified\n distribution. Each of these sequences will have the TAL4 motif (in both forward and reverse compliment) embedded\n  between one and three times, at random positions.\n\nThe embedding pipeline:\n\n\n```python\nimport simdna\nfrom simdna import synthetic\n# create a PWM object to represent the motif\nthepwm = simdna.pwm.PWM('TAL4').addRows(matrix_of_letter_probabilities).finalise(pseudocountProb=0.001)\n# the class that samples from the pwm\npwmsampler = synthetic.PwmSampler(thepwm)\n# a wrapper that will randomly take the reverse complement of an embeddable string returned by pwm sampler;\n# this allows embedding motifs in both orientations\nrc_pwmsampler = synthetic.ReverseComplementWrapper(pwmsampler)\n# a wrapper that embeds a returned element at; SubstringEmbedder samples the positionGenerator each time it embeds\n# the positionGenerator can return the same or different positions each time; here it draws a random position from a\n# uniform distribution; SubstringEmbedder will not overwrite a previously embedded element\nmult_rc_pwmembedder = synthetic.SubstringEmbedder(rc_pwmsampler, positionGenerator=synthetic.UniformPositionGenerator())\n# a wrapper that embeds a returned element multiple times; SubstringEmbedder samples the quantityGenerator each \n# time it is called and embeds that many elements in the given sequence\nrepeatedpwmembedder = synthetic.RepeatedEmbedder(\n                          mult_rc_pwmembedder, \n                          quantityGenerator=synthetic.UniformIntegerGenerator(minVal=1, maxVal=3)\n                      )\n```\n\nThe background generator:\n```python\n# the background generator (“zero order” refers to the order of the markov model; this randomly samples each\n# nucelotide independently)\nbggen = synthetic.ZeroOrderBackgroundGenerator(seqLength=200, \n                                               discreteDistribution={'A': 0.27, 'C': 0.23, 'G': 0.23, 'T': 0.27})\n```\n\nPutting it all together:\n```python\n# this combines the background and the pipeline for generation\nseq_sim = synthetic.EmbedInABackground(backgroundGenerator=bggen, embedders=[repeatedpwmembedder])\n# create a generator to run the pipeline, from generating a background through the embedding pipeline N times\nsequence_set = synthetic.GenerateSequenceNTimes(seq_sim, 1000)\n# actually generate and save the sequences frmo the pipeline\nsynthetic.printSequences(\"sequences.simdata\", sequence_set, \n                         includeFasta=True, includeEmbeddings=True, prefix=\"myprefix\")\n```\n\n### Reading a simdata file\n\nThe simdaata file encodes all the sequences, as well as all of the embedded motifs in each sequence. These can beread\n using the `read_simdata_file` function:\n\n```python\nimport simdna.synthetic\ndata = simdna.synthetic.read_simdata_file(\"sim.simdata\")  \n\nfor sequence, embeddings in zip(data.sequences, data.embeddings):\n    ...\n```\n\nThis code allows iterating over all generated sequences and the motifs embedded in those sequences. This is helpful\n in actually using the simulated sequences to perform computational experiments.\n\n## Examples\nPlease see the scripts folder for example scripts generating simulations and the scripts_test folder for example arguments.\n- densityMotifSimulation.py generates a simulated dataset where multiple instances of motifs are present per sequence, as determined by a poisson distribution which could optionally be subject to zero-inflation.\n- motifGrammarSimulation.py illustates how to set up a simulation where two motifs have a fixed-spacing or variable-spacing grammar (set `--generationSetting` to twoMotifsFixedSpacing or twoMotifsVariableSpacing as desired).\n- emptyBackground.py just generates a background sequence with no motifs embedded.\n\n\n## Creating Custom Simulations\n\n### Loading motifs\n\nThe SimDNA package comes with the ENCODE and HOCOMO databases of motifs so any motif in those databases can be used with\n the LoadedMotifs class to load PWMs (as in the examples). ENCODE, Homer, and Jaspar\n motifs. To load your own motifs you can create a single file with\n multiple motifs from in any of these formats load it accordingly.\n\nEncode:\n```\nloadedMotifs = synthetic.LoadedEncodeMotifs({path})\n```\n\nHomer:\n```\nloadedMotifs = synthetic.LoadedHomerMotifs({path})\n```\n\nJaspar:\n\n```python3\nloadedMotifs = synthetic.LoadedJasparRawPMFMotifs({path})\n```\n\n### Background Sequence\n\nSimDNA afford several ways of generating background sequences to embed elements into. \n\n\nThe first and simplest of these is simply generating a set of completely randomized background sequences according to a set of probabilities for each individual nucleotide. \n\n```python3\nbackground_gen = synthetic.ZeroOrderBackgroundGenerator(seqLength={seqLength}, \n                                                        discreteDistribution={nucleotide_distribution})\n```\n\nThe second, slightly more sophisticated randomized generator uses a first order markov chain to guarantee a given dinucleotide frequency distribution in the generated sequences.\n\n```python3\nbackground_gen = synthetic.FirstOrderBackgroundGenerator(seqLength={seqLength},\n                                                         priorFrequencies={nuc_distribution},\n                                                         dinucFrequencies={dinuc_distribution})\n```\n\nA third method is to supply a sequence and dinucleotide shuffle that sequence. \n\n### PWM Sampler\n\nPwmSampler includes three ways to sample PWMs: first is simply sampling randomly from the PWM; second is sampling the\n best hit of the PWM; third sampling only motifs achieving some minimum logodds score relative to a background via\n  the minScore argument.\n\n### Embeddables\n\nIn addition to the basic string embeddable there is also a PairEmbeddable which allows embedding two embeddables with\n a given seperation. PairEmbeddables can be nested to allows embedding any number of motifs with a fixed spacing\n . Being able to set a given spacing is useful in contexts where yo want to simulate motif-motif interactions, and\n  other more complex elements of cis-regulatory grammars.\n\n### Embedders\n\nAn embedder is an object that embeds an embeddable. Embedders take a EmbeddableGenerator and a PositionGenerator and\n draw a position to embed the embeddable in the background string passed into the _embed function.\nThe most common embedder is the SubstringEmbedder, which will be the main embedder used to add motifs to background\n sequences. A RepeatedEmbedder takes an embedder and embeds it a number of times drawn from a QuantityGenerator. An\n  XOREmbedder takes two embedders and with probability embedes on or the other. RandomSubsetOfEmbedders takes a\n   list of embedders and draws from a quantity generator to select a number of them to embed, then selecting from the\n    list that many at random. Finally AllEmbedders takes a list of embedders and embeds them all.\n\n#### PositionGenerators\n\nThe main types of PositionGenerator are the FixedPositionGenerator and UniformPositionGenerator generators, but\n SimDNA also supports a NormalDistributionPositionGenerator (which acts as a truncated normal centered in the\n  sequence -- with optional offsets) and InsideCentralBP and OutsideCentralBP generators whcih can be used to uniformly sample from subsets of the\n  sequence.\n\n#### QuantityGenerators\n\nSimDNA supports FixedQuantityGenerator, which returns a fixed vale; ChooseValueFromASet, which\n randomly samples from a given set with given probabiliities; and UniformIntegerGenerator\n, PoissonQuantityGenerator, BernoulliQuantityGenerator each of which sample from the specified distribution. There\n are also two important wrapper generators that manipulate the other generators: MinMaxWrapper which forces a\n  generated sample to be within a set range and ZeroInflater which with probability returns a 0 instead of the\n   quantity generated.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkundajelab%2Fsimdna","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkundajelab%2Fsimdna","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkundajelab%2Fsimdna/lists"}