{"id":13773161,"url":"https://github.com/evoldoers/biomake","last_synced_at":"2026-03-09T20:32:02.671Z","repository":{"id":1634731,"uuid":"2358957","full_name":"evoldoers/biomake","owner":"evoldoers","description":"GNU-Make-like utility for managing builds and complex workflows","archived":false,"fork":false,"pushed_at":"2023-11-14T23:00:19.000Z","size":960,"stargazers_count":110,"open_issues_count":27,"forks_count":9,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-12-17T07:48:03.692Z","etag":null,"topics":["bioinformatics","gnu-make","makefiles","prolog","swi-prolog","workflows"],"latest_commit_sha":null,"homepage":"","language":"Prolog","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"RyanMcG/lein-npm","license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/evoldoers.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2011-09-10T00:48:26.000Z","updated_at":"2025-11-01T08:46:51.000Z","dependencies_parsed_at":"2022-07-07T12:54:52.049Z","dependency_job_id":"6d138585-42a9-4a97-8d26-b65bf92dc2d5","html_url":"https://github.com/evoldoers/biomake","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/evoldoers/biomake","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evoldoers%2Fbiomake","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evoldoers%2Fbiomake/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evoldoers%2Fbiomake/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evoldoers%2Fbiomake/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/evoldoers","download_url":"https://codeload.github.com/evoldoers/biomake/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evoldoers%2Fbiomake/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30310764,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-09T20:05:46.299Z","status":"ssl_error","status_checked_at":"2026-03-09T19:57:04.425Z","response_time":61,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bioinformatics","gnu-make","makefiles","prolog","swi-prolog","workflows"],"created_at":"2024-08-03T17:01:12.178Z","updated_at":"2026-03-09T20:32:02.647Z","avatar_url":"https://github.com/evoldoers.png","language":"Prolog","readme":"[![Build Status](https://travis-ci.org/evoldoers/biomake.svg?branch=master)](https://travis-ci.org/evoldoers/biomake)\n[![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)\n\nBiomake\n=======\n\nBiomake is a [make](https://www.gnu.org/software/make/)-compatible utility for managing builds (or analysis workflows) involving multiple\ndependent files.\nIt supports most of the functionality (and syntax) of GNU Make, along with neat extensions like\ncluster-based job processing, multiple wildcards per target, MD5 checksums instead of timestamps,\nand declarative logic programming in Prolog.\n\nIndeed: [Prolog](https://en.wikipedia.org/wiki/Prolog).\nNo knowledge of the dark logical arts is necessary to use Biomake; the software can be\nrun directly off a GNU Makefile. However, if you know (or are prepared to\nlearn) a little Prolog, you can do a lot more.\nMakefiles are logic programs: their power comes from combining a _declarative_ specification of dependencies\nwith _procedural_ shell scripts to build targets.\nProlog is a simple but expressive language for logic programming\nthat allows Makefile rules to be extended in sophisticated and flexible ways.\n\nGetting Started\n---------------\n\n1. Install SWI-Prolog from http://www.swi-prolog.org\n\n2. Get the latest biomake source from github. No installation steps are\nrequired. Just add it to your path (changing the directory if necessary):\n\n    `export PATH=$PATH:$HOME/biomake/bin`\n\n3. Get (minimal) help from the command line:\n\n    `biomake -h`\n\n4. Create a 'Makefile' or a 'Makeprog' (see below)\n\nAlternate installation instructions\n-----------------------------------\n\nIf you want to install biomake system-wide, instead of adding it to your path, type `make install` (or `bin/biomake install`) in the top level directory of the repository.\nThis will copy the repository into `/usr/local/share` and create a symlink to `/usr/local/bin`.\n(If you just want to create the symlink and leave the repository where it is, type `make symlink` instead.)\n\nYou can also try `make test` (or, equivalently, `biomake test`) to run the test suite.\n\nThe program can also be installed via the SWI-Prolog pack system.\nJust start SWI and type:\n\n    ?- pack_install('biomake').\n\nCommand-line\n------------\n\n    biomake [OPTIONS] [TARGETS]\n\nOptions\n-------\n\n```\n-h,--help \n    Show help\n-v,--version \n    Show version\n-n,--dry-run,--recon,--just-print \n    Print the commands that would be executed, but do not execute them\n-B,--always-make \n    Always build fresh target even if dependency is up to date\n-f,--file,--makefile GNUMAKEFILE\n    Use a GNU Makefile as the build specification [default: Makefile]\n-p,--prog,--makeprog MAKEPROG\n    Use MAKEPROG as the (Prolog) build specification [default: Makeprog]\n-m,--eval,--makefile-syntax STRING\n    Evaluate STRING as GNU Makefile syntax\n-P,--eval-prolog,--makeprog-syntax STRING\n    Evaluate STRING as Prolog Makeprog syntax\n-I,--include-dir DIR\n    Specify search directory for included Makefiles\n--target TARGET\n    Force biomake to recognize a target even if it looks like an option\n-T,--translate,--save-prolog FILE\n    Translate GNU Makefile to Prolog Makeprog syntax\n-W,--what-if,--new-file,--assume-new TARGET\n    Pretend that TARGET has been modified\n-o,--old-file,--assume-old TARGET\n    Do not remake TARGET, or remake anything on account of it\n-k,--keep-going \n    Keep going after error\n-S,--no-keep-going,--stop \n    Stop after error\n-t,--touch \n    Touch files (and update MD5 hashes, if appropriate) instead of running recipes\n-N,--no-dependencies \n    Do not test or rebuild dependencies\n-D,--define Var Val\n    Assign Makefile variables from command line\nVar=Val \n    Alternative syntax for '-D Var Val'\n-s,--quiet,--silent \n    Silent operation; do not print recipes as they are executed\n--one-shell \n    Run recipes in single shell (loosely equivalent to GNU Make's .ONESHELL)\n-y,--sync,--sync-dir URI\n    Synchronize current working directory to a remote URI. If no --sync-exec is specified, S3-form URIs (s3://mybucket/my/path) are handled using the AWS CLI tool; other URIs will be passed to rsync.\n-x,--sync-exec COMMAND\n    Specify executable for --sync.\n-H,--md5-hash \n    Use MD5 hashes instead of timestamps\n-C,--no-md5-cache \n    Recompute MD5 checksums whenever biomake is restarted\n-M,--no-md5-timestamp \n    Do not recompute MD5 checksums when timestamps appear stale\n-Q,--queue-engine ENGINE\n    Queue recipes using ENGINE (supported: poolq,sge,pbs,slurm,test)\n-j,--jobs JOBS\n    Number of job threads (poolq engine)\n--qsub-exec PATH\n    Path to qsub (sge,pbs) or sbatch (slurm)\n--qdel-exec PATH\n    Path to qdel (sge,pbs) or scancel (slurm)\n--queue-args 'ARGS'\n    Queue-specifying arguments for qsub/qdel (sge,pbs) or sbatch/scancel (slurm)\n--qsub-args,--sbatch-args 'ARGS'\n    Additional arguments for qsub (sge,pbs) or sbatch (slurm)\n--qsub-use-biomake,--sbatch-use-biomake \n    Force qsub/sbatch to always call biomake recursively\n--qsub-biomake-args,--sbatch-biomake-args 'ARGS'\n    Arguments passed recursively to biomake by qsub/sbatch (default: '-N')\n--qsub-header,--sbatch-header 'HEADER'\n    Header for qsub (sge,pbs) or sbatch (slurm)\n--qsub-header-file,--sbatch-header-file 'FILENAME'\n    Header file for qsub (sge,pbs) or sbatch (slurm)\n--qdel-args,--scancel-args 'ARGS'\n    Additional arguments for qdel (sge,pbs) or scancel (slurm)\n--flush,--qsub-flush \u003ctarget or directory\u003e\n    Erase all jobs for given target/dir\n-d \n    [developers] Print debugging messages. Equivalent to '--debug verbose'\n--debug MSG\n    [developers] Richer debugging messages. MSG can be verbose, bindrule, build, pattern, makefile, makeprog, md5...\n--trace PREDICATE\n    [developers] Print debugging trace for given predicate\n--no-backtrace \n    [developers] Do not print a backtrace on error\n```\n\nEmbedding Prolog in Makefiles\n-----------------------------\n\nBrief overview:\n\n- Prolog can be embedded within `prolog` and `endprolog` directives\n- `$(bagof Template,Goal)` expands to the space-separated `List` from the Prolog `bagof(Template,Goal,List)`\n- Following the target list with `{target_goal}` causes the rule to match only if `target_goal` is satisfied. The target goal will be tested _before_ any dependencies are built. The special variable `TARGET`, if used, will be bound to the target filename (i.e. `$@`)\n- Following the dependency list with `{deps_goal}` causes the recipe to be executed only if `deps_goal` is satisfied. The deps goal will be tested _after_ any dependencies are built (so it can examine the dependency files). The special variables `TARGET` and `DEPS`, if used, will be bound to the target and dependency-list (i.e. `$@` and `$^`, loosely speaking; except the latter is a true Prolog list, not encoded as a string with whitespace separators as in GNU Make)\n\nExamples\n--------\n\nThis assumes some knowledge of GNU Make and [Makefiles](https://www.gnu.org/software/make/manual/html_node/index.html).\n\nUnlike makefiles, biomake allows multiple variables in pattern\nmatching. Let's say we have a program called `align` that compares two\nfiles producing some output (e.g. biological sequence alignment, or\nontology alignment). Assume our file convention is to suffix \".fa\" on\nthe inputs.  We can write a `Makefile` with the following:\n\n    align-$X-$Y: $X.fa $Y.fa\n        align $X.fa $Y.fa \u003e $@\n\nNow if we have files `x.fa` and `y.fa` we can type:\n\n    biomake align-x-y\n\nProlog extensions allow us to do even fancier things with logic.\nSpecifically, we can embed arbitrary Prolog, including both database facts and\nrules. We can use these rules to control flow in a way that is more\npowerful than makefiles.\n\nLet's say we only want to run a certain program when the inputs match a certain table in our database.\nWe can embed Prolog in our Makefile as follows:\n\n    prolog\n    sp(mouse).\n    sp(human).\n    sp(zebrafish).\n    endprolog\n\n    align-$X-$Y: $X.fa $Y.fa {sp(X),sp(Y)}\n        align $X.fa $Y.fa \u003e $@\n\nThe lines beginning `sp` between `prolog` and `endprolog` define the set of species that we want the rule to apply to.\nThe rule itself consists of 4 parts:\n\n * the target (`align-$X-$Y`)\n * the dependencies (`$X.fa` and `$Y.fa`)\n * a Prolog goal, enclosed in braces (`{sp(X),sp(Y)}`), that is used as an additional logic test of whether the rule can be applied\n * the command (`align ...`)\n\nIn this case, the Prolog goal succeeds with 9 solutions, with 3\ndifferent values for `X` and `Y`. If we type...\n\n    biomake align-platypus-coelacanth\n\n...it will not succeed, even if the .fa files are on the filesystem. This\nis because the goal `{sp(X),sp(Y)}` cannot be satisfied for these two values of `X` and `Y`.\n\nTo get a list of all matching targets,\nwe can use the special BioMake function `$(bagof...)`\nwhich wraps the Prolog predicate [bagof/3](http://www.swi-prolog.org/pldoc/man?predicate=bagof/3).\nThe following example also uses the Prolog predicates\n[format/2](http://www.swi-prolog.org/pldoc/man?predicate=format/2)\nand\n[format/3](http://www.swi-prolog.org/pldoc/man?predicate=format/3),\nfor formatted output:\n\n~~~~\nprolog\n\nsp(mouse).\nsp(human).\nsp(zebrafish).\n\nordered_pair(X,Y) :- sp(X),sp(Y),X@\u003cY.\n\nmake_filename(F) :-\n  ordered_pair(X,Y),\n  format(atom(F),\"align-~w-~w\",[X,Y]).\n\nendprolog\n\nall: $(bagof F,make_filename(F))\n\nalign-$X-$Y: $X.fa $Y.fa { ordered_pair(X,Y),\n                           format(\"Matched ~w \u003c-- ~n\",[TARGET,DEPS]) },\n    align $X.fa $Y.fa \u003e $@\n~~~~\n\nNow if we type...\n\n    biomake all\n\n...then all non-identical ordered pairs will be compared\n(since we have required them to be _ordered_ pairs, we get e.g. \"mouse-zebrafish\" but not \"zebrafish-mouse\";\nthe motivation here is that the `align` program is symmetric, and so only needs to be run once per pair).\n\nIn these examples, the goals between braces are tested _after_ the dependencies.\nThis means that any Prolog code in these braces can safely examine the dependency files\n(for example, you could constrain a rule to apply only if a dependency file was below a certain size,\nor in a certain file format).\nYou can also place a Prolog goal (in braces) between the target list and the colon;\nit will then be tested after the target name has been matched,\nbut _before_ trying to build any dependencies.\nIn such a goal, you can use the `TARGET` variable but not the `DEPS` variable.\n\nProgramming directly in Prolog\n------------------------------\n\nIf you are a Prolog wizard who finds embedding Prolog in Makefiles too cumbersome, you can use a native Prolog-like syntax.\nBiomake looks for a Prolog file called `Makeprog` (or `Makespec.pro`) in your\ncurrent directory. (If it's not there, it will try looking for a\n`Makefile` in GNU Make format. The following examples describe the\nProlog syntax.)\n\nAssume you have two file formats, \".foo\" and \".bar\", and a `foo2bar`\nconverter.\n\nAdd the following rule to your `Makeprog`:\n\n    '%.bar' \u003c-- '%.foo',\n        'foo2bar $\u003c \u003e $@'.\n\nUnlike makefiles, whitespace is irrelevant (except inside the quote-delimited shell and filesystem expressions). However, you\ndo need the quotes, and remember the closing \".\",\nas this is Prolog syntax.\n\nIf you prefer to stick with GNU Make syntax,\nthe above `Makeprog` is equivalent to the following `Makefile`:\n\n    %.bar: %.foo\n    \t   foo2bar $\u003c \u003e $@\n\nTo convert a pre-existing file \"x.foo\" to \"x.bar\" type:\n\n    biomake x.bar\n\nLet's say we can go from a .bar to a .baz using a `bar2baz`\nconverter. We can add an additional rule:\n\n    '%.baz' \u003c-- '%.bar',\n        'bar2baz $\u003c \u003e $@'.\n\nNow if we type...\n\n    touch x.foo\n    biomake x.baz\n\n...we get something like the following output:\n\n~~~~\n% Checking dependencies: x.baz \u003c-- [x.bar]\n%  Checking dependencies: x.bar \u003c-- [x.foo]\n%   Nothing to be done for x.foo\n%  Target x.bar not materialized - build required\n foo2bar x.foo \u003e x.bar\n%  x.bar built\n% Target x.baz not materialized - build required\nbar2baz x.bar \u003e x.baz\n% x.baz built\n~~~~\n\nThe syntax in the makeprog above is designed to be similar to the automatic variable syntax\nalready used in makefiles. You can bypass this and use Prolog\nvariables. The following form is functionally equivalent:\n\n    '$(Base).bar' \u003c-- '$(Base).foo',\n        'foo2bar $(Base).foo \u003e $(Base).bar'.\n\nThe equivalent `Makefile` would be this...\n\n    $(Base).bar: $(Base).foo\n    \tfoo2bar $(Base).foo \u003e $(Base).bar\n\n...although strictly speaking, this is only equivalent if you are using Biomake;\nGNU Make's treatment of this Makefile isn't quite equivalent, since unbound variables\ndon't work the same way in GNU Make as they do in Biomake\n(Biomake will try to use them as wildcards for pattern-matching,\nwhereas GNU Make will just replace them with the empty string - which is also the default behavior\nfor Biomake if they occur outside of a pattern-matching context).\n\nFollowing the GNU Make convention, variable names must be enclosed in\nparentheses unless they are single letters.\n\nAutomatic translation to Prolog\n-------------------------------\n\nYou can parse a GNU Makefile (including Biomake-specific extensions, if any)\nand save the corresponding Prolog syntax using the `-T` option\n(long-form `--translate`).\n\nHere is the translation of the Makefile from the previous section (lightly formatted for clarity):\n\n~~~\nsp(mouse).\nsp(human).\nsp(zebrafish).\n\nordered_pair(X,Y):-\n sp(X),\n sp(Y),\n X@\u003cY.\n\nmake_filename(F):-\n ordered_pair(X,Y),\n format(atom(F),\"align-~w-~w\",[X,Y]).\n\n\"all\" \u003c-- \"$(bagof F,make_filename(F))\".\n\n\"align-$X-$Y\" \u003c--\n [\"$X.fa\",\"$Y.fa\"],\n {ordered_pair(X,Y),\n  format(\"Matched ~w \u003c-- ~n\",[TARGET,DEPS])},\n \"align $X.fa $Y.fa \u003e $@\".\n~~~\n\nNote how the list of dependencies in the second rule, which contains more than one dependency (`$X.fa` and `$Y.fa`), is enclosed in square brackets, i.e. a Prolog list (`[\"$X.fa\",\"$Y.fa\"]`).\nThe same syntax applies to rules which have lists of multiple targets, or multiple executables.\n\nThe rule for target `all` in this translation involves a call to the Biomake function `$(bagof ...)`,\nbut (as noted) this function is just a wrapper for the Prolog `bagof/3` predicate.\nThe automatic translation is not smart enough to remove this layer of wrapping,\nbut we can do so manually, yielding a clearer program:\n\n~~~\nsp(mouse).\nsp(human).\nsp(zebrafish).\n\nordered_pair(X,Y):-\n sp(X),\n sp(Y),\n X@\u003cY.\n\nmake_filename(F):-\n ordered_pair(X,Y),\n format(atom(F),\"align-~w-~w\",[X,Y]).\n\n\"all\", {bagof(F,make_filename(F),DepList)} \u003c-- DepList, {true}.\n\n\"align-$X-$Y\" \u003c--\n [\"$X.fa\",\"$Y.fa\"],\n {ordered_pair(X,Y),\n  format(\"Matched ~w \u003c-- ~n\",[TARGET,DEPS])},\n \"align $X.fa $Y.fa \u003e $@\".\n~~~\n\nMake-like features\n------------------\n\nBiomake supports most of the functionality of GNU Make, including\n- different [flavors of variable](https://www.gnu.org/software/make/manual/make.html#Flavors) (recursive, expanded, etc.)\n- various ways of [setting variables](https://www.gnu.org/software/make/manual/html_node/Setting.html)\n- [appending to variables](https://www.gnu.org/software/make/manual/html_node/Appending.html)\n- [multi-line variables](https://www.gnu.org/software/make/manual/html_node/Multi_002dLine.html)\n- [automatic variables](https://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html) such as `$\u003c`, `$@`, `$^`, `$(@F)`, etc.\n- [substitution references](https://www.gnu.org/software/make/manual/html_node/Substitution-Refs.html)\n- [computed variable names](https://www.gnu.org/software/make/manual/html_node/Computed-Names.html)\n- [all the text functions](https://www.gnu.org/software/make/manual/html_node/Text-Functions.html)\n- [all the filename functions](https://www.gnu.org/software/make/manual/html_node/File-Name-Functions.html)\n- [the shell function](https://www.gnu.org/software/make/manual/html_node/Shell-Function.html)\n- [user-defined functions](https://www.gnu.org/software/make/manual/html_node/Call-Function.html)\n- [errors and warnings](https://www.gnu.org/software/make/manual/html_node/Make-Control-Functions.html)\n- many of the same [command-line options](https://www.gnu.org/software/make/manual/html_node/Options-Summary.html)\n- [conditional syntax](https://www.gnu.org/software/make/manual/html_node/Conditionals.html) and [conditional functions](https://www.gnu.org/software/make/manual/html_node/Conditional-Functions.html)\n- the [include](https://www.gnu.org/software/make/manual/html_node/Include.html) directive\n- [wildcards in dependency lists](https://www.gnu.org/software/make/manual/html_node/Wildcards.html)\n- [phony targets](https://www.gnu.org/software/make/manual/html_node/Phony-Targets.html)\n- various other quirks of GNU Make syntax e.g. single-line recipes, forced rebuilds\n\nCurrently unsupported features of GNU Make\n------------------------------------------\n\nThe following features of GNU Make are not (yet) implemented:\n\n- [Order-only prerequisites](https://www.gnu.org/software/make/manual/html_node/Prerequisite-Types.html)\n- [Directory search](https://www.gnu.org/software/make/manual/html_node/Directory-Search.html)\n- Many of the [special built-in targets](https://www.gnu.org/software/make/manual/html_node/Special-Targets.html), with some exceptions:\n    - `.PHONY` is implemented\n    - `.SILENT` is implemented\n    - `.NOTPARALLEL` is implemented\n    - `.ONESHELL` is implemented\n    - `.IGNORE` is implemented\n    - `.DELETE_ON_ERROR` is implemented\n    - `.SECONDARY` is implicit and `.INTERMEDIATE` is unsupported: Biomake never removes intermediate files (unless `.DELETE_ON_ERROR` is specified)\n    - `.PRECIOUS` is implicit for all targets\n    - `.SECONDEXPANSION` is implicit\n    - `.SUFFIXES` is unsupported (or implicit with no dependencies), since suffix rules are unsupported\n    - other special targets not mentioned in the above list are not supported (they'll just be parsed as regular targets, i.e. ignored)\n- [Multiple rules per target](https://www.gnu.org/software/make/manual/html_node/Multiple-Rules.html)\n- [Static pattern rules](https://www.gnu.org/software/make/manual/html_node/Static-Pattern.html)\n- [Double-colon rules](https://www.gnu.org/software/make/manual/html_node/Double_002dColon.html)\n- [Suffix rules](https://www.gnu.org/software/make/manual/html_node/Suffix-Rules.html)\n- Modifiers in recipe lines are only partially supported:\n    - The [+ sign to force execution during dry runs](https://www.gnu.org/software/make/manual/html_node/Instead-of-Execution.html) is _not_ supported\n    - The [- sign to suppress errors in recipes](https://www.gnu.org/software/make/manual/html_node/Errors.html) _is_ supported\n    - The [@ sign to execute recipe lines silently](https://www.gnu.org/software/make/manual/html_node/Echoing.html) _is_ supported\n- The [export](https://www.gnu.org/software/make/manual/html_node/Variables_002fRecursion.html) keyword is supported, but \"unexport\" and \"override\" are _not_ supported\n- [Target-specific variable assignments](https://www.gnu.org/software/make/manual/html_node/Target_002dspecific.html) (see issue [#79](https://github.com/evoldoers/biomake/issues/79))\n\nPlease [submit a GitHub issue](https://github.com/evoldoers/biomake/issues) if any of these are important to you.\n\nOther differences from GNU Make\n-------------------------------\n\nThere are slight differences in the way variables are expanded, which arise from the fact that Biomake\ntreats variable expansion as a post-processing step (performed at the last possible moment) rather than a pre-processing step (which is how GNU Make does it - at least partly).\n\nSpecifically, Biomake parses the Makefile, reading all variable and recipe declarations into memory, and only when the build begins are variables expanded.\nThe only exception to this is when variables are used in [conditional syntax](https://www.gnu.org/software/make/manual/html_node/Conditionals.html),\nto control which parts of the Makefile are actually read:\nthese variables are expanded at parse-time.\n\nIn contrast, GNU Make expands variables in dependency lists at parse time (along with conditional syntax),\nbut expands variables in recipe bodies later.\n\nThis can cause differences between GNU and Biomake in situations where variables change value throughout the Makefile.\nThese situations are usually counter-intuitive anyway, as the following example illustrates.\nThis Makefile, which might naively be expected to print `hello everybody`,\nin fact prints `hello world` when run with `make test`, but `goodbye world` when run with `biomake test`:\n\n~~~~\nA = hello\nB = everybody\n\ntest: $A\n\t@echo $B\n\nA = goodbye\nB = world\n\nhello goodbye:\n\t@echo $@\n~~~~\n\nThis example gets even more counterintuitive if we wrap the `test` recipe with conditional syntax.\nIt still gives the same results, though: `hello world` when run with `make test`, and `goodbye world` when run with `biomake test`.\n\n~~~~\nA = hello\nB = everybody\n\nifeq ($B,everybody)\ntest: $A\n\t@echo $B\nelse\ntest:\n\t@echo Curioser and curioser\nendif\n\nA = goodbye\nB = world\n\nhello goodbye:\n\t@echo $@\n~~~~\n\nAnother consequence is that, when using Biomake, variable expansions must be aligned with the overall syntactic structure; they cannot span multiple syntactic elements.\nAs a concrete example, GNU Make allows this sort of thing:\n\n~~~~\nRULE = target: dep1 dep2\n$(RULE) dep3\n~~~~\n\nwhich (in GNU Make, but not biomake) expands to\n\n~~~~\ntarget: dep1 dep2 dep3\n~~~~\n\nThat is, the expansion of the `RULE` variable spans both the target list and the start of the dependency list.\nTo emulate this behavior faithfully, Biomake would have to do the variable expansion in a separate preprocessing pass - which would mean we couldn't translate variables directly into Prolog.\nWe think it's worth sacrificing this edge case in order to maintain the semantic parallel between Makefile variables and Prolog variables, which allows for some powerful constructs.\n\nThe implementation of [conditional syntax](https://www.gnu.org/software/make/manual/html_node/Conditionals.html)\n(`ifeq`, `ifdef` and the like) similarly requires that syntax to be aligned with the overall structure:\nyou can only place a conditional at a point where a variable assignment, recipe, or `include` directive could go\n(i.e. at the top level of the `Makefile` grammar).\nConditional syntax _is_ implemented as a preprocessing step.\n\nUnlike GNU Make, Biomake does not offer domain-specific language extensions in [Scheme](https://www.gnu.org/software/guile/)\n(even though this is one of the cooler aspects of GNU Make), but you can program it in Prolog instead - it's quite hackable.\n\nDetailed build logic\n--------------------\n\nThe build logic for biomake should _usually_ yield the same results as GNU Make, though there may be subtle differences.\nThe GNU Make [algorithm](https://www.gnu.org/software/make/manual/html_node/Implicit-Rule-Search.html) differs in the details.\n\nBefore attempting to build a target `T` using a rule `R`, Biomake performs the following steps:\n- It tries to match the target name `T` to one of the target names in `R`\n- It tests whether the Prolog _target goal_ (if there is one) is satisfied\n- It checks whether there is a _theoretical path_ to all the dependencies. A theoretical path to a dependency `D` exists if either of the following is true:\n    - There is a rule that could be used to build `D`, the target goal for that rule is satisfied, and there is a theoretical path to all the dependencies of that rule;\n    - File `D` already exists, and the only applicable rules to rebuild `D`, if any exist at all, are wildcard (pattern) rules; that is, there are no rules that _explicitly and uniquely_ rebuild `D`.\n- It attempts to build all the dependencies\n- It tests whether the Prolog _deps goal_ (if there is one) is satisfied\n- It tests whether the target is stale. Details depend on the various options:\n    - Command-line options for marking targets as stale or new (`-W`, `-B`, `-o`) can override any of the following behavior\n    - If using the queueing engine, or if doing a dry-run (`-n`), targets are flagged as stale if any of their dependency tree has been rebuilt (or submitted to the queue for a rebuild);\n    - If using MD5 signatures (and _not_ the queueing engine), a target is stale if its MD5 checksum appears to be out of date;\n    - If using MD5 _and_ queues, the MD5 signature will not be checked until the queueing engine executes the job (which is guaranteed to happen after any dependencies are rebuilt). Otherwise the dependencies might change after the MD5 checksum was tested. This is accomplished by wrapping the recipe script with a recursive call to biomake; so biomake has to be available on the worker machines, and not just the cluster head. (The same is true, incidentally, when using a cluster to execute any rule that has a Prolog deps goal: the submitted job is wrapped by biomake, in order that the goal can be tested after the dependencies are built.)\n    - Otherwise (no queues and no MD5), Biomake looks at the file timestamps and/or the dependency tree.\n\nIf any of these tests fail, Biomake will backtrack and attempt to build the target using a different rule, or a different pattern-match to the same rule.\nIf all the tests pass, Biomake will commit to using the rule, and will attempt to execute the recipe using the shell (or the queueing engine).\n\nNote that the target goal is tested multiple times (to plan theoretical build paths) and so should probably not have side effects.\nThe deps goal is tested later, and only once for every time the rule is bound, so it is a bit safer for the deps goal to have side effects.\n\nFailure during execution of the recipe (or execution of any recipes in the dependency tree) will never cause Biomake to backtrack; it will either halt, or (if the `-k` command-line option was specified) soldier on obliviously.\n\nArithmetic functions\n--------------------\n\nBiomake provides a few extra functions for arithmetic on lists:\n\n- `$(iota N)` returns a space-separated list of numbers from `1` to `N`\n- `$(iota S,E)` returns a space-separated list of numbers from `S` to `E`\n- `$(add X,L)` adds `X` to every element of the space-separated list `L`\n- `$(multiply Y,L)` multiplies every element of the space-separated list `L` by `Y`\n- `$(divide Z,L)` divides every element of the space-separated list `L` by `Z`\n\nMD5 hashes\n----------\n\nInstead of using file timestamps, which are fragile (especially on networked filesystems),\nBiomake can optionally use MD5 checksums to decide when to rebuild files.\nTurn on this behavior with the `-H` option (long form `--md5-hash`).\n\nBiomake uses the external program `md5` to do checksums (available on OS X), or `md5sum` (available on Linux).\nIf neither of these are found, Biomake falls back to using the SWI-Prolog md5 implementation;\nthis does however require loading the entire file into memory (which may be prohibitive for large files).\n\nQueues\n------\n\nTo run jobs in parallel, locally or on a cluster, you need to specify a queueing engine\nusing the `-Q` option (long form `--queue-engine`). Note that, unlike with GNU Make, multi-threading is not activated\nsimply by specifying the number of threads with `-j`; you need `-Q` as well.\n\nThere are several queueing engines currently supported:\n\n- `-Q poolq` uses an internal thread pool for running jobs in parallel on the same machine that `biomake` is running on\n- `-Q sge` uses [Sun Grid Engine](https://en.wikipedia.org/wiki/Oracle_Grid_Engine) or compatible (e.g. [Open Grid Scheduler](http://gridscheduler.sourceforge.net/))\n- `-Q pbs` uses [PBS](https://en.wikipedia.org/wiki/Portable_Batch_System)\n- `-Q slurm` uses [Slurm](https://slurm.schedmd.com/)\n- `-Q test` just runs the jobs synchronously. Used for testing purposes only\n\nFor Sun Grid Engine, PBS and Slurm, the paths to the relevant job control executables, and any arguments to those executables\n(such as the name of the queue that jobs should be run on), can be controlled using various command-line arguments.\nIn particular, the `--qsub-args` command-line option (applying to all recipes)\nand the `QsubArgs` Prolog variable (on a per-recipe basis, in the target goal)\ncan be used to pass parameters such as the queue name.\n\nHere's an example of using `QsubArgs`:\n\n~~~~\nmy_target { QsubArgs = '--cores-per-socket=4' } : my_dependency\n    do_something \u003e$@\n~~~~\n\nNote that `QsubArgs` has to be set in the target goal, not the deps goal\n(since the job is submitted to the queueing engine before the dependencies are guaranteed to have been built).\n\nSimilarly, you can use the `QsubHeader` variable (or the `--qsub-header` command-line option) to add header lines to the wrapper script that is submitted to the queue engine\n(for example, to provide queue configuration directives),\nor you can use `QsubHeaderFile` (or `--qsub-header-file`) to specify the filename of a header file to include.\n\nThe names of these Prolog variables for fine-grained queue configuration (`QsubArgs`, `QsubHeader`, `QsubHeaderFile`) are the same for Slurm as for SGE and PBS,\neven though the batch submission command for Slurm is `sbatch` and not `qsub`.\n\nMore\n----\n\nIdeas for future development:\n\n* a web-based build environment (a la Galaxy)\n* semantic web enhancement (using NEPOMUK file ontology)\n* using other back ends and target sources (sqlite db, REST services)\n* cloud-based computing\n* metadata\n","funding_links":[],"categories":["Build Systems"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevoldoers%2Fbiomake","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fevoldoers%2Fbiomake","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevoldoers%2Fbiomake/lists"}