Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/mobily-enterprises/docco-next

Docco-compatible literate documentation generator, written in ES6
https://github.com/mobily-enterprises/docco-next

Last synced: 6 months ago
JSON representation

Docco-compatible literate documentation generator, written in ES6

Awesome Lists containing this project

README 

        

# Docco Next



[Docco Next](https://github.com/mobily-enterprises/docco-next) facilitates

[literate programming](https://en.wikipedia.org/wiki/Literate_programming)

in several languages. It's written in modern Javascript, and runs in Node.



See the [generated documentation as HTML](https://mobily-enterprises.github.io/docco-next/)



Install and use

---------------



To use Docco Next run `npm install -g docco-next` and run it passing it a list of files

(e.g. `docco src/*js`)



By default, every file will be converted into formatted HTML and will be placed in the

default destination directory (`docs` by default). It's also possible to

convert files to Markdown, rather than HTML. Note that files will

retain their original names and will change their extension to `html`.



Is it literate programming?

---------------------------



Formally speaking, literate programs put documentation strictly first.



In simple words, a literate file will look like a Markdown file, and its

code is whichever Markdown code is included in the file.

This means that anything that is not Markdown code (and is therefore indented in) is

considered documentation, and programs are organised so that the documentation

actually makes sense.



A notable example is CoffeeScript, which supports .litcoffee files natively.



However, not many languages support literate programming. Most languages will

need a proprocessor, for example to convert the `.js.md` file into `.js`.

[lit](https://github.com/vijithassar/lit) is one of such tools. In Node, you

can use Rich Harris's [lit-node](https://github.com/Rich-Harris/lit-node)

to require .md files directly.



Docco Next works with normal source code (such as `.js` or `.c`) as well as literate

source code (such as literate CoffeeScript, `.litcoffee`, or `.js.md`).



With pure literate style, all you have to do is add the .md extension to your

source file.



When processing source files (such as `.js` or `.c`), one line comments are considered

documentation and are parsed using Markdown. The code (that is, anything that is

not a one-line comment)  is processed by [Shiki](https://shiki.matsu.io/).



In both cases, the end result is a set of HTML pages with your documentation.



Similar projects

------------------



Here is a list of _active_ projects which achieve similar goals:



* [Docco](https://github.com/jashkenas/docco) by Jeremy Ashkenas (the same

wise author of Underscore and Backbone). The program that

inspired many others (including me) to use literate programming techniques and

implement code that facilitates it



* [pycco](https://github.com/pycco-docs/pycco) by Nick Fitzgerald. A Python

implementation of literate programming



* [Marginalia](https://github.com/gdeer81/marginalia) by Gary Deer. A **Clojure**

implementation of literate programing. The project was originally started by Michael Fogus.



This file is the full source code of Docco Next, and it explains how it works in

pure literate style.



Required files and starting configuration

-----------------------------------------



The following libraries are required by Docco Next:

* `ejs`. Used to convert the layout master file into HTML

* `globby`. Used to [glob](https://en.wikipedia.org/wiki/Glob_(programming)) for filenames when the shell doesn't otherwise support this

* `fs-extra`. Used for all I/O operations

* `marked`. Used to convert Markdown into HTML

* `commander`. Used to interpret command line parameters

* `shiki`. Used to highlight source code



In Javascript terms, this becomes:



```javascript

const path = require('path')

const ejs = require('ejs')

const globby = require('globby')

const fs = require('fs-extra')

const marked = require('marked').marked

const commander = require('commander')

const shiki = require('shiki')



```



On startup, the `version` variable is worked out straight from the `package.json`

file, which is loaded using `require`



```javascript

const pkg = require('./package.json')

const version = pkg.version



```



Docco Next manages several languages, stored in `layouts/languages.json`.

Users can add more languages if desired. However, the list of default languages

(and their relevant information) is stored in `layouts/languages.json`



```javascript

const defaultLanguages = require('./layouts/languages.json')



```



By default, `marked` is run with very basic options (basically, just turning

`smartypants` on). Users can add more options, but this is the starting point



```javascript

const defaultMarkedOptions = { smartypants: true }



```



Configuration

---------------



The `run` function is the entry point of the script when it's run by the

command line (that is, it's not being used as a library).

The `docco` command in `bin` simply runs `require('../docco.js').run()`.



The package `commander` will be used to parse command-line parameters.

Commander will generate a configuration object based on how it's configured

using its `option` method.



For example the line:



    .option('-L, --languages [file]', 'use a custom languages.json')



Means that if Docco Next is run with `--languages ./some/file.json`, the config

object will include `{ languages: "./some/file.json"}`.



Commander also provides the handy `helpInformation()` method, which will print

out how the command is used.



This is the full set of options available in Docco Next:



```javascript

async function run (args = process.argv) {

  commander

    .name('docco')

    .version(version)

    .usage('[options] files')

    .option('-L, --languages [file]', 'use a custom languages.json')

    .option(

      '-l, --layout [name]',

      'choose a layout (default, parallel or classic)'

    )

    .option('-s, --shiki-theme [shikiTheme]', 'choose a shiki theme')

    .option('-o, --output [path]', 'output to a given folder')

    .option('-c, --css [file]', 'use a custom css file')

    .option('-p, --plugin [file]', 'use a custom plugin file')

    .option('-t, --template [file]', 'use a custom .ejs template')

    .option(

      '-e, --inputExtension [ext]',

      'assume a file extension for all inputs'

    )

    .option('-m, --marked [file]', 'use custom marked options')

    .option(

      '-x, --outputExtension [ext]',

      'set default file extension for all outputs'

    )

    .parse(args)

  if (commander.args.length) {

    const config = { ...commander.opts(), args: commander.args }

    await cmdLineNormalise(config)

    configure(config)

    await cmdLineSanityCheck(config)



    await documentAll(config)

  } else {

    return console.log(commander.helpInformation())

  }

}



```



Three configuration functions are called:



* `cmdLineNormalise()` -- expands some command line options to objects

* `configure()` -- enriches configuration options with full paths etc.

* `cmdLineSanityCheck()` -- checks that files and folders actually exist.



Here is an explanation of what each one does, followed by their source code.



### **Step 1: `cmdLineNormalise()`**



It's important to understand that Docco Next can be used as a library as well as

a command line program.

Two of the command line options, `marked` and `languages`, are external JSON

files. When used as a library, Docco Next will expect `marked` and `languages` to be

objects. However, when run as a command line program, those options will

contain a string with a (JSON) file name instead.



The `cmdLineNormalise()` function is used to convert those strings into

Javascript objects which will depend on the contents of the corresponding

JSON files

The fuction also assigns `config.args` (which is the list of files to be

converted) to `config.sources`.



To sum up: when using Docco Next as a library, `config.languages` and `config.maked`

will need to be objects. However, from the command line, they will be the paths of

JSON files, and will be converted to objects by `cmdLineNormalise()`, which

reads:



```javascript

async function cmdLineNormalise (config) {

  if (config.languages) {

    if (!(await fileExists(config.languages))) {

      console.error('Languages file not found:', config.languages)

      process.exit(5)

    }

    const languages = await fs.readFile(config.languages)

    config.languages = JSON.parse(languages)

  }



  if (config.plugin) {

    if (!(await fileExists(config.plugin))) {

      console.error('Plugin file not found:', config.plugin)

      process.exit(5)

    }

    config.plugin = require(path.join(process.cwd(), config.plugin))

  } else {

    config.plugin = {}

  }



  if (!config.outputExtension) config.outputExtension = 'html'



  if (config.marked) {

    if (!(await fileExists(config.marked))) {

      console.error('Marked file not found:', config.marked)

      process.exit(6)

    }

    const marked = await fs.readFile(config.marked)

    config.marked = JSON.parse(marked)

  }



  config.sources = await globby(config.args)

}



```



### **Step 2: `configure()`**



This function is different to the other two `cmdLineNormalise()` and

`cmdLineSanityCheck()`: rather than being a command line-only normalisation

function, it's actually a function that is run _every time_ an API call is

invoked. (Note that once it's run once, it sets the `config.configured`

property in the `config` object, and will check this property so that it will

ever do anything only the first time it's invoked)



This function is responsible to set sane defaults in the `config` property,

so that each function doesn't need to worry with wasteful checking.



For example `config.output` will be set as `docs` by default.



Also, for passed properties such as `languages` or `marked`, this function

makes sure that the passed options are _added_ to sane defaults. For example

by passing the `marked` property in the config object, this function makes

sure that the default `smartypants: true` is on, and any configuration

options are added on _top_ of what the user has passed. Or, that the

config.languages contains the used-defined languages, _as well as_ the

default ones.



This function also sets some indirect configuration options that are not meant

to be passed by the developer, but that are a result of the configuration.



Finally, this function also makes sure that the passed config makes sense:

`config.sources` is scanned and any source with an unknown language (that is,

a language not in the `languages` array) is filtered out (with a warning).



You can see this function as insurance that there is a solid, valid `config`

object that every call can use. This is why every single function that uses

`config` will call `configure` first.



```javascript

function configure (config) {

  if (config.configured) return

  config.configured = true



  config.output = config.output || 'docs'



  config.languages = properObjectWithKeys(config.languages)

    ? { ...defaultLanguages, ...config.languages }

    : defaultLanguages



  config.marked = properObjectWithKeys(config.marked)

    ? { ...defaultMarkedOptions, ...config.marked }

    : defaultMarkedOptions



  config.layout = config.layout || 'default'

  if (config.layout.match(/^[a-zA-Z0-9]+$/)) {

    config.layout = path.join(__dirname, 'layouts', config.layout)

  }



  if (!config.css) config.css = path.join(config.layout, 'docco.css')



  config.public = path.join(config.layout, 'public')



  if (!config.template) {

    config.template = path.join(config.layout, 'docco.ejs')

  }



  config.sources = config.sources || {}



  config.sources = config.sources.filter((source) => {

    const there = getLanguage(source, config)

    if (!there) {

      console.warn(

        `docco: file not processed, language not supported: (${path.basename(

          source

        )})`

      )

    }

    return there

  })

}



```



### **Step 3: `cmdLineSanityCheck()`**



The `cmdLineSanityCheck()` is a command line-only function, which

is used to make sure that paths specified in the `config` object

have corresponding files or directories.



For example `config.output` represents the directory where all output files

will be written. If that directory doesn't exist, Docco Next (as run from the command

line) will refuse to work.

The same applies to `config.layout` (the path to the layout, which is effectively

the "theme" used), `config.css` (the alternative CSS file used), and all files

specified in the `sources` array.



```javascript

async function cmdLineSanityCheck (config) {

  if (config.output && !(await dirExists(config.output))) {

    console.error('Output directory not found:', config.output)

    process.exit(1)

  }

  if (config.layout && !(await dirExists(config.layout))) {

    console.error('Layout directory not found:', config.layout)

    process.exit(2)

  }



  /*

  if (config.css && !await fileExists(config.css)) {

    console.error('CSS file not found:', config.css)

    process.exit(3)

  }

  */



  if (config.sources) {

    for (const source of config.sources) {

      if (source && !(await fileExists(source))) {

        console.error('source file not found:', source)

        process.exit(5)

      }

    }

  }

}



```



Going through each file via documentOne() and documentAll()

-----------------------------------------------------------



Up to this point, it's all been about preparing the ground for the program

to run: processing of command line options, sanitising the configuration object,

and so on.



It's time to get to the actual work: scanning the `sources` array, and actually

get the documentation create and copied, which happens thanks to the

calls `documentAll()` and `documentOne()`.



First of all, some important file-related utility functions are created -- some

of them are very generic, while some others are very specific to Docco Next.



The functions are:



* `dirExists()` -- checks if a directory exists

* `fileExists()` -- checks if a file exists

* `finalPath()` -- works out the final path of a file, depeding of `config.output`

  and allowing a change of extension

* `copyAsset()` -- copies a file or a directory into `config.output`, preserving

  the path relative to the destination. So `some/other/file.html` will be copied

  to `docs/some/other/file.html`

* `write()` -- simply writes a file to its destination



There are also some utility functions that are not file-related:



* `properObjectWithKeys()` -- returns `true` if the passed variable

  is an proper (not-null) object that is not empty

* `getLanguage(source, config)` - TODO: explain what it actually does



Here is the code for those functions:



```javascript

async function dirExists (dir) {

  if (await fs.pathExists(dir)) {

    const stat = await fs.lstat(dir)

    return stat.isDirectory()

  }

  return false

}



async function fileExists (dir) {

  if (await fs.pathExists(dir)) {

    const stat = await fs.lstat(dir)

    return stat.isFile() || stat.isSymbolicLink()

  }

  return false

}



function finalPath (source, config) {

  const ext = config.outputExtension

  return path.join(

    config.output,

    path.dirname(source),

    path.basename(source, path.extname(source)) + '.' + ext

  )

}



async function copyAsset (file, type, config = {}) {

  configure(config)

  if (!file) return

  if (type === 'file' && !(await fileExists(file))) return

  if (type === 'directory' && !(await dirExists(file))) return

  return fs.copy(file, path.join(config.output, path.basename(file)))

}



async function write (source, path, contents) {

  console.log(`docco: ${source} -> ${path}`)

  await fs.outputFile(path, contents)

}



function properObjectWithKeys (o) {

  return typeof o === 'object' && o !== null && Object.keys(o).length

}



function getLanguage (source, config = {}) {

  configure(config)



  let codeExt, codeLang, lang



  const ext =

    config.inputExtension || path.extname(source) || path.basename(source)

  lang = config.languages[ext]

  if (!lang) return

  if (lang.name === 'markdown') {

    codeExt = path.extname(path.basename(source, ext))

    if (codeExt) {

      codeLang = config.languages[codeExt]

      if (codeLang) {

        lang = { ...codeLang, literate: true }

      }

    }

  }

  /* Add commentMatcher */

  lang.commentMatcher = RegExp(`^\\s*${lang.symbol}\\s?`)

  /* Add commentFilter */

  /* Ignore [hashbangs](http://en.wikipedia.org/wiki/Shebang_%28Unix%29) and interpolations... */

  lang.commentFilter = /(^#![/]|^\s*#\{)/



  return lang

}



```



Now that everything _really is_ ready, it's time to (finally!) go through

the `source` array and run `documentOne()` on each one of them. Also,

if the sources are being converted to HTML, the `css` file and the `public` directory

for that layout are copied over using the `copyAsset` utility function



Note that `configure()` is run. Since `configure()` only ever does anything

the first time it's run, and since it was already run in the `run()` function,

it may seem superfluous to run it again. However, keep in mind that

Docco Next exports an API. So, every call that receives `config` will always

run `configure(config)` to make sure that sane defaults are set.



```javascript

async function documentAll (config = {}) {

  configure(config)

  await fs.mkdirs(config.output)



  for (const source of config.sources) {

    await documentOne(source, config)

  }



  await copyAsset(config.css, 'file', config)

  await copyAsset(config.public, 'directory', config)

}



```



`documentOne` is the centrepiece of the program: it uses the important

parsing and manipulation functions (which are also available as API) to

write the HTML file to the specified output path (`docs` by default)

This function will:



* Read a file

* figures out which language it's written in

* If a language is literate (see: `.litcoffee` or `.md`), convert it to

  code first using `litToCode()`. This is to ensure that `parse()` (the next

  call) always receives "normal", runnable code (which is what it expects)

* Run `parse()` which will parse the file into an array of `sections`, where

  each element has the properties `docsText` and `codeText`

* Run `formatAsHtml()` which  returns a formatted, finalised, ready-to-go

  HTML string based on the array of sections passed

* Write the file to its destination directory



Here is the source code:



```javascript

async function documentOne (source, config = {}) {

  configure(config)

  /* console.log(source) */



  const buffer = await fs.readFile(source)

  let lines = buffer.toString().split('\n')

  const path = finalPath(source, config)



  config.lang = getLanguage(source, config)

  if (!config.lang) {

    console.warn(

      `docco: file not processed, language not supported: (${path.basename(

        source

      )})`

    )

    return

  }

  if (config.lang.literate) {

    lines = litToCode(lines, config)

  }

  const sections = parse(source, lines, config)



  const result = await formatAsHtml(source, sections, config)



  await write(source, path, result)

}



```



The actual parsing and manipulation

-------------------------------------



As explained above, `documentOne()` is Docco Next's centrepiece, calling

`litToCode()`, `parse()`, `formatAsHtml`, and finally `write()`.



These functions are indeed the core functionalities of Docco Next.



## litToCode()



This function takes literal code and returns "proper" source code that is

ready to be executed. It does this by simply de-intending Markdown-indented

code, and adding a comment marker to any other line. So, for example:



    This is a literate .md file that contains code. This is the full

    extent of the program:



        console.log("Hello world!")



    This is it!



Is transformed into actually executable code:



    # This is a literate .md file that contains code. This is the full

    # extent of the program:



    console.log("Hello world!")



    # This is it!



This step will only be taken in cases where the input file is

literate code, which is true for those languages in `languages.json` where

the flag `literate` is set to true, _or_ when the file is

for example `something.js.md` -- that is, a Markdown file that

contains Javascript



```javascript

function litToCode (lines, config) {

  configure(config)



  const lang = config.lang

  const retLines = []

  const markdownIndented = /^([ ]{4}|[ ]{0,3}\t)/

  let inCode = lines[0] && markdownIndented.exec(lines[0])



  /** Remembering that a source code line are those without leading "   ":

   * add a comment marker at the beginning of each non-code line

   * Take out the leading "    " from every code line

   **/

  for (const [i, line] of lines.entries()) {

    /**

      Empty lines are a special case:

        - in code, they must stay blank.

        - In doc, they myst include the comment marker

    **/

    const emptyLine = /^\s*$/.test(line)

    /* Non-empty lines are added depending on them being code or doc. */

    if (!emptyLine) {

      inCode = markdownIndented.exec(line)

      retLines[i] = inCode

        ? line.slice(inCode[0].length)

        : lang.symbol + ' ' + line

      /* The concept of "empty" changes whether we are in code or not */

    } else {

      retLines[i] = inCode ? '' : lang.symbol

    }

  }

  return retLines

}



```



## parse()



This function takes a string representing the source code, and returns

an array of `sections`. Each section has two properties: `docsText` (the

documentation text, formatted with Markdown) and `codeText` (the code

displayed underneath the text in `docsText`).



Basically, each "section" is made up of two chunks: one formatted as

Markdown, _followed_ by a piece of code.



There are two main cases: one for non-empty lines, and one for empty lines.



Non-empty lines can be either "comments" (which will end up in `docsText`) or

code (which will end up in `codeText`). It's crucial to understand that

the flow works by adding comment lines to `docsText`, then (once a code line

is found) adding code lines to `codeText`, and when a comment line is

found again a new section is created, the two variables are reset, and everythig

starts again.



In more technical terms, the flow starts with empty `docsText` and

`codeText`; the function will append comment lines to `docsText` (with

`codeText` being empty) until a code line is encountered.

At that point it will append to `codeText`; however, when the next

comment line (destined to `docsText`) is encountered, since `codeText` is

_not_ empty, a new section containing `docsText` and `codeText` is created

and both those variables are cleared; the commented line that triggered the

new section is then added to `docsText` (with a now empty `codeText`),

and everything starts again. Basically, in the flow _you know if you are in the

middle of code if `codeText` is empty_.



Empty lines are... empty, both for doc and code. To know where to add

the empty line (`codeText` or `docsText`), the same assumption as above is

used: if there is anything in `codeText`, it means that the parser is

in the middle of a code section, and therefore the empty line will be

added to that code section.



```javascript

function parse (source, lines, config = {}) {

  let codeText, docsText

  const sections = []



  configure(config)



  const lang = config.lang

  docsText = codeText = ''

  let lineNumber = 0

  let startLineNumber = 1

  for (let line of lines) {

    lineNumber++

    /* If the line is not empty, it will either go in the code section */

    /* or the docs section, depending on whether the comment character was */

    /* found at the beginning of the line */

    if (line) {

      /* Case #1: it's a "comment" */

      /*  Text will go in docsText as documentation */

      if (line.match(lang.commentMatcher) && !line.match(lang.commentFilter)) {

        /*

          DETOUR: If there is code in codeText already, close off that section

          the section by pushing it into `sections` and zeroing

         `docsText` and `codeText`

        */

        if (codeText) {

          sections.push({ docsText, codeText, startLineNumber })

          docsText = codeText = ''

        }



        /* Add the line to the documentation (docsText) taking out the leading */

        /* comment marker */

        if (lang.symbol) {

          line = line.replace(lang.commentMatcher, '')

        }

        docsText += line + '\n'



        /* If the line was a new markdown section (`===`,  `---` or `##`), */

        /* close off that section */

        if (/^(---+|===+|#+.*)$/.test(line)) {

          sections.push({ docsText, codeText })

          docsText = codeText = ''

        }

        /* Case #2: it's not a comment */

        /* Note that from this moment on `codeText` is no longer empty, */

        /* which means that the next comment line (destined to docsText) will */

        /* trigger a new section */

      } else {

        if (codeText === '') {

          startLineNumber = lineNumber

        }

        codeText += line + '\n'

      }

      /* If it's an empty line, it will go either in the */

      /* code section or in the docs section. */

      /* We know we are in the code section by checking if */

      /* there is any code in codeText yet */

    } else {

      if (codeText) codeText += line + '\n'

      else docsText += line + '\n'

    }

  }

  sections.push({ docsText, codeText, startLineNumber })



  return sections

}



function codeToHtml (highlighter, code, language, lineNumber) {

  const htmlEscapes = {

    '&': '&',

    '<': '<',

    '>': '>',

    '"': '"',

    "'": '''

  }



  function escapeHtml (html) {

    return html.replace(/[&<>"']/g, (chr) => htmlEscapes[chr])

  }



  const FontStyle = {

    NotSet: -1,

    None: 0,

    Italic: 1,

    Bold: 2,

    Underline: 4

  }



  /* See: https://github.com/shikijs/shiki/blob/f322a2b97470b25b26a4d8c1cf4892b059eb69db/packages/shiki/src/renderer.ts */

  function renderToHtml (lines, options = {}) {

    const bg = options.bg || 'transparent'



    const hasLineNumbers = typeof options.lineNumber === 'number'

    const lineNumber = options.lineNumber || 1



    const numberOfNonEmptyLines = lines.filter((l) => l.length > 0).length

    if (numberOfNonEmptyLines === 0) {

      return ''

    }



    let html = ''



    html += `
`

    if (options.langId) {

      html += `
${options.langId}
`

    }

    html +=

      ''



    lines.forEach((l, idx) => {

      const lineClasses = ['line', l.length > 0 ? undefined : 'empty-line']

        .filter(Boolean)

        .join(' ')

      const currentLineId = `L${lineNumber + idx}`



      html +=

        `'



      l.forEach((token) => {

        const cssDeclarations = [`color: ${token.color || options.fg}`]

        if (token.fontStyle & FontStyle.Italic) {

          cssDeclarations.push('font-style: italic')

        }

        if (token.fontStyle & FontStyle.Bold) {

          cssDeclarations.push('font-weight: bold')

        }

        if (token.fontStyle & FontStyle.Underline) {

          cssDeclarations.push('text-decoration: underline')

        }

        html += `${escapeHtml(

          token.content

        )}`

      })

      html += '\n'

    })

    html = html.replace(/\n*$/, '') /* Get rid of final new lines */

    html += '
'



    return html

  }



  const tokens = highlighter.codeToThemedTokens(code, language)



  return renderToHtml(tokens, { lineNumber })

}



```



## formatAsHtml()



This function takes the sections and format them as

HTML, alternating `docsText` and (Markdown-indented) `codeText`



This function is essentially split into two subfunctions; they are both

run passing on all of the parameters:



* `formatSections()`

* `makeHtmlBlob()`



The first one, `formatSections()`, will take the `sections` array; remember:

each element has two properties, the leading `docsText` and the trailing

`codeText`. After running `formatSections()`, each element will also have

`docsHtml` and `codeHtml` (their respective HTML versions). This is done using

`markdown` for the docs, and `shiki` for the code.



If a plugin was

specified, the following functions from the plugin may be run:



* `plugin.beforeMarked` --  run before feeding the text to Marked. This 

  allows users to extend Markdown as neeed.  

* `plugin.afterHtml` -- run after the HTML has been generated

* `plugin.finalPass` -- run once the whole file has been generated and

  it's ready to be written on the disk.



Since Markdown documentation can _also_ contain code (by indenting 4 spaces),

the `shiki` option is set for Markdown, instructing it what to do when

a code block is encountered: obviously, the `shiki` library

will be used to format it.



The second functtion, `makeHtmlBlob()`, actually creates the final HTML code

using the formatted sections as a starting point. The conversion is done

by using the EJS template provided, and passing it important variables:



* `sources` -- it's the list of sources, useful to create table of contents

* `css` -- it's the path to the CSS file, relative to the processed file.

* `title` -- the title of the file. If the file starts with a markdown

  heading, this variable will have the contents of that heading; otherwise,

  it will have the file name.

* `firstSectionIsTitle` -- if true, the `title` variable is indeed the first

  section's markdown heading. Templates can use this information to write

  the logic around the title.

* `sections` -- array of the various sections, which include `docsHtml` and `codeHtml`.

  This array is used by the EJS template to know what the file actually

  contains

* `finalPath` and `relativeToThisFile` -- two functions often used together

  in templates to know how to (HTML) link to another file in sources. For

  example `relativeToThisFile(finalPath(source))`.



One note about the `_getTemplate()` function. The aim of the function is

to load the template file, and return an EJS compiler. The file itself

is memoized, so that calling `_getTemplate()` doesn't result in multiple

reloading of the same file (since `formatAsHtml()` is potentially called

multiple times, once for each passed file). Memoization avoids using a

global variable.



```javascript

async function formatAsHtml (source, sections, config = {}) {

  configure(config)



  const lang = config.lang



  /* Format sections, as HTML (from Markdown) or as highlighted code */

  await formatSections(source, sections, config, lang)



  /* return the HTML blob */

  return makeHtmlBlob(source, sections, config, lang)



  /* Format and highlight the various section of the code, using */

  async function formatSections (source, sections, config = {}, lang) {

    const highlighter = await shiki.getHighlighter({

      theme: config.shikiTheme || 'min-light'

    })



    /* [Markdown](https://github.com/markedjs/marked) and Shiki */

    /* Set options specified by the user, using to `smartypants: true` */

    /* as a starting point */

    marked.setOptions(config.marked)



    /* Code might happen within the markdown documentation as well! If that */

    /* is the case, it will highlight code either using the language specified */

    /* within the Markdown codeblock, or the default language used for the processes */

    /* file */

    marked.setOptions({

      highlight: function (code, language) {

        if (!language) language = lang.name



        try {

          return codeToHtml(highlighter, code, language, undefined)

        } catch (error) {

          console.warn(

            `${source}: language '${language}' not recognised, code block not highlighted`

          )

          return code

        }

      }

    })



    for (const section of sections) {

      let code = codeToHtml(

        highlighter,

        section.codeText,

        lang.name,

        section.startLineNumber

      )



      code = code.replace(/\s+$/, '')

      if (code !== '') section.codeHtml = `${code}`

      else section.codeHtml = ''

      if (config.plugin.beforeMarked) {

        const newText = await config.plugin.beforeMarked(section.docsText)

        section.docsText = newText

      }

      section.docsHtml = marked(section.docsText)



      if (config.plugin.afterHtml) {

        const newHtml = await config.plugin.afterHtml(section.docsHtml)

        section.docsHtml = newHtml

      }

    }

  }



  /* Once all of the code has finished highlighting, we can **write** the resulting */

  /* documentation file by passing the completed HTML sections into the template, */

  /* and rendering it to the specified output path. */

  async function makeHtmlBlob (source, sections, config = {}) {

    let first



    async function _getTemplate (template) {

      if (formatAsHtml._template) return formatAsHtml._template



      template = (await fs.readFile(template)).toString()

      template = formatAsHtml._template = ejs.compile(template)

      return template

    }



    function relativeToThisFile (file) {

      const from = path.resolve(path.dirname(thisFile))

      const to = path.resolve(path.dirname(file))

      return path.join(path.relative(from, to), path.basename(file))

    }



    function includeText (source) {

      return (s, silentFail) => {

        let contents

        let file

        if (path.isAbsolute(s)) {

          file = s

        } else {

          file = s

        }

        try {

          contents = fs.readFileSync(file)

          return contents

        } catch (e) {

          if (silentFail && e.code === 'ENOENT') return ''

          if (e.code === 'ENOENT') {

            console.error('Could not load included file:', file)

          } else {

            console.log(e)

          }

          process.exit(100)

        }

      }

    }



    const thisFile = finalPath(source, config)



    /* Work out `title`, which will be either the first heading in the */

    /* documentation, or (as a last resort) the file name */

    const firstSection = sections.find((s) => {

      return s.docsText.length > 0

    })

    let lexed

    if (firstSection) {

      lexed = marked.lexer(firstSection.docsText)

      first = lexed[0]

    }

    const maybeTitle = first && first.type === 'heading' && first.depth === 1

    const title = maybeTitle ? first.text : path.basename(source)

    const firstSectionIsTitle = maybeTitle && lexed.length === 1



    /* If the first section is the title, then get rid of it  */

    /* since the title is already being displayed by the template anyway */

    if (firstSectionIsTitle) {

      sections.shift()

    }



    /* The `css` variable will be available in the template as a relative */

    /* link to the CSS file */

    const css = relativeToThisFile(

      path.join(config.output, path.basename(config.css))

    )



    const template = await _getTemplate(config.template)



    /* Make up the HTML based on the template */

    let html = template({

      lang,

      includeText: includeText(source),

      source,

      sources: config.sources,

      css,

      firstSectionIsTitle,

      title,

      sections,

      finalPath: (path) => finalPath(path, config),

      relativeToThisFile,



      hasTitle: firstSectionIsTitle, // compatibility to Docco's original API

      destination: (path) => finalPath(path, config), // compatibility to Docco's original API

      relative: relativeToThisFile // compatibility to Docco's original API

    })



    /* Run the final pass */

    if (config.plugin.finalPass) {

      html = await config.plugin.finalPass(html)

    }



    return html

  }

}



```



Public API

----------



These functions are available once the module is `required()`. They are

self contained and can be used to take Docco Next to different directions.



```javascript

exports = module.exports = {

  copyAsset,

  run,

  parse,

  formatAsHtml,

  litToCode,

  documentOne,

  documentAll,

  version

}



/*

`rm -rf dir2/*; node --inspect-brk bin/docco  -o dir2 ./doccoOrig.js sub/doccoOrig.js`

`rm -rf dir2/*; node bin/docco  -l default -L /tmp/extras.json -o dir2 ./docco.js

docco: ./docco.js -> dir2/docco.html

*/



```