{"id":15642016,"url":"https://github.com/bwplotka/mdox","last_synced_at":"2025-04-09T05:11:11.593Z","repository":{"id":40400623,"uuid":"291308731","full_name":"bwplotka/mdox","owner":"bwplotka","description":"Format your docs; autogenerate from flags or Go structs or even generate versioned website directly from markdown!  ","archived":false,"fork":false,"pushed_at":"2024-10-01T17:40:15.000Z","size":630,"stargazers_count":74,"open_issues_count":36,"forks_count":11,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-01T07:52:11.737Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bwplotka.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-29T16:36:33.000Z","updated_at":"2025-03-26T12:28:21.000Z","dependencies_parsed_at":"2023-01-29T01:31:06.887Z","dependency_job_id":"42d9385a-61e4-4ffa-935c-545cf8fe59f8","html_url":"https://github.com/bwplotka/mdox","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bwplotka%2Fmdox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bwplotka%2Fmdox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bwplotka%2Fmdox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bwplotka%2Fmdox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bwplotka","download_url":"https://codeload.github.com/bwplotka/mdox/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247980837,"owners_count":21027808,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-03T11:53:32.483Z","updated_at":"2025-04-09T05:11:11.577Z","avatar_url":"https://github.com/bwplotka.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mdox\n\n[![go.dev reference](https://img.shields.io/badge/go.dev-reference-007d9c?logo=go\u0026logoColor=white\u0026style=flat-square)](https://pkg.go.dev/github.com/bwplotka/mdox) [![Latest Release](https://img.shields.io/github/release/bwplotka/mdox.svg?style=flat-square)](https://github.com/bwplotka/mdox/releases/latest) [![CI](https://github.com/bwplotka/mdox/workflows/go/badge.svg)](https://github.com/bwplotka/mdox/actions?query=workflow%3Ago) [![Go Report Card](https://goreportcard.com/badge/github.com/bwplotka/mdox)](https://goreportcard.com/report/github.com/bwplotka/mdox) [![Slack](https://img.shields.io/badge/join%20slack-%23mdox-brightgreen.svg)](https://cloud-native.slack.com/archives/mdox)\n\n`mdox` (spelled as `md docs`) is a CLI for maintaining automated, high-quality project documentation and website leveraging [GitHub Flavored Markdown](https://github.github.com/gfm/) and git.\n\nThis project can be used both as CLI as well as a library.\n\n## Goals\n\nAllow projects to have self-updating up-to-date documentation available in both markdown (e.g readable from GitHub) and static HTML. Hosted in the same repository as code and integrated with Pull Requests CI, hosting CD, and code generation.\n\n## Features\n\n* Enhanced and consistent formatting for markdown files in [GFM](https://github.github.com/gfm/) format, focused on readability.\n* Auto generation of code block content based on `mdox-exec` directives (see [#code-generation](#code-generation)). Useful for:\n  * Generating help output from CLI --help\n  * Generating example YAML from Go configuration struct (+comments)\n* Robust and fast relative and remote link checking. (see [#link-validation-configuration](#link-validation-configuration))\n* Website integration:\n  * \"Localizing\" links to relative docs if specified (useful for multi-domain websites or multi-version doc). (see [#link-localization](#link-localization))\n    * This allows smooth integration with static document websites like [Docusaurus](https://docusaurus.io/) or [hugo](https://gohugo.io) based themes!\n  * Flexible pre-processing allowing easy to use GitHub experience as well as website. (see [#transform-usage](#transformation))\n* Allows profiling(using [fgprof](https://github.com/felixge/fgprof)) and exports metrics(saves to file in [OpenMetrics](https://openmetrics.io/) format) for easy debugging\n\n## Usage\n\n### Formatting and Link Checking\n\nJust run `mdox fmt` and pass markdown files (or glob matching those).\n\nFor example, this README is formatted by the CI on every PR using [`mdox fmt -l *.md` command](https://github.com/bwplotka/mdox/blob/9e183714070f464b1ef089da3df8048aff1abeda/Makefile#L59).\n\n```bash mdox-exec=\"mdox fmt --help\"\nusage: mdox fmt [\u003cflags\u003e] \u003cfiles\u003e...\n\nFormats in-place given markdown files uniformly following GFM (GitHub Flavored\nMarkdown: https://github.github.com/gfm/). Example: mdox fmt *.md\n\n\nFlags:\n  -h, --[no-]help              Show context-sensitive help (also try --help-long\n                               and --help-man).\n      --[no-]version           Show application version.\n      --log.level=info         Log filtering level.\n      --log.format=clilog      Log format to use.\n      --profiles.path=PROFILES.PATH  \n                               Path to directory where CPU and heap profiles\n                               will be saved; If empty, no profiling will be\n                               enabled.\n      --metrics.path=METRICS.PATH  \n                               Path to directory where metrics are saved in\n                               OpenMetrics format; If empty, no metrics will be\n                               saved.\n      --[no-]check             If true, fmt will not modify the given files,\n                               instead it will fail if files needs formatting\n      --[no-]soft-wraps        If true, fmt will preserve soft line breaks for\n                               given files\n      --[no-]code-fmt          Reformat code snippets\n      --[no-]code.disable-directives  \n                               If false, fmt will parse custom fenced\n                               code directives prefixed with 'mdox-gen' to\n                               autogenerate code snippets. For example:\n                               \n                                 ```\u003clang\u003e mdox-exec=\"\u003cexecutable + arguments\u003e\"\n                               \n                               This directive runs executable with arguments\n                               and put its stderr and stdout output inside code\n                               block content, replacing existing one.\n      --anchor-dir=ANCHOR-DIR  Anchor directory for all transformers. PWD is\n                               used if flag is not specified.\n      --links.localize.address-regex=LINKS.LOCALIZE.ADDRESS-REGEX  \n                               If specified, all HTTP(s) links that target a\n                               domain and path matching given regexp will be\n                               transformed to relative to anchor dir path (if\n                               exists). Absolute path links will be converted to\n                               relative links to anchor dir as well.\n  -l, --[no-]links.validate    If true, all links will be validated\n      --links.validate.config-file=\u003cfile-path\u003e  \n                               Path to YAML file for skipping\n                               link check, with spec defined in\n                               github.com/bwplotka/mdox/pkg/linktransformer.ValidatorConfig\n      --links.validate.config=\u003ccontent\u003e  \n                               Alternative to 'links.validate.config-file'\n                               flag (mutually exclusive). Content of YAML file\n                               for skipping link check, with spec defined in\n                               github.com/bwplotka/mdox/pkg/linktransformer.ValidatorConfig\n      --[no-]cache.clear       If true, entire cache database will be dropped\n                               and rebuilt when mdox is run. Useful in case\n                               cache needs to be cleared immediately from GitHub\n                               Actions or other CI runner cache.\n\nArgs:\n  \u003cfiles\u003e  Markdown file(s) to process.\n\n```\n\n#### Code Generation\n\nIt's not uncommon that documentation is explaining code or configuration snippets. One of the challenges of such documentation is keeping it up to date. This is where `mdox` code block directives comes handy! To ensure mdox will auto update code snippet add `mdox-exec=\"\u003cwhatever command you want take output from\u003e\"` after language directive on code block.\n\nFor example this Readme contains `mdox --help` which is has to be auto generated on every PR:\n\n```markdown\n```bash mdox-exec=\"mdox fmt --help\"\n...\n```\n\nThis also enables auto updating snippets of code in code blocks using tools like `sed`. For example, below code block directive will auto update and insert lines 3 to 6 from main.go into code block.\n\n```markdown\n```go mdox-exec=\"sed -n '3,6p' main.go\"\n...\n```\n\nSome commands might have non-zero exit codes. mdox will fail commands in such cases(otherwise errors might get formatted into markdown) but the expected exit code can also be passed as a code block directive! For example, below code block executes `go --help` which has 2 as its exit code,\n\n```markdown\n```text mdox-exec=\"go --help\" mdox-expect-exit-code=2\n...\n```\n\nYou can disable this feature by specifying `--code.disable-directives`\n\n#### Link Validation Configuration\n\nBy default, mdox checks all links (both relative and remote) in passed markdown files! However, in some cases, link checks might fail even when links are working, such as with rate limiting, Cloudflare protections, or something else(like localhost links in docs).\n\nThis might lead to failed CI checks which aren't desirable. So, you can use the `links.validate.config-file` flag to pass in YAML configuration file for selective link checking using special validators and link regexes.\n\nFor example,\n\n```yaml mdox-exec=\"cat examples/.mdox.validate.yaml\"\nversion: 1\ntimeout: '1m'\nparallelism: 100\nhost_max_conns: 2\nrandom_delay: '1s'\n\nvalidators:\n  - regex: '(^http[s]?:\\/\\/)(www\\.)?(github\\.com\\/)bwplotka\\/mdox(\\/pull\\/|\\/issues\\/)'\n    type: 'githubPullsIssues'\n\n  - regex: 'localhost'\n    type: 'ignore'\n\n  - regex: 'thanos\\.io'\n    type: 'roundtrip'\n```\n\nAs seen above, mdox supports validate configuration supports a few parameters and passing an array of link validators with types and regexes. The supported configuration parameters are:\n\n* `timeout`: The HTTP client's timeout. Defaults to \"10s\".\n* `parallelism`: The maximum amount of concurrent HTTP requests. Defaults to 100.\n* `host_max_conns`: The maximum amount of HTTP connections open per host. Defaults to 2.\n* `random_delay`: A random delay between 0 and this value is added between requests. It takes values like \"500ms\", \"1s\", \"1m\", or \"1m30s\". Defaults to no delay.\n\nThere are three types of validators:\n\n* `ignore`: This type of validator makes sure that `mdox` does not check links with provided regex. This is the most common use case.\n* `githubPullsIssues`: This is a smart validator which only accepts a specific type of regex of the form `(^http[s]?:\\/\\/)(www\\.)?(github\\.com\\/){ORG}\\/{REPO}(\\/pull\\/|\\/issues\\/)`. It performs smart validation on GitHub PR and issues links, by fetching GitHub API to get the latest pull/issue number and matching regex. This makes sure that mdox doesn't get rate limited by GitHub, even when checking a large number of GitHub links(which is pretty common in documentation)!\n* `roundtrip`: All links are checked with the roundtrip validator by default(no need for including into config explicitly) which means that each link is visited and fails if http status code is not 200(even after retries).\n\nRelative link checking *is not* affected by this configuration, as it is expected that such links will work.\n\nYAML can be passed in directly as well using `links.validate.config` flag! For more details [go.dev reference](https://pkg.go.dev/github.com/bwplotka/mdox) or [Go struct](https://github.com/bwplotka/mdox/blob/main/pkg/mdformatter/linktransformer/config.go).\n\n### Link localization\n\nIt is expected for documentation to contain remote links to the project website. However, in such cases, it creates problems for multi-version docs or multi-domain websites (links would need to be updated for each version which is cumbersome). Also, it would not be navigatable locally or through GitHub (would always redirect to the website) and requires additional link checking.\n\nThis is where the `links.localize.address-regex` flag comes in handy!\n\nIt ensures that all HTTP(s) links that target a domain and path matching given regex will be transformed by `mdox` to relative links which are relative to anchor dir path (if exists). Also, all absolute path links will be converted to relative links to anchor dir as well.\n\nSo passing in regex such as `--links.localize.address-regex=\"https:\\/\\/example\\.\\/.*` will allow mdox to transform links like `https://example.com/getting-started.md/` to simply `getting-started.md`.\n\n### Transformation\n\nmdox allows various types of markdown file transformation which are useful for website pre-processing and is often required when using static site generators like Hugo. It helps in generating front/backmatter, renaming, and moving files, and converts links to work on websites.\n\nJust run `mdox transform --config-file=.mdox.yaml` and pass in YAML configuration.\n\n```bash mdox-exec=\"mdox transform --help\"\nusage: mdox transform [\u003cflags\u003e]\n\nTransform markdown files in various ways. For example pre-process markdown files\nto allow it for use for popular static HTML websites based on markdown source\ncode and front matter options.\n\n\nFlags:\n  -h, --[no-]help                Show context-sensitive help (also try\n                                 --help-long and --help-man).\n      --[no-]version             Show application version.\n      --log.level=info           Log filtering level.\n      --log.format=clilog        Log format to use.\n      --profiles.path=PROFILES.PATH  \n                                 Path to directory where CPU and heap profiles\n                                 will be saved; If empty, no profiling will be\n                                 enabled.\n      --metrics.path=METRICS.PATH  \n                                 Path to directory where metrics are saved in\n                                 OpenMetrics format; If empty, no metrics will\n                                 be saved.\n      --config-file=\u003cfile-path\u003e  Path to Path to the YAML\n                                 file with spec defined in\n                                 github.com/bwplotka/mdox/pkg/transform.Config\n      --config=\u003ccontent\u003e         Alternative to 'config-file' flag\n                                 (mutually exclusive). Content of Path\n                                 to the YAML file with spec defined in\n                                 github.com/bwplotka/mdox/pkg/transform.Config\n\n```\n\nFor example,\n\n```yaml mdox-exec=\"cat examples/.mdox.yaml\"\nversion: 1\n\ninputDir: \"docs\"\noutputDir: \"website/docs-pre-processed/tip\"\nextraInputGlobs:\n  - \"CHANGELOG.md\"\n  - \"static\"\n\ngitIgnored: true\nlocalLinksStyle:\n  hugo:\n    indexFileName: \"_index.md\"\n\ntransformations:\n\n  - glob: \"../CHANGELOG.md\"\n    path: /thanos/CHANGELOG.md\n    popHeader: true\n    frontMatter:\n      template: |\n        title: \"{{ .Origin.FirstHeader }}\"\n        type: docs\n        lastmod: \"{{ .Origin.LastMod }}\"\n    backMatter:\n      template: |\n        Found a typo, inconsistency or missing information in our docs?\n        Help us to improve [Thanos](https://thanos.io) documentation by proposing a fix [on GitHub here](https://github.com/thanos-io/thanos/edit/main/{{ .Origin.Filename }}) :heart:\n\n  - glob: \"getting-started.md\"\n    path: /thanos/getting-started.md\n    frontMatter:\n      template: |\n        type: docs\n        title: \"{{ .Origin.FirstHeader }}\"\n        lastmod: \"{{ .Origin.LastMod }}\"\n        slug: \"{{ .Target.FileName }}\"\n\n  - glob: \"../static/**\"\n    path: /favicons/**\n```\n\nAs seen above,\n\n* `inputDir`: It's a relative (to PWD) path that assumes input directory for markdown files and assets.\n* `outputDir`: It's a relative (to PWD) output directory where you can expect all files to land in. Typically that can be a `content` dir which Hugo uses as an input.\n* `extraInputGlobs`: It allows you to bring files from outside of `inputDir`.\n* `linkPrefixForNonMarkdownResources`: It specifies the link to be glued onto relative links which don't point to markdown or image files.\n* `gitIgnored`: It specifies whether `outputDir` should be git-ignored.\n* `localLinksStyle`: It sets the linking style to be applied. If empty, mdox assumes default style.\n\n* `transformation`: Array of transformations to apply for files in `inputDir` and `extraInputGlobs`. This consists of,\n  * `glob`: It is matched against the relative path of the file in the `inputDir` using https://github.com/gobwas/glob.\n  * `path`: It is an optional different path for the file to be moved into. If not specified, the file will be moved to the exact same position as it is in `inputDir`.\n  * `popHeader`: If set to true, it pops the first header of md file. True by default for files in the root of `inputDir`\n  * `frontMatter`: Optional template for constructing frontmatter of markdown file.\n  * `backMatter`: Optional template for constructing backmatter of markdown file(content appended to end like edit links)\n\nYAML can be passed in directly as well using `--config` flag! For more details [go.dev reference](https://pkg.go.dev/github.com/bwplotka/mdox) or [Go struct](https://github.com/bwplotka/mdox/blob/main/pkg/transform/config.go).\n\n### Installing\n\nRequirements to build this tool:\n\n* Go 1.19+\n* Linux or macOS\n\n```shell\ngo install github.com/bwplotka/mdox@latest\n```\n\nor via [bingo](https://github.com/bwplotka/bingo) if want to pin it:\n\n```shell\nbingo get -u github.com/bwplotka/mdox\n```\n\n## Production Usage\n\n* [Thanos](https://github.com/thanos-io/thanos)\n* [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator)\n* [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus)\n* [Observatorium](https://github.com/observatorium/observatorium)\n* [RedHat Observability Group Handbook](https://github.com/rhobs/handbook)\n* [Bingo](https://github.com/bwplotka/bingo)\n* [effiecientgo/tools](https://github.com/efficientgo/tools)\n* [effiecientgo/e2e](https://github.com/efficientgo/e2e)\n\n## Contributing\n\nAny contributions are welcome! Just use GitHub Issues and Pull Requests as usual. We follow [Thanos Go coding style](https://thanos.io/tip/contributing/coding-style-guide.md/) guide.\n\nHave questions or feedback? Join our [slack channel](https://cloud-native.slack.com/archives/mdox)!\n\n## Initial Author\n\n[@bwplotka](https://bwplotka.dev)\n\nNote: This project was a part of [GSoC'21](https://summerofcode.withgoogle.com/projects/#5053843303301120) (mentor: [@bwplotka](https://bwplotka.dev), mentee: [@saswatamcode](https://saswatamcode.tech)).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbwplotka%2Fmdox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbwplotka%2Fmdox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbwplotka%2Fmdox/lists"}