{"id":13822559,"url":"https://github.com/zimbatm/mdsh","last_synced_at":"2025-04-09T03:13:07.395Z","repository":{"id":34171943,"uuid":"170899273","full_name":"zimbatm/mdsh","owner":"zimbatm","description":"`$ mdsh` # a markdown shell pre-processor","archived":false,"fork":false,"pushed_at":"2025-04-01T22:14:09.000Z","size":184,"stargazers_count":133,"open_issues_count":14,"forks_count":12,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-01T23:24:16.499Z","etag":null,"topics":["markdown","pre-commit-hook","pre-processing","shell"],"latest_commit_sha":null,"homepage":"https://zimbatm.github.io/mdsh/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zimbatm.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-15T17:00:14.000Z","updated_at":"2025-04-01T14:15:23.000Z","dependencies_parsed_at":"2024-02-27T11:02:09.062Z","dependency_job_id":"fdd23ebb-84ba-44a4-b1bd-6a6fca6248b8","html_url":"https://github.com/zimbatm/mdsh","commit_stats":{"total_commits":127,"total_committers":8,"mean_commits":15.875,"dds":0.4803149606299213,"last_synced_commit":"fbe779b825c7eb326b1b72e3293150a5dd1cf66a"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zimbatm%2Fmdsh","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zimbatm%2Fmdsh/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zimbatm%2Fmdsh/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zimbatm%2Fmdsh/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zimbatm","download_url":"https://codeload.github.com/zimbatm/mdsh/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247968374,"owners_count":21025823,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["markdown","pre-commit-hook","pre-processing","shell"],"created_at":"2024-08-04T08:02:06.128Z","updated_at":"2025-04-09T03:13:07.362Z","avatar_url":"https://github.com/zimbatm.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# `$ mdsh` - a markdown shell pre-processor\n\n[![Build Status](https://github.com/zimbatm/mdsh/actions/workflows/ci.yaml/badge.svg)](https://github.com/zimbatm/mdsh/actions/workflows/ci.yaml?branch=master) [![crates.io](https://img.shields.io/crates/v/mdsh.svg)](https://crates.io/crates/mdsh)\n\nThe mdsh project describes a Markdown language extension that can be used to\nautomate some common tasks in README.md files. Quite often I find myself\nneeding to embed a snippet of code or markdown from a different file. Or I\nwant to show the output of a command. In both cases this can be done manually,\nbut what all you had to do was run `mdsh` and have the file updated\nautomatically?\n\nSo the goal of this tool is first to extend the syntax of Markdown in a\nnatural way. Something that you might type. And if the `mdsh` tool is run, the\nrelated blocks get updated in place. Most other tools would produce a new file\nbut we really want a sort of idempotent operation here.\n\nIn the end this gives a tool that is a bit akin to literate programming or\njupyer notebooks but for shell commands. It adds a bit of verbosity to the\nfile and in exchange it allows to automate the refresh of those outputs.\n\n## Usage\n\n`$ mdsh --help`\n\n```\nMarkdown shell pre-processor. Never let your READMEs and tutorials get out of sync again.\n\nExits non-zero if a sub-command failed.\n\nUsage: mdsh [OPTIONS]\n\nOptions:\n  -i, --inputs \u003cINPUTS\u003e\n          Path to the markdown files. `-` for stdin\n          \n          [default: ./README.md]\n\n  -o, --output \u003cOUTPUT\u003e\n          Path to the output file, `-` for stdout [defaults to updating the input file in-place]\n\n      --work_dir \u003cWORK_DIR\u003e\n          Directory to execute the scripts under [defaults to the input file’s directory]\n\n      --frozen\n          Fail if the output is different from the input. Useful for CI.\n          \n          Using `--frozen`, you can guarantee that developers update documentation when they make a change. Just add `mdsh --frozen` as a check to your continuous integration setup.\n\n      --clean\n          Remove all generated blocks\n\n  -h, --help\n          Print help (see a summary with '-h')\n\n  -V, --version\n          Print version\n```\n\n## Syntax Extensions\n\n### Inline Shell Code\n\nSyntax regexp:\n```regexp\n^`[$\u003e] ([^`]+)`\\s*$\n```\n\nInline Shell Code are normal `inline code` that:\n\n* start at the beginning of a line\n* include either `$` or `\u003e` at the beginning of their content\n* contain a shell command\n\nWhen those are enountered, the command is executed by `mdsh` and output as\neither a fenced code block (`$`) or markdown code (`\u003e`).\n\n* `$` runs the command and outputs a code block\n* `\u003e` runs the command and outputs markdown\n\nExamples:\n\n~~~\n`$ seq 4 | sort -r`\n\n```\n4\n3\n2\n1\n```\n~~~\n\n~~~\n`\u003e echo 'I *can* include markdown. \u003ccode\u003eHehe\u003c/code\u003e.'`\n\n\u003c!-- BEGIN mdsh --\u003e\nI *can* include markdown. \u003ccode\u003eHehe\u003c/code\u003e.\n\u003c!-- END mdsh --\u003e\n~~~\n\n### Multiline Shell Code\n\nSyntax regexp:\n```regexp\n^```[$^]\\n.*\\n```$\n```\n\nMultiline Shell Code are normal multiline code that:\n\n* start at the beginning of a line\n* include `$` or `^` as \"language\"\n* contain a shell command\n\nWhen those are enountered, the command is executed by `mdsh` and output as\neither a fenced code block (`$`) or markdown code (`\u003e`).\n\n* `$` runs the command and outputs a code block\n* `\u003e` runs the command and outputs markdown\n\nExamples:\n\n~~~\n```$ as bash\nseq 3 | sort -r\nseq 2 | sort -r\n```\n\n```bash\n3\n2\n1\n2\n1\n```\n~~~\n\n~~~\n```\u003e\necho 'I *can* include markdown. \u003ccode\u003eHehe\u003c/code\u003e.'\n```\n\n\u003c!-- BEGIN mdsh --\u003e\nI *can* include markdown. \u003ccode\u003eHehe\u003c/code\u003e.\n\u003c!-- END mdsh --\u003e\n~~~\n\n### Variables\n\nSyntax regexp:\n```regexp\n^`! ([\\w_]+)=([^`]+)`\\s*$\n```\n\nVariables allow you to set new variables in the environment and reachable by\nthe next blocks that are being executed.\n\nThe value part is being evaluated by bash and can thus spawn sub-shells.\n\nExamples:\n\n`! user=bob`\n\nNow the `$user` environment variable is available:\n\n`$ echo hello $user`\n\n```\nhello bob\n```\n\nNow capitalize the user\n\n`! USER=$(echo $user | tr '[[:lower:]]' '[[:upper:]]')`\n\n`$ echo hello $USER`\n\n```\nhello BOB\n```\n\n### Link Includes\n\nSyntax regexp:\n```regexp\n^\\[[$\u003e] ([^\\]]+)]\\([^\\)]+\\)\\s*$\n```\n\nLink Includes work similarily to code blocks but with the link syntax.\n\n* `$` loads the file and embeds it as a code block\n* `\u003e` loads the file and embeds it as markdown\n\nExamples:\n\n~~~\n[$ code.rb](samples/code.rb) as ruby\n\n```ruby\nrequire \"pp\"\n\npp ({ foo: 3 })\n```\n~~~\n\n~~~\n[\u003e example.md](samples/example.md)\n\n\u003c!-- BEGIN mdsh --\u003e\n*this is part of the example.md file*\n\u003c!-- END mdsh --\u003e\n~~~\n\n### ANSI escapes\n\nANSI escape sequences are filtered from command outputs:\n\n`$ echo $'\\e[33m'yellow`\n\n```\nyellow\n```\n\n### Commented-out commands\n\nSometimes it's useful not to render the command that is being shown. All the\ncommands support being hidden inside of a HTML comment like so:\n\n~~~\n\u003c!-- `$ echo example` --\u003e\n\n```\nexample\n```\n~~~\n\n### Fenced code type\n\nIf you want GitHub to highlight the outputted code fences, it's possible to\npostfix the line with `as \u003ctype\u003e`. For example:\n\n~~~\n`$ echo '{ key: \"value\" }'` as json\n\n```json\n{ key: \"value\" }\n```\n~~~\n\n## Installation\n\nThe best way to install `mdsh` is with the rust tool cargo.\n\n```bash\ncargo install mdsh\n```\n\nIf you are lucky enough to be a nix user:\n\n```bash\nnix-env -f https://github.com/NixOS/nixpkgs/archive/master.tar.gz -iA mdsh\n```\n\nIf you are a nix + flakes user:\n\n```bash\nnix profile install github:zimbatm/mdsh\n```\n\n## Running without installation\n\nIf you are a nix + flakes user:\n\n```bash\nnix run github:zimbatm/mdsh -- --help\n```\n\n### Pre-commit hook\n\nThis project can also be installed as a [pre-commit](https://pre-commit.com/)\nhook.\n\nAdd to your project's `.pre-commit-config.yaml`:\n\n```yaml\n-   repo: https://github.com/zimbatm/mdsh.git\n    rev: main\n    hooks:\n    -   id: mdsh\n```\n\nMake sure to have rust available in your environment.\n\nThen run `pre-commit install-hooks`\n\n## Known issues\n\nThe tool currently lacks in precision as it doesn't parse the Markdown file,\nit just looks for the desired blocks by regexp. It means that in some cases it\nmight misintepret some of the commands. Most existing Markdown parsers are\nused to generate HTML in the end and are thus not position-preserving. Eg:\npulldown-cmark\n\nThe block removal algorithm doesn't support output that contains triple\nbacktick or `\u003c!-- END mdsh --\u003e`.\n\n## Related projects\n\n* \u003chttp://chriswarbo.net/essays/activecode/\u003e is the closest to this project. It\n  has some interesting Pandoc filters that capture code blocks into outputs.\n  The transformation is not in-place like `mdsh`.\n* [Literate Programming](https://en.wikipedia.org/wiki/Literate_programming)\n  is the practice of interspesing executable code into documents. There are\n  many language-specific implementations out there. `mdsh` is a bit like a\n  bash literate programming language.\n* [Jupyter Notebooks](https://jupyter.org/) is a whole other universe of\n  documentation and code. It's great but stores the notebooks as JSON files. A\n  special viewer program is required to render them to HTML or text.\n\n## User Feedback\n\n### Issues\n\nIf you have any problems with or questions about this project, please contact\nuse through a [GitHub issue](https://github.com/zimbatm/mdsh/issues).\n\n### Contributing\n\nYou are invited to contribute new features, fixes or updates, large or small;\nwe are always thrilled to receive pull requests, and do our best to process\nthem as fast as we can.\n\n## License\n\n[\u003e LICENSE](LICENSE)\n\n\u003c!-- BEGIN mdsh --\u003e\nMIT License\n\nCopyright (c) 2019 zimbatm and contributors\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n\u003c!-- END mdsh --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzimbatm%2Fmdsh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzimbatm%2Fmdsh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzimbatm%2Fmdsh/lists"}