{"id":15284543,"url":"https://github.com/gbingersoll/sciengdox","last_synced_at":"2025-04-12T23:35:06.613Z","repository":{"id":57464381,"uuid":"200911916","full_name":"gbingersoll/sciengdox","owner":"gbingersoll","description":"A Python package for creating scientific and engineering documents via pandoc including inline-executable Python code.","archived":false,"fork":false,"pushed_at":"2023-02-11T02:34:44.000Z","size":5787,"stargazers_count":3,"open_issues_count":10,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-12T23:34:41.917Z","etag":null,"topics":["documentation","engineering","numpy","pandoc","python","science","scipy"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gbingersoll.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-06T19:27:49.000Z","updated_at":"2022-03-01T20:34:22.000Z","dependencies_parsed_at":"2023-01-25T13:46:03.843Z","dependency_job_id":null,"html_url":"https://github.com/gbingersoll/sciengdox","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gbingersoll%2Fsciengdox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gbingersoll%2Fsciengdox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gbingersoll%2Fsciengdox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gbingersoll%2Fsciengdox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gbingersoll","download_url":"https://codeload.github.com/gbingersoll/sciengdox/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248647258,"owners_count":21139081,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["documentation","engineering","numpy","pandoc","python","science","scipy"],"created_at":"2024-09-30T14:58:07.642Z","updated_at":"2025-04-12T23:35:06.557Z","avatar_url":"https://github.com/gbingersoll.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sciengdox\n\nA python package for creating scientific and engineering documents\nvia [`pandoc`](https://pandoc.org/) including inline-executable Python code.\n\n\n## Key Features\n\n1. [Pandoc filter](https://pandoc.org/filters.html) for converting pandoc\n   markdown to other formats (especially HTML and PDF).\n2. Codeblock execution\n3. Helper functions for generating tables, SVG\n   [matplotlib](https://matplotlib.org/) plots, etc.\n\n\n# Motivation\n\nThis is inspired by [`pweave`](http://mpastell.com/pweave/),\n[`codebraid`](https://github.com/gpoore/codebraid),\n[`knitr`](https://yihui.name/knitr/), and cousins, but I always seemed to have\nto do some pre/post-processing to get things the way I want them.  I already use\nother pandoc filters (e.g. pandoc-citeproc, pandoc-crossref), so why not simply\nhave another pandoc filter that will execute inline code and insert the results?\n\nAnother key is getting quality diagrams from scientific python code.  For\nexample, pweave automatically inserts generated images, but there doesn't seem\nto be a way to get SVG images without, again, pre- and post-processing in\nanother script.  SVG plots are, obviously, scalable and work much better for web\nand PDF outputs.\n\n\n# Development\n\nUse [`poetry`](https://python-poetry.org/) for local environment management.\nAfter cloning the repository:\n\n```shell\n$ cd \u003cproject-repo\u003e\n$ poetry install\n$ poetry shell\n```\n\nTo package and release:\n\n```shell\n$ poetry build\n$ poetry publish\n```\n\nBe sure to\n[configure your credentials](https://python-poetry.org/docs/repositories/#configuring-credentials)\nprior to publishing.\n\nSee also [this page](https://packaging.python.org/tutorials/packaging-projects/).\n\n# Use and Example\n\nAn example Pandoc markdown file can be found in `example`.  To process this\nfile, you need to have [`pandoc`](https://pandoc.org/) installed and in your\npath.  You also need to install the Pandoc filters\n[pandoc-crossref](https://github.com/lierdakil/pandoc-crossref) and\n[pandoc-citeproc](https://github.com/jgm/pandoc-citeproc) which provide nice\ncross-referencing and reference/bibliography handling.\n\n## Installation\n\nWhen working with macOS or Linux or\n[Linux on Windows via WSL](https://gist.github.com/gbingersoll/9e18afb9f4c3acd8674f5595c7e010f5)\n`pandoc` and the filters can be installed via [Homebrew](https://brew.sh/).  (On\nLinux/WSL, install [linuxbrew](https://docs.brew.sh/Homebrew-on-Linux).)  Then\nsimply run:\n\n```shell\n$ brew install pandoc\n$ brew install pandoc-crossref\n$ brew install pandoc-citeproc\n$ brew install librsvg\n```\n\nThen, of course, you need to install this filter and some other helpers for the\nexample.  The example helpers can be installed into your Python virtual\nenvironment by running:\n\n```shell\n$ poetry install -E examples\n```\n\n### Windows-specific Install\n\nTo set up an environment for Windows from scratch including terminals, editors,\nPython, etc., see\n[this gist](https://gist.github.com/gbingersoll/c3033f8cb41c3eb865563c0711a30545).\nAdditional installation steps to use this library include installing `pandoc`\nand additional filters and utilities.\n\nInstall `pandoc` by [downloading the installer](https://pandoc.org/installing.html)\nand following the standard instructions.  This should also get you\n`pandoc-citeproc.exe` for managing citations.\n\nInstall `pandoc-crossref` (for managing intra-document cross-references) by\n[downloading](https://github.com/lierdakil/pandoc-crossref/releases) the zipped\nWindows release.  Unzip it, and move `pandoc-crossref.exe` to a location that is\non your system path.  For example, you can move to next to `pandoc-citeproc.exe`\nin `C:\\Program Files\\Pandoc`.\n\nFinally, to handle embedding SVG images in PDF documents, this library relies on\n`rsvg-convert`.  This can be installed via\n[Chocolatey](https://chocolatey.org/).  Install the Chocolatey package manager\nif you do not already have it, and then run:\n\n```shell\n$ choco install rsvg-convert\n```\n\nInstead of (or in addition to) Chocolately, you can also install the\n[Scoop](https://scoop.sh/) installer.  Scoop does not currently have a formula\nfor `rsvg-convert`, but it can also be installed from\n[SourceForge](https://sourceforge.net/projects/tumagcc/files/rsvg-convert-dll-2.40.16.7z/download?use_mirror=phoenixnap)\nif you do not want to use Chocolatey.\n\n\n#### UTF-8 Note\n\nThe underlying Pandoc filter for executing Python code embedded in your\ndocuments relies on inter-process communication with a Python REPL behind the\nscenes.  The default inter-process character encoding for Python on Windows is\n[CP-1252](https://en.wikipedia.org/wiki/Windows-1252), and this can cause\nproblems if your Python scripts generate output with special characters (and if\nyou are doing any scientific or engineering writing, they definitely will).\n\nFortunately, this is easily worked-around by setting a Windows environment\nvariable `PYTHONIOENCODING` to `utf-8`.  After setting this, be sure to restart\nany open terminal windows for the change to take effect.\n\n#### Matplotlib Note\n\nIf you use `matplotlib` for generating plots in inline Python code in your\ndocument, you should explicity set the `Agg` backend early in your document (see\nthe `example/example.md` in this repo).  Without this, document conversion can\nhang when the `svg_figure` helper function is called.\n\nSomewhere near the top of your Markdown document, add an executable Python code\nblock (without `.echo` so it won't appear in the output) that includes:\n\n```python\nimport matplotlib\nmatplotlib.use('Agg')\n```\n\n#### Panflute Version Note\n\nThis plugin relies on the\n[`panflute`](https://github.com/sergiocorreia/panflute) Python package as a\nbridge between Python and `pandoc`'s Haskell.  The `panflute`\n[README](https://github.com/sergiocorreia/panflute#supported-pandoc-versions)\nlists API compatibility requirements between versions of `panflute` and versions\nof `pandoc`.  Double-check this if you run into errors that mention `panflute`\nwhen compiling a document.\n\nIf you are running an older version of `pandoc` (e.g. 2.9.2) and start a new\nproject, you will need to explicitly install the compatible `panflute` version\nin your environment with e.g. `poetry add panflute@1.12.5`.  Or\nalternatively install a `pandoc` version 2.11.x or later.\n\n### PDF Generation\n\nTo generate PDF files through Pandoc, you need to have `xelatex` installed.\nOn Linux/WSL:\n\n```shell\n$ sudo apt-get install texlive-xetex\n```\n\nOn macOS:\n\n```shell\n$ brew install --cask mactex\n```\n\nOn Windows (without WSL):\n\n[Download the MikTeX installer](https://miktex.org/download) and install as\nusual.  Then ensure that the binary folder is in your path (e.g. \n`C:\\Users\\\u003cusername\u003e\\AppData\\Local\\Programs\\MiKTeX 2.9\\miktex\\bin\\x64\\`).  Note\nthat the first time you generate a document, MikTex will prompt you to install a\nlot of packages, so watch for a MikTeX window popping up (possibly behind other\nwindows) and follow the prompts.\n\n\n### Fonts\n\nThe example templates rely on having a few fonts installed.\nThe fonts to get are the Google\n[Source Sans Pro](https://fonts.google.com/specimen/Source+Sans+Pro),\n[Source Code Pro](https://fonts.google.com/specimen/Source+Code+Pro), and\n[Source Serif Pro](https://fonts.google.com/specimen/Source+Serif+Pro) families.\n\nOn macOS or Windows (without WSL), these can simply be downloaded and installed\nas you would any other font.  On Linux via WSL, you can install these normally\non the Windows side and then synchronize the Windows font folder to the Linux\nside.  To do this, edit (using `sudo`) `/etc/fonts/local.conf` and add:\n\n```xml\n\u003c?xml version=\"1.0\"?\u003e\n\u003c!DOCTYPE fontconfig SYSTEM \"fonts.dtd\"\u003e\n\u003cfontconfig\u003e\n    \u003cdir\u003e/mnt/c/Windows/Fonts\u003c/dir\u003e\n\u003c/fontconfig\u003e\n```\n\nThen update the font cache on the Linux side:\n\n```shell\n$ sudo fc-cache -fv\n```\n\n### Stylesheets\n\nThe example file uses an HTML template that includes a CSS stylesheet that is\ngenerated from [SCSS](https://sass-lang.com/documentation/syntax).  To compile\nthis automatically, you need to have\n[SASS installed](https://sass-lang.com/install).\n\nOn macOS, this can be installed via Homebrew:\n\n```shell\n$ brew install sass/sass/sass\n```\n\nOn macOS/Linux/WSL/Windows it can be installed as a Node.js package (assuming\nyou already have [Node.js/npm](https://nodejs.org/) installed):\n\n```shell\n$ npm install -g sass\n```\n\n## Building\n\nThis Python library provides a script, `compiledoc`, that will appear in your\n`poetry` or `pipenv` virtual environment's path (or globally) once the library\nis installed.  In general, you provide an output directory and an input markdown\nfile, and it will build an HTML output when the `--html` flag is used (and also\nby default).\n\n```shell\n$ compiledoc -o output --html mydoc.md\n```\n\nTo build a PDF (via `xelatex`):\n\n```shell\n$ compiledoc -o output --pdf mydoc.md\n```\n\nTo build a Markdown file with executable Python output included (e.g. for\ndebugging purposes), specify `--md`.  This will generate a file in the output\ndirectory with (perhaps confusingly) the same name as the input:\n\n```shell\n$ compiledoc -o output --md mydoc.md\n```\n\nTo build everything, specify `--all`:\n\n```shell\n$ compiledoc -o output --all mydoc.md\n```\n\nTo see all available command line options (for specifying templates, paths to\nrequired external executables, static files like images and bibliography files,\netc.):\n\n```shell\n$ compiledoc --help\n```\n\n## Building the Example\n\nOnce everything is setup, compile the example HTML file by running:\n\n```shell\n$ cd example\n$ compiledoc -o output example.md\n```\n\nOpen `example/output/example.html` in your browser or use e.g. the [Live\nServer](https://marketplace.visualstudio.com/items?itemName=ritwickdey.LiveServer)\nplugin for VS Code.\n\n## Auto Regen\n\nTo autoregenerate the document (e.g. the HTML version, the output of which is\nwatched by the\n[Live Server](https://marketplace.visualstudio.com/items?itemName=ritwickdey.LiveServer)\n), you can use [Watchman](https://facebook.github.io/watchman/).\n\nTo create a trigger on a particular directory (`doc/` in this example) with a\n`notebook.md` file (change this to suit your purposes), copy the following into\na temporary `trigger.json` file:\n\n```json\n[\n    \"trigger\",\n    \"doc/\",\n    {\n        \"name\": \"build_html\",\n        \"expression\": [\n            \"anyof\",\n            [\n                \"match\",\n                \"notebook.md\",\n                \"wholename\"\n            ]\n        ],\n        \"command\": [\n            \"poetry\",\n            \"run\",\n            \"compiledoc\",\n            \"-o\",\n            \"output\",\n            \"--html\",\n            \"notebook.md\"\n        ]\n    }\n]\n```\n\nThen from your project root directory run:\n\n```shell\nwatchman -j \u003c trigger.json\nrm trigger.json\n```\n\nIt is also recommended that you add a `.watchmanconfig` file to the watched\ndirectory (e.g. `doc/`; also add `.watchmanconfig` to your `.gitignore`) with\nthe following contents:\n\n```json\n{\n  \"settle\": 3000\n}\n```\n\nThe settle parameter is in milliseconds.\n\nTo turn off watchman:\n\n```shell\nwatchman shutdown-server\n```\n\nTo turn it back on:\n\n```shell\ncd \u003cproject-root\u003e\nwatchman watch doc/\n```\n\nTo watch the Watchman:\n\n```shell\ntail -f /usr/local/var/run/watchman/\u003cusername\u003e-state/log\n```\n\n(Note that on Windows/WSL, to get `tail` to work the way you expect, you need to\nadd `---disable-inotify` to the command; and yes, that's three `-` for some\nreason.)\n\n## Older pandoc Versions\n\nFor `pandoc` 2.9 and earlier, the citation manager `pandoc-citeproc` was a\nseparate filter that gets added to the compliation pipeline.  The path to this\nfilter can be specified on the command line to `compiledoc` with the\n`--pandoc-citeproc PATH` flag.\n\nIn newer versions of `pandoc` (2.11 and beyond), the citeproc filter is built-in\nto pandoc and is run by adding `--citeproc` to the `pandoc` command-line.  The\n`compiledoc` script adds this by default unless the flag `--use-pandoc-citeproc`\nis added, in which case the older filter will be used.\n\nIf you do not with to run `citeproc` at all, you can add the flag\n`compiledoc --no-citeproc` to skip citation processing altogether.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgbingersoll%2Fsciengdox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgbingersoll%2Fsciengdox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgbingersoll%2Fsciengdox/lists"}