{"id":13941187,"url":"https://github.com/JP-Ellis/mathematica-notebook-filter","last_synced_at":"2025-07-20T04:30:49.936Z","repository":{"id":57636914,"uuid":"102358234","full_name":"JP-Ellis/mathematica-notebook-filter","owner":"JP-Ellis","description":"Filter Mathematica notebooks to commit them more easily","archived":false,"fork":false,"pushed_at":"2019-09-18T06:01:52.000Z","size":148,"stargazers_count":50,"open_issues_count":2,"forks_count":3,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-05-23T04:48:47.511Z","etag":null,"topics":["mathematica","version-control"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JP-Ellis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-04T12:19:21.000Z","updated_at":"2024-01-24T17:36:37.000Z","dependencies_parsed_at":"2022-08-30T09:41:04.359Z","dependency_job_id":null,"html_url":"https://github.com/JP-Ellis/mathematica-notebook-filter","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/JP-Ellis/mathematica-notebook-filter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JP-Ellis%2Fmathematica-notebook-filter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JP-Ellis%2Fmathematica-notebook-filter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JP-Ellis%2Fmathematica-notebook-filter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JP-Ellis%2Fmathematica-notebook-filter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JP-Ellis","download_url":"https://codeload.github.com/JP-Ellis/mathematica-notebook-filter/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JP-Ellis%2Fmathematica-notebook-filter/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266067254,"owners_count":23871324,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mathematica","version-control"],"created_at":"2024-08-08T02:01:13.539Z","updated_at":"2025-07-20T04:30:49.652Z","avatar_url":"https://github.com/JP-Ellis.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# mathematica-notebook-filter\n\n`mathematica-notebook-filter` is a program written in\n[Rust](https://www.rust-lang.org/) that parses Mathematica notebook files and\nstrips them of superfluous information so that they can be committed into\nversion control systems more easily.  Instructions to integrate this program\ninto version control systems can be found [below](#integration) and can be set\nup so that this is all done transparently without modifying the file on disk.\n\n[![Crates.io](https://img.shields.io/crates/v/mathematica-notebook-filter.svg)](https://crates.io/crates/mathematica-notebook-filter)\n[![Travis](https://img.shields.io/travis/JP-Ellis/mathematica-notebook-filter/master.svg)](https://travis-ci.org/JP-Ellis/mathematica-notebook-filter)\n[![Codecov](https://img.shields.io/codecov/c/github/JP-Ellis/mathematica-notebook-filter/master.svg)](https://codecov.io/gh/JP-Ellis/mathematica-notebook-filter)\n\nLicensed under [GPLv3](https://www.gnu.org/licenses/gpl-3.0.html).\n\n*This program has not been rigorously tested.  It works for me on all my\nNotebooks, but there may still be some situations which have not been accounted\nfor.  If you use this program, please let me know (both good and bad feedback).*\n\n## Introduction\n\nVersion control systems (such as [git](https://git-scm.com/) and\n[mercurial](https://www.mercurial-scm.org/) among many others) provide a\nfantastic way to keep track of changes to files in such a way that multiple\npeople can collaborate on them without accidentally overwriting other people's\nchanges.  Version control systems primarily keep track of source code and if two\npeople change the same file, it is possible to compare the two files\nside-by-side so that the changes can be merged.\n\nAlthough binary files (such as compiled outputs, images, PDFs, ...) can be\nincluded in a version control system too, it is generally not possible or\nmeaningful to compare two sets of changes to one binary file.  As a result,\nbinary files are quite opaque to version control systems and it is inadvisable\nto store binary files in a version control system if they will be changed\nfrequently.\n\nThis is specifically an issue for Mathematica notebooks as they store both\ninputs and outputs in the same file.  A quite typical example of this is the\nsimple input:\n\n```mathematica\nPlot[Sin[x] / x, {x, -4 Pi, 4 Pi}]\n```\n\nwhich, when plotted, is stored in the Notebook file as:\n\n```mathematica\nGraphicsBox[{{{{}, {},\n  TagBox[\n       {RGBColor[0.368417, 0.506779, 0.709798], AbsoluteThickness[1.6],\n        Opacity[1.], LineBox[CompressedData[\"\u003comitted\u003e\"]],\n        LineBox[CompressedData[\"\u003comitted\u003e\"]]},\n       Annotation[#,\n        \"Charting`Private`Tag$5185#1\"]\u0026 ], {}}, {{}, {}, {}}}, {}, {}},\n   AspectRatio-\u003eNCache[GoldenRatio^(-1), 0.6180339887498948],\n   Axes-\u003e{True, True},\n   AxesLabel-\u003e{None, None},\n   AxesOrigin-\u003e{0, 0},\n   DisplayFunction-\u003eIdentity,\n   Frame-\u003e{{False, False}, {False, False}},\n   FrameLabel-\u003e{{None, None}, {None, None}},\n   FrameTicks-\u003e{{Automatic,\n      Charting`ScaledFrameTicks[{Identity, Identity}]}, {Automatic,\n      Charting`ScaledFrameTicks[{Identity, Identity}]}},\n   GridLines-\u003e{None, None},\n   GridLinesStyle-\u003eDirective[\n     GrayLevel[0.5, 0.4]],\n   ImagePadding-\u003eAll,\n   Method-\u003e{\n    \"DefaultBoundaryStyle\" -\u003e Automatic, \"DefaultMeshStyle\" -\u003e\n     AbsolutePointSize[6], \"ScalingFunctions\" -\u003e None,\n     \"CoordinatesToolOptions\" -\u003e {\"DisplayFunction\" -\u003e ({\n         (Identity[#]\u0026 )[\n          Part[#, 1]],\n         (Identity[#]\u0026 )[\n          Part[#, 2]]}\u0026 ), \"CopiedValueFunction\" -\u003e ({\n         (Identity[#]\u0026 )[\n          Part[#, 1]],\n         (Identity[#]\u0026 )[\n          Part[#, 2]]}\u0026 )}},\n   PlotRange-\u003e\n    NCache[{{(-4) Pi, 4 Pi}, {-0.21723358083481298`,\n      0.9999892952885239}}, {{-12.566370614359172`,\n     12.566370614359172`}, {-0.21723358083481298`, 0.9999892952885239}}],\n   PlotRangeClipping-\u003eTrue,\n   PlotRangePadding-\u003e{{\n      Scaled[0.02],\n      Scaled[0.02]}, {\n      Scaled[0.05],\n      Scaled[0.05]}},\n   Ticks-\u003e{Automatic, Automatic}]\n```\n\nNote that the above snippet was significantly abbreviated as the compressed\nbase-64 encoded data is an additional 300 lines or so.\n\nFor the version control system, this large output is extremely cumbersome as a\nsmall change in the input (such as replacing `Sin[x]` with `Sin[2 x]`) will\nproduce a 300+ line diff.  The purpose of `mathematica-notebook-filter` is\nspecifically to avoid such large diffs and try and make them much more\nmeaningful.  It does so by parsing the Mathematica notebook file format and\nremoving all the output cells and metadata.  The program is implemented in\n[Rust](https://www.rust-lang.org/) and distributed on\n[crates.io](https://crates.io/crates/mathematica-notebook-filter).\n\nHaving said that, it should be noted that Mathematica unfortunately does not\nstore the input in a very simple form as it not only stores the plain\nMathematica expression, but also stores formatting information.  As a concrete\nexample, an input cell with the above plot function will be stored in the\nNotebook file as:\n\n```mathematica\nCell[BoxData[\n RowBox[{\"Plot\", \"[\",\n  RowBox[{\n   FractionBox[\n    RowBox[{\"Sin\", \"[\", \"x\", \"]\"}], \"x\"], \",\",\n   RowBox[{\"{\",\n    RowBox[{\"x\", \",\",\n     RowBox[{\n      RowBox[{\"-\", \"4\"}], \"Pi\"}], \",\",\n     RowBox[{\"4\", \"Pi\"}]}], \"}\"}]}], \"]\"}]], \"Input\"]\n```\n\nThe change of `Sin[x]` to `Sin[2 x]` results in the cell now being stored as:\n\n```mathematica\nCell[BoxData[\n RowBox[{\"Plot\", \"[\",\n  RowBox[{\n   FractionBox[\n    RowBox[{\"Sin\", \"[\",\n     RowBox[{\"2\", \"x\"}], \"]\"}], \"x\"], \",\",\n   RowBox[{\"{\",\n    RowBox[{\"x\", \",\",\n     RowBox[{\n      RowBox[{\"-\", \"4\"}], \"Pi\"}], \",\",\n     RowBox[{\"4\", \"Pi\"}]}], \"}\"}]}], \"]\"}]], \"Input\"]\n```\n\nThis program, at least at this stage, will *not* strip the extra formatting\ninformation.  If you wish to avoid the above, then you should save your\nnotebooks as scripts files (with extension `.wl` or `.m`).\n\n## Usage Notes\n\n`mathematica-notebook-filter` parses Mathematica notebook files (usually stored\nwith the extension `.nb`) and strips all generated outputs and other metadata.\nBy default, the program reads from standard input and outputs to standard\noutput.  Additional usage information can be obtained from\n\n```sh\nmathematica-notebook-filter --help\n```\n\nAlthough it is possible to use `mathematica-notebook-filter` manually, it is\ndesigned to be integrated with version control systems (see\n[below](#Integration) for instructions) such that Notebooks are first piped\nthrough the filter before the diffs are generated.  This is specifically\ndesigned so that original file is left untouched with all outputs and metadata\nremaining, and the filter effectively makes the version control system blind to\nthe extra content.\n\nIf you wish to run it manually, a simple call would be:\n\n```sh\nmathematica-notebook-filter -i my_notebook.nb -o my_notebook_cleaned.nb\n```\n\nIf both input and output files are identical, the program will first output to a\ntemporary file and only after successfully parsing the whole input will the\noriginal file be replaced.\n\nThis program does *not* parse the Wolfram language in general and is specific to\n*full* Mathematica notebooks; thus it makes some fairly strong assumptions about\nthe functions that will be found and their order.  It only parses a single\nNotebook at a time and will stop after the end of the first Notebook.  If an\nerror is encountered during the parsing, `mathematica-notebook-filter` will exit\nwith a non-zero code and the output will be left incomplete.\n\nIt also should be re-iterated that the best way to commit Mathematica code to a\nversion control system is to save the code in script files (`.wl` or `.m`).\nWhen doing so, Mathematica save the file in a very simple format (essentially a\nplain text file), without the superfluous formatting information and without\noutputs.  This unfortunately has the disadvantage that the Notebook interface is\nnot available.\n\nAlso note that Mathematica notebooks allow you to copy-paste graphics (such as\ngenerated plots) and use them as inputs.  If you do so, the version control\nsystem will be forced to include the full plot in the diff, thereby defeating\nthe point of `mathematica-notebook-filter`.  An alternative to copy-pasting\noutputs is to store the output into a variable, or use `%` (and `%%`, `%%%`,\n...) to refer to the previous output (though make sure to only use `%` within\nthe one cell and not across cells as `%` refers to the last generated output,\nnot the previous output in the Notebook order).\n\n## Installation\n\nThis program is written in [Rust](https://www.rust-lang.org/).  Probably the\neasiest way to install Rust is to use the [rustup.rs](https://www.rustup.rs/)\nscript.  Once set up, it should simply be a matter of running\n\n```sh\ncargo install mathematica-notebook-filter\n```\n\nThis will download, compile, and install `mathematica-notebook-filter` in your\nCargo home direction (`~/.cargo` by default on Linux).  Assuming you have\ncorrectly set up your PATH variable (which rustup.rs should have done\nautomatically), then you can execute the program by typing\n`mathematica-notebook-filter`.\n\n## Integration\n\n### Git\n\nIt is possible to set *attributes* based on pattern globs.  In this instance, we\nwant to make sure that all `*.nb` files are processed by this filter before\nbeing committed.  To globally set the attribute, add to `~/.gitattributes`:\n\n```text\n*.nb    filter=dropoutput_nb\n```\n\nand to your `~/.gitconfig`:\n\n```text\n[filter \"dropoutput_nb\"]\n    clean = mathematica-notebook-filter\n    smudge = cat\n```\n\n### Other\n\nPull requests to add instructions for other version control system are welcome.\n\n\n## Disclaimer\n\nThe Wolfram Research organization unfortunately does not appear to offer any\nspecification to their language or their file formats.  As a result, this filter\nwas entirely developed by inspecting outputs generated by Mathematica.\nSpecifically, this was developed using Mathematica 11.1 and thus there is no\nguarantee that this filter will work with past or future version of the Notebook\nfile format.\n\nIf you find a bug, please feel free to open an issue though please provide\nenough information to reproduce the bug or a minimal example of a Notebook file\nthat causes the issue.\n\n## Contributing\n\nPull requests to improve compatibility with other versions (or to fix bugs) are\nvery welcome.  If you find a bug, please feel free to open an issue and make\nsure to provide enough information to reproduce the bug or a minimal example of\na Notebook file that causes the issue.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJP-Ellis%2Fmathematica-notebook-filter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FJP-Ellis%2Fmathematica-notebook-filter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJP-Ellis%2Fmathematica-notebook-filter/lists"}