{"id":42212887,"url":"https://github.com/foliant-docs/foliantcontrib.utils.preprocessor_ext","last_synced_at":"2026-01-27T01:06:52.203Z","repository":{"id":57431788,"uuid":"187176109","full_name":"foliant-docs/foliantcontrib.utils.preprocessor_ext","owner":"foliant-docs","description":"[DEPRECATED]","archived":false,"fork":false,"pushed_at":"2021-07-20T13:35:34.000Z","size":16,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-11-03T07:18:37.468Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/foliant-docs.png","metadata":{"files":{"readme":"README.md","changelog":"changelog.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-05-17T08:17:59.000Z","updated_at":"2021-07-20T13:35:49.000Z","dependencies_parsed_at":"2022-09-02T12:40:22.342Z","dependency_job_id":null,"html_url":"https://github.com/foliant-docs/foliantcontrib.utils.preprocessor_ext","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/foliant-docs/foliantcontrib.utils.preprocessor_ext","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foliant-docs%2Ffoliantcontrib.utils.preprocessor_ext","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foliant-docs%2Ffoliantcontrib.utils.preprocessor_ext/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foliant-docs%2Ffoliantcontrib.utils.preprocessor_ext/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foliant-docs%2Ffoliantcontrib.utils.preprocessor_ext/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/foliant-docs","download_url":"https://codeload.github.com/foliant-docs/foliantcontrib.utils.preprocessor_ext/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foliant-docs%2Ffoliantcontrib.utils.preprocessor_ext/sbom","scorecard":{"id":406115,"data":{"date":"2025-08-11","repo":{"name":"github.com/foliant-docs/foliantcontrib.utils.preprocessor_ext","commit":"02d4f133081346e06bf480d453464ec1dcab740e"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":0,"reason":"Found 0/8 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-18T21:18:04.254Z","repository_id":57431788,"created_at":"2025-08-18T21:18:04.254Z","updated_at":"2025-08-18T21:18:04.254Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28794640,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-26T21:49:50.245Z","status":"ssl_error","status_checked_at":"2026-01-26T21:48:29.455Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-27T01:06:51.685Z","updated_at":"2026-01-27T01:06:52.179Z","avatar_url":"https://github.com/foliant-docs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![](https://img.shields.io/pypi/v/foliantcontrib.utils.preprocessor_ext.svg)](https://pypi.org/project/foliantcontrib.utils.preprocessor_ext/) [![](https://img.shields.io/github/v/tag/foliant-docs/foliantcontrib.utils.preprocessor_ext.svg?label=GitHub)](https://github.com/foliant-docs/foliantcontrib.utils.preprocessor_ext)\n\n**This package is deprecated and will be removed in future. Preprocessor extensions moved to the [main utils package](https://github.com/foliant-docs/foliantcontrib.utils).**\n\n# Overview\n\nExtension for basic Preprocessor class with useful methods and functions.\n\n# Usage\n\nUsually when we create preprocessors we inherit from `BasePreprocessor`:\n\n```python\n\nclass Preprocessor(BasePreprocessor):\n  ...\n```\n\nTo use all features of preprocessor_ext we should inherit from `BasePreprocessorExt` instead:\n\n```python\n\nclass Preprocessor(BasePreprocessorExt):\n  ...\n```\n\n# Features\n\n## Simplified tag processing workflow\n\nUsually to process tags in Markdown-files we write something like this:\n\n```python\nclass Preprocessor(BasePreprocessor):\n  ...\n  def _process_tags(content):\n    def sub(match):\n      # process the tag\n      return processed_string\n\n    self.pattern.sub(sub, content)\n\n  def apply(self):\n    self.logger.info('Applying preprocessor')\n    for markdown_file_path in self.working_dir.rglob('*.md'):\n        with open(markdown_file_path, encoding='utf8') as markdown_file:\n            content = markdown_file.read()\n\n        processed_content = self._process_tags(content)\n\n        if processed_content:\n            with open(markdown_file_path, 'w') as markdown_file:\n                markdown_file.write(processed_content)\n        self.logger.info('Preprocessor applied')\n```\n\nBasePreprocessorExt saves us from writing these many lines of the code which appear unchanged in most preprocessors.\n\nSo now instead of all that code we will write:\n\n```python\nclass Preprocessor(BasePreprocessorExt):\n  ...\n  def _process_tag(match):\n    # process the tag\n    return processed_string\n\n  def apply(self):\n    self._process_tags_for_all_files(func=self._process_tag, buffer=False)\n        self.logger.info('Preprocessor applied')\n```\n\nNote the `buffer=False` parameter (which in this case is excessive because it is `False` by default). If `buffer=True`, markdown files processing will be buffered, e.g. they won't be updated until all of them are processed.\n\nAs a bonus when using this workflow we get additional capabilities of logging and outputting warnings.\n\n### Issuing warnings\n\nWhen using `_process_tags_for_all_files` method to process tags we can also take advantage of `_warning` method.\n\nWhat this method does:\n1. Prints warning message to user and adds the md-file name to this message.\n2. Logs this message.\n3. May also add to logged message context of the tag where the problem occured.\n4. May also add to logged message the error traceback.\n5. If debug=true, #3 and #4 are also added to the message which is printed to user.\n\nSimple example:\n\n```python\nclass Preprocessor(BasePreprocessorExt):\n  ...\n  def _process_tag(match):\n    try:\n      config = open(self.options.get('config'))\n    except FileNotFoundError as e:\n      self._warning('Config file not found! Using default', error=e)\n    ...\n```\n\nHere if the exception gets triggered, user will see something like this in console:\n\n```bash\nParsing config\nDone\n\nApplying preprocessor my_preprocessor\nWARNING: [index.md] Config file not found! Using default\n\nDone\n\nApplying preprocessor _unescape\nDone\n\n────────────────────\nResult: slug.pre\n\n```\n\nAs we see, we've only supplied the message, but the preprocessor also added the `WARNING:` prefix and current file name to console. If we'd run the make command in debug mode (with `-d --debug` flag), we would also see full traceback of the error. In any case, traceback is stored in log.\n\n### Getting tag context\n\nSometimes we want to show the context of the tag, where we met some problems. By context I mean some words before the tag, some contents of the tag body and some words after the tag. It's really useful for debugging large md-files, with context you usually can identify the place in the document which causes errors.\n\nFor this purpose `BasePreprocessorExt` class has a handy method called `get_tag_context`. Give it the match object you are currently working with and it will return you the string with tag context.\n\nFor example:\n\n```python\nclass Preprocessor(BasePreprocessorExt):\n  ...\n  def _process_tag(match):\n    try:\n      config = open(self.options.get('config'))\n    except FileNotFoundError as e:\n      context = self.get_tag_context(match)\n      print(f'Config not found, check the tag:\\n{context}')\n    ...\n```\n\nThis will print:\n\n```bash\nParsing config\nDone\n\nApplying preprocessor my_preprocessor\n\nConfig not found, check the tag:\n...amet, consectetur adipisicing elit. Dolores ipsum non nisi voluptatum alias.\n\n\u003cmy_tag param=\"value\" config=\"wrong/path\"\u003e\n    Tag body consectetur adipisicing elit. Voluptatem.\n\u003c/template\u003e\n\nEnd of document.\n```\n\nNow user can easily understand where's the problem in his document.\n\n`get_tag_context` function accepts two parameters:\n\n`limit` (default: `100`) — number of characters included in context from before the tag, after the tag, and of the tag body;\n`full_tag` (default: `False`) — if this is True, the tag body is copied into context without cropping (useful for relatively small expected tag bodies).\n\n### Sending context to warning\n\nOne last thing to use the full power of `BasePreprocessorExt` warnings:\n\n`self._warning` accepts the context parameter. You can send there the context string. The context is only shown to user in the console if the debug mode is on, but it is always saved in the log file.\n\nExample:\n\n```python\nclass Preprocessor(BasePreprocessorExt):\n  ...\n  def _process_tag(match):\n    try:\n      config = open(self.options.get('config'))\n    except FileNotFoundError as e:\n      self._warning('Config file not found! Using default',\n                    context=self.get_context(match),\n                    error=e)\n    ...\n```\n\nNow if we catch this exception in normal mode, we will only get the md-filename and the message in the console. But if we run it in debug mode, we will get a full python traceback and the context of the tag. And a happy user.\n\n### allow_fail decorator\n\nOften we don't want the whole preprocessor to crash if there are some problems in just one tag of the document. We can easily achieve this using the `allow_fail` decorator, which is included in the `preprocessor_ext` module. Decorate your function, which is then sent to `_process_tags_for_all_files` method:\n\n```python\nfrom foliant.preprocessors.utils.preprocessor_ext import (BasePreprocessorExt,\n                                                          allow_fail)\n\n\nclass Preprocessor(BasePreprocessorExt):\n  ...\n  @allow_fail()\n  def _process_tag(match):\n    # process the tag\n    return processed_string\n\n  def apply(self):\n    self._process_tags_for_all_files(func=self._process_tag)\n        self.logger.info('Preprocessor applied')\n```\n\nNow in case _any_ error occurs in the `_process_tag` function, preprocessor will issue warning, show it to user, save it into log and skip the tag.\n\nThe `allow_fail` decorator accepts one argument, the error message, which will be shown to user in case of exception. It defaults to: _Failed to process tag. Skipping._\n\n## Simplified file processing workflow\n\nIf your preprocessor doesn't have tags, you're probably doing somehing like this:\n\n```python\nclass Preprocessor(BasePreprocessor):\n  ...\n  def _process_file(content):\n    processed = content\n    # do something with the content\n    return processed\n\n  def apply(self):\n    self.logger.info('Applying preprocessor')\n    for markdown_file_path in self.working_dir.rglob('*.md'):\n        with open(markdown_file_path, encoding='utf8') as markdown_file:\n            content = markdown_file.read()\n\n        processed_content = self._process_tags(content)\n\n        if processed_content:\n            with open(markdown_file_path, 'w') as markdown_file:\n                markdown_file.write(processed_content)\n        self.logger.info('Preprocessor applied')\n```\n\nBasePreprocessorExt saves us from writing these many lines of the code which appear unchanged in most preprocessors.\n\nSo now instead of all that code we will write:\n\n```python\nclass Preprocessor(BasePreprocessorExt):\n  ...\n  def _process_file(match):\n    processed = content\n    # do something with the content\n    return processed\n\n  def apply(self):\n    self._process_all_files(func=self._process_tag, buffer=False)\n        self.logger.info('Preprocessor applied')\n```\n\nNote the `buffer=False` parameter (which in this case is excessive because it is `False` by default). If `buffer=True`, markdown files processing will be buffered, e.g. they won't be updated until all of them are processed.\n\nAs a bonus we have `self.current_filepath` set to the path of currently processing file and `self.current_filename` — to the chapter name, as it is would be stated in `chapters` foliant.yml section.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffoliant-docs%2Ffoliantcontrib.utils.preprocessor_ext","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffoliant-docs%2Ffoliantcontrib.utils.preprocessor_ext","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffoliant-docs%2Ffoliantcontrib.utils.preprocessor_ext/lists"}