{"id":15048294,"url":"https://github.com/github/html-pipeline","last_synced_at":"2025-10-04T08:31:17.604Z","repository":{"id":65974851,"uuid":"47654784","full_name":"github/html-pipeline","owner":"github","description":"HTML processing filters and utilities","archived":true,"fork":true,"pushed_at":"2022-03-18T10:58:25.000Z","size":770,"stargazers_count":16,"open_issues_count":0,"forks_count":11,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-09-29T00:21:25.592Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"gjtorikian/html-pipeline","license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/github.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-12-08T22:58:15.000Z","updated_at":"2024-07-31T03:15:49.000Z","dependencies_parsed_at":"2023-02-19T18:01:21.619Z","dependency_job_id":null,"html_url":"https://github.com/github/html-pipeline","commit_stats":null,"previous_names":[],"tags_count":35,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/github%2Fhtml-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/github%2Fhtml-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/github%2Fhtml-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/github%2Fhtml-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/github","download_url":"https://codeload.github.com/github/html-pipeline/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235232489,"owners_count":18957057,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T21:10:24.668Z","updated_at":"2025-10-04T08:31:12.326Z","avatar_url":"https://github.com/github.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HTML::Pipeline [![Build Status](https://travis-ci.org/jch/html-pipeline.svg?branch=master)](https://travis-ci.org/jch/html-pipeline)\n\nGitHub HTML processing filters and utilities. This module includes a small\nframework for defining DOM based content filters and applying them to user\nprovided content. Read an introduction about this project in\n[this blog post](https://github.com/blog/1311-html-pipeline-chainable-content-filters).\n\n- [Installation](#installation)\n- [Usage](#usage)\n  - [Examples](#examples)\n- [Filters](#filters)\n- [Dependencies](#dependencies)\n- [Documentation](#documentation)\n- [Extending](#extending)\n  - [3rd Party Extensions](#3rd-party-extensions)\n- [Instrumenting](#instrumenting)\n- [Contributing](#contributing)\n  - [Contributors](#contributors)\n  - [Releasing A New Version](#releasing-a-new-version)\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'html-pipeline'\n```\n\nAnd then execute:\n\n```sh\n$ bundle\n```\n\nOr install it yourself as:\n\n```sh\n$ gem install html-pipeline\n```\n\n## Usage\n\nThis library provides a handful of chainable HTML filters to transform user\ncontent into markup. A filter takes an HTML string or\n`Nokogiri::HTML::DocumentFragment`, optionally manipulates it, and then\noutputs the result.\n\nFor example, to transform Markdown source into Markdown HTML:\n\n```ruby\nrequire 'html/pipeline'\n\nfilter = HTML::Pipeline::MarkdownFilter.new(\"Hi **world**!\")\nfilter.call\n```\n\nFilters can be combined into a pipeline which causes each filter to hand its\noutput to the next filter's input. So if you wanted to have content be\nfiltered through Markdown and be syntax highlighted, you can create the\nfollowing pipeline:\n\n```ruby\npipeline = HTML::Pipeline.new [\n  HTML::Pipeline::MarkdownFilter,\n  HTML::Pipeline::SyntaxHighlightFilter\n]\nresult = pipeline.call \u003c\u003c-CODE\nThis is *great*:\n\n    some_code(:first)\n\nCODE\nresult[:output].to_s\n```\n\nPrints:\n\n```html\n\u003cp\u003eThis is \u003cem\u003egreat\u003c/em\u003e:\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode\u003esome_code(:first)\n\u003c/code\u003e\u003c/pre\u003e\n```\n\nTo generate CSS for HTML formatted code, use the [pygments.rb](https://github.com/tmm1/pygments.rb#usage) `#css` method. `pygments.rb` is a dependency of the `SyntaxHighlightFilter`.\n\nSome filters take an optional **context** and/or **result** hash. These are\nused to pass around arguments and metadata between filters in a pipeline. For\nexample, if you don't want to use GitHub formatted Markdown, you can pass an\noption in the context hash:\n\n```ruby\nfilter = HTML::Pipeline::MarkdownFilter.new(\"Hi **world**!\", :gfm =\u003e false)\nfilter.call\n```\n\n### Examples\n\nWe define different pipelines for different parts of our app. Here are a few\nparaphrased snippets to get you started:\n\n```ruby\n# The context hash is how you pass options between different filters.\n# See individual filter source for explanation of options.\ncontext = {\n  :asset_root =\u003e \"http://your-domain.com/where/your/images/live/icons\",\n  :base_url   =\u003e \"http://your-domain.com\"\n}\n\n# Pipeline providing sanitization and image hijacking but no mention\n# related features.\nSimplePipeline = Pipeline.new [\n  SanitizationFilter,\n  TableOfContentsFilter, # add 'name' anchors to all headers and generate toc list\n  CamoFilter,\n  ImageMaxWidthFilter,\n  SyntaxHighlightFilter,\n  EmojiFilter,\n  AutolinkFilter\n], context\n\n# Pipeline used for user provided content on the web\nMarkdownPipeline = Pipeline.new [\n  MarkdownFilter,\n  SanitizationFilter,\n  CamoFilter,\n  ImageMaxWidthFilter,\n  HttpsFilter,\n  MentionFilter,\n  EmojiFilter,\n  SyntaxHighlightFilter\n], context.merge(:gfm =\u003e true) # enable github formatted markdown\n\n\n# Define a pipeline based on another pipeline's filters\nNonGFMMarkdownPipeline = Pipeline.new(MarkdownPipeline.filters,\n  context.merge(:gfm =\u003e false))\n\n# Pipelines aren't limited to the web. You can use them for email\n# processing also.\nHtmlEmailPipeline = Pipeline.new [\n  PlainTextInputFilter,\n  ImageMaxWidthFilter\n], {}\n\n# Just emoji.\nEmojiPipeline = Pipeline.new [\n  PlainTextInputFilter,\n  EmojiFilter\n], context\n```\n\n## Filters\n\n* `MentionFilter` - replace `@user` mentions with links\n* `AbsoluteSourceFilter` - replace relative image urls with fully qualified versions\n* `AutolinkFilter` - auto_linking urls in HTML\n* `CamoFilter` - replace http image urls with [camo-fied](https://github.com/atmos/camo) https versions\n* `EmailReplyFilter` - util filter for working with emails\n* `EmojiFilter` - everyone loves [emoji](http://www.emoji-cheat-sheet.com/)!\n* `HttpsFilter` - HTML Filter for replacing http github urls with https versions.\n* `ImageMaxWidthFilter` - link to full size image for large images\n* `MarkdownFilter` - convert markdown to html\n* `PlainTextInputFilter` - html escape text and wrap the result in a div\n* `SanitizationFilter` - whitelist sanitize user markup\n* `SyntaxHighlightFilter` - [code syntax highlighter](#syntax-highlighting)\n* `TextileFilter` - convert textile to html\n* `TableOfContentsFilter` - anchor headings with name attributes and generate Table of Contents html unordered list linking headings\n\n## Dependencies\n\nFilter gem dependencies are not bundled; you must bundle the filter's gem\ndependencies. The below list details filters with dependencies. For example,\n`SyntaxHighlightFilter` uses [github-linguist](https://github.com/github/linguist)\nto detect and highlight languages. For example, to use the `SyntaxHighlightFilter`,\nadd the following to your Gemfile:\n\n```ruby\ngem 'github-linguist'\n```\n\n* `AutolinkFilter` - `rinku`\n* `EmailReplyFilter` - `escape_utils`, `email_reply_parser`\n* `EmojiFilter` - `gemoji`\n* `MarkdownFilter` - `github-markdown`\n* `PlainTextInputFilter` - `escape_utils`\n* `SanitizationFilter` - `sanitize`\n* `SyntaxHighlightFilter` - `github-linguist`\n* `TextileFilter` - `RedCloth`\n\n_Note:_ See [Gemfile](/Gemfile) `:test` block for version requirements.\n\n## Documentation\n\nFull reference documentation can be [found here](http://rubydoc.info/gems/html-pipeline/frames).\n\n## Extending\nTo write a custom filter, you need a class with a `call` method that inherits\nfrom `HTML::Pipeline::Filter`.\n\nFor example this filter adds a base url to images that are root relative:\n\n```ruby\nrequire 'uri'\n\nclass RootRelativeFilter \u003c HTML::Pipeline::Filter\n\n  def call\n    doc.search(\"img\").each do |img|\n      next if img['src'].nil?\n      src = img['src'].strip\n      if src.start_with? '/'\n        img[\"src\"] = URI.join(context[:base_url], src).to_s\n      end\n    end\n    doc\n  end\n\nend\n```\n\nNow this filter can be used in a pipeline:\n\n```ruby\nPipeline.new [ RootRelativeFilter ], { :base_url =\u003e 'http://somehost.com' }\n```\n\n### 3rd Party Extensions\n\nIf you have an idea for a filter, propose it as\n[an issue](https://github.com/jch/html-pipeline/issues) first. This allows us discuss\nwhether the filter is a common enough use case to belong in this gem, or should be\nbuilt as an external gem.\n\nHere are some extensions people have built:\n\n* [html-pipeline-asciidoc_filter](https://github.com/asciidoctor/html-pipeline-asciidoc_filter)\n* [jekyll-html-pipeline](https://github.com/gjtorikian/jekyll-html-pipeline)\n* [nanoc-html-pipeline](https://github.com/burnto/nanoc-html-pipeline)\n* [html-pipeline-bity](https://github.com/dewski/html-pipeline-bitly)\n* [html-pipeline-cite](https://github.com/lifted-studios/html-pipeline-cite)\n* [tilt-html-pipeline](https://github.com/bradgessler/tilt-html-pipeline)\n* [html-pipeline-wiki-link'](https://github.com/lifted-studios/html-pipeline-wiki-link) - WikiMedia-style wiki links\n* [task_list](https://github.com/github/task_list) - GitHub flavor Markdown Task List\n* [html-pipeline-rouge_filter](https://github.com/JuanitoFatas/html-pipeline-rouge_filter) - Syntax highlight with [Rouge](https://github.com/jneen/rouge/)\n* [html-pipeline-nico_link](https://github.com/rutan/html-pipeline-nico_link) - An HTML::Pipeline filter for [niconico](http://www.nicovideo.jp) description links\n* [html-pipeline-gitlab](https://gitlab.com/gitlab-org/html-pipeline-gitlab) - This gem implements various filters for html-pipeline used by GitLab\n* [html-pipeline-youtube](https://github.com/st0012/html-pipeline-youtube) - An HTML::Pipeline filter for YouTube links\n* [html-pipeline-flickr](https://github.com/st0012/html-pipeline-flickr) - An HTML::Pipeline filter for Flickr links\n* [html-pipeline-vimeo](https://github.com/dlackty/html-pipeline-vimeo) - An HTML::Pipeline filter for Vimeo links\n* [html-pipeline-hashtag](https://github.com/mr-dxdy/html-pipeline-hashtag) - An HTML::Pipeline filter for hashtags\n* [html-pipeline-linkify_github](https://github.com/jollygoodcode/html-pipeline-linkify_github) - An HTML::Pipeline filter to autolink GitHub urls\n\n## Instrumenting\n\nFilters and Pipelines can be set up to be instrumented when called. The pipeline\nmust be setup with an [ActiveSupport::Notifications]\n(http://api.rubyonrails.org/classes/ActiveSupport/Notifications.html)\ncompatible service object and a name. New pipeline objects will default to the\n`HTML::Pipeline.default_instrumentation_service` object.\n\n``` ruby\n# the AS::Notifications-compatible service object\nservice = ActiveSupport::Notifications\n\n# instrument a specific pipeline\npipeline = HTML::Pipeline.new [MarkdownFilter], context\npipeline.setup_instrumentation \"MarkdownPipeline\", service\n\n# or set default instrumentation service for all new pipelines\nHTML::Pipeline.default_instrumentation_service = service\npipeline = HTML::Pipeline.new [MarkdownFilter], context\npipeline.setup_instrumentation \"MarkdownPipeline\"\n```\n\nFilters are instrumented when they are run through the pipeline. A\n`call_filter.html_pipeline` event is published once the filter finishes. The\n`payload` should include the `filter` name. Each filter will trigger its own\ninstrumentation call.\n\n``` ruby\nservice.subscribe \"call_filter.html_pipeline\" do |event, start, ending, transaction_id, payload|\n  payload[:pipeline] #=\u003e \"MarkdownPipeline\", set with `setup_instrumentation`\n  payload[:filter] #=\u003e \"MarkdownFilter\"\n  payload[:context] #=\u003e context Hash\n  payload[:result] #=\u003e instance of result class\n  payload[:result][:output] #=\u003e output HTML String or Nokogiri::DocumentFragment\nend\n```\n\nThe full pipeline is also instrumented:\n\n``` ruby\nservice.subscribe \"call_pipeline.html_pipeline\" do |event, start, ending, transaction_id, payload|\n  payload[:pipeline] #=\u003e \"MarkdownPipeline\", set with `setup_instrumentation`\n  payload[:filters] #=\u003e [\"MarkdownFilter\"]\n  payload[:doc] #=\u003e HTML String or Nokogiri::DocumentFragment\n  payload[:context] #=\u003e context Hash\n  payload[:result] #=\u003e instance of result class\n  payload[:result][:output] #=\u003e output HTML String or Nokogiri::DocumentFragment\nend\n```\n\n## FAQ\n\n### 1. Why doesn't my pipeline work when there's no root element in the document?\n\nTo make a pipeline work on a plain text document, put the `PlainTextInputFilter`\nat the beginning of your pipeline. This will wrap the content in a `div` so the\nfilters have a root element to work with. If you're passing in an HTML fragment,\nbut it doesn't have a root element, you can wrap the content in a `div`\nyourself. For example:\n\n```ruby\nEmojiPipeline = Pipeline.new [\n  PlainTextInputFilter,  # \u003c- Wraps input in a div and escapes html tags\n  EmojiFilter\n], context\n\nplain_text = \"Gutentag! :wave:\"\nEmojiPipeline.call(plain_text)\n\nhtml_fragment = \"This is outside of an html element, but \u003cstrong\u003ethis isn't. :+1:\u003c/strong\u003e\"\nEmojiPipeline.call(\"\u003cdiv\u003e#{html_fragment}\u003c/div\u003e\") # \u003c- Wrap your own html fragments to avoid escaping\n```\n\n### 2. How do I customize a whitelist for `SanitizationFilter`s?\n\n`SanitizationFilter::WHITELIST` is the default whitelist used if no `:whitelist`\nargument is given in the context. The default is a good starting template for\nyou to add additional elements. You can either modify the constant's value, or\nre-define your own constant and pass that in via the context.\n\n## Contributing\n\nPlease review the [Contributing Guide](https://github.com/jch/html-pipeline/blob/master/CONTRIBUTING.md).\n\n1. [Fork it](https://help.github.com/articles/fork-a-repo)\n2. Create your feature branch (`git checkout -b my-new-feature`)\n3. Commit your changes (`git commit -am 'Added some feature'`)\n4. Push to the branch (`git push origin my-new-feature`)\n5. Create new [Pull Request](https://help.github.com/articles/using-pull-requests)\n\nTo see what has changed in recent versions, see the [CHANGELOG](https://github.com/jch/html-pipeline/blob/master/CHANGELOG.md).\n\n### Contributors\n\nThanks to all of [these contributors](https://github.com/jch/html-pipeline/graphs/contributors).\n\nProject is a member of the [OSS Manifesto](http://ossmanifesto.org/).\n\n### Releasing A New Version\n\nThis section is for gem maintainers to cut a new version of the gem.\n\n* create a new branch named `release-x.y.z` where `x.y.z` follows [semver](http://semver.org)\n* update lib/html/pipeline/version.rb to next version number X.X.X\n* update CHANGELOG.md. Prepare a draft with `script/changelog`\n* push branch and create a new pull request\n* after tests are green, merge to master\n* on the master branch, run `script/release`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgithub%2Fhtml-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgithub%2Fhtml-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgithub%2Fhtml-pipeline/lists"}