{"id":47592106,"url":"https://github.com/matecat/subfiltering","last_synced_at":"2026-04-01T17:37:28.112Z","repository":{"id":43007213,"uuid":"370287528","full_name":"matecat/subfiltering","owner":"matecat","description":"Subfiltering is a component used by Matecat and MyMemory for converting strings between the database, external services, and the UI layers. It provides a pipeline of filters to safely transform content across these layers while preserving XLIFF tags, HTML placeholders, and special entities.","archived":false,"fork":false,"pushed_at":"2026-03-18T16:33:24.000Z","size":421,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2026-03-19T06:24:40.403Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/matecat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-05-24T08:53:00.000Z","updated_at":"2026-03-18T14:43:58.000Z","dependencies_parsed_at":"2024-04-24T22:26:06.505Z","dependency_job_id":"24496590-9a30-4489-8e79-01b246284f4d","html_url":"https://github.com/matecat/subfiltering","commit_stats":{"total_commits":56,"total_committers":4,"mean_commits":14.0,"dds":0.4821428571428571,"last_synced_commit":"ccd41dca496e9212d4ca870db405f9877869127a"},"previous_names":[],"tags_count":83,"template":false,"template_full_name":null,"purl":"pkg:github/matecat/subfiltering","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matecat%2Fsubfiltering","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matecat%2Fsubfiltering/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matecat%2Fsubfiltering/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matecat%2Fsubfiltering/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/matecat","download_url":"https://codeload.github.com/matecat/subfiltering/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matecat%2Fsubfiltering/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31290537,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T13:12:26.723Z","status":"ssl_error","status_checked_at":"2026-04-01T13:12:25.102Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-01T17:37:27.582Z","updated_at":"2026-04-01T17:37:28.098Z","avatar_url":"https://github.com/matecat.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Matecat Subfiltering\n\n[![Build Status](https://app.travis-ci.com/matecat/subfiltering.svg?token=qBazxkHwP18h3EWnHjjF\u0026branch=master)](https://app.travis-ci.com/matecat/subfiltering)\n[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=matecat_subfiltering\u0026metric=alert_status)](https://sonarcloud.io/summary/new_code?id=matecat_subfiltering)\n[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=matecat_subfiltering\u0026metric=coverage)](https://sonarcloud.io/summary/new_code?id=matecat_subfiltering)\n[![Reliability Rating](https://sonarcloud.io/api/project_badges/measure?project=matecat_subfiltering\u0026metric=reliability_rating)](https://sonarcloud.io/summary/new_code?id=matecat_subfiltering)\n[![Maintainability Rating](https://sonarcloud.io/api/project_badges/measure?project=matecat_subfiltering\u0026metric=sqale_rating)](https://sonarcloud.io/summary/new_code?id=matecat_subfiltering)\n\n\nSubfiltering is a component used by Matecat and MyMemory for converting strings between the database, external services, and the UI layers. \nIt provides a pipeline of filters to safely transform content across these layers while preserving XLIFF tags, HTML placeholders, and special entities.\n\n## Overview\nEmbedding XML in a REST JSON payload is notoriously hard to render safely and legibly in a web browser. \nBrowsers, frameworks, and JSON serializers all have opinions about angle brackets, entities, and special characters. \nThe result is typically a mix of double-encoding, broken markup, or inline codes that translators can accidentally damage.\n\nThis library solves that by introducing reversible “layers” and a transformation pipeline that makes XML- and XLIFF-rich content safe for transport and UI display, while guaranteeing you can restore the exact original.\n\nWhat makes XML in JSON hard for the browser\n- Angle brackets and entities: Raw \u003c and \u003e conflict with HTML, and HTML/JS frameworks may escape or re-escape entities differently than you expect.\n- Inline codes in text: XLIFF inline tags (ph, pc, etc.), HTML/XML snippets, ICU, or sprintf tokens can be misinterpreted or edited improperly when shown as-is.\n- Safety vs. readability: You need to prevent XSS and layout breakage, but you also need a UI where users can read and edit the text around inline codes.\n\n- Use it when:\n  - Your source text includes variables, placeholders, XML, or HTML tags.\n  - You must accept user edits while preventing structural damage to tags.\n\n- What it gives you:\n    - Converts inline tags to robust placeholders with base64 “memory” of the original, then restores exactly after the round-trip.\n    - Prevents double-encoding and protects structural elements.\n\nIn short, this library is a bridge between “XML-correct” and “browser-safe,” letting you serve and accept JSON payloads that are straightforward to display and edit in the web UI,\nwhile guaranteeing that your original XML/XLIFF structure is preserved perfectly end to end.\n\n## How the library addresses it\n\n- Normalizes and preserves XLIFF tags across transformations.\n- Encodes/decodes special characters and placeholders for safe round-trips.\n- Converts between three processing layers:\n  - Layer 0 (Database): A database-safe XML form, suitable for persistence, export, and exact reconstruction.\n  - Layer 1 (External services): A transport-safe form tailored for MT/TM systems that aren’t XML-aware.\n  - Layer 2 (UI): A browser/UI-friendly form that replaces raw tags with safe placeholders and base64-backed metadata.\n\n- UI-friendly placeholders\n  - XML/XLIFF/HTML tags are converted to stable placeholders with an embedded, base64-encoded “memory” of the original tag.\n  - The UI can display and move placeholders without exposing raw markup, reducing the risk of accidental tag damage.\n\n- Reversible roundtrips\n  - When the browser sends edited text back, the library restores Layer 2 content to Layer 0, reconstructing the exact original tags from the placeholders.\n  - The same applies for Layer 0 ↔ Layer 1 when calling external services.\n\n\n- Supports XLIFF 2.x dataRef replacement, aligning inline codes from `\u003coriginalData\u003e` with inline tags in segments.\n  - If your XLIFF uses originalData with dataRef/dataRefStart/dataRefEnd, the library will create meaningful placeholders for the UI and then restore real XLIFF tags afterward.\n  - This keeps both the JSON payload and browser rendering safe without losing fidelity.\n\n\n## Installation\n\nInstall via Composer:\n\n```shell\nbash composer require matecat/subfiltering\n```\n\nRequirements:\n\n- PHP 7.4+\n- PHPUnit 9.x for running tests (dev)\n\n## Filters\n\nTwo concrete filters are provided (both implement `AbstractFilter`):\n\n- `Matecat\\SubFiltering\\MateCatFilter`\n- `Matecat\\SubFiltering\\MyMemoryFilter`\n\nCreate instances using the static `getInstance` factory:\n\n```php\n\u003c?php\n\nuse Matecat\\SubFiltering\\MateCatFilter;\nuse Matecat\\SubFiltering\\Contracts\\FeatureSetInterface;\nuse Matecat\\SubFiltering\\Mocks\\FeatureSet; // Example implementation lives under tests/ (use your own in production)\n\n$featureSet = new FeatureSet(); // must implement FeatureSetInterface\n\n// Optional parameters:\n// - $source (string): source language (e.g., 'en-US')\n// - $target (string): target language (e.g., 'it-IT')\n// - $dataRefMap (array): map for XLIFF 2 dataRef replacement (see section below)\n$filter = MateCatFilter::getInstance($featureSet, 'it-IT', 'en-US', []);\n```\n\nThe first argument MUST be a concrete implementation of `Matecat\\SubFiltering\\Contracts\\FeatureSetInterface`.\n\n## DataRef replacement (XLIFF 2)\n\nXLIFF 2.0/2.1 allows binding inline tags to `\u003coriginalData\u003e` via:\n\n- `\u003cph\u003e`, `\u003csc\u003e`, `\u003cec\u003e` using `dataRef`\n- `\u003cpc\u003e` using `dataRefStart` and `dataRefEnd`\n\nThis library can automatically introduce an `equiv-text` attribute (base64-encoded original value) based on a provided dataRef map, and convert `\u003cpc\u003e` pairs to Matecat-compatible `\u003cph\u003e` placeholders for UI consumption. On the way back, it restores the original XLIFF structure.\n\n- Full documentation and examples: docs/dataRef.md\n\nHow to provide the map:\n\n- Build an associative array where keys are data ids from `\u003coriginalData\u003e\u003cdata id=\"...\"\u003evalue\u003c/data\u003e\u003c/originalData\u003e`.\n- Pass that array as the fourth parameter when instantiating the filter.\n\nExample:\n\n```php\n\u003c?php\n\nuse Matecat\\SubFiltering\\MateCatFilter;\nuse Matecat\\SubFiltering\\Mocks\\FeatureSet;\n\n$dataRefMap = [\n    'source1' =\u003e '${AMOUNT}',\n    'source2' =\u003e '${RIDER}',\n];\n\n$filter = MateCatFilter::getInstance(new FeatureSet(), 'en-US', 'it-IT', $dataRefMap);\n\n// When converting to Layer 2 (UI), the filter will:\n// - add equiv-text to \u003cph\u003e/\u003csc\u003e/\u003cec\u003e using the map\n// - convert \u003cpc\u003e ranges to UI placeholders with originalData captured\n// When converting back to Layer 1/0, it restores the original XLIFF tags.\n```\n\nNote:\n\n- If a dataRef key exists but its value is null or empty, it is treated as the literal string `NULL`.\n- If the dataRef map is empty, the component still preserves inline codes by encoding original tags as Matecat placeholders to keep them safe in the UI.\n\nSee [docs/dataRef.md](https://github.com/matecat/subfiltering/blob/master/docs/dataRef.md) for concrete before/after string examples and behavior details.\n\n## Basic usage\n\nOnce you have a filter instance, use the methods below to convert between layers.\n\n`MateCatFilter` methods:\n\n- `fromLayer0ToLayer2`\n- `fromLayer1ToLayer2`\n- `fromLayer2ToLayer1`\n- `fromLayer2ToLayer0`\n- `fromLayer0ToLayer1`\n- `fromLayer1ToLayer0`\n- `fromRawXliffToLayer0`\n- `fromLayer0ToRawXliff`\n\n`MyMemoryFilter` methods:\n\n- `fromLayer0ToLayer1`\n- `fromLayer1ToLayer0`\n\nWhere:\n\n- Layer 0 = Database\n- Layer 1 = External services (MT/TM)\n- Layer 2 = Matecat UI\n\n### Example: DB to UI and back (with dataRef map)\n\n```php\n\u003c?php\n\nuse Matecat\\SubFiltering\\MateCatFilter;\nuse Matecat\\SubFiltering\\Mocks\\FeatureSet;\n\n$featureSet = new FeatureSet();\n\n$dataRefMap = [\n    'd1' =\u003e '_',\n    'd2' =\u003e '**',\n];\n\n$filter = MateCatFilter::getInstance($featureSet, 'en-US', 'it-IT', $dataRefMap);\n\n// Example Layer 0 content holding XLIFF inline codes\n$layer0 = \"Hi %s .\";\n\n// 1) Layer 0 -\u003e Layer 2 (UI)\n$ui = $filter-\u003efromLayer0ToLayer2($layer0);\n// 'Hi \u003cph id=\"mtc_1\" ctype=\"x-sprintf\" equiv-text=\"base64:JXM=\"/\u003e .'\n\n// 2) User edits happen in UI ...\n\n// 3) Layer 2 -\u003e Layer 0 (restore original XLIFF structure)\n$backToDb = $filter-\u003efromLayer2ToLayer0($ui);\n````\n\n### Example: External service roundtrip\n\n```php\n\u003c?php\n\nuse Matecat\\SubFiltering\\MateCatFilter;use Matecat\\SubFiltering\\Mocks\\FeatureSet;\n\n$filter = MateCatFilter::getInstance(new FeatureSet(), 'en-US', 'de-DE', []);\n\n$layer0 = 'Text with \u003cph id=\"1\" equiv-text=\"\u0026amp;lt;br/\u0026amp;gt;\"/\u003e and placeholders.';\n\n// Prepare for MT/TM\n$layer1 = $filter-\u003efromLayer0ToLayer1($layer0);\n// 'Text with \u003cph id=\"mtc_1\" ctype=\"x-original_ph\" x-orig=\"PHBoIGlkPSIxIiBlcXVpdi10ZXh0PSImbHQ7YnIvJmd0OyIvPg==\" equiv-text=\"base64:Jmx0O2JyLyZndDs=\"/\u003e and placeholders.'\n\n// ... send $layer1 to MT/TM and get $translatedLayer1 back ...\n\n// Restore for DB\n$layer0Restored = $filter-\u003efromLayer1ToLayer0($layer1);\n\n```\n\n### Injecting custom handlers into the pipeline\n\nGoal Show how to inject only a subset of supported injectable handlers into the transformation pipeline so they run alongside the built-in handlers.\nKey points\n\n- Handlers are classes that extend the base handler and implement a transform method.\n- You do not manually construct handlers; the pipeline instantiates them and injects the Pipeline instance via setPipeline.\n- You inject handlers by passing an array of class names to the filter factory method. Unknown classes are ignored. The sorter normalizes the final execution order.\n\nExample:\n\n```php\n\u003c?php\n\nuse Matecat\\SubFiltering\\MateCatFilter;\nuse Matecat\\SubFiltering\\Enum\\InjectableFiltersTags;\n\n// Example 1: enable only a subset of supported injectable handlers.\n// Only handlers known to the sorter will be kept and ordered.\n$featureSet = new YourFeatureSetImplementation(); // implements FeatureSetInterface\n\n$filter = MateCatFilter::getInstance(\n    $featureSet,\n    'en-US',\n    'it-IT',\n    [], // dataRef map\n    [\n        InjectableFiltersTags::markup,       // supported\n        InjectableFiltersTags::single_curly, // supported (disabled by default, but its injection is allowed and thus, enabled here)\n        'foobar'                             // any unsupported class would be ignored\n    ]\n);\n\n$input = 'You have {NUM_RESULTS, plural, =0 {no results} one {1 result} other {# results}} for \"{SEARCH_TERM}\".';\n\n// 'You have {NUM_RESULTS, plural, =0 {no results} one {1 result} other {# results}} for \"\u003cph id=\"mtc_1\" ctype=\"x-curly-brackets\" equiv-text=\"base64:e1NFQVJDSF9URVJNfQ==\"/\u003e\".'\n$l1 = $filter-\u003efromLayer0ToLayer1($input);\n\n$l2 = $filter-\u003efromLayer0ToLayer2($input);\n\n```\n\n### Disable all injectable handlers by passing null\n\nExample:\n\n```php\n\u003c?php\n\nuse Matecat\\SubFiltering\\MateCatFilter;\n\n// Example 2: disable all injectable handlers by passing null.\n// Only the fixed, non-injectable pipeline steps will run.\n$featureSet = new YourFeatureSetImplementation(); // implements FeatureSetInterface\n\n$filterNoInjectables = MateCatFilter::getInstance(\n    $featureSet,\n    'en-US',\n    'it-IT',\n    [],\n    null // no injectable handlers\n);\n\n$string    = 'This is \u0026amp;lt;b\u0026amp;gt;bold\u0026amp;lt;/b\u0026amp;gt; text.';\n\n$l1_no = $filter-\u003efromLayer0ToLayer1($input);\n// 'This is \u0026amp;lt;b\u0026amp;gt;bold\u0026amp;lt;/b\u0026amp;gt; text.'\n\n$l2_no = $filterNoInjectables-\u003efromLayer0ToLayer2($input);\n````\n\n## FeatureSet\n\nYou must provide a `FeatureSetInterface` implementation to adjust the pipeline per transformation. A simple, working example lives under the tests/ folder. In your application, implement only the features you need and register them via your FeatureSet.\n\n## Running tests\n\n```shell\nbash composer install ./vendor/bin/phpunit\n```\n\n## Support\n\nPlease open issues and feature requests on GitHub:\nhttps://github.com/matecat/subfiltering/issues\n\n## Authors\n\n- **Domenico Lupinetti** - https://github.com/ostico\n- **Mauro Cassani** - https://github.com/mauretto78\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE.md) file for details\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatecat%2Fsubfiltering","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatecat%2Fsubfiltering","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatecat%2Fsubfiltering/lists"}