{"id":50290261,"url":"https://github.com/shisa-ai/textguard","last_synced_at":"2026-05-28T05:33:53.341Z","repository":{"id":350556054,"uuid":"1207340258","full_name":"shisa-ai/textguard","owner":"shisa-ai","description":"Hostile-text normalization, inspection, and cleaning for LLM-adjacent systems.","archived":false,"fork":false,"pushed_at":"2026-04-10T21:53:42.000Z","size":330,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-28T05:33:51.278Z","etag":null,"topics":["cli","llm","prompt-injection","python","sanitization","security","text-processing","unicode","unicode-normalization","yara"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shisa-ai.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-04-10T20:46:03.000Z","updated_at":"2026-05-04T13:50:13.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/shisa-ai/textguard","commit_stats":null,"previous_names":["shisa-ai/textguard"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/shisa-ai/textguard","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shisa-ai%2Ftextguard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shisa-ai%2Ftextguard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shisa-ai%2Ftextguard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shisa-ai%2Ftextguard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shisa-ai","download_url":"https://codeload.github.com/shisa-ai/textguard/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shisa-ai%2Ftextguard/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33596316,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-28T02:00:06.440Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","llm","prompt-injection","python","sanitization","security","text-processing","unicode","unicode-normalization","yara"],"created_at":"2026-05-28T05:33:52.492Z","updated_at":"2026-05-28T05:33:53.329Z","avatar_url":"https://github.com/shisa-ai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# textguard\n\nHostile-text normalization, inspection, and cleaning for LLM-adjacent systems.\n\n`textguard` extracts the reusable text-defense work from [`shisad`](https://github.com/shisa-ai/shisad) into a standalone Python package that can scan and clean untrusted text inputs — prompts, Markdown, skill files, and other content — without dragging in daemon or framework dependencies.\n\n- Legitimate multilingual Unicode text is a first-class use case, so we do **NOT** convert everything to ASCII by default.\n- Lossy transforms are always explicit opt-in.\n- All decode paths have bounded depth and expansion limits.\n- Findings carry severity levels (`info`, `warn`, `error`) — there is no opaque aggregate risk score.\n- The package does not silently download models or make network requests.\n\n## Install\n\n```bash\npip install textguard            # core only, zero dependencies\npip install 'textguard[yara]'    # + pattern-based detection\npip install 'textguard[all]'     # everything\n```\n\nOr with uv:\n\n```bash\nuv pip install textguard\nuv tool install textguard        # CLI-only, isolated\nuvx textguard scan SKILL.md      # one-shot, no install\n```\n\n## CLI\n\nThe CLI works like a UNIX text utility:  pipe text through it, use it in shell scripts, or point it at files. Default usage is comparable to `cat` or `sed`: read from stdin or files, write cleaned output to stdout.\n\nBoth the CLI and [Python API](#python-api) are designed for agentic workflows. Scan untrusted inputs before they reach your model, or clean them inline as part of a pipeline.\n\n```bash\n# Scan: read-only, report findings\ntextguard scan SKILL.md\ntextguard scan SKILL.md --json\ntextguard scan docs/*.md --json \u003e report.json\n\n# Clean: output sanitized text\ntextguard clean SKILL.md              # cleaned text to stdout\ntextguard clean SKILL.md -i           # overwrite in place\ntextguard clean SKILL.md -o out.md    # write to file\ntextguard clean SKILL.md -i --report  # overwrite, human-readable report to stderr\ncat untrusted.txt | textguard clean - # pipe from stdin\n\n# Presets\ntextguard clean SKILL.md --preset strict\ntextguard clean SKILL.md --preset ascii\n\n# Optional backends\ntextguard scan --yara-rules ./rules/ SKILL.md\ntextguard scan --no-yara-bundled SKILL.md\ntextguard scan --split-tokens SKILL.md\ntextguard scan --promptguard ~/.local/share/textguard/models/promptguard2/ SKILL.md\n```\n\n### Presets\n\nPresets control how aggressive cleaning is:\n\n| Preset | Normalization | Strips | Decodes | Use case |\n|--------|--------------|--------|---------|----------|\n| **default** | NFC | Tag chars, soft hyphens, whitespace collapse, combining mark cap | No | Safe for all multilingual text including CJK |\n| **strict** | NFKC | All invisibles, bidi, variation selectors, tag chars, soft hyphens | All seven layers | Skill files, prompts, contexts where hidden content is suspect |\n| **ascii** | NFKC + ASCII transliteration | Everything non-ASCII | All seven layers | When you explicitly want ASCII-only output |\n\nThe default preset preserves legitimate multilingual text. NFKC is not the default because it destroys semantic content in Japanese and other scripts. Strict and ascii presets opt into progressively more aggressive cleaning.\n\nScan-time analysis is intentionally more aggressive than clean-time rewriting. `scan()` always strips hostile formatting and unwinds bounded encodings for analysis; presets control what `clean()` rewrites into the returned output.\n\nSplit-token smuggling detection is opt-in. Enable it with `TextGuard(split_tokens=True)`, `TEXTGUARD_SPLIT_TOKENS=1`, config file `split_tokens = true`, or CLI `--split-tokens`.\n\nExit codes from `scan` reflect the strongest signal in the result: structural findings map to `0` none, `1` info, `2` warn, `3` error; semantic tiers map to `0` none, `1` medium, `2` high, `3` critical. Runtime failures across subcommands return `4`.\n\n## Protection Tiers\n\ntextguard is organized into three tiers. Each adds detection capability on top of the previous.\n\n| Tier | Install | What it detects | Footprint |\n|------|---------|----------------|-----------|\n| **Core** | `textguard` | Invisible chars, bidi abuse, tag chars, soft hyphens, variation selectors, zalgo, homoglyphs/mixed-script, encoding layer abuse (URL, HTML entity, ROT13, base64, Unicode escapes, hex escapes, Punycode) | stdlib-only, small |\n| **YARA** | `textguard[yara]` | Pattern-based detection: prompt injection phrases, tool spoofing tags, custom signatures. Runs against both raw and decoded text. | +~6 MB (`yara-python`) |\n| **PromptGuard** | `textguard[promptguard]` | Semantic prompt injection / jailbreak classification via ONNX model. | +~27 MB wheels (`onnxruntime`, `transformers`) + ~295 MB model on first fetch |\n\nCore has zero runtime dependencies. Heavy backends are always optional extras.\n\n## Model Management\n\nPromptGuard requires a ~295 MB ONNX model pack ([shisa-ai/promptguard2-onnx](https://huggingface.co/shisa-ai/promptguard2-onnx)). textguard never downloads models silently.\n\n```bash\n# Fetch, verify, and install to the XDG data dir\ntextguard models fetch promptguard2\n\n# Point textguard to it\nexport TEXTGUARD_PROMPTGUARD_MODEL=~/.local/share/textguard/models/promptguard2\n\n# Or pass directly\ntextguard scan --promptguard ~/.local/share/textguard/models/promptguard2 SKILL.md\n```\n\n`textguard models fetch promptguard2` downloads from Hugging Face via stdlib HTTP, verifies the SSH ed25519 signature against the bundled public key, and checks SHA-256 file hashes from the manifest before installing under `~/.local/share/textguard/models/promptguard2/` (or `XDG_DATA_HOME` if set).\n\n## Python API\n\nThe two primary calls are `scan()` and `clean()` — they do what you'd expect. `scan()` inspects text and returns findings; `clean()` rewrites the text with hostile content removed.\n\n```python\nfrom textguard import scan, clean, TextGuard\n\n# Quick functional API — uses defaults, zero config\nresult = scan(text)          # ScanResult\ncleaned = clean(text)        # CleanResult\n\n# Configured instance — reusable, carries backend state\nguard = TextGuard(\n    preset=\"strict\",\n    split_tokens=True,\n    yara_rules_dir=\"./rules/\",\n    promptguard_model_path=\"~/.local/share/textguard/models/promptguard2/\",\n)\nresult = guard.scan(text)    # ScanResult\ncleaned = guard.clean(text)  # CleanResult\nsemantic = guard.score_semantic(text)  # SemanticResult\n```\n\nTop-level `scan()` and `clean()` are thin wrappers around `TextGuard` with default settings. For repeated calls or backend-enabled scanning, create a `TextGuard` instance.\n\n### Results\n\n`scan()` returns a `ScanResult` — findings, decode metadata, and optional semantic classification:\n\n```python\nresult = scan(text)\nfor f in result.findings:\n    print(f\"{f.severity}: {f.kind} at offset {f.offset} — {f.detail}\")\n```\n\n`clean()` returns a `CleanResult` — the cleaned text plus a report of what changed:\n\n```python\ncleaned = clean(text)\nprint(cleaned.text)          # the safe output\nfor c in cleaned.changes:\n    print(f\"  {c.kind}: {c.detail}\")\n```\n\n\n## Dependencies \u0026 Supply Chain\n\n- **Core runtime**: The core is stdlib-only and has **no runtime dependencies**.\n- **Unicode data**: vendored Unicode data for script ranges and confusables is generated by manual script run.\n- **Optional YARA**: `yara-python\u003e=4.5.4`\n- **Optional PromptGuard**: `onnxruntime\u003e=1.24.4`, `transformers\u003e=5.5.3`\n- Pinned via floor versions in `pyproject.toml`. Exact resolution through committed `uv.lock` with hashes.\n\nThe [pypi package](https://pypi.org/project/textguard) is published through Github CI with OIDC and SBOM.\n\n## Related Projects\n\nOur work depends most on these public contributions:\n\n- [Unicode 17.0 Scripts](https://www.unicode.org/Public/17.0.0/ucd/Scripts.txt)\n- [Unicode 17.0 Confusables](https://www.unicode.org/Public/security/17.0.0/confusables.txt)\n- [YARA](https://virustotal.github.io/yara/) / [yara-python](https://github.com/VirusTotal/yara-python)\n- [Llama Prompt Guard 2](https://github.com/meta-llama/PurpleLlama/tree/main/Llama-Prompt-Guard-2)\n\nWhile textguard fills a somewhat unique niche (hostile Unicode + encoding smuggling for LLM inputs), there are some foundational and related projects:\n\n- Unicode security references:\n  - https://www.unicode.org/reports/tr36/ — the foundational doc on Unicode attack surfaces\n  - https://www.unicode.org/reports/tr39/ — where confusables and mixed-script detection are formally specified\n- Unicode / text normalization:\n  - https://github.com/rspeer/python-ftfy — fixes mojibake and encoding problems\n  - https://github.com/vhf/confusable_homoglyphs — Python lib built on the same Unicode confusables data\n  - https://github.com/avian2/unidecode — ASCII transliteration\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshisa-ai%2Ftextguard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshisa-ai%2Ftextguard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshisa-ai%2Ftextguard/lists"}