https://github.com/recogito/tei-standoffconverter-js
Convert between TEI/XML and plaintext without losing markup context.
https://github.com/recogito/tei-standoffconverter-js
ner nlp tei tei-xml
Last synced: 5 months ago
JSON representation
Convert between TEI/XML and plaintext without losing markup context.
- Host: GitHub
- URL: https://github.com/recogito/tei-standoffconverter-js
- Owner: recogito
- License: mit
- Created: 2025-04-29T05:45:48.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-05-15T07:48:08.000Z (about 1 year ago)
- Last Synced: 2025-05-15T08:41:55.707Z (about 1 year ago)
- Topics: ner, nlp, tei, tei-xml
- Language: TypeScript
- Homepage:
- Size: 178 KB
- Stars: 9
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Recogito TEI/XML Standoff Converter
A JavaScript/TypeScript utility that bridges the gap between TEI/XML documents and plaintext processing tools.
This library creates a reversible mapping between TEI/XML markup and character offsets in plaintext, allowing you to apply text analysis tools to TEI documents without losing markup context.
```
TEI/XML:
This is a sample text.
Plaintext: This is a sample text.
^----^
Identified entity
Result: This is a sample text.
```
The core logic was ported to TypeScript from the excellent Python [standoffconverter](https://github.com/standoff-nlp/standoffconverter) by [@millawell](https://github.com/millawell).
## Installation
```sh
npm install @recogito/standoff-converter
```
## Why?
Text analysis tools (e.g. for Named Entity Recognition) typically work with plaintext only. However, TEI/XML documents contain rich structural markup that must be stripped away before processing. When analysis tools identify entities or features at specific character positions in the plaintext, it's hard to map those positions back to the original TEI markup structure.
This library:
- Creates a linearized representation that maintains the relationship between plaintext character positions and XML markup.
- Allows you to process the plaintext with any text analysis tool.
- Maps the identified features back to the exact location in the original XML.
- Allows you to modify the TEI/XML structure while preserving all existing markup.
Perfect for enriching TEI documents with automatically extracted entities, annotations, or other textual features!
## Features
- Extract plaintext from TEI/XML while preserving a bidirectional mapping between character offsets and markup.
- Convert between plaintext character offsets and TEI XPointer expressions.
- Insert new inline tags at specific character positions (e.g. add `` or `` tags based on NER results).
- Preserve all original markup when serializing changes back to TEI/XML.
- Works in both Node.js and browser environments.
## Usage in Node
This library works in Node (using [xmldom](https://github.com/xmldom/xmldom) and [xpath](https://github.com/goto100/xpath) internally).
```ts
import { parseXML } from '@recogito/standoff-converter';
const xml = `
Sample TEI Document
This is a sample paragraph with markup.
`;
const parsed = parseXML(xml);
// Get plaintext
const text = parsed.text();
// XPointer expression from character position
const xpointer = parsed.getXPointer(550);
// Character position from XPointer expression
const position = parsed.getCharacterOffset('//TEI/text[1]/body[1]/p[1]/::5');
// Add inline tag at character position
parsed.addInline(5, 7, 'rs', { resp: 'aboutgeo' });
// Modified markup as a DOM Element
const el = parsed.xml();
// Modified markup serialized to string
const xml = parsed.xmlString();
```
## Usage in the Browser
You can use this library in the browser in combination with [CETEIcean](https://github.com/TEIC/CETEIcean).
```ts
import { parseXML } from '@recogito/standoff-converter';
window.onload = async function () {
const CETEIcean = new CETEI();
CETEIcean.getHTML5('paradise-lost.xml', data => {
document.getElementById('orig').appendChild(data);
const el = document.getElementById('orig').firstChild;
// Parse CETEIcean content
const parsed = parseXML(el);
// Get XPointer expressions from plaintext character offsets
console.log(parsed.getXPointer(550));
// Get character offsets from an XPointer expression (format: path::offset)
const xpointer = '//text[@xml:id="text-1"]/body[1]/div[1]/p[4]/hi[1]::5';
console.log(parsed.getCharacterOffset(xpointer));
// Add inline tags at character positions
parsed.addInline(550, 560, 'tei-note', { type: 'comment', resp: 'aboutgeo' });
// Serialize back to TEI/XML
const teiElement = parsed.toXML();
document.getElementById('serialized').appendChild(teiElement);
});
};
```
## API
### Core Functions
| Function | Description | Parameters | Return Value |
|----------------|-------------|------------|--------------|
| `parseXML(input)` | Parse TEI/XML | `input`: XML string or Element | `parsed` instance |
| `parsed.text()` | Get plaintext | None | `string` |
| `parsed.tokens` | Access linearized token array | - | `Array` of token objects |
| `parsed.getXPointer(offset)` | Convert plaintext character offset to XPointer | `offset`: number | `string` XPointer expression |
| `parsed.getCharacterOffset(xpointer)` | Convert XPointer to character offset | `xpointer`: string | `number` |
| `parsed.addInline(start, end, tagName, attrs)` | Insert inline tag at character positions | `start`: number
`end`: number
`tagName`: string
`attrs`: object | `void` |
| `parsed.xml()` | Get TEI/XML (DOM Element) | None | `Element` |
| `parsed.xmlString()` | Get XML (serialized string) | None | `string` |
### Recogito-Specific Functions
| Function/Method | Description | Parameters | Return Value |
|----------------|-------------|------------|--------------|
| `parsed.annotations(standOffId?)` | Get standoff annotations from all or a specific TEI `` element | `standOffId?`: string | `Array` of standoff annotation objects |
| `parsed.addStandOff(id)` | Add a new TEI `` element | `id`: string | `string` annotation ID |
| `parsed.addAnnotation(standOffId, annotation)` | Add Recogito annotation to `standOff` element | `standOffId`: string
`annotation`: standoff annotation | `void` |
| `parsed.addStandOffTag(standOffId, start, end, tag)` | Add Recogito annotation to `standOff` element that represents a simple (NER) tag | `standOffId`: string
`start`: number
`end`: number
`tag`: string or `{ id, label }`
| `void` |
## Known Issues
- **standOff Anchors and inline markup modifications**. Using `.addInline` will change the TEI markup. This has the potential to break XPath anchors for annotations in `` blocks. In order to prevent this, we would need to do before/after checks for affected anchors, and update them accordingly, so that they remain in sync with the changed TEI document.