https://github.com/flotwig/node-streaming-markov-chain-builder
A Markov chain builder that accepts input text as a stream and outputs a stream of n-grams.
https://github.com/flotwig/node-streaming-markov-chain-builder
Last synced: 6 months ago
JSON representation
A Markov chain builder that accepts input text as a stream and outputs a stream of n-grams.
- Host: GitHub
- URL: https://github.com/flotwig/node-streaming-markov-chain-builder
- Owner: flotwig
- Created: 2019-11-11T18:15:26.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2021-01-09T22:01:23.000Z (over 5 years ago)
- Last Synced: 2025-10-02T01:15:21.618Z (10 months ago)
- Language: TypeScript
- Size: 10.7 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# streaming-markov-chain-builder
[![Gitlab pipeline status (self-hosted)][ci-badge]][ci]
[![npm bundle size][size-badge]][npm]
[![npm][npm-badge]][npm]
[`streaming-markov-chain-builder`][npm] is a Markov chain builder that accepts input text as a stream and outputs a stream of n-grams.
## Installation
```shell
npm i --save streaming-markov-chain-builder
```
## Usage
```js
const { MarkovBuilder } = require('streaming-markov-chain-builder')
const builder = MarkovBuilder({
// number of "context words" to add to each individual word
// defaults to 1
order: 1
// optional - return true if this `word` should be considered 'proper'
// see `src/is-proper.ts` for the default implementation, exported as { isProperFn }
isProperFn: (word) => { return _.isUpperCase(word[0]) }
// optional - given a single line, return a list of sub-sentences
// see `src/sentence-splitter.ts` for the default implementation, exported as { sentenceSplitterFn }
sentenceSplitterFn: (line) => { return line.split(/[\.\?!]/g) }
})
// now, you can start ingesting data by writing it...
builder.write('the quick brown fox jumped over the lazy dog')
// or by streaming it in from a file...
fs.createReadStream('/tmp/corpus.txt').pipe(builder)
// since MarkovBuilder is a Transform stream, you can consume the output by reading from it...
builder.on('data', (ngram) => {
// see below for structure of the `ngram`
})
// you can also pipe the Transform stream to a consumer that accepts object-mode streams
builder.pipe(storage)
```
## Ngram structure
```typescript
export type MarkovNgram = {
// list of the words in this ngram
ngram: string[],
// is this a sentence starter?
sentenceStart: boolean,
// is this a sentence ender?
sentenceEnd: boolean,
// is ngram[0] proper?
startsWithProper: boolean
}
```
[ci-badge]: https://img.shields.io/gitlab/pipeline/flotwig/node-streaming-markov-chain-builder?gitlab_url=https%3A%2F%2Fci.chary.us
[ci]: https://ci.chary.us/flotwig/node-streaming-markov-chain-builder/pipelines
[size-badge]: https://img.shields.io/bundlephobia/min/streaming-markov-chain-builder
[npm-badge]: https://img.shields.io/npm/v/streaming-markov-chain-builder
[npm]: https://www.npmjs.com/package/streaming-markov-chain-builder