https://github.com/poppa/string-walker

Navigate within a string
https://github.com/poppa/string-walker

parser string-manipulation strings tokenizer

Last synced: 2 months ago
JSON representation

Navigate within a string

• Host: GitHub
• URL: https://github.com/poppa/string-walker
• Owner: poppa
• License: mit
• Created: 2019-09-19T16:37:13.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2024-06-18T16:59:05.000Z (7 months ago)
• Last Synced: 2024-10-15T11:03:56.815Z (3 months ago)
• Topics: parser, string-manipulation, strings, tokenizer
• Language: TypeScript
• Size: 1.12 MB
• Stars: 0
• Watchers: 3
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# string-walker

This is a simple class that lets you navigate within a string. This is useful

when creating simple tokenizers for instance.

# Examples

**Print each character in the string**

```ts

import { StringWalker } from 'string-walker'

const str = 'My name is Pontus'

const s = new StringWalker(str)

while (!s.isEof()) {

  console.log(s.currentChar())

  s.next()

}

/*

Outputs:

M

y

n

a

m

e

i

s

P

o

n

t

u

s

*/

```

**Tokenize a simple key/value file**

Say we have a key/value file that allows for a syntax like:

```

key1: strvalue

key2 :  12

key3:some value

```

We don't care about the whitespace between the key, separator and value

```ts

import { StringWalker } from 'string-walker'

const str = `

key1: strvalue

key2 :  12

key3:some value`

const tokens: Array<{ type: string, value: string }> = []

const s = new StringWalker(str)

let state: 'key' | 'value' | 'none' = 'none'

while (!s.isEof()) {

  // Consume whitespace

  if (['\n', ' ', '\t'].includes(s.currentChar())) {

    s.consume(['\n', ' ', '\t'])

  }

  switch (state) {

    // When state is none we're expecting a key

    case 'none': {

      // Find the position of either space, tab or :

      const end = s.findNextOf([' ', '\t', ':'])

      // If end is NaN we got to the end of the string without finding

      // what we are looking for

      if (isNaN(end)) {

        throw new Error('Syntax error')

      }

      // The key is located from the current position to `end`

      const key = s.substring(s.position, end)

      tokens.push({ type: 'key', value: key })

      // Now move the cursor to where we found the character

      s.moveTo(end)

      state = 'key'

      break

    }

    // When state is `key` we're expecting the next token to be a delimiter

    case 'key': {

      if (s.currentChar() !== ':') {

        throw new Error(`Syntax error: Expected : got ${s.currentChar()}`)

      }

      tokens.push({ type: 'delimiter', value: ':' })

      // Next we're expecting a value

      state = 'value'

      // Move the cursor to the next position

      s.next()

      break

    }

    case 'value': {

      // The value should stretch to the end of the line

      let end = s.findNext('\n')

      // If end is NaN we've reached the end of the string

      if (isNaN(end)) {

        end = s.length

      }

      const val = s.substring(s.position, end)

      tokens.push({ type: 'value', value: val })

      s.moveTo(end)

      // Now we're expecting either whitespace to the end or a new key

      state = 'none'

      break

    }

    // We should never get here

    default:

      throw new Error('Unknown stuff')

  }

}

console.log(tokens)

/*

Expected output:

[

  { type: 'key', value: 'key1' },

  { type: 'delimiter', value: ':' },

  { type: 'value', value: 'strvalue' },

  { type: 'key', value: 'key2' },

  { type: 'delimiter', value: ':' },

  { type: 'value', value: '12' },

  { type: 'key', value: 'key3' },

  { type: 'delimiter', value: ':' },

  { type: 'value', value: 'some value' }

]

*/

```