https://github.com/jsh6789/ijp
Parser for JSON data generated by LLMs.
https://github.com/jsh6789/ijp
incremental-parsing json json-parser large-language-model llm python unlicense
Last synced: 6 months ago
JSON representation
Parser for JSON data generated by LLMs.
- Host: GitHub
- URL: https://github.com/jsh6789/ijp
- Owner: jsh6789
- License: unlicense
- Created: 2025-01-06T15:32:14.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2025-01-06T16:03:24.000Z (over 1 year ago)
- Last Synced: 2025-09-16T03:33:45.651Z (10 months ago)
- Topics: incremental-parsing, json, json-parser, large-language-model, llm, python, unlicense
- Language: Python
- Homepage:
- Size: 11.7 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Incremental JSON Parser
An incremental JSON parser geared towards handling JSON data generated by Large Language Models (LLMs). It is ideal in any situation where you need to process JSON data containing long string values as it arrives in small chunks.
## Installation from Git
To install the Incremental JSON Parser, clone the repository and use `pip` to install the package:
```bash
git clone https://github.com/jsh6789/ijp.git
cd ijp
pip install .
```
## Usage
The example below shows how to use the parser in your own scripts:
```python
from ijp import IncrementalJSONParser
json_string = '''
{
"price": 19.99,
"itemNo": "3735272",
"modulars" : [
{
"zone": "A",
"section": 29,
"position": 10
},
{
"zone": "A",
"section": 29,
"position": 15
}
]
}'''
# For demonstratory purpose: break the JSON into chunks.
chunk_size = 4
chunk_list = [
json_string[i:i + chunk_size] for i in range(0, len(json_string), chunk_size)
]
with IncrementalJSONParser() as parser:
for chunk in chunk_list:
parser.send(chunk)
for token in parser:
print(token)
```
This will output:
```
(['price'], 'float', 19.99)
(['itemNo'], 'stringpart', '37')
(['itemNo'], 'stringpart', '3527')
(['itemNo'], 'stringpart', '2')
(['itemNo'], 'string', '3735272')
(['modulars', 0, 'zone'], 'stringpart', 'A')
(['modulars', 0, 'zone'], 'string', 'A')
(['modulars', 0, 'section'], 'int', 29)
(['modulars', 0, 'position'], 'int', 10)
(['modulars', 1, 'zone'], 'stringpart', 'A')
(['modulars', 1, 'zone'], 'string', 'A')
(['modulars', 1, 'section'], 'int', 29)
(['modulars', 1, 'position'], 'int', 15)
```
More examples demonstrating uses for this parser can be found in the 'examples' directory.
## License
This project is in the public domain. See the accompanying UNLICENSE file for more info.