awesome-llm-json

Resource list for generating JSON using LLMs via function calling, tools, CFG. Libraries, Models, Notebooks, etc.
https://github.com/imaurer/awesome-llm-json

Last synced: 3 minutes ago
JSON representation

Leaderboards
- HuggingFace Open LLM Leaderboard
- Code Generation on HumanEval
- Berkeley Function-Calling Leaderboard (BFCL) - calling capabilities including over 2k question-function-answer pairs across languages like Python, Java, JavaScript, SQL, and REST API, focusing on simple, multiple, and parallel function calls, as well as function relevance detection.
Local LLMs
- Local LLM Repositories
  - Open Assistant - A chat-based ChatGPT-like large language model. (2023-04-15, Pythia, LLAMA, Apache 2.0 License)
- Local LLM Spaces, Models & Datasets
- Local LLM Resources
LLM-based Tools
- Local LLM Resources
  - TextSynth
Non-English Models & Datasets
- Local LLM Resources
  - Polpaca
Autonomous Agents
- Autonomous Agent Resources
Python Libraries
- Instructor
- DSPy - blocks/8-typed_predictors.md) to leverage [Pydantic](https://github.com/pydantic/pydantic) for enforcing type constraints on inputs and outputs, improving upon string-based fields.
- FuzzTypes
- guidance - 2.0) enables constrained generation, interleaving Python logic with LLM calls, reusable functions, and calling external tools. Optimizes prompts for faster generation.
- LangChain
- LiteLLM
- LlamaIndex - defined Pydantic programs for specific output types.
- Marvin - 2.0) is a lightweight toolkit for building reliable natural language interfaces with self-documenting tools for tasks like entity extraction and multi-modal support.
- Pydantic
- SGLang - 2.0) allows specifying JSON schemas using regular expressions or Pydantic models for constrained decoding. Its high-performance runtime accelerates JSON decoding.
- SynCode - guided generation of Large Language Models (LLMs). It supports CFG for Python, Go, Java, JSON, YAML, and many more.
- Mirascope
- Magnetic
- Formatron - string templates that support regular expressions, context-free grammars, JSON schemas, and Pydantic models. Formatron integrates seamlessly with various model inference libraries.
- PydanticAI
- Transformers-cfg - free grammar (CFG) support via an EBNF interface. It enables grammar-constrained generation with minimal changes to existing code of transformers and supports JSON mode and JSON Schema.
Jupyter Notebooks
Videos
- GPT Extracting Unstructured Data with Datasette and GPT-4 Turbo - 04-09, Simon Willison) showcases the datasette-extract plugin's ability to populate database tables from unstructured text and images, leveraging GPT-4 Turbo's API for data extraction.
- Hermes 2 Pro Overview - 03-18, Prompt Engineer) introduces Hermes 2 Pro, a 7B parameter model excelling at function calling and structured JSON output. Demonstrates 90% accuracy in function calling and 84% in JSON mode, outperforming other models.
- Mistral AI Function Calling - 02-24, Sophia Yang) demonstrates connecting LLMs to external tools, generating function arguments, and executing functions. Could be extended to generate or manipulate JSON data.
- Function Calling in Ollama vs OpenAI - 02-13, [Matt Williams](https://twitter.com/Technovangelist)) clarifies that models generate structured output for parsing and invoking functions. Compares implementations, highlighting Ollama's simpler approach and using few-shot prompts for consistency.
- LLM Engineering: Structured Outputs - 02-12, [Jason Liu](https://twitter.com/jxnlco), [Weights & Biases Course](https://www.wandb.courses/)) offers a concise course on handling structured JSON output, function calling, and validations using Pydantic, covering essentials for robust pipelines and efficient production integration.
- Pydantic is all you need - 10-10, [Jason Liu](https://twitter.com/jxnlco), [AI Engineer Conference](https://www.ai.engineer/)) discusses the importance of Pydantic for structured prompting and output validation, introducing the Instructor library and showcasing advanced applications for reliable and maintainable LLM-powered applications.
- LLM Structured Output for Function Calling with Ollama - 03-25, Andrej Baranovskij) demonstrates function calling-based data extraction using Ollama, Instructor and [Sparrow agent](https://github.com/katanaml/sparrow).
Prompting Tools
- Autonomous Agent Resources
  - AimStack
Terminology
- JSON Mode
- Tool Usage
- Tool Usage
- Guided Generation - Free Grammar](https://en.wikipedia.org/wiki/Context-free_grammar).
- GPT Actions
- Structured Outputs
- Tool Usage
- Function Calling
Hosted Models
- Function Calling
- Function Calling - generation/json-mode) [Pricing](https://docs.endpoints.anyscale.com/pricing/) [Announcement (2023)](https://www.anyscale.com/blog/anyscale-endpoints-json-mode-and-function-calling-features) |
- Function Calling - us/pricing/details/cognitive-services/openai-service/#pricing) [Mistral Pricing](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/000-000.mistral-ai-large-offer?tab=PlansAndPrice) |
- Function Calling - R (2024-03-11)](https://txt.cohere.com/command-r/) [Command-R+ (2024-04-04)](https://txt.cohere.com/command-r-plus-microsoft-azure/) |
- Function Calling
- API Docs - 03-18)](https://rysana.com/inversion) |
- Function Calling - mode) [Pricing](https://together.ai/pricing/) [Announcement 2024-01-31](https://www.together.ai/blog/function-calling-json-mode) |
- API Docs - use) |
- Function Calling
- Function Calling - generation/json-mode) [Pricing](https://openai.com/pricing) [Announcement (2023-06-13)](https://openai.com/blog/function-calling-and-other-api-updates) |
- Function Calling - generation/json-mode) [Pricing](https://openai.com/pricing) [Announcement (2023-06-13)](https://openai.com/blog/function-calling-and-other-api-updates) |
- Function Calling - generation/json-mode) [Pricing](https://docs.endpoints.anyscale.com/pricing/) [Announcement (2023)](https://www.anyscale.com/blog/anyscale-endpoints-json-mode-and-function-calling-features) |
- many open-source models - generation-inference/conceptual/guidance#guidance) For [free locally](https://huggingface.co/docs/text-generation-inference/basic_tutorials/consuming_tgi), or via [dedicated](https://huggingface.co/docs/inference-endpoints/index) or [serverless](https://huggingface.co/docs/api-inference/index) endpoints. |
- Function Calling - generation/json-mode) [Pricing](https://docs.endpoints.anyscale.com/pricing/) [Announcement (2023)](https://www.anyscale.com/blog/anyscale-endpoints-json-mode-and-function-calling-features) |
- API Docs - use) |
- API Docs - use) |
Local Models
- Hermes 2 Pro - Mistral 7B - 03-13, Nous Research) is a 7B parameter model that excels at function calling, JSON structured outputs, and general tasks. Trained on an updated OpenHermes 2.5 Dataset and a new function calling dataset, it uses a special system prompt and multi-turn structure. Achieves 91% on function calling and 84% on JSON mode evaluations.
- Gorilla OpenFunctions v2 - 02-27, Apache 2.0 license, [Charlie Cheng-Jie Ji et al.](https://gorilla.cs.berkeley.edu//blogs/7_open_functions_v2.html)) interprets and executes functions based on JSON Schema Objects, supporting multiple languages and detecting function relevance.
- NexusRaven-V2 - 12-05, Nexusflow) is a 13B model outperforming GPT-4 in zero-shot function calling by up to 7%, enabling effective use of software tools. Further instruction-tuned on CodeLlama-13B-instruct.
- Functionary - 08-04, [MeetKai](https://meetkai.com/)) interprets and executes functions based on JSON Schema Objects, supporting various compute requirements and call types. Compatible with OpenAI-python and llama-cpp-python for efficient function execution in JSON generation tasks.
- C4AI Command R+ - 03-20, CC-BY-NC, Cohere) is a 104B parameter multilingual model with advanced Retrieval Augmented Generation (RAG) and tool use capabilities, optimized for reasoning, summarization, and question answering across 10 languages. Supports quantization for efficient use and demonstrates unique multi-step tool integration for complex task execution.
- Hugging Face TGI - generation-inference/supported_models).
- Mistral 7B Instruct v0.3 - 05-22, Apache 2.0) an instruct fine-tuned version of Mistral with added function calling support.
Blog Articles
- Structured Generation Improves LLM performance: GSM8K Benchmark - 03-15, .txt Engineering) demonstrates consistent improvements across 8 models, highlighting benefits like "prompt consistency" and "thought-control."
- LoRAX + Outlines: Better JSON Extraction with Structured Generation and LoRA - 03-03, Predibase Blog) combines Outlines with LoRAX v0.8 to enhance extraction accuracy and schema fidelity through structured generation, fine-tuning, and LoRA adapters.
- FU, Show Me The Prompt. Quickly understand inscrutable LLM frameworks by intercepting API calls - 02-14, Hamel Husain) provides a practical guide to intercepting API calls using mitmproxy, gaining insights into tool functionality, and assessing necessity. Emphasizes minimizing complexity and maintaining close connection with underlying LLMs.
- Coalescence: making LLM inference 5x faster - 02-02, .txt Engineering) shows how structured generation can be made faster than unstructured generation using a technique called "coalescence", with a caveat regarding how it may affect the quality of the generation.
- Pushing ChatGPT's Structured Data Support To Its Limits - 12-21, Max Woolf) delves into leveraging ChatGPT's capabilities using paid API, JSON schemas, and Pydantic. Highlights techniques for improving output quality and the benefits of structured data support.
- Using OpenAI functions and their Python library for data extraction - 07-09, Simon Willison) demonstrates extracting structured data using OpenAI Python library and function calling in a single API call, with a code example and suggestions for handling streaming limitations.
- Why use Instructor? - 11-18, Jason Liu) explains the benefits of the library, offering a readable approach, support for partial extraction and various types, and a self-correcting mechanism. Recommends additional resources on the Instructor website.
- Using grammars to constrain llama.cpp output - 09-06, Ian Maurer) integrates context-free grammars with llama.cpp for more accurate and schema-compliant responses, particularly for biomedical data.
- Why Pydantic became indispensable for LLMs - 01-19, [Adam Azzam](https://twitter.com/aaazzam)) explains Pydantic's emergence as a critical tool, enabling sharing data models via JSON schemas and reasoning between unstructured and structured data. Highlights the importance of quantizing the decision space and potential issues with LLMs overfitting to older schema versions.
- How fast can grammar-structured generation be? - 04-12, .txt Engineering) demonstrates an almost cost-free method to generate text that follows a grammar. It is shown to outperform `llama.cpp` by a factor of 50x on the C grammar.

Programming Languages

Python 12 Jupyter Notebook 3

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

awesome-llm-json

Leaderboards

Local LLMs

Local LLM Repositories

Local LLM Spaces, Models & Datasets

Local LLM Resources

LLM-based Tools

Local LLM Resources

Non-English Models & Datasets

Local LLM Resources

Autonomous Agents

Autonomous Agent Resources

Python Libraries

Jupyter Notebooks

Videos

Prompting Tools

Autonomous Agent Resources

Terminology

Hosted Models

Local Models

Blog Articles