{"id":20313613,"url":"https://github.com/weaviate/structured-rag","last_synced_at":"2026-03-10T00:31:36.160Z","repository":{"id":250956600,"uuid":"827428095","full_name":"weaviate/structured-rag","owner":"weaviate","description":"Experimental Code for StructuredRAG: JSON Response Formatting with Large Language Models","archived":false,"fork":false,"pushed_at":"2025-04-09T22:41:15.000Z","size":3345,"stargazers_count":105,"open_issues_count":0,"forks_count":6,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-26T08:55:53.763Z","etag":null,"topics":["dspy","retrieval-augmented-generation","structured-generation"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2408.11061","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/weaviate.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-07-11T16:15:25.000Z","updated_at":"2025-04-09T22:41:19.000Z","dependencies_parsed_at":"2024-08-07T19:05:21.614Z","dependency_job_id":"8eb72c5c-ce9f-438f-aaf1-66aa6368dfd6","html_url":"https://github.com/weaviate/structured-rag","commit_stats":null,"previous_names":["weaviate/structured-output-benchmarker"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/weaviate/structured-rag","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weaviate%2Fstructured-rag","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weaviate%2Fstructured-rag/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weaviate%2Fstructured-rag/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weaviate%2Fstructured-rag/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/weaviate","download_url":"https://codeload.github.com/weaviate/structured-rag/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weaviate%2Fstructured-rag/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30318395,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-09T20:05:46.299Z","status":"ssl_error","status_checked_at":"2026-03-09T19:57:04.425Z","response_time":61,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dspy","retrieval-augmented-generation","structured-generation"],"created_at":"2024-11-14T18:12:01.041Z","updated_at":"2026-03-10T00:31:36.138Z","avatar_url":"https://github.com/weaviate.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# StructuredRAG: JSON Response Formatting with Large Language Models\n\n## News 📯\n\nWeaviate Podcast #119 with Will Kurt and Cameron Pfiffer from dottxt.ai is live on [YouTube](https://www.youtube.com/watch?v=3PdEYG6OusA) and [Spotify](https://spotifycreators-web.app.link/e/b8MEmkkbrSb)\n\nOur research paper is live on [ArXiv](https://arxiv.org/abs/2408.11061)!\n\nWeaviate Podcast #108 with Zhi Rui Tam, lead author of \"Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models\", is live on [YouTube](https://www.youtube.com/watch?v=UsVIX9NJ_a4) and [Spotify](https://spotifyanchor-web.app.link/e/KkmrH99LkOb)!\n\nLarge Language Models have become extremely powerful at Zero-Shot Instruction Following. This benchmarker aims to target how well LLMs can follow the instruction of formatting its output in a particular JSON template. It is extremely important that these outputs follow these instructions for building reliably LLM systems such as metadata extraction, reasoning, report generation, agents, and more!\n\nThis benchmarker firstly compares `f-String` prompting with the `Follow the Format (FF)` method used in DSPy.\n\nThis benchmarker secondly compares `Gemini` with `Llama3 (Ollama)`.\n\nThe benchmarker explores different RAG inspired tasks with structured outputs as follows:\n\n| Output Type                        | Task                        | Example                                                                                                                                           |\n|-----------------------------|-----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|\n| `string`                      | GenerateAnswer              | {\"answer\": \"The National Gallery of Art, Washington D.C., and the Pinacoteca di Brera, Milan, Italy.\"}                                             |\n| `integer`                     | RateContext                 | {\"context_score\": 5}                                                                                                                              |\n| `boolean`                     | AssessAnswerability         | {\"answerable_question\": True}                                                                                                                     |\n| `List[string]`                | ParaphraseQuestions         | {\"paraphrased_questions\": [\"Where can some of Vincenzo Civerchio’s works be found?\", \"Where are some pieces by Vincenzo Civerchio displayed?\", \"Where can I find some of Vincenzo Civerchio’s art?\"]} |\n| `RAGASmetrics`                 | RAGAS                       | {\"faithfulness_score\": 2.5, \"answer_relevance_score\": 1.0, \"context_relevance_score\": 3.5}\n| `AnswerWithConfidence`        | GenerateAnswerWithConfidence| {\"answer\": \"The National Gallery of Art, Washington D.C., and the Pinacoteca di Brera, Milan, Italy.\", \"confidence\": 5}                            |\n| `List[AnswerWithConfidence]`  | GenerateAnswersWithConfidence| [{\"answer\": \"National Gallery of Art, Washington D.C.\", \"confidence\": 5}, {\"answer\": \"Pinacoteca di Brera, Milan, Italy\", \"confidence\": 4}]         |\n\n## Additional Models\n\n```python\nclass RAGASmetrics(BaseModel):\n  faithfulness_score: float\n  answer_relevance_score: float\n  context_relevance_score: float\n\nclass AnswerWithConfidence(BaseModel):\n  answer: str\n  confidence: float\n```\n\nThe WikiQuestions dataset can also be found on [HuggingFace Datasets](https://huggingface.co/datasets/weaviate/Wiki-Answerable-Questions)!\n\n![Experimental Results](./structured_rag/run_test/result_visualization/success_rates_per_test.png)\n\n## Citation\nPlease consider citing our paper if you find this work useful:\n\n```bibtex\n@misc{shorten2024,\n      title={StructuredRAG: JSON Response Formatting with Large Language Models}, \n      author={Connor Shorten and Charles Pierse and Thomas Benjamin Smith and Erika Cardenas and Akanksha Sharma and John Trengrove and Bob van Luijt},\n      year={2024},\n      eprint={2408.11061},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2408.11061}, \n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweaviate%2Fstructured-rag","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fweaviate%2Fstructured-rag","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweaviate%2Fstructured-rag/lists"}