{"id":30437412,"url":"https://github.com/mattbriggs/retrieval-json-vs-md","last_synced_at":"2025-08-23T03:48:51.433Z","repository":{"id":304903832,"uuid":"933350735","full_name":"mattbriggs/retrieval-json-vs-md","owner":"mattbriggs","description":"Developing measurement algorithm and process for evaluating AI-Gen (information retrieval) from Markdown/HTML and YAML/JSON-LD content. ","archived":false,"fork":false,"pushed_at":"2025-07-15T23:42:42.000Z","size":2572,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-17T00:06:38.419Z","etag":null,"topics":["ai","measuring"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mattbriggs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-15T18:41:24.000Z","updated_at":"2025-07-15T23:42:45.000Z","dependencies_parsed_at":"2025-07-17T07:08:09.210Z","dependency_job_id":"508d3848-9b11-488d-b3bd-b595a0c94583","html_url":"https://github.com/mattbriggs/retrieval-json-vs-md","commit_stats":null,"previous_names":["mattbriggs/retrieval-json-vs-md"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/mattbriggs/retrieval-json-vs-md","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mattbriggs%2Fretrieval-json-vs-md","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mattbriggs%2Fretrieval-json-vs-md/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mattbriggs%2Fretrieval-json-vs-md/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mattbriggs%2Fretrieval-json-vs-md/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mattbriggs","download_url":"https://codeload.github.com/mattbriggs/retrieval-json-vs-md/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mattbriggs%2Fretrieval-json-vs-md/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271740076,"owners_count":24812636,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","measuring"],"created_at":"2025-08-23T03:48:50.531Z","updated_at":"2025-08-23T03:48:51.404Z","avatar_url":"https://github.com/mattbriggs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Prepare, set-up, and run two vector search pipelines to compare HTML vs JSON-LD\n\n## Introduction\n\nThis study examines how different content structures—**structured JSON-LD (Schema.org)** and **unstructured HTML**—impact **AI-driven information retrieval**. The goal is to assess which format enables more **accurate, efficient, and AI-ready** search results, using **F1 score** as the primary evaluation metric.\n\n### Business Context\n\nCompanies increasingly rely on AI-powered search and recommendation systems to extract insights from large-scale content repositories. Ensuring content is structured in a way that enhances machine understanding can directly impact:\n- **Customer self-service efficiency** (e.g., chatbots, FAQ retrieval)\n- **Internal knowledge management** (e.g., employee portals, document search)\n- **Search engine optimization (SEO)** and structured data compliance\n\nThis study seeks to provide **data-driven insights** on whether structuring content as **JSON-LD** (used in knowledge graphs) improves information retrieval over **raw HTML** (often used in traditional web search).\n\n---\n\n## Study Design \u0026 Methodology\n\n### Content Sources \u0026 Formats\n\nThis study evaluates the same content in two different formats:\n\n1. **HTML-based FAQ pages**: Generated from Markdown (MD), representing traditional web content.\n2. **JSON-LD structured content**: Derived from YAML, adhering to **Schema.org's FAQPage** schema, used in knowledge graphs and AI-driven search systems.\n\n### Evaluation approach\nThe effectiveness of each format is measured by how accurately it answers **golden dataset questions**—a standardized set of business-relevant queries. The study follows these key steps.\n\n### Step 1: Set up a virtualenv and install the requirements\n\nOn a macOS:\n\n```bash\npython -m venv ENV\nsource ./ENV/bin/activate\npip install -r requirements.txt\n```\n\nYou may need to run `install-nltk-corpus.py` to have access to the training data for the NLTK.\n\n    ```bash\n    python install-nltk-corpus.py\n    ```\n\n### Step 2: Data preparation\n\n1. Retrieve data from the web with articles that contain both HTML and JSON-LD (FAQPage) payloads.\n    Input: A list of URLS.\n    Output: Two folders HTML and JSONLD that contain the data.\n\n    For instructions: [Prepare, set-up, and run two vector search pipelines to compare HTML vs JSON-LD](faqpage-scrapper-readme.md)\n\n2. Create the Golden Questions.\n    Input: Two folders HTML and JSONLD that contain the data.\n    Output: A JSON file of questions to use with both the HTML and JSONLD test.\n\n3. You may want to reduce the number of golden questions to make the assessment more manageable. Make sure you are using the same set for both tests.\n\n### Step 3: Set up Weaviate in a Docker container\n\nFollow the guide that explains how to set up and run Weaviate in a Docker container on macOS. It also covers installing dependencies and verifying the setup. See [Setting Up Weaviate with Docker on macOS](docker_weaviate_readme.md)\n\n### Step 3: Query execution and performance measurement\n\n1. Generate the F1 scores for the golden question set using HTML.\n\n    ```bash\n    python html_load_faq_weaviate.py\n    ```\n\n1. Generate the F1 scores for the golden question set using JSON-LD.\n\n```bash\npython json_load_faq_weaviate.py\n```\n\n## Note on the F1 score in this study\n\nBoth scripts calculate the F1 score using cosine similarity rather than the traditional precision-recall F1-score. They encode both the retrieved answer (from Weaviate) and the expected answer (from the golden questions dataset) into vector embeddings using SentenceTransformers (`all-MiniLM-L6-v2`) and then compute the cosine similarity between them using `util.pytorch_cos_sim`. This similarity score, ranging from 0 (completely different) to 1 (identical), is treated as the F1 score in the evaluation. The final performance metric is the average cosine similarity across all questions, providing a measure of how well Weaviate retrieves semantically relevant answers.\n\n\n## Expected Business Insights\nThis study provides empirical data to help organizations understand:\n\n1. **Does structured JSON-LD data improve AI-powered retrieval?**\n   - If JSON-LD (Neo4J) outperforms HTML (Weaviate), it validates that **structured content enhances machine understanding**.\n   \n2. **How effective is AI-driven vector search for unstructured data?**\n   - If Weaviate performs well, businesses may not need full schema compliance for effective AI search.\n   \n3. **How should businesses structure content for AI-readiness?**\n   - The results can help companies **decide on content formatting strategies** for AI-based search, chatbots, and knowledge management systems.\n\n## Conclusion\n\nThis study compares **structured vs. unstructured information retrieval** to determine the best strategy for **AI-powered business search applications**. By applying **F1 score measurement**, businesses can make **data-driven decisions** on how to format content for better AI readiness.\n\nThe findings can provide **practical recommendations** on:\n\n - Whether structured **JSON-LD knowledge graphs** should be prioritized over traditional **HTML FAQs**.\n - How **vector search AI** can compensate for **unstructured content**.\n - Best practices for **optimizing enterprise search** using **graph vs. AI-based retrieval**.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmattbriggs%2Fretrieval-json-vs-md","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmattbriggs%2Fretrieval-json-vs-md","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmattbriggs%2Fretrieval-json-vs-md/lists"}