{"id":50949748,"url":"https://github.com/tcorral/prompfoo-issue-loading-external-files-for-nested-assertions","last_synced_at":"2026-06-18T00:03:11.203Z","repository":{"id":340115834,"uuid":"1164587927","full_name":"tcorral/prompfoo-issue-loading-external-files-for-nested-assertions","owner":"tcorral","description":"This repository is for demonstration of the problem loading external files for nested assertions","archived":false,"fork":false,"pushed_at":"2026-02-23T10:16:00.000Z","size":135,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-23T18:40:26.648Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tcorral.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-23T09:00:36.000Z","updated_at":"2026-02-23T10:16:04.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/tcorral/prompfoo-issue-loading-external-files-for-nested-assertions","commit_stats":null,"previous_names":["tcorral/prompfoo-issue-loading-external-files-for-nested-assertions"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/tcorral/prompfoo-issue-loading-external-files-for-nested-assertions","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tcorral%2Fprompfoo-issue-loading-external-files-for-nested-assertions","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tcorral%2Fprompfoo-issue-loading-external-files-for-nested-assertions/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tcorral%2Fprompfoo-issue-loading-external-files-for-nested-assertions/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tcorral%2Fprompfoo-issue-loading-external-files-for-nested-assertions/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tcorral","download_url":"https://codeload.github.com/tcorral/prompfoo-issue-loading-external-files-for-nested-assertions/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tcorral%2Fprompfoo-issue-loading-external-files-for-nested-assertions/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34470324,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-17T02:00:05.408Z","response_time":127,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-18T00:03:11.027Z","updated_at":"2026-06-18T00:03:11.194Z","avatar_url":"https://github.com/tcorral.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Promptfoo: External file references for nested assertions\n\nThis repository demonstrates a feature request for [Promptfoo](https://www.promptfoo.dev/) to support external file references for nested assertion properties.\n\n## The problem\n\nCurrently, Promptfoo only allows external file references (`file://`) for the top-level `value` property in assertions. This means:\n\n- You cannot reference external files for assertion `type`, `provider`, or other nested properties\n- You must either inline all assertion configurations or duplicate them across test cases\n- RReusing assertions across multiple tests violates the DRY principle\n- Building modular assertion libraries is not possible\n\n## The solution\n\nThis feature request proposes supporting external file references throughout the entire assertion structure, enabling:\n\n- Defining assertions once and reusing them across multiple test cases\n- Multi-level file organization with nested references\n- True modularity and reusability of assertion configurations\n- Better maintainability of large test suites\n\n## Repository structure\n\n```\n├── assertions/\n│   ├── working/              # Current working approach (file refs in value only)\n│   │   ├── is-valid-json-default.yaml\n│   │   ├── is-valid-json-edge-cases.yaml\n│   │   ├── llm-rubric-warm-greeting.txt\n│   │   └── edge-cases.js\n│   └── not-working/          # Demonstration of the limitation\n│       ├── expected/         # What currently works (sort of)\n│       │   └── default.yaml\n│       └── ideal/            # What we want (nested file refs)\n│           ├── default.yaml\n│           └── edge-cases.yaml\n├── tests/\n│   ├── original.yaml         # Working: inline assertions\n│   ├── expected.yaml         # Not working: nested file refs in test\n│   ├── ideal.yaml            # Ideal: cleaner nested file refs\n│   └── workaround.yaml       # Workaround: use labels\n├── prompts/\n│   └── prompt.txt            # Sample prompt\n├── promptfooconfig.yaml      # Promptfoo configuration\n└── package.json              # Project dependencies\n```\n\n## Test scenarios\n\n### 1. Original working approach (`tests/original.yaml`)\n\nWhat works: Inlining all assertion properties with file references only in `value`\n\n```yaml\n- vars: { name: \"Alice\", time_of_day: \"morning\" }\n  assert:\n    - type: is-json\n      value: file://assertions/working/is-valid-json-default.yaml\n    - type: llm-rubric\n      provider: ollama:mistral:7b\n      value: file://assertions/working/llm-rubric-warm-greeting.txt\n```\n\nRun it:\n```bash\nnpm run eval:original\n```\n\nPros:\n- Works out of the box\n- File references work for assertion validation details\n\nCons:\n- Must inline `type` and `provider` in each test case\n- Assertion configurations are duplicated\n- Not truly modular\n\n---\n\n### 2. Expected approach (`tests/expected.yaml`)\n\nWhat doesn't work: Referencing a file that contains multiple assertions\n\n```yaml\n# tests/expected.yaml\n- vars: { name: \"Alice\", time_of_day: \"morning\" }\n  assert:\n    - value: file://assertions/not-working/expected/default.yaml\n\n# assertions/not-working/expected/default.yaml\n- type: is-json\n  value:\n    required: [\"greeting\"]\n    type: object\n    properties:\n      greeting:\n        type: string\n- type: llm-rubric\n  provider: ollama:mistral:7b\n  value: |\n    The greeting should be:\n    1. Warm and personalized - addressing the person by name\n    2. Contextually appropriate to the time of day\n    3. Professional and helpful\n    Grade as PASS if it meets all three criteria, FAIL otherwise.\n```\n\nRun it:\n```bash\nnpm run eval:expected\n```\n\nStatus: This currently doesn't work properly because file references are only supported for the `value` property.\n\n---\n\n### 3. Ideal approach (`tests/ideal.yaml`)\n\nWhat we want: Full support for nested file references\n\n```yaml\n# tests/ideal.yaml\n- vars: { name: \"Alice\", time_of_day: \"morning\" }\n  assert:\n    - value: file://assertions/not-working/ideal/default.yaml\n\n# assertions/not-working/ideal/default.yaml\n- type: is-json\n  value: file://assertions/working/is-valid-json-default.yaml\n- type: llm-rubric\n  provider: ollama:mistral:7b\n  value: file://assertions/working/llm-rubric-warm-greeting.txt\n```\n\nRun it (currently doesn't work):\n```bash\nnpm run eval:ideal\n```\n\nBenefits:\n- Fully modular assertion definitions\n- Assertions are defined once, reused everywhere\n- Multi-level file organization\n- Easy to maintain and update\n- Scalable for large test suites\n\n---\n\n## Use cases\n\n### 1. Shared assertion libraries\n\nCreate a centralized library of reusable assertions:\n\n```\nassertions/\n├── json-schemas/\n│   ├── greeting-response.yaml\n│   ├── error-response.yaml\n│   └── user-profile.yaml\n├── llm-rubrics/\n│   ├── customer-service.txt\n│   ├── technical-accuracy.txt\n│   └── tone-and-style.txt\n└── edge-cases/\n    └── default-validations.yaml\n```\n\nThen reference them in test files without duplication.\n\n### 2. Test organization by feature\n\n```\ntests/\n├── greeting-feature/\n│   ├── basic.yaml\n│   ├── edge-cases.yaml\n│   └── localization.yaml\n└── user-profile/\n    ├── creation.yaml\n    ├── updates.yaml\n    └── validation.yaml\n```\n\nAll tests in a feature can reference the same assertions from `assertions/` directory.\n\n### 3. Assertion versioning\n\nMaintain multiple versions of assertions and reference different ones:\n\n```\nassertions/\n├── llm-rubrics/\n│   ├── warmth-v1.txt\n│   ├── warmth-v2.txt\n│   └── warmth-latest.txt  # Points to v2\n```\n\n---\n\n## Getting started\n\n### Prerequisites\n\n- Node.js 18+\n- Promptfoo CLI installed (or use `npm install`)\n- Ollama running with `qwen2.5` model (or change provider in `promptfooconfig.yaml`)\n\n### Installation\n\n```bash\nnpm install\n```\n\n### Run tests\n\n```bash\n# Original working approach\nnpm run eval:original\n\n# Expected approach (currently doesn't work)\nnpm run eval:expected\n\n# Ideal approach (currently doesn't work)\nnpm run eval:ideal\n\n# Workardound approach\nnpm run eval:workaround\n\n# View results\nnpm run view\n```\n\n---\n\n## Feature request details\n\nThis repository is a demonstration for a Promptfoo feature request:\n\nTitle: Support external file references for nested assertion properties (e.g., `type`, `provider`, `value`)\n\nGitHub Issue Link: https://github.com/promptfoo/promptfoo/issues/7823\n\n---\n\n## Workarounds\n\nUntil this feature is implemented, you can:\n\n1. Inline all assertions (not modular but works)\n2. Use JavaScript assertions with inline logic\n3. Create wrapper scripts that generate test files dynamically\n4. Maintain separate test configurations for different scenarios\n\nSee [workaround](https://github.com/tcorral/prompfoo-issue-loading-external-files-for-nested-assertions/blob/main/tests/workaround.yaml)\n\n---\n\n## Contributing\n\nThis repository demonstrates the feature request. If you're interested in:\n\n- Using this feature: Comment on the GitHub issue or 👍 react to show interest\n- Implementing this feature: Check the Promptfoo repository for contribution guidelines\n- Improving this demo: Feel free to fork and enhance the examples\n\n---\n\n## Resources\n\n- [Promptfoo Documentation](https://www.promptfoo.dev/)\n- [Promptfoo GitHub Repository](https://github.com/promptfoo/promptfoo)\n\n---\n\n## Author\n\nCreated by [Tomas Corral Casas](https://github.com/tcorral)\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftcorral%2Fprompfoo-issue-loading-external-files-for-nested-assertions","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftcorral%2Fprompfoo-issue-loading-external-files-for-nested-assertions","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftcorral%2Fprompfoo-issue-loading-external-files-for-nested-assertions/lists"}