{"id":43292245,"url":"https://github.com/batfish/lab-validation","last_synced_at":"2026-02-01T18:34:23.121Z","repository":{"id":304298022,"uuid":"1016522313","full_name":"batfish/lab-validation","owner":"batfish","description":"Lab validation framework for validating Batfish network analysis results against real device data","archived":false,"fork":false,"pushed_at":"2025-09-18T06:23:14.000Z","size":6763,"stargazers_count":2,"open_issues_count":66,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-18T06:31:28.660Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/batfish.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-07-09T06:23:35.000Z","updated_at":"2025-09-18T06:23:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"e6504dad-e112-4b82-9002-1478d49999c5","html_url":"https://github.com/batfish/lab-validation","commit_stats":null,"previous_names":["batfish/lab-validation"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/batfish/lab-validation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/batfish%2Flab-validation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/batfish%2Flab-validation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/batfish%2Flab-validation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/batfish%2Flab-validation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/batfish","download_url":"https://codeload.github.com/batfish/lab-validation/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/batfish%2Flab-validation/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28985818,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-01T18:17:03.387Z","status":"ssl_error","status_checked_at":"2026-02-01T18:16:57.287Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-01T18:34:23.048Z","updated_at":"2026-02-01T18:34:23.111Z","avatar_url":"https://github.com/batfish.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Batfish Lab Validation Framework\n\nThe open source lab validation infrastructure that powers Batfish's empirical validation methodology, as described in [\"Validating the Validator\"](https://batfish.org/2020/12/18/validating-the-validator.html).\n\n## Overview\n\nThis framework implements comprehensive multi-level benchmarking to validate Batfish network analysis accuracy against real device behavior. It provides the testing infrastructure that ensures Batfish models faithfully predict network device state across multiple vendors and complex network scenarios.\n\n**Note**: This is an open-sourced subset of Intentionet's internal validation infrastructure. GitHub issues were automatically migrated during open sourcing and may contain outdated references. The lab creation and collection tooling has not yet been open sourced. Contributions and bug fixes are welcome!\n\n## Validation Approach\n\n**Multi-Level Benchmarking**: The framework implements three levels of validation as described in the Intentionet methodology:\n\n- **Feature-Level**: Individual network features tested in isolation (route maps, ACLs, routing protocols)\n- **Device-Level**: Complex feature interactions on single devices, testing vendor-specific behaviors\n- **Network-Level**: Multi-device topologies validating end-to-end behaviors across vendors\n\n**Empirical Testing**: 96 network labs with real device configurations and operational data capture, covering 12+ vendors including Arista EOS, Cisco IOS-XE/NX-OS, Juniper JunOS, and others.\n\n**Continuous Validation**: Automated testing compares Batfish predictions against actual device state using sophisticated cost-based matching algorithms.\n\n## Getting Started\n\n### Requirements\n\n- Python 3.10+\n- Pybatfish\n- Batfish server running locally (see [development setup instructions](https://github.com/batfish/batfish/tree/master/docs/development))\n\n### Developer Setup\n\n```bash\n# Clone and set up environment\ngit clone https://github.com/batfish/lab-validation.git\ncd lab-validation\n\n# Recommended: Use a Python virtual environment.\n\n# Install in development mode\npip install -e .\npip install -r requirements-dev.txt\n```\n\n### Running Lab Tests\n\n```bash\n# Run tests for a specific lab\npytest lab_tests/ --labname=eos_bgp_aggregate\n\n# Run all lab validation tests\npython run_all_labs.py\n\n# Run all labs with filtering and options\npython run_all_labs.py --lab-filter nxos --verbose --fail-fast\n\n# Run tests with coverage\npytest --cov=lab_validation --cov-report=term-missing\n```\n\n### Example Lab Structure\n\nEach lab in `snapshots/` contains:\n\n- **configs/**: Device configuration files\n- **show/**: Captured device show command outputs\n- **validation/**: Expected failure definitions (sickbay.yaml)\n\n## How It Works\n\nThe validation process implements the methodology described in the Intentionet blog post:\n\n1. **Lab Environment**: Network configurations deployed in emulated environments\n2. **Data Capture**: Device show commands capture actual operational state\n3. **Batfish Analysis**: Network snapshot analyzed to predict device behavior\n4. **Empirical Comparison**: Cost-based matching algorithms compare predicted vs. actual state\n5. **Continuous Improvement**: Test failures tracked through GitHub issues drive model refinements\n\n### Architecture Components\n\n- **Multi-Vendor Validators**: 12+ vendor-specific implementations (Arista, Cisco, Juniper, etc.)\n- **Sophisticated Parsers**: Vendor-specific command output parsing with complex grammar handling\n- **Cost-Based Matching**: Advanced algorithms for pairing device data with Batfish predictions\n- **Sickbay System**: Expected failure management with GitHub issue integration for model improvement tracking\n\n## Development\n\n### Testing Framework\n\n```bash\n# Run all tests (unit + lab integration)\npytest\n\n# Run specific validator unit tests\npytest tests/test_arista_validator.py -v\n\n# Run specific lab integration test\npytest lab_tests/test_labs.py --labname=eos_bgp_aggregate -v\n\n# Code quality checks\npre-commit run --all-files\n```\n\n### Lab Coverage\n\nThe framework includes 96 network labs covering:\n\n- **BGP**: Route aggregation, EVPN, multi-VRF scenarios\n- **OSPF**: Multi-area, stub areas, route redistribution\n- **Interfaces**: VLAN, VXLAN, port-channels, VRFs\n- **Multi-vendor**: Cross-vendor interoperability testing\n- **Data Center**: EVPN/VXLAN spine-leaf architectures\n\n## Contributing\n\nThis framework provides the empirical foundation for Batfish's network modeling accuracy. Contributions help improve network analysis for the entire community.\n\n1. Fork the repository\n2. Create a feature branch\n3. Add tests for any changes\n4. Ensure all tests pass: `pytest \u0026\u0026 pre-commit run --all-files`\n5. Submit a pull request\n\n## Related Projects\n\n- **[Batfish](https://github.com/batfish/batfish)**: Open source network configuration analysis engine\n- **[Pybatfish](https://github.com/batfish/pybatfish)**: Python SDK for Batfish\n- **[Intentionet Blog](https://batfish.org/2020/12/18/validating-the-validator.html)**: \"Validating the Validator\" methodology\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbatfish%2Flab-validation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbatfish%2Flab-validation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbatfish%2Flab-validation/lists"}