{"id":40943305,"url":"https://github.com/stratosphereips/mipseval","last_synced_at":"2026-01-22T04:37:30.316Z","repository":{"id":303749437,"uuid":"935926324","full_name":"stratosphereips/MIPSEval","owner":"stratosphereips","description":null,"archived":false,"fork":false,"pushed_at":"2025-12-09T09:30:45.000Z","size":24230,"stargazers_count":13,"open_issues_count":1,"forks_count":4,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-12-10T07:23:56.604Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stratosphereips.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":".github/SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-02-20T08:45:24.000Z","updated_at":"2025-12-09T09:30:34.000Z","dependencies_parsed_at":"2025-07-23T15:08:52.730Z","dependency_job_id":null,"html_url":"https://github.com/stratosphereips/MIPSEval","commit_stats":null,"previous_names":["stratosphereips/mipseval"],"tags_count":64,"template":false,"template_full_name":"stratosphereips/awesome-code-template","purl":"pkg:github/stratosphereips/MIPSEval","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratosphereips%2FMIPSEval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratosphereips%2FMIPSEval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratosphereips%2FMIPSEval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratosphereips%2FMIPSEval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stratosphereips","download_url":"https://codeload.github.com/stratosphereips/MIPSEval/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratosphereips%2FMIPSEval/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28654886,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-22T01:17:37.254Z","status":"online","status_checked_at":"2026-01-22T02:00:07.137Z","response_time":144,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-22T04:37:30.242Z","updated_at":"2026-01-22T04:37:30.300Z","avatar_url":"https://github.com/stratosphereips.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# MIPSEval\n\nMulti-turn Injection Planning System for LLM Evaluation\n\nMIPSEval is a modular framework for simulating and evaluating the behavior of Large Language Models (LLMs) in adversarial or structured multi-turn conversational scenarios. It supports both OpenAI-hosted models and locally hosted models.\n\nMIPSEval uses LLMs to design a conversation strategy as well as execute it, making it fully automated. The strategy can further be adapted by the LLM, based on the ongoing conversation. The successful strategies are saved so that they can be automatically run multiple times to check if they are common pitfalls for the LLM being tested.\n\n![LLM Attacker Evaluator Diagram](images/LLM%20Attacker_Evaluator%20Diagram%20-%20MIPSEval%20Diagram.jpg)\n\n## Features\n\n- Modular structure with planner, executor, and judge components  \n- Multi-turn conversation handling  \n- Configurable attack logic via YAML  \n- Supports both OpenAI and local LLMs  \n- JSONL logging of interaction history  \n- Fully automated evaluation\n- Strategy and execution are performed by LLMs\n- 3 prompt types: Benign, Probing, and Malicious\n- Strategies are updated based on the ongoing conversation\n- LLM is used to judge success\n- High variety of malicious tasks and jailbreaks/prompt injections\n- Working in explore or exploit mode\n- Evolving of successful strategies\n- Any LLM can be tested with MIPSEval\n- An extensible framework that allows evaluation of other aspects of LLMs\n\n## Installation\n\n```bash\ngit clone https://github.com/stratosphereips/mipseval.git\ncd mipseval\npip install -r requirements.txt\ncd src\n```\n\nBefore running the tool, RAG of prompt injections and jailbreaks needs to be set up. It can be done with the following command in src folder:\n```\npython add_json_to_rag.py\n```\n\nThe definitions and examples of jailbreaks and prompt injection that are used for RAG are provided in the `prompt_injections_and_jailbreaks.json` file.\n\nYou must also create a `.env` file with your API key (if using OpenAI):\n\n```\nOPENAI_API_KEY=your_openai_api_key\n```\n\nIf the target LLM is a local model, the URL must be set in the `llm_executor.py` script, specifically in the `call_local_api` function.\n\n## Usage\n\nRun the application using:\n\n```bash\npython mipseval.py -e .env -c path/to/config.yaml -p openai -t openai '[-j conversation_history.jsonl]'\n```\n\nFor local model usage:\n\n```bash\npython mipseval.py -e .env -c path/to/config.yaml -p local -t local '[-j conversation_history.jsonl]'\n```\n\nDefault OpenAI models used to run MIPSEval are gpt-4o for Planner and gpt-4o-mini for executioner. This can be changed in ```setup.py``` for executioner and ```llm_planner.py``` for planner (in get_step_for_evaluator function). Testing was done with the default models used.\n\n### Command-Line Arguments\n\n| Argument           | Description                                        | Required |\n|--------------------|----------------------------------------------------|----------|\n| `-e`, `--env`       | Path to environment file (.env)                   | Yes      |\n| `-c`, `--config`    | Path to YAML configuration file                   | Yes      |\n| `-p`, `--provider`  | MIPSEval Model provider: `openai` or `local`               | Yes      |\n| `-t`, `--target`  | Target Model provider: `openai` or `local`               | Yes      |\n| `-j`, `--json_history` | Optional path to conversation log `.jsonl`   | No       |\n\n\n## Output\n\nConversations are logged in JSONL format. Three files are created:\n- Conversation History\n- Strategies\n- Victorious Strategies\n\n\n## Demo\n\n![Demo](images/MIPSEval.gif)\n\n\n## License\n\nThis project is licensed under the GNU GPL License. See the `LICENSE` file for details.\n\n## Contributing\n\nContributions are welcome. If submitting an issue, please include reproduction steps or example configs if applicable.\n\n\n# About\n\nThis tool was developed at the Stratosphere Laboratory at the Czech Technical University in Prague.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstratosphereips%2Fmipseval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstratosphereips%2Fmipseval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstratosphereips%2Fmipseval/lists"}