{"id":21168484,"url":"https://github.com/jetbrains-research/coqpilot","last_synced_at":"2025-04-09T23:16:31.764Z","repository":{"id":207287472,"uuid":"687187890","full_name":"JetBrains-Research/coqpilot","owner":"JetBrains-Research","description":"VSCode extension that is designed to help automate writing of Coq proofs.","archived":false,"fork":false,"pushed_at":"2025-04-07T19:21:36.000Z","size":19711,"stargazers_count":99,"open_issues_count":1,"forks_count":5,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-09T23:16:24.696Z","etag":null,"topics":["code-generation","coq","formal-methods","llm","theorem-prover"],"latest_commit_sha":null,"homepage":"https://coqpilot.net","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-2.1","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JetBrains-Research.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-04T20:30:30.000Z","updated_at":"2025-04-04T09:58:30.000Z","dependencies_parsed_at":"2023-12-21T02:06:34.974Z","dependency_job_id":"16791c77-a4f1-49a6-be31-975dacdea457","html_url":"https://github.com/JetBrains-Research/coqpilot","commit_stats":null,"previous_names":["jetbrains-research/coqpilot"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JetBrains-Research%2Fcoqpilot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JetBrains-Research%2Fcoqpilot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JetBrains-Research%2Fcoqpilot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JetBrains-Research%2Fcoqpilot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JetBrains-Research","download_url":"https://codeload.github.com/JetBrains-Research/coqpilot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248125593,"owners_count":21051771,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["code-generation","coq","formal-methods","llm","theorem-prover"],"created_at":"2024-11-20T15:14:03.912Z","updated_at":"2025-04-09T23:16:31.737Z","avatar_url":"https://github.com/JetBrains-Research.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CoqPilot ![Version](https://img.shields.io/badge/version-v2.4.3-blue?style=flat-square)\n\n*Authors:* Andrei Kozyrev, Gleb Solovev, Nikita Khramov, and Anton Podkopaev, [Programming Languages and Tools Lab](https://lp.jetbrains.com/research/plt_lab/) at JetBrains Research.\n\n`CoqPilot` is a [Visual Studio Code](https://code.visualstudio.com/) extension that is designed to help automate writing of Coq proofs. It uses Large Language Models to generate multiple potential proofs and then uses [coq-lsp](https://github.com/ejgallego/coq-lsp) to typecheck them. It substitutes the proof in the editor only if a valid proof is found. \n\n# Table of Contents\n\n- 🚀 [CoqPilot Overview](#coqpilot-version)\n- 📋 [Requirements](#requirements)\n- 📚 [Related papers](#related-papers)\n- 🔍 [Brief Technical Overview](#brief-technical-overview)\n- 💡 [Example Usage](#example-usage)\n- 🛠 [Installation](#installation)\n  - ▶️ [Coq-LSP Installation](#coq-lsp-installation)\n  - 🤖 [Building Locally](#building-locally)\n- ⚙️ [Extension Settings](#extension-settings)\n- 📐 [Guide to Model Configuration](#guide-to-model-configuration)\n  - 🎛 [How VSCode settings work](#how-vscode-settings-work)\n  - 🧰 [Model Configuration](#model-configuration)\n- 📌 [Contributed Commands](#contributed-commands)\n- 📊 [Benchmark](#benchmark)\n- 🧩 [Integrating other solutions](#integrating-other-solutions)\n  - 🧠 [Tactician](#tactician)\n  - 🔨 [CoqHammer](#coqhammer)\n- 🔜 [Future Plans](#future-plans)\n- 📜 [Release Notes](#release-notes)\n\n## Requirements\n\n* `coq-lsp` version `0.2.2+8.19` is currently required to run the extension.\n\n## Related papers\n\n- **[ASE Demo'24]** *CoqPilot, a plugin for LLM-based generation of proofs*\n  \u003cbr /\u003e\n  [[Paper](https://dl.acm.org/doi/10.1145/3691620.3695357) | [arXiv](https://arxiv.org/abs/2410.19605) | [Video (5min)](https://www.youtube.com/watch?v=oB1Lx-So9Lo) | [Video (10min)](https://www.youtube.com/watch?v=P-LHXf7vntM)]\n- **[AITP'24 \u0026 CoqWS'24]** *CoqPilot, a plugin for LLM-based generation of proofs*\n  \u003cbr /\u003e\n  [[Extended Abstract](https://coq-workshop.gitlab.io/2024/files/EA2.pdf) | [CoqWS Slides](https://coq-workshop.gitlab.io/2024/files/SL2.pdf)]\n\n\n## Brief technical overview\n\n`CoqPilot` fetches proofs from multiple completion services. Now we support: \n- a service that always returns a list of pre-defined in the settings tactics/coq sentances.\n- an [OpenAI](https://openai.com) gpt service.\n- a service that fetches completions from the model, running locally in LM Studio.\n- a service that uses Grazie platform (only for JetBrains employees for now).\n\nFor each service, an array of models could be defined through the settings. Each model will be used for generation independantly. This brings freedom to the user to experiment with different model parameters, e.g. temperature, prompt, etc.\n\nWhen `CoqPilot` completion command is issued, it parses the currently opened file, extracts theorems that have complete proofs and processes them into a message history for the LLM. It helps LLM to keep the style and hallucinate less. \n\nFor each `admit.` present in the file, an independent completion process is issued. If a valid proof is found, it is substituted in the editor. `CoqPilot` also allows a multi-round fixing procedure for the proofs from the LLM. I.e. if the proof was incorrect, compiler message could be automatically sent to the LLM with a request to repair it. It can now be configured in the settings. One can set the number of attempts for the consequtive proof fixing with compiler feedback.\n\nAs soon as at least one valid proof is found, it is substituted in the editor and the process is finished.\n\n**Notice:** By default, CoqPilot sets only `PredefinedProofs` and `OpenAI` services. The first one tries `auto.` tactic and the second one has one model -- `gpt-3.5`. By default the `apiKey` for OpenAI is not set, i.e. set to `None`. Do not forget to change that in the settings before using this service.\n\n**Notice:** File `settings.json` declares not all the settings, but those that are overriden from the defaults. Keep that in mind, if you want, for example, to turn off the `OpenAI` service. For that, you would need to override the corresponding setting with an empty array, but not delete this property from the file.\n\n## Example usage\n\n`CoqPilot` only runs on an opened `coq` file. User can:\n- Run `CoqPilot` with some chosen selection to try substitute all admits in this selection.\n\n\u003cimg src=\"./etc/gif/solve-in-selection.gif\"/\u003e\n\n- Run `CoqPilot` to try substitute all admits in the file.\n- Run `CoqPilot` to substitute the proof for the admit if there is one under the cursor.\n\n## Installation\n\n### Coq-LSP installation\n\nTo run the extension, you must install a `coq-lsp` server. Depending on the system used in your project, you should install it using `opam` or `nix`. A well-configured `nix` project should have the `coq-lsp` server installed as a dependency. To install `coq-lsp` using `opam`, you can use the following commands: \n```bash\nopam pin add coq-lsp 0.2.2+8.19\nopam install coq-lsp\n```\nFor more information on how to install `coq-lsp` please refer to [coq-lsp](https://github.com/ejgallego/coq-lsp). \n\nEither way around, if the [coq-lsp](https://github.com/ejgallego/coq-lsp) extension works well and you can see the goals and theorems in the VSCode, then `CoqPilot` should work as well. However, using [coq-lsp](https://github.com/ejgallego/coq-lsp) as a plugin for Coq support is not mandatory for `CoqPilot` to work.\n\nIf your installation of `coq-lsp` is not in the default path, you can specify the path to the `coq-lsp` server in the settings using the `coqpilot.coqLspServerPath` setting. Default value should work well for `opam`.\n\n**IMPORTANT**: If you are using `nix` in your project, make sure to **UPDATE** the path of the `coq-lsp` server in the settings. The default path is set to `coq-lsp`, which is the default path for `opam`. If you are using `nix`, you should run the following command from inside of the `nix-shell`: \n```bash\nwhich coq-lsp\n```\nAnd then copy the path to the `coq-lsp` server and paste it into the `coqpilot.coqLspServerPath` setting. \n\nIn the benchmark the same rule applies, but the path to the `coq-lsp` server should be set as an environment variable `COQ_LSP_PATH`: \n```bash\nexport COQ_LSP_PATH=$(which coq-lsp)\n```\n\n\n### Building locally\n\nFirst, clone the CoqPilot repository and navigate into its directory.\n```bash\ngit clone https://github.com/JetBrains-Research/coqpilot.git\ncd coqpilot\n```\n\nTo build the extension locally, you'll need Node.js installed. The recommended way to manage Node.js versions is by using `nvm`. From the CoqPilot root directory, execute:\n```bash\nnvm use\n```\nIf you prefer not to use `nvm`, ensure you install the Node.js version specified in the [`.nvmrc`](.nvmrc) file by any other method you prefer.\n\nOnce Node.js is installed, the remaining setup will be handled by the `npm` package manager. Run the following commands:\n```bash\nnpm install\nnpm run compile\n```\n\nTo run the extension from the VSCode, you can press `F5` or click on `Run extension` in the `Run and Debug` section. It will open a new window with the extension running.\n\nTo run all tests properly (i.e. with rebuilding the resources and the code first), execute the following task:\n```bash\nnpm run clean-test\n```\n\nTo run specific tests, you can use `npm run test -- -g=\"grep pattern\"`.\n\n\u003c!-- ## Architecture\n\nThe extension's architecture overview is stored in the [ARCHITECTURE.md](https://github.com/JetBrains-Research/coqpilot/blob/refactor/ARCHITECTURE.md) file. It will be extended and updated as the project evolves. --\u003e\n\n## Extension Settings\n\nThis extension contributes the following settings:\n\n* `coqpilot.contextTheoremsRankerType` : The type of theorems ranker that will be used to select theorems for proof generation (when context is smaller than taking all of them). Either randomly, by Jacard index (similarity metric) or by distance from the theorem, with the currently observed admit. \n* `coqpilot.loggingVerbosity` : Verbosity of the logs. Could be `info`, `debug`.\n* `coqpilot.coqLspServerPath` : Path to the coq-lsp server. By default, it is set to `coq-lsp`.\n\n* `coqpilot.predefinedProofsModelsParameters`, `coqpilot.openAiModelsParameters`, `coqpilot.grazieModelsParameters` and `coqpilot.lmStudioModelsParameters`:\n\nEach of these settings are modified in `settings.json` and contain an array of models from this service. Each model will be used for generation independantly. Multiple models for a single service could be defined. For example, you can define parameters for two OpenAI gpt models. One would be using `gpt-3.5` and the other one `gpt-4`. CoqPilot will first try to generate proofs using the first model, and if it doesn't succeed, it will try the second one. This way CoqPilot iterates over all services (currently 4 of them) and for each service it iterates over all models. \n\n## Guide to Model Configuration\n\n### How VSCode settings work\n\nA common way to change the settings, contributed by the extension, is to open the `settings.json` file, or click `Edit in settings.json` on some field in settings UI. Say, by default extension contributes field (setting) `A` with default state `a'`. When you click edit, this field is being copied to the `settings.json` file with the value `a'`: \n```json\n{\n    \"A\": \"a'\"\n}\n```\nFrom that moment and until you completely remove this field from the `settings.json` file, this will be the source of truth for this setting. Once again, if you want to set the value of the setting `A` back to the default, you have to remove this field from the file completely.\n\n### Model configuration\n\nAs mentioned in the previous section, at the moment, four services are supported. \n\nBy default, only `PredefinedProofs` and `OpenAI` services are enabled. The first one tries `auto.` tactic and the second one has one model -- `gpt-3.5`. Models for other services are defaulted with empty arrays. That denotes that we do not create any models from these services. \n\nEach and every service is configured with an array of independent models. This was made to easily experiment with different models and their parameters. \n\nThe simplest service to configure is `PredefinedProofs`: \n```json\n{\n    \"coqpilot.predefinedProofsModelsParameters\": [\n        {\n            \"modelId\": \"predefined proofs\",\n            \"tactics\": [\n                \"reflexivity.\",\n                \"simpl. reflexivity.\",\n                \"auto.\"\n            ]\n        }\n    ]\n}\n```\nThe `modelId` property may be any string you like, but it should be unique for each model. This way, CoqPilot will be able to correctly tell you which model might have configuration issues.\n\nThe most commonly used service is `OpenAI` (`Grazie` and `LmStudio` are configured very similarly). \n```json\n{\n    \"coqpilot.openAiModelsParameters\": [\n        {\n            \"modelId\": \"openai-gpt-4o-mini\",\n            \"modelName\": \"gpt-4o-mini\",\n            \"temperature\": 1,\n            \"apiKey\": \"***your-api-key***\",\n            \"choices\": 15,\n            \"systemPrompt\": \"Generate proof...\",\n            \"maxTokensToGenerate\": 2048,\n            \"tokensLimit\": 4096,\n            \"maxContextTheoremsNumber\": 100,\n            \"multiroundProfile\": {\n                \"maxRoundsNumber\": 1,\n                \"proofFixChoices\": 1,\n                \"proofFixPrompt\": \"Unfortunately, the last proof is not correct...\",\n                \"maxPreviousProofVersionsNumber\": 100\n            }\n        }\n    ],\n}\n```\nDon't forget to set up the `apiKey` field, by default it is set to `None`. Moreover, make sure that your OpenAI key is valid and has enough credits to run the models. If you create a free version of the key, it will not work (it has some weird limitations like 5 requests per inf). You can check you key here: https://platform.openai.com/playground/chat. If the playground works, the key is probably valid.\n\nMulti-round profile setting configures the number of attempts to fix the proof if it is incorrect. If the proof is incorrect, the compiler message is sent to the LLM with a request to repair it. The number of round attempts for one proof is set by `maxRoundsNumber`. The number of choices for the proof fixing is set by `proofFixChoices`. By default, values are set to 1 and that means that **NO** attempts to fix the proof are made. That means that proof is only being generated once. That's equivalent to say that multi-round fixing is turned off. 0 is not a valid value for `maxRoundsNumber` nor for `proofFixChoices`.  \n\nAnother thing to keep in mind: We are still in beta and changes in settings may occur pretty often. When that happens, and your re-defined settings (which are stored aside from the extension) are not updated (by hand), this can lead to exceptions. Keep in mind that if you get an error or plugin does not start after the update, you may want double check the settings. Easy way is remove the setting completely in `settings.json`, e.g. `openAiModelsParameters`, than go to the UI, click `Edit in settings.json` on the `openAiModelsParameters` field. It will fill up with updated default values. Afterwards you can re-define the settings as you want.\n\n## Contributed Commands\n\n* `coqpilot.perform_completion_under_cursor`: Try to generate proof for the goal under the cursor.\n* `coqpilot.perform_completion_for_all_admits`: Try to prove all holes (admitted goals) in the current file.\n* `coqpilot.perform_completion_in_selection`: Try to prove holes (admitted goals) in selection. \n\n## Benchmark\n\nTo run benchmarks on some project, apart from installing and building CoqPilot manually as described above, you will need to download the necessary projects that are used as datasets for the benchmarks. These projects are added as submodules to the repository. To download them, run the following commands:\n```bash\ngit submodule init\ngit submodule update\n```\nAfter that, you need to build the projects. Be careful, the actively maintained way to build this projects is `nix`. Moreover, when adding your own projects, make sure that they are built using `coq-8.19.0`.\n\n### New framework (beta)\n\nThe new benchmarking framework with extended capabilities is now available. \nHowever, it is still in the testing phase, so some bugs and missing features may be present.\n\nTo use it, follow the instructions in the [`BENCHMARKING_FRAMEWORK_GUIDE.md`](etc/docs/benchmark/BENCHMARKING_FRAMEWORK_GUIDE.md).\n\n### Legacy framework\n\nThe process of running the benchmark is not perfectly automated and we are working on it. For now, one project (one unit containing nix environment) shall be ran at a time. Let's say you are going to run the benchmark on the `imm` project. You will have to do the following: \n\n\u003c!-- 0. Go the the `imm` subdirectory and add a `_CoqProject` file in the root with the following: \n    ```\n    -I result/lib/coq/8.19/user-contrib/imm\n    -R result/lib/coq/8.19/user-contrib/imm imm\n    ```\n    This is needed for the lsp-server to correctly resolve file dependencies. --\u003e\n\n1. Install nix, as specified in the [here](https://nixos.org/download.html). \n\n2. Install needed caches: \n    ```bash\n    nix-env -iA nixpkgs.cachix \u0026\u0026 cachix use coq \u0026\u0026 cachix use coq-community \u0026\u0026 cachix use math-comp\n    cachix use weakmemory\n    ```\n\n3. Go to the `imm` subdirectory, apply the nix environment (without it the project will **NOT** build) and build the project: \n    ```bash\n    cd dataset/imm \n    nix-shell \n    make\n    ```\n4. Make sure the `_CoqProject` was successfully generated in the root of your project. Return to the project root not exiting the nix-shell. Run the benchmark: \n    ```bash\n    cd ../../\n    npm run benchmark\n    ```    \n\nIf you aim to run the benchmark with the use of `Tactician`, you should install the `imm` project and `Tactician` with `opam`. For `opam` installation instruction, you can refer to the [Tactician website](https://coq-tactician.github.io/manual/installation/) and [imm repository](https://github.com/weakmemory/imm/tree/master?tab=readme-ov-file#installation-via-opam-supported-up-to-the-15-version-of-imm) (this part of the README file is a little outdated, but the installation process for `coq-8.19.0` is still the same).\n\n## Integrating other solutions\n\nAs CoqPilot supports adding predefined commands to try as completion both in the plugin and the benchmark, you can integrate `Coq` generation methods, that contribute a specific tactic and are triggered from OCaml. \n\n### Tactician\n\n[Tactician](https://coq-tactician.github.io) is a tactic learner and prover for the Coq Proof Assistant. To install: \n```bash\nopam pin coq-tactician https://github.com/coq-tactician/coq-tactician.git#coq8.19\nopam install coq-tactician\ntactician enable\n```\n\nTo use completion tactics from `Tactician` you need to add an import: \n```coq\nFrom Tactician Require Import Ltac1.\n```\n\nAfter that, add the `synth.` tactic to the predefined tactics in the settings. \n\nNeural `Graph2Tac` completion unfortunately requires `coq \u003c 8.12~`. \n\n### CoqHammer\n\n[CoqHammer](https://coqhammer.github.io) is an automated reasoning tool for Coq. To install: \n```bash\nopam install coq-hammer\n```\n\nTo install `CoqHammer` using `nix`, you can use the following command [nix configuration file](https://github.com/vbgl/nixpkgs/tree/coq-coq-hammer).\n\nImport the tactics: \n```coq\nFrom Hammer Require Import Hammer.\n```\n\nThen add the `hammer.`, `sauto.` or any other tactic from `CoqHammer` to the predefined tactics in the settings.\n\n## Future plans\n\n- Currently the user needs to manually enter the nix shell to get the correct environment for the benchmarks. We are working on automating this process.\n\n## Release Notes\n\nRelease notes could be found in the [CHANGELOG.md](https://github.com/JetBrains-Research/coqpilot/blob/main/CHANGELOG.md) file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjetbrains-research%2Fcoqpilot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjetbrains-research%2Fcoqpilot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjetbrains-research%2Fcoqpilot/lists"}