{"id":14966267,"url":"https://github.com/mit-pdos/perennial","last_synced_at":"2025-04-12T22:39:14.844Z","repository":{"id":41207536,"uuid":"173480464","full_name":"mit-pdos/perennial","owner":"mit-pdos","description":"Verifying concurrent crash-safe systems","archived":false,"fork":false,"pushed_at":"2025-04-09T18:38:49.000Z","size":25046,"stargazers_count":176,"open_issues_count":10,"forks_count":38,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-04-09T18:49:26.356Z","etag":null,"topics":["concurrency","coq","verification"],"latest_commit_sha":null,"homepage":"","language":"Coq","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mit-pdos.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-02T17:54:02.000Z","updated_at":"2025-04-09T18:32:55.000Z","dependencies_parsed_at":"2024-01-11T04:39:25.298Z","dependency_job_id":"d3d22cb5-128f-4879-bda3-27455d18ebf4","html_url":"https://github.com/mit-pdos/perennial","commit_stats":{"total_commits":6913,"total_committers":32,"mean_commits":216.03125,"dds":0.659482135107768,"last_synced_commit":"11abaa58606535886591ccf25967f4103500f20a"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-pdos%2Fperennial","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-pdos%2Fperennial/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-pdos%2Fperennial/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-pdos%2Fperennial/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mit-pdos","download_url":"https://codeload.github.com/mit-pdos/perennial/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248643043,"owners_count":21138353,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","coq","verification"],"created_at":"2024-09-24T13:36:07.532Z","updated_at":"2025-04-12T22:39:14.822Z","avatar_url":"https://github.com/mit-pdos.png","language":"Coq","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Verifying concurrent, crash-safe systems with Perennial\n\n[![CI](https://github.com/mit-pdos/perennial/actions/workflows/ci.yml/badge.svg)](https://github.com/mit-pdos/perennial/actions/workflows/ci.yml)\n\nPerennial is a system for verifying correctness for systems with both\nconcurrency and crash-safety requirements, including recovery procedures. For\nexample, think of file systems, concurrent write-ahead logging like Linux's jbd2\nlayer, and persistent key-value stores like RocksDB.\n\nPerennial uses [Goose](https://github.com/goose-lang/goose) to enable verification\nof programs written in (a subset of) Go.\n\nThis repository also includes a proof of correctness for GoJournal, the verified journaling system used in\n[go-nfsd](https://github.com/mit-pdos/go-nfsd), including a simple NFS\nserver built on top.\n\n## Compiling\n\nWe develop Perennial using Coq master and maintain compatibility with the\nlatest version of Coq.\n\nThis project uses git submodules to include several dependencies. You should run `git submodule update --init --recursive` to set that up.\n\nTo compile just run `make` with Coq on your `$PATH`.\n\nWe compile with [coqc.py](etc/coqc.py), a Python wrapper around `coqc` to get\ntiming information. The wrapper requires Python3 and the `argparse` library. You\ncan also compile without timing information with `make TIMED=false`.\n\n## Maintaining dependencies\n\nThere are a few dependencies managed as submodules in `external/`. To update\nthem, run `git submodule update --init --remote`, then commit the resulting\nchange with git.\n\nThe dependencies are frozen at a particular version to avoid breaking the\nPerennial build when there are incompatible upstream changes.\nWe use Dependabot to do daily checks for dependency updates.\n\n## Compilation times\n\nPerennial takes about 120 CPU minutes to compile. Compiling in parallel with\n`make -j4` is significantly faster, and can cut the time down to 45-50 minutes.\n\nIncremental builds are better, after Iris and some core libraries are compiled.\n\nWhen you make a change to a dependency, you can keep working without fully\ncompiling the dependency by compiling `vos` interface files, which skips proofs.\nThe simplest way to do this is just to run `make vos`, but it's fastest to pass\na specific target, like `make src/program_proof/wal/proof.required_vos`, which\nonly builds the vos dependencies to work on the `wal/proof.v` file.\n\nIf you're working on Goose and only need to re-check the Goose output, you can run `make interpreter` to run the interpreter's semantics tests, or directly compile just the Goose output files.\n\nCoq also has a feature called `vok` files, where `coqc` compiles a `vos` file\nwithout requiring its dependencies to be built. The process does not produce a\nself-contained `vo` file, but emits an empty `vok` file to record that checking\nis complete. This allows checking individual files completely and in parallel.\nUsing `vos` and `vok` files can significantly speed up the edit-compile-debug\ncycle. Note that technically `vok` checking isn't the same as regular compilation - it doesn't check universe constraints in the same way.\n\n## Updating Goose output\n\nThis repo has committed versions of the output of Goose, to avoid making Go and\nGoose a dependency for compilation. You can update these using the\n`./etc/update-goose.py` script, which records exactly how to generate the output\nfor the various Goose projects we have. Use `./etc/update-goose.py --help` to get all the options. The script only translates the projects you pass\na path to.\n\n## Source organization\n\n`src/`\n\n- `program_logic/`\n  The main library that implements the crash safety reasoning in Perennial. This\n  includes crash weakest preconditions, crash invariants, idempotence, and crash\n  refinement reasoning.\n\n- `goose_lang/`\n  A lambda calculus with memory, concurrency, and an \"FFI\" to some external\n  world, which can be instantiated to provide system calls specified using a\n  relation over states.\n\n  This language is the target of Goose and thus models Go and also implements\n  the Iris `language` and Perennial `crash_lang` interfaces for proofs using\n  Perennial.\n\n  This directory includes a lifting to ghost state that supports more standard\n  points-to facts, compared to the semantics which specifies transitions over\n  the entire state. It also proves Hoare triples using these resources for the\n  primitives of the language.\n\n  - `typing.v`\n    A type system for GooseLang. This type system is used as part of the\n    `typed_mem/` machinery. TODO: there's much more to say here.\n\n  - `lib/`\n    GooseLang is partly a shallow embedding - many features of Go are\n    implemented as implementations. These features are divided into several\n    libraries. Each library has an implementation and a proof file. For example,\n    `map/impl.v` implements operations maps using GooseLang's sums while\n    `map/map.v` proves Hoare triples for the implementation. Separating the\n    implementation allows us to run the implementation in the GooseLang\n    interpreter without compiling the proofs.\n    - `typed_mem/`\n      Implements support for flattening products over several contiguous\n      locations, which is the foundation for supporting struct fields as\n      independent entities. The proofs build up reasoning about the `l ↦[t] v`\n      assertion, which says `l` points to a value `v` of type `t` (in the\n      GooseLang type system). If `t` is a composite type, this assertion owns\n      multiple physical locations.\n    - `struct/`\n      Implements struct support. Structs are essentially tuples with names for\n      the fields. The theorems proven here culminate in a way to split a typed\n      points-to for a struct into its individual fields.\n    - `slice/`\n      Implements Go slices using a tuple of an array, length, and capacity.\n    - `map/`\n      Implements Go maps as a linked list of key-value pairs, terminating in a\n      default value.\n    - `loop/`\n      Implements a combinator for loops on top of the basic recursion support.\n    - `lock/`\n      Implements locks and condition variables using a spin lock, which is\n      implemented using `CmpXchg`.\n    - `encoding/`\n      Implements uint64 and uint32 little-endian encoding.\n  - `examples/`\n    The Goose unit tests; these are auto-generated from the Goose repo, from\n    `internal/examples/`.\n  - `interpreter/`\n    An interpreter for sequential GooseLang, equipped with a proof of partial\n    correctness: if the interpreter produces a result, the semantics also can.\n\n    This is used to implement tests in `generated_test.v`. These tests are Go\n    functions which return booleans that should be true; we check this using Go\n    test, and compare against the interpreter's behavior.\n\n  - `ffi/`\n\n    Two different FFIs to plug into `GooseLang` - `disk.v` is the one we\n    actually use, while `append_log.v` is the FFI-based specification for the\n    `append_log` example.\n\n* `program_proof/`\n\n  The proofs about programs we have so far.\n\n  - `append_log_proof.v` Hoare triples about the `append_log` example, which is\n    implemented in the Goose repo at `internal/examples/append_log/`.\n\n  - `examples/` Examples we wrote for POPL\n\n  - `wal/`, `txn/`, and `buftxn/` proof of the transaction system library in\n    goose-nfsd\n\n  - `simple/` proof of a simple NFS server\n\n* `Helpers/`\n\n  - `Integers.v`\n    Wrapper around `coqutil`'s word library for u64, u32, and u8.\n  - `Transitions.v`\n    A library for writing relations in a monadic, combinator style.\n  - `NamedProps.v`\n    An Iris library for naming hypotheses within definitions and using them to\n    automatically destruct propositions.\n  - other files are basically standard library extensions\n\n* `algebra/`\n\n  Additional CMRAs for Iris ghost variables\n\nIt's also worth noting that `external/Goose` contains committed copies of the\nGoose output on some real code we have. This includes\n`github.com/tchajed/marshal` and `github.com/mit-pdos/goose-nfsd`. The directory\nstructure here mirrors the way Go import paths work.\n\n## Publications\n\nPerennial 1 is described in our SOSP paper, \"[Verifying concurrent, crash-safe\nsystems with Perennial](https://www.chajed.io/papers/perennial:sosp2019.pdf)\".\nThe actual codebase was quite different at the time of this paper; it notably\nused a shallow embedding of Goose and did not have WPCs or any of the associated\nprogram logic infrastructure. See the tag `sosp2019` or the `shallow` branch.\n\nGoose is briefly described in a CoqPL extended abstract and associated talk,\n\"[Verifying concurrent Go code in Coq with\nGoose](https://www.chajed.io/papers/goose:coqpl2020.pdf)\".\n\nThe verified interpreter and test framework for Goose is described in Sydney Gibson's masters thesis, \"[Waddle: A proven interpreter and test framework for a subset of the Go semantics](https://pdos.csail.mit.edu/papers/gibsons-meng.pdf)\".\n\nThe proof of GoJournal's correctness is described in the OSDI paper,\n\"[GoJournal: a verified, concurrent, crash-safe journaling system](https://www.chajed.io/papers/gojournal:osdi2021.pdf)\". The framework\nhas evolved in several ways since then. See the tag `osdi21` for the version\nused there.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmit-pdos%2Fperennial","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmit-pdos%2Fperennial","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmit-pdos%2Fperennial/lists"}