{"id":13569360,"url":"https://github.com/emichael/dslabs","last_synced_at":"2025-05-14T12:08:30.419Z","repository":{"id":40364613,"uuid":"134776151","full_name":"emichael/dslabs","owner":"emichael","description":"Distributed Systems Labs and Framework","archived":false,"fork":false,"pushed_at":"2025-01-22T07:01:07.000Z","size":19647,"stargazers_count":1372,"open_issues_count":11,"forks_count":368,"subscribers_count":32,"default_branch":"master","last_synced_at":"2025-04-05T10:04:31.246Z","etag":null,"topics":["distributed-systems","university-project"],"latest_commit_sha":null,"homepage":"https://ellismichael.com/dslabs/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/emichael.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-24T22:46:09.000Z","updated_at":"2025-03-28T18:17:57.000Z","dependencies_parsed_at":"2023-02-08T05:17:18.564Z","dependency_job_id":"a17e03af-fa49-4156-8f14-f37fd4c1d353","html_url":"https://github.com/emichael/dslabs","commit_stats":{"total_commits":294,"total_committers":14,"mean_commits":21.0,"dds":0.108843537414966,"last_synced_commit":"43a03e487958aa75f0d80cf5fa9137477a8ffc17"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emichael%2Fdslabs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emichael%2Fdslabs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emichael%2Fdslabs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emichael%2Fdslabs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/emichael","download_url":"https://codeload.github.com/emichael/dslabs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248568142,"owners_count":21125976,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-systems","university-project"],"created_at":"2024-08-01T14:00:39.089Z","updated_at":"2025-04-12T12:39:17.535Z","avatar_url":"https://github.com/emichael.png","language":"Java","funding_links":[],"categories":["Java","分布式开发"],"sub_categories":[],"readme":"**DO NOT DISTRIBUTE OR PUBLICLY POST SOLUTIONS TO THESE LABS. MAKE ALL FORKS OF\nTHIS REPOSITORY WITH SOLUTION CODE PRIVATE.**\n\n\n# Distributed Systems Labs and Framework\n\n[Ellis Michael](http://ellismichael.com/)  \n*University of Washington*\n\n\nDSLabs is a new framework for creating, testing, model checking, visualizing,\nand debugging distributed systems lab assignments.\n\nThe best way to understand distributed systems is by implementing them. And as\nthe old saying goes, \"practice doesn't make perfect, perfect practice makes\nperfect.\" That is, it's one thing to write code which usually works; it's\nanother thing entirely to write code which works in all cases. The latter\nendeavor is far more useful for understanding the complexities of the\ndistributed programming model and specific distributed protocols.\n\nTesting distributed systems, however, is notoriously difficult. One thing we\nfound in previous iterations of the distributed systems class at UW is that many\nstudents would write implementations which passed all of our automated tests but\nnevertheless were incorrect, often in non-trivial ways. Some of these bugs would\nonly manifest themselves in later assignments, while others would go entirely\nunnoticed by our tests. We were able to manually inspect students' submissions\nand uncover some of these errors, but this approach to grading does not scale\nand does not provide the immediate feedback of automated tests.\n\nThe DSLabs framework and labs are engineered around the goal of helping students\nunderstand and correctly implement distributed systems. The framework provides a\nsuite of tools for creating automated tests, including model checking tests\nwhich systematically explore the state-space of students' implementations. These\ntests are much more likely to catch many common distributed systems bugs,\nespecially bugs which rely on precise orderings of messages. Moreover, when a\nbug is found, these search-based tests output a trace which generates the error,\nmaking debugging dramatically simpler. Finally, DSLabs is integrated with a\nvisual debugging tool, which allows students to graphically explore executions\nof their systems and visualize invariant-violating traces found by the\nmodel-checker.\n\n\n### Programming Model\nThe DSLabs framework is built around message-passing state machines (also known\nas I/O automata or distributed actors), which we call *nodes*. These basic units\nof a distributed system consist of a set of message and timer handlers; these\nhandlers define how the node updates its internal state, sends messages, and\nsets timers in response to an incoming message or timer. These nodes are run in\nsingle-threaded event loops, which take messages from network and timers from\nthe node's timer queue and call the node's handlers for those events.\n\nThis model of computation is typically the one we use when introduce distributed\nsystems for the first time and the one we use when we want to reason about\ndistributed protocols and prove their correctness. The philosophy behind this\nframework is that by creating for students a programming environment which\nmirrors the mathematical model distributed protocols are described in, we put\nthem on the best footing to be able to reason about their own implementations.\n\n\n### Testing and Model Checking\nThe lab infrastructure has a suite of tools for creating automated test cases\nfor distributed systems. These tools make it easy to express the scenarios the\nsystem should be tested against (e.g., varying client workloads, network\nconditions, failure patterns, etc.) and then run students' implementations on an\nemulated network (it is also possible to replace the emulated network interface\nwith an interface to the actual network).\n\nWhile executing certain scenarios is useful in uncovering bugs in students'\nimplementations, it is difficult to test all possible scenarios that might\noccur. Moreover, once these tests uncover a problem, it is a challenge to\ndiscover its root cause. Because the DSLabs framework has its node-centric view\nof distributed computation, it enables a more thorough form of testing –\n*model checking*.\n\nModel checking a distributed system is conceptually simple. First, the initial\nstate of the system is configured. Then, we say that one state of the system,\ns₂, (consisting of the internal state of all nodes, the state of their timer\nqueues, and the state of the network) is the successor of another state s₁ if it\ncan be obtained from s₁ by delivering a single message or timer that is pending\nin s₁. A state might have multiple successor states. Model checking is the\nsystematic exploration of this state graph, the simplest approach being\nbreadth-first search. The DSLabs model-checker lets us define *invariants* that\nshould be preserved (e.g. linearizability) and then search though all possible\nordering of events to make sure those invariants are preserved in students'\nimplementations. When an invariant violation is found, the model-checker can\nproduce a *minimal trace* which leads to the invariant violation.\n\nWhile model checking distributed systems is useful and has been used extensively\nin industry and academia to find bugs in distributed systems, exploration of the\nstate graph is still a fundamentally hard problem – the size of the graph is\ntypically exponential as a function of depth. To extend the usefulness of model\nchecking even further, the test infrastructure lets us prune the portion of the\nstate graph we explore for an individual test, guiding the search towards common\nproblems while still exploring all possible executions in the remaining portion\nof the state space.\n\nThe DSLabs model is built to be usable by students and be as _transparent as is\npractical_. Students will be required to make certain accommodations for the\nmodel checker's sake, but we try to limit these and provide tools that help\nvalidate the model checker's assumptions and debug model checking performance\nissues. Moreover, the model checker itself is not designed with state-of-the-art\nperformance as its only goal. Building a model checker that can test student\nimplementations of runnable systems built in a general-purpose language such as\nJava requires striking a balance between usability and performance.\n\n\n### Visualization\nThis framework is integrated with a visual debugger. This tool allows students\nto interactively explore executions of the distributed systems they build. By\nexploring executions of their distributed system, students can very quickly test\ntheir own hypotheses about how their nodes should behave, helping them discover\nbugs in their protocols and gain a deeper understanding for the way their\nsystems work. Additionally, the tool is used to visualize the\ninvariant-violating traces produced by the model-checker.\n\n\n## Assignments\nWe currently have four individual assignments in this framework. In these\nprojects, students incrementally build a distributed, fault-tolerant, sharded,\ntransactional key/value store!\n- Lab 0 provides a simple ping protocol as an example.\n- Lab 1 has students implement an exactly-once RPC protocol on top of an\n  asynchronous network. They re-use the pieces they build in lab 1 in later labs.\n- Lab 2 introduces students to fault-tolerance by having them implement a\n  primary-backup protocol.\n- Lab 3 asks students to take the lessons learned in lab 2 and implement Paxos.\n- Lab 4 has students build a sharded key/value store out of multiple replica\n  groups, each of which uses Paxos internally for replication. They finish by\n  implementing a two-phase commit protocol to handle multi-key updates.\n\nParts of this sequence of assignments (especially labs 2 and 4) are adapted from\nthe [MIT 6.824 Labs](http://nil.csail.mit.edu/6.824/2015/). The finished product\nis a system whose core design is very similar to production storage systems like\nGoogle's Spanner.\n\nWe have used the DSLabs framework and assignments in distributed systems classes\nat the University of Washington.\n\n\n## Directory Overview\n- `framework/src` contains the interface students program against.\n- `framework/tst` contains in the testing infrastructure.\n- `framework/tst-self` contains the tests for the interface and testing\n  infrastructure.\n- `labs` contains a subdirectory for each lab. The lab directories each have a\n  `src` directory initialized with skeleton code where students write their\n  implementations, as well as a `tst` directory containing the tests for that\n  lab.\n- `handout-files` contains files to be directly copied into the student\n  handout, including the main `README` and `run-tests.py`.\n- `grading` contains scripts created by previous TAs for the course to batch\n  grade submissions.\n- `www` contains the DSLabs website which is built with Jekyll.\n\nThe `master` branch of this repository is not setup to be distributed to\nstudents as-is. The `Makefile` has targets to build the `handout` directory and\n`handout.tar.gz`, which contain a single JAR with the compiled framework,\ntesting infrastructure, and all dependencies. The `handout` branch of this\nrepository is an auto-built version of the handout.\n\n\n## Contributing\nThe main tools for development are the same as the students' dependencies — Java\n17 and Python 3. You will also need a few utilities such as `wget` to build with\nthe provided `Makefile`; MacOS users will need `gtar` and `gcp` provided by the\n`coreutils` Homebrew package, and `gsed` provided by its own Homebrew package.\n\nIntelliJ files are provided and include a code style used by this project. In\norder to provide IntelliJ with all of the necessary libraries, you must run\n`make dependencies` once after cloning the repository and whenever you add to or\nmodify the project's dependencies. You will also need the Lombok IntelliJ\nplugin.\n\nThis project uses\n[`google-java-format`](https://github.com/google/google-java-format) to format\nJava files. You should run `make format` before committing changes. If you want\nIntelliJ to apply the same formatting, you will need the `google-java-format`\nIntelliJ plugin, and you will need to apply the necessary [post-install\nsettings](https://github.com/google/google-java-format/blob/master/README.md#intellij-jre-config)\nin IntelliJ.\n\nIf you add fields to any student-visible classes (all classes in the `framework`\npackage as well as `SearchState` and related classes), you should take care to\nensure that `toString` prints the correct information and that the classes are\ncloned correctly. See `dslabs.framework.testing.utils.Cloning` for more details.\nAlso see Lombok's `@ToString` annotation for more information about customizing\nits behavior. In particular, note that `transient` and `static` fields are\nignored by default by all cloning, serialization, and `toString` methods.\n\n\n## Acknowledgements\nThe framework and labs have been improved thanks to valuable contributions from:\n- Alex Saveau\n- Andrew Wei\n- Arman Mohammed\n- Doug Woos\n- Guangda Sun\n- James Wilcox\n- John Depaszthory\n- Kaelin Laundry\n- Logan Gnanapragasam\n- Nick Anderson\n- Paul Yau\n- Sarang Joshi\n- Thomas Anderson\n\nThe lab assignments, especially labs 2 and 4, were adapted with permission from\nthe MIT 6.824 labs developed by Robert Morris and colleagues.\n\n\n## Contact\nBug reports and feature requests should be submitted using the GitHub issues\ntool. Email Ellis Michael (emichael@cs.washington.edu) with any other questions.\n\nIf you use these labs in a course you teach, I'd love to hear from you!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femichael%2Fdslabs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Femichael%2Fdslabs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femichael%2Fdslabs/lists"}