{"id":19825795,"url":"https://github.com/jepsen-io/redpanda","last_synced_at":"2025-05-01T14:30:38.167Z","repository":{"id":69265238,"uuid":"428776683","full_name":"jepsen-io/redpanda","owner":"jepsen-io","description":"Tests for the Redpanda distributed queue","archived":false,"fork":false,"pushed_at":"2024-11-13T16:08:40.000Z","size":305,"stargazers_count":5,"open_issues_count":0,"forks_count":4,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-11-29T09:18:13.970Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jepsen-io.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-16T18:48:46.000Z","updated_at":"2024-11-13T16:08:44.000Z","dependencies_parsed_at":"2024-10-19T01:40:03.742Z","dependency_job_id":null,"html_url":"https://github.com/jepsen-io/redpanda","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jepsen-io%2Fredpanda","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jepsen-io%2Fredpanda/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jepsen-io%2Fredpanda/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jepsen-io%2Fredpanda/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jepsen-io","download_url":"https://codeload.github.com/jepsen-io/redpanda/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251889939,"owners_count":21660416,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T11:08:39.052Z","updated_at":"2025-05-01T14:30:38.137Z","avatar_url":"https://github.com/jepsen-io.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# jepsen.redpanda\n\nTests for the Redpanda distributed queue.\n\n## Installation\n\nYou'll need a Jepsen environment: see\n[https://github.com/jepsen-io/jepsen#setting-up-a-jepsen-environment](Jepsen's\nsetup documentation) for details. In short, this means a cluster of 3+ Debian\nBuster *DB nodes* to run instances of Redpanda, and a *control node* to run\nthis test harness. The control node needs a JDK, JNA, Leiningen, Gnuplot, and\nGraphviz. On the control node, plan on ~24G of memory and about 100 GB of disk, if you want to run a few hundred ~1000 second tests back to back.\n\nOnce you have a Jepsen cluster set up, you'll need a copy of this test harness.\nThat can be a tarball or git clone of this repository, or you can compile this\ntest to a fat jar using `lein uberjar`, copy that jar to your control node,\nand invoke it using `java -jar \u003cjar-file\u003e \u003ctest args ...\u003e`. In these docs,\nwe'll assume you've got a copy of this directory and are invoking tests via\n`lein run`.\n\n## Quickstart\n\nFor these examples, we'll assume you have a file `~/nodes` which has your DB\nnode hostnames, one per line, and that your user with sudo access on each node\nis named `admin. This is the setup you'd get out of the box from the [Jepsen\nCloudformation deployment on the AWS\nMarketplace](https://aws.amazon.com/marketplace/pp/prodview-ykhheuyq5qdnq). All\nthese commands should be run in the top-level directory of this repository.\n\nTo run a very brief test, just to make sure everything works:\n\n```\nlein run test --nodes-file ~/nodes --username admin\n```\n\nThis will likely print something like:\n\n```clj\n ...\n :valid? true}\n\n\nEverything looks good! ヽ(‘ー`)ノ\n```\n\nTo observe inconsistent offsets and/or duplicates in Redpanda 21.10.1, try:\n\n```\nlein run test --nodes-file ~/nodes --username admin -s --nemesis pause,kill --time-limit 300 --test-count 5\n```\n\n`-s` asks for safer options than the defaults: it turns on idempotence, sets\nauto-offset-reset=earliest, etc. `--nemesis pause,kill` asks the nemesis to\npause and kill random nodes throughout the test. We also increase the time of\neach test to 300 seconds, and run up to 5 tests in a row, stopping as soon as\nthe test detects an error. This might emit something like:\n\n```clj\n :workload {:valid? false\n            :duplicate {:count 1,\n                        :errs ({:key 8, :value 711, :count 2})},\n```\n\nOr we could run the test against version 21.11.2, with process crashes and membership changes, and for significantly longer. This should, after a few hours, demonstrate lost/stale messages, reported as `unseen`:\n\n```\nlein run test --nodes-file ~/nodes --username admin -s --no-ww-deps --nemesis kill,membership --time-limit 1000 --test-count 10 --version 21.11.2-1-f58e69b6\n```\n\nTo start a web server which lets you browse the results of these tests on port\n8080, run:\n\n```\nlein run serve\n```\n\n## Usage\n\n`lein run test` runs a single test--possibly multiple times in a row.\n\n`lein run test-all` runs a whole slew of different tests, including multiple\nworkloads, versions, combinations of nemeses, with and without transactions.\nYou can constrain any of these--for example, to compare behavior under network\npartitions across two different versions, both with and without transactions:\n\n```clj\nlein run test-all --nodes-file ~/nodes --username admin --nemesis partition --time-limit 300 --versions 21.10.1-1-e7b6714a,21.11.2-1-f58e69b6\n```\n\n`lein run serve` runs a web server to browse results of tests from the local\n`store/` directory. There's nothing magic about this directory--you can tar it\nup and copy it around, and it'll still work fine. You can copy one store\ndirectory's contents into another and that'll work too, which might be helpful\nfor CI or farming out tests to lots of clusters.\n\nAll three of these commands provide docs for all of their options: try `lein\nrun test -h` for help on running a single test.\n\n## Understanding Results\n\nIn each test directory in `store/` you'll find several files. For starters, there's one directory for every DB node, which contains:\n\n- `data.tar.bz2`: a tarball of that node's Redpanda data directory\n- `redpanda.log`: the stdout/stderr logs of the Redpanda server\n- `tcpdump.pcap`: If running with `--tcpdump`, a packet capture file of this node's kafka-protocol network traffic.\n\nThree files show the chronology of the test. `jepsen.log` has the full output\nof the test run--everything printed to the console. `history.edn` is a\nmachine-readable file with all the logical operations Jepsen performed during\nthe test. `history.txt` is the same, but as a tab-aligned text file.\n\n`latency-raw.png` and `latency-quantiles.png` show the latency of each\noperation over time. Colors denote successful (ok), failed (fail), or unknown\n(info) operations. Shape denotes the logical function `:f` of that\noperation--for example, a `poll`, `subscribe`, `txn`. Nemesis activity is shown as colored bars along the top of the plot, and vertical lines whenever the nemesis changes something.\n\n`rate.png` shows the rate of requests per second, over time, with the same shapes and colors as the latency plot.\n\n`unseen.png` shows the number of messages which were committed but not read by\nany poller over time, broken out by key (partition-topic). Two final vertical\nlines show the start and stop of the final polling period, when the cluster has\nhealed and we're trying to read all writes. This graph should go to zero\nrelatively quickly during the recovery window: if it doesn't, that suggests\nsome writes have been lost or are severely delayed.\n\n`realtime-lag` plots shows how far behind the most recent committed write each\nconsumer is, over time. These come in several families, aggregated by key, by\nworker thread (which maps to one client at any given time), or by neither.\nWhenever we subscribe or assign a fresh topic, this might jump up, but it\nshould head down as consumers catch up to the most recent values. You can use\nthese plots to identify whether a single key or single thread has gotten stuck,\nand whether other threads or keys are making progress.\n\n`test.fressian` and `test.jepsen` have binary representations of the test in\nits entirety.\n\n`elle/` shows cycles between transactions, both as text files and as svg graphs. These are categorized by anomaly type--e.g. G0, G1c, etc.\n\n`orders` appears whenever we see Weird Things Happen on some key. It contains\none SVG file for each key that did something odd, and shows the view of each\n`ok` operation into that particular key's offsets. Time flows top to bottom,\nand offsets are arranged left to right. Each number is the value of the message\nat that particular offset. Hovering over any row shows more information. This\nis helpful for understanding when there are reorderings or message loss.\n\n`results.edn` has the results of the checker.\n\n### Results In Depth\n\n`results.edn` is a map with several keys. At each layer, the `:valid?` key says\nwhether the results (or part thereof) were considered valid, or if an anomaly was detected.\n\nThe `stats` part of the analysis provides overall statistics on the number of\noperations, how many were successful (ok), failed (fail), or indeterminate\n(info). These are also broken down by function, so you can see the behavior of\n(e.g.) just `poll` operations.\n\n```clj\n{:stats {:valid? true,\n         :count 14772,\n         :ok-count 14653,\n         :fail-count 0,\n         :info-count 119,\n         :by-f {:assign {:valid? true,\n                         :count 1901,\n                         :ok-count 1901,\n                         :fail-count 0,\n                         :info-count 0},\n                :crash {:valid? false,\n                        :count 8,\n                        :ok-count 0,\n                        :fail-count 0,\n                        :info-count 8},\n                :debug-topic-partitions {:valid? true,\n                                         :count 8,\n                                         :ok-count 8,\n                                         :fail-count 0,\n                                         :info-count 0},\n                :poll {:valid? true,\n                       :count 6551,\n                       :ok-count 6551,\n                       :fail-count 0,\n                       :info-count 0},\n                :send {:valid? true,\n                       :count 6304,\n                       :ok-count 6193,\n                       :fail-count 0,\n                       :info-count 111}}},\n```\n\n`clock` and `perf` are trivially true; they generate the clock, latency, and\nrate plots as a side effect.\n\n```clj\n :clock {:valid? true},\n :perf {:latency-graph {:valid? true},\n        :rate-graph {:valid? true},\n        :valid? true},\n```\n\n`ex` tracks exceptions thrown by Jepsen Clients during the test. When\nexceptions appear here, you may want to add them to the error handling in that\nparticular client. It is, in general, always safe to allow them to\nthrow---explicitly handling these errors improves test specificity and\nperformance.\n\n`assert` looks for assertion errors thrown by the Redpanda server, by parsing\nlogfiles.\n\n`workload` contains workload-specific results.\n\n#### Queue results\n\nThe queue workload includes two important keys: `:error-types`, which shows all\n\"interesting\" behaviors observed during the test, and `:bad-error-types`, which\nare those we think are specifically illegal given the test being run. For\nexample:\n\n            :error-types (:G0\n                          :duplicate\n                          :int-nonmonotonic-poll\n                          :int-poll-skip\n                          :poll-skip),\n            :bad-error-types (:duplicate\n                              :int-nonmonotonic-poll\n                              :int-poll-skip\n                              :poll-skip)},\n\nHere we detected a G0 anomaly, but because those happen *normally* in the Kafka\ntransaction model, we didn't flag it as a \"bad\" error. The duplicate, internal\nnonmonotonic polls and poll skips, and external poll skips, caused this test to\nfail.\n\nEach error type has a corresponding key in `:errors` part of the workload\nresults. Examples of each type of error follow. For simplicity, these examples\nare drawn from the test suite's internal tests; they use keys like `:x` rather\nthan integers, but should otherwise be structurally alike to those reported by\nthe real test harness.\n\n`:inconsistent-offsets` signifies that a single offset in some key's log\ncontained multiple values. The offset-value mapping, called a `version order`,\nis derived from both send and poll operations. This example tells us that key\n`:x`, at offset 0, contained both values `1` and `2`. `:index` is a dense\noffset, without gaps.\n\n```clj\n{:inconsistent-offsets\n ({:key :x, :offset 0, :index 0, :values #{1 2}})}\n```\n\n`:G1a` reports cases where an operation definitely failed, but one of its\nwrites appeared in a poll operation. For instance, this error shows that on key\n`:x`, value `2` was written by a send operation which failed, but that send was\nlater observed by `:reader`.\n\n```clj\n{:G1a\n ({:key :x,\n   :value 2,\n   :writer\n   {:index 1,\n    :time 1,\n    :process 0,\n    :type :fail,\n    :f :send,\n    :value [[:send :x 2] [:send :y 3]]},\n   :reader\n   {:index 3,\n    :time 3,\n    :process 1,\n    :type :ok,\n    :f :poll,\n    :value [[:poll {:x [[0 2]]}]]}})}\n```\n\n`:lost-write` finds cases where a known-successful send occurs at offset a, and\nsome offset b (such that a \u003c b) is polled, and the message at offset a never\nappears to any poll. This example shows that on key `:x`, value `:a` was lost.\nIt was written at index 0 in the log for `:x`, and the highest index which was\nobserved in some poll was 2. The writer of value `:a` is given, and so is the\nreader which polled index 2. Because we expect pollers (across all consumers,\nat least) observe values without gaps, this read of offset 2 implies we should\nalso have read offset 0--and yet no such read was found in this history.\n\n```clj\n{:lost-write\n ({:key :x,\n   :value :a,\n   :index 0,\n   :max-read-index 2,\n   :writer\n   {:index 1,\n    :time 1,\n    :process 0,\n    :type :ok,\n    :f :send,\n    :value [[:send :x [0 :a]]]},\n   :max-read\n   {:index 7,\n    :time 7,\n    :process 0,\n    :type :ok,\n    :f :poll,\n    :value [[:poll {:x [[2 :c]]}]]}}\n```\n\nNote that this analysis is sophisticated enough to reason about inconsistent\noffsets conservatively. Not all parts of the checker do this, but lost-writes\nis careful to keep track of multiple indexes for a given message value, and\nmultiple values at a given index.\n\nThe lost-write checker also helps verify transactional atomicity: reading one\npart of a transaction lets the checker prove that all the other writes must\nhave been written also---even if the writing transaction was itself\nindeterminate.\n\n`:poll-skip` finds places where a poller unexpectedly jumps over some offsets in\nthe log during two successive calls to `poll` performed by *different*\noperations on the same client, and there was no call to `assign` or `subscribe`\nbetween those polls which would have caused the consumer to forget the offset\nit was tracking. This example shows that on key `:x`, a single consumer jumped\n2 indexes forward in the log, skipping over value `:c`. The two operations are\nshown: the first polled `:a` and `:b` at indexes 1 and 2, and the second polled\n`:d` at index 4. The checker here knew that there existed a value `:c` at index\n3 between these two polls, which went unobserved. `:delta` is the number of\nindexes between the two polls--if pollers read in perfect order, we'd expect\nthis to always be 1.\n\n```clj\n{:poll-skip\n ({:key :x,\n   :delta 2,\n   :skipped (:c),\n   :ops\n   [{:index 1,\n     :time 1,\n     :process 0,\n     :type :ok,\n     :f :poll,\n     :value [[:poll {:x [[1 :a] [2 :b]]}]]}\n    {:index 7,\n     :time 7,\n     :process 0,\n     :type :ok,\n     :f :poll,\n     :value [[:poll {:x [[4 :d]]}]]}]})}\n```\n\n`:int-poll-skip` finds the same thing, but inside a single operation. This is\nhelpful for detecting anomalies that could occur inside a single transaction.\nFor instance, this error shows that on key `x`, a single transaction polled values `:a` and `:d` in sequence, which skipped over `:b`.\n\n```clj\n{:int-poll-skip\n ({:key :x,\n   :values [:a :d],\n   :delta 2,\n   :skipped (:b),\n   :op\n   {:index 3,\n    :time 3,\n    :process 0,\n    :type :ok,\n    :f :poll,\n    :value [[:poll {:x [[1 :a] [4 :d]]}]]}})}\n```\n\n`:nonmonotonic-poll` finds cases where a single consumer, without changing its\nassign/subscribe mapping for some key, performed subsequent calls to `poll` and\nobserved values in the second poll which started at or before the previous poll\nfinished. For instance, this error shows two successive operations on the same consumer which polled values `:c` then `:b`, jumping back one index.\n\n```clj\n{:nonmonotonic-poll\n ({:key :x,\n   :values [:c :b],\n   :delta -1,\n   :ops\n   [{:index 3,\n     :time 3,\n     :process 0,\n     :type :ok,\n     :f :poll,\n     :value [[:poll {:x [[1 :a] [2 :b] [3 :c]]}]]}\n    {:index 7,\n     :time 7,\n     :process 0,\n     :type :ok,\n     :f :poll,\n     :value [[:poll {:x [[2 :b] [3 :c] [4 :d]]}]]}]})}\n```\n\n`:int-nonmonotonic-poll` does the same, but inside a single operation. For\ninstance, this error shows that on key `:x`, a single process went from reading offset 3 (`:c`) to offset 1 (`:a`), jumping two indices backwards in the log.\n\n```clj\n{:int-nonmonotonic-poll\n ({:key :x,\n   :values [:c :a],\n   :delta -2,\n   :op\n   {:index 3,\n    :time 3,\n    :process 0,\n    :type :ok,\n    :f :poll,\n    :value [[:poll {:x [[3 :c] [1 :a]]}]]}})}\n```\n\n`:int-send-skip` looks for a single transaction which performs two subsequent\nsends to the same key, and some other offset lands *between* those writes. This\nis another way to detect G0 cycles, where writes interleave with one another.\nIn this example, a single transaction wrote `:a` and `:c` to key `:x`, skipping\nover message `:b` from another transaction. This shows a lack of write\nisolation between Kafka/Redpanda transactions, and appears to be normal\nbehavior.\n\n```clj\n{:int-send-skip\n ({:key :x,\n   :values [:a :c],\n   :delta 2,\n   :skipped (:b),\n   :op\n   {:index 1,\n    :time 1,\n    :process 0,\n    :type :ok,\n    :f :send,\n    :value [[:send :x [1 :a]] [:send :x :c]]}})},\n```\n\n`:nonmonotonic-send` finds cases where two subsequent operations on the same\nproducer sent values to the same key, and the latter message wound up at an\noffset *prior* to the former message. For instance, this case shows that on key\n`:x`, two calls to `send` on the same producer, split across two different\noperations, wrote offsets out of order. The second operation's first send of\n`:a` landed three indices before the first operation's final send of `:d`.\n\n```clj\n{:nonmonotonic-send\n ({:key :x,\n   :values [:d :a],\n   :delta -3,\n   :ops\n   [{:index 1,\n     :time 1,\n     :process 0,\n     :type :ok,\n     :f :send,\n     :value [[:send :x [3 :c]] [:send :x [4 :d]]]}\n    {:index 5,\n     :time 11,\n     :process 0,\n     :type :ok,\n     :f :send,\n     :value [[:send :x [1 :a]] [:send :x [2 :b]]]}]})}\n```\n\n`:int-nonmonotonic-send` is the same thing, but inside a single transaction.\nHere, two calls to send within a single transaction received offsets out of\norder on key `:x`.\n\n```clj\n{:int-nonmonotonic-send\n ({:key :x,\n   :values [:c :a],\n   :delta -1,\n   :op\n   {:index 1,\n    :time 1,\n    :process 0,\n    :type :ok,\n    :f :send,\n    :value [[:send :x [3 :c]] [:send :x [1 :a]]]}}\n```\n\n`:duplicate` occurs when a single value appears at multiple offsets in some key's log. Since we only ever insert unique values, and do not (above the level of the Kafka producer's internal retries) ever retry, we expect that each value appear at most once per key. This example shows that on key `:x`, value `:a` appeared at two distinct offsets.\n\n```clj\n{:duplicate ({:key :x, :value :a, :count 2})}\n```\n\n`:unseen` reports the number of messages which were successfully committed but\nhave not appeared in any poll, as of the last poll in the history. This checker\ncannot distinguish between a lost write vs one which is simply very delayed. To\nmitigate this weakness, we try *really hard* to read everything at the end of a\ntest---but that's still not a guarantee that `:unseen` errors are truly lost.\nSure, they texted \"omw\" three hours ago and still haven't shown up to Show\nTunes at Sidetrack, but they might not be dead. Maybe they're watching a sixth\nepisode of Golden Girls and trying to figure out which high tops to wear. We\ndon't judge.\n\nThis example includes the time (in nanoseconds since the start of the test) of\nthe final unseen inference, a map `:unseen` of keys to the number of unseen\nmessages on each key, and a map `:messages` of keys to the specific messages\nunseen. This test failed to observe one committed message on key 6, and 5 on\nkey 23.\n\n```clj\n{:unseen {:time 1256465220303,\n          :unseen {6 1, 23 5},\n          :messages {6 (311),\n                     23 (360 361 362 363 365)}}}\n```\n\n`:G0` finds write cycles: a cluster of transactions such that each wrote some\nmessage both before *and* after every other transaction in the cluster. A data\nstructure representation is included in the workload. Here, for instance, a\npair of transactions had a cycle where T1 wrote before T2 on key `:x`, and\nvice-versa on key `:y`. The `:cycle` key shows the operations involved: [T1,\nT2, T1]. The `:steps` field explains the relationships between successive pairs\nin that cycle. The first relationship was a write-write (`:ww`) dependency on key `:x`, where the first transaction wrote message `:a` and the second wrote message `:b`.\n\n```clj\n{:G0\n [{:cycle\n   [{:index 2,\n     :time 2,\n     :process 0,\n     :type :ok,\n     :f :send,\n     :value [[:send :x [0 :a]] [:send :y [1 :a]]]}\n    {:index 3,\n     :time 3,\n     :process 1,\n     :type :ok,\n     :f :send,\n     :value [[:send :x [1 :b]] [:send :y [0 :b]]]}\n    {:index 2,\n     :time 2,\n     :process 0,\n     :type :ok,\n     :f :send,\n     :value [[:send :x [0 :a]] [:send :y [1 :a]]]}],\n   :steps\n   ({:type :ww,\n     :key :x,\n     :value :a,\n     :value' :b,\n     :a-mop-index 0,\n     :b-mop-index 0}\n    {:type :ww,\n     :key :y,\n     :value :b,\n     :value' :a,\n     :a-mop-index 1,\n     :b-mop-index 1}),\n   :type :G0}]}\n```\n\nThis can be a bit hard to understand from the data structure representation,\nbut you'll find corresponding plain-English and visual diagrams explaining this\ncycle in `elle/G0.txt` and `elle/g0/`.\n\n`:G1c` finds circular information flow: clusters where a cycle exists composed\nof write-read and write-write dependencies. As in G0, write-write dependencies\nare inferred from offsets written. Write-read dependencies are inferred\nwhenever one transaction polls another's sent messages. This G1c is comprised\nentirely of write-read dependencies---note the `:wr` types in each step. The\nfirst transaction sent `:a` to key `:x`, which was read by the second\ntransaction. The second transaction sent `:b` to key `:y`, which was read by\nthe first. Like G0, textual and visual explanations of this anomaly are\navailable in the `elle/` directory.\n\n```clj\n{:G1c\n [{:cycle\n   [{:index 2,\n     :time 2,\n     :process 0,\n     :type :ok,\n     :f :txn,\n     :value [[:send :x [0 :a]] [:poll {:y [[0 :b]]}]]}\n    {:index 3,\n     :time 3,\n     :process 1,\n     :type :ok,\n     :f :txn,\n     :value [[:send :y [0 :b]] [:poll {:x [[0 :a]]}]]}\n    {:index 2,\n     :time 2,\n     :process 0,\n     :type :ok,\n     :f :txn,\n     :value [[:send :x [0 :a]] [:poll {:y [[0 :b]]}]]}],\n   :steps\n   ({:type :wr, :key :x, :value :a, :a-mop-index 0, :b-mop-index 1}\n    {:type :wr, :key :y, :value :b, :a-mop-index 0, :b-mop-index 1}),\n   :type :G1c}]}\n```\n\nNote that both G0 and G1c cycles involving write-write edges may or may not be\n\"real\" depending on how you interpret Kafka's `producer.send` semantics. If you\nprefer to ignore these write dependencies, pass `--no-ww-deps` to the test, and\nit'll *only* infer write-read edges.\n\nQueue tests also generate an additional file, `consume-counts.edn`. This file\nattempts---perhaps poorly---to tell whether a history offered \"exactly once\nsemantics\". It looks at all successful operations which performed a poll\noperation while using `subscribe` (not `assign`, which we expect leads to\nduplicate polls!), and counts the number of times each value was polled. Under\nexactly-once semantics, I think this number should always be 1, but we were\nnever able to get this to work.\n\nThis file has two keys. `:distribution` is a map of counts (i.e. how many times\na record was polled) to the number of times that count occurred. `:dup-counts`\nis a map of keys (topic-partitions) to values to the number of times that value\nwas polled, for any values which were polled multiple times. For examples, this test had 17133 messages which were polled once, and 174 which were polled twice. Key 2 had a single duplicate, message 79, which was saw twice.\n\n```clj\n{:distribution {1 17133, 2 174},\n :dup-counts\n {2 {79 2},\n  3\n  {111 2,\n   112 2,\n   113 2,\n   114 2,\n   ...}\n ...}}\n```\n\n\n## What's Here\n\n### Overall Structure\n\n`project.clj` defines this test's version, dependencies, JVM options, and entry\npoint; it's read by Leiningen. Source code for the test suite lives in `src/`.\nTests for that testing code live in `test/`. In both of these directories,\nfolder structure maps to namespaces: the file `src/jepsen/redpanda/core.clj`\ndefines the `jepsen.redpanda.core` namespace. The `store/` directory stores the\nresults of any tests you might run.\n\nThe top-level namespace for this test is `jepsen.redpanda.core`, which defines\nCLI options and constructs tests to run, then passes them to Jepsen for\nexecution. `jepsen.redpanda.db.redpanda` and `jepsen.redpanda.db.kafka` defines\ndatabase setup and teardown for Redpanda and Kafka, respectively.\n`jepsen.redpanda.client` is for working with Kafka clients.\n`jepsen.redpanda.nemesis` handles fault injection: most notably, the cluster\nmembership state machine. Workloads live in `jepsen.redpanda.workload.queue`\nand jepsen.redpanda.workload.list-append`.\n\n### Workloads\n\nThis test comprises two workloads.\n\nThe main workload, `queue`, performs both transactional and non-transactional\nsends and polls, mixed with calls to `assign` or `subscribe`. At the end of the\ntest, it tries to read everything that was written via a series of final polls.\nIt has a sophisticated family of analyzers which look for duplicate writes,\ninconsistent offsets, places where consumers or producers jump forward or\nbackwards in offsets, aborted reads, and some basic cycles like G0 and G1c.\n\nThe second workload, `list-append`, is much simpler: it performs sends much\nlike `queue`, but for any read, attempts to read the *entire* topic-partition\nbased on the most recent offset.\n\nBoth workloads spread their operations across a rotating pool of\ntopic-partitions, creating new topics once existing topics reach a certain\nthreshold of writes. You can choose which workload is run via `-w queue` or `-w\nlist-append`.\n\n### Faults\n\n`--nemesis pause,kill` performs randomized process pauses and kill -9. The\nother fault types are `clock`, which jitters clocks around, `partition`, which\npartitions the network between DB nodes (but not between clients and servers!),\nand `membership`, which adds and politely removes nodes from the cluster.\n\nNote that clock skew tests will only work on nodes which have real\nclocks---Docker and LXC can't change the system clock.\n\nNote also that the membership nemesis in the main branch only works with new\nAPIs introduced after 21.11.2; tests using `--nemesis membership` will still\nrun with versions 21.11.2, but won't actually remove nodes, and will complain a\nlot. Use `git checkout compat-21.11.2` for the last version of the test suite\nwhich ran with 21.11.2's membership APIs. Note that this membership nemesis may\ndrive the cluster into unsafe regimes, so watch its activity in `jepsen.log`\ncarefully when verifying a failure.\n\n`--nemesis-interval 5` sets the mean interval between nemesis operations to 5\nseconds; see `latency.png` to get a sense of how this affects availability.\n``-db-targets` controls how DB-node related faults choose their targets;\n`--db-targets all`, for instance, kills and pauses *every* node at once,\nwhereas `--db-targets one` only kills or pauses a single node at a time.\n`--partition-targets` controls how Jepsen chooses the topology of network\npartitions.\n\n### Tests for Tests\n\nThis test harness also comes with its own tests--mainly for the queue\nworkload's various analyzers. These tests live in `test/`, and can be run via\n`lein test`---not to be confused with `lein run test`, which runs the Jepsen\ntest itself. Use these if you wind up changing the checkers somehow, to make\nsure they still detect the anomalies they ought to. These tests can also be\nhelpful in understanding why the various analyzer functions work the way they\ndo, and what they consider an anomaly.\n\n## Using the REPL\n\nSometimes you need to explore a test's history in more detail. `lein repl` will\nspawn a Clojure repl with all the test suite's code available. To start with,\nyou might want a few namespaces and functions available:\n\n```clj\njepsen.redpanda.core=\u003e (require '[jepsen.store :as s] '[jepsen.checker :refer [check]] '[jepsen.redpanda.workload.queue :as q] '[jepsen.redpanda.client :as c])\n```\n\nWe can load a test from disk using `jepsen.store/test`. It can take a path to a\nparticular test's directory in `store/`. As a shortcut, clicking the title of a\ntest directory in the web interface will copy this path as a string, so you can\npaste it right into the REPL.\n\n```clj\njepsen.redpanda.core=\u003e (def t (s/test \"/home/aphyr/redpanda/store/2022-01-19.deb queue subscribe acks=all retries=1000 aor=earliest default-r=3 auto-topics=false idem=true pause/20220120T220302.000-0500\"))\n```\n\nWas this test valid?\n\n```clj\njepsen.redpanda.core=\u003e (:valid? (:results t))\nfalse\n```\n\nNo! Why not?\n\n```clj\njepsen.redpanda.core=\u003e (-\u003e\u003e t :results :workload pprint)\n... eight billion lines ...\n     :time 63860531157,\n     :process 131,\n     :index 7195}}]}}\n```\n\nWell, that's a lot to read. What keys are in this map?\n\n```clj\njepsen.redpanda.core=\u003e (-\u003e\u003e t :results :workload keys)\n(:valid? :unseen :poll-skip :info-txn-causes :worst-realtime-lag :int-send-skip :lost-write :nonmonotonic-poll :error-types :int-poll-skip :int-nonmonotonic-poll)\n```\n\nRight, let's look at the error types:\n\n```clj\njepsen.redpanda.core=\u003e (-\u003e\u003e t :results :workload :error-types)\n[:int-nonmonotonic-poll :int-poll-skip :int-send-skip :lost-write :nonmonotonic-poll :poll-skip :unseen]\n```\n\nAll *kinds* of cool stuff here. How many lost-write errors?\n\n```clj\njepsen.redpanda.core=\u003e (-\u003e\u003e t :results :workload :lost-write :count)\n4\n```\n\nWhat was the first?\n\n```clj\njepsen.redpanda.core=\u003e (-\u003e\u003e t :results :workload :lost-write :errs first pprint)\n{:key 54,\n :value 436,\n :index 401,\n :max-read-index 977,\n :writer\n {:type :ok,\n  :f :txn,\n  :value [[:send 54 [1114 436]] [:poll {}] [:poll {}]],\n  :time 800986095314,\n  :process 648,\n  :rebalance-log\n  [{:type :revoked, :keys [55 4]} {:type :revoked, :keys [53]}],\n  :index 91528},\n :max-read\n {:type :ok,\n  :f :txn,\n  :value\n  [[:poll ...]\n   [:poll ...]\n   [:send 56 [1725 661]]\n   [:send 55 [2245 800]]],\n  :time 828187300771,\n  :process 590,\n  :index 95996}}\n```\n\nOK, so *something* went wrong on key 54, around value 436. What operation wrote that? The result claims it has the writer here, but just to double-check, let's look at the history ourselves.\n\n```clj\njepsen.redpanda.core=\u003e (-\u003e\u003e t :history (q/writes-of-key-value 54 436) pprint)\n({:type :invoke,\n  :f :txn,\n  :value [[:send 54 436] [:poll] [:poll]],\n  :time 791552742537,\n  :process 648}\n {:type :ok,\n  :f :txn,\n  :value [[:send 54 [1114 436]] [:poll {}] [:poll {}]],\n  :time 800986095314,\n  :process 648,\n  :rebalance-log\n  [{:type :revoked, :keys [55 4]} {:type :revoked, :keys [53]}]})\n```\n\nHere's the invocation and the completion of the operation which wrote key 54.\nSure enough, it appears to have suceeded! Was it ever read?\n\n```clj\njepsen.redpanda.core=\u003e (-\u003e\u003e t :history (q/reads-of-key-value 54 436) pprint)\n()\n```\n\nNothing ever read it. What about *nearby* messages? We know this write\nostensibly went to offset 1114--let's look at everything that interacted with\nthe local neighborhood of, say, five offsets before and after 1114.\n\n```clj\njepsen.redpanda.core=\u003e (-\u003e\u003e t :history (q/around-key-offset 54 1114 5) pprint)\n({:type :ok,\n  :f :txn,\n  :value ([:send 54 [1117 427]]),\n  :time 791596972858,\n  :process 778}\n {:type :ok,\n  :f :txn,\n  :value ([:send 54 [1110 434]]),\n  :time 791613396199,\n  :process 584}\n {:type :ok,\n  :f :txn,\n  :value ([:send 54 [1114 436]]),\n  :time 800986095314,\n  :process 648,\n  :rebalance-log\n  [{:type :revoked, :keys [55 4]} {:type :revoked, :keys [53]}]}\n {:type :ok,\n  :f :txn,\n  :value ([:poll {54 ([1110 434] [1117 427])}]),\n  :time 828187300771,\n  :process 590}\n ... lots more polls ...\n {:f :poll,\n  :value ([:poll {54 ([1110 434] [1117 427])}]),\n  :poll-ms 1000,\n  :time 1039009647614,\n  :process 868,\n  :type :ok})\n```\n\nSo we successfully wrote 427 to offset 1117, 434 to offset 1110, and 436 to\noffset 1114. Yet somehow when we went to read anything in this neighborhood, we\nonly observed values 434 and 427---no 436! This very much looks to be a lost\nwrite! Note that around-key-offset has *trimmed* the polls and sends inside\neach operation in order to show us only those parts relevant to this particular\nkey and region of offsets. The real poll operations here have hundreds of\nmessages each, so this is much easier to read.\n\nYou'll find several more functions for slicing and dicing history operations in\n`jepsen.redpanda.workload.queue`.\n\n## FAQ\n\nIf you're running in containers Redpanda may fail to start, citing a need to\nincrease `/proc/sys/fs/aio-max-nr` on the host OS--individual containers can't\nalter it themselves. Try\n\n```\nsudo sh -c 'echo 10000000 \u003e /proc/sys/fs/aio-max-nr'\n```\n\n## License\n\nCopyright © 2021, 2022, 2024 Jepsen. LLC\n\nThis program and the accompanying materials are made available under the\nterms of the Eclipse Public License 2.0 which is available at\nhttp://www.eclipse.org/legal/epl-2.0.\n\nThis Source Code may also be made available under the following Secondary\nLicenses when the conditions for such availability set forth in the Eclipse\nPublic License, v. 2.0 are satisfied: GNU General Public License as published by\nthe Free Software Foundation, either version 2 of the License, or (at your\noption) any later version, with the GNU Classpath Exception which is available\nat https://www.gnu.org/software/classpath/license.html.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjepsen-io%2Fredpanda","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjepsen-io%2Fredpanda","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjepsen-io%2Fredpanda/lists"}