{"id":21340063,"url":"https://github.com/imandra-ai/cme-mdp","last_synced_at":"2025-10-09T11:11:01.246Z","repository":{"id":97326444,"uuid":"75570580","full_name":"imandra-ai/cme-mdp","owner":"imandra-ai","description":"Imandra Modelling Language CME MDP Model","archived":false,"fork":false,"pushed_at":"2020-05-12T11:41:09.000Z","size":2280,"stargazers_count":13,"open_issues_count":0,"forks_count":3,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-07-12T14:46:53.894Z","etag":null,"topics":["automated-reasoning","cme","formal-methods","formal-verification","imandra","imandra-markets","market-data","market-data-handler","mdp","ocaml","protocol"],"latest_commit_sha":null,"homepage":"https://www.imandra.ai","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/imandra-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-12-04T23:08:00.000Z","updated_at":"2024-01-13T23:56:05.000Z","dependencies_parsed_at":"2023-03-13T16:15:31.338Z","dependency_job_id":null,"html_url":"https://github.com/imandra-ai/cme-mdp","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/imandra-ai/cme-mdp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imandra-ai%2Fcme-mdp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imandra-ai%2Fcme-mdp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imandra-ai%2Fcme-mdp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imandra-ai%2Fcme-mdp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/imandra-ai","download_url":"https://codeload.github.com/imandra-ai/cme-mdp/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imandra-ai%2Fcme-mdp/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279001290,"owners_count":26083058,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automated-reasoning","cme","formal-methods","formal-verification","imandra","imandra-markets","market-data","market-data-handler","mdp","ocaml","protocol"],"created_at":"2024-11-22T00:48:56.514Z","updated_at":"2025-10-09T11:11:01.241Z","avatar_url":"https://github.com/imandra-ai.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Aesthetic Integration](https://storage.googleapis.com/imandra-assets/images/docs/iml_cme_mdp_model.svg)\n---\n## Overview\n\nThe CME Group Market Data Platform (MDP) disseminates market data for CME Group\nmarkets.  The client state is updated in real time with an ordered stream of\nIncremental Refresh messages.  These messages are encoded using Simple Binary\nEncoding (SBE), gathered in packets, and are sent over two redundant UDP\nconnections: Incremental Refresh A and Incremental Refresh B. This redundancy\nis needed to reduce the chance of packet loss in the UDP.\n\nFurthermore, there is another pair of UDP feeds: Market Recovery A and Market Recovery B.\nThe recovery feeds are used to disseminate market data snapshots of the active\nbooks. These snapshot messages can be used to recover from a possible loss of\nIncremental Refresh packets from both of the incremental channels.\n\n\nGenerating test cases JSON files using `imandra-analyser`\n```bash\nimandra-analyser-cli strategy.yaml | python extract_packets.py generatedJsons\n\n```\n\nStarting the packets and transitions visualizer on http://localhost:8000\n\n```\ncd visualizer/\npython -m SimpleHTTPServer\n```\n\n## Imandra model of the CME book-builder  \nWe model the CME book-builder as an infinite state machine. The `feed_state`\ndatatype contains all of the information about CME book-builder state:\n\n- `feed_sec_type` : the ID of the security that the exchange is working on.\n- `books` : current state of books in the exchange\n- `channels` : state of the communication channels. Contains a sequence of\n  incoming unprocessed packets in the chronological order, a sequence of\n  already processed packets, latest received snapshot, an ordered set of cached\n  packets for recovery and a pair of `cycle_history` structures for channels A\n  and B.\n- `feed_status` : the current status of the feed can be either `Normal` or\n  `InRecovery`.\n- `internal_changes` : stores all the internal changes in the algorithm - how\n  books are updated with every message, current state of the cache and cycle\n  histories, whether the status is `Normal` or `InRecovery`, e.t.c.\n- `cur_time`: the timestamp of last processed packet\n\nThe model takes the next packet from the list of unprocessed packets and\niterates over the messages in it. \n\nIn the `Normal` state, the message sequence number is checked and the message\nis either processed normally or, if the message is off-sequence, added to cache\nwith the feed transitioning to the `InRecovery` state. \n\nWhile `InRecovery`, incremental update messages are added to cache, and\nsnapshot messages are saved as a `last_snapshot`. After each message a recovery\nis attempted:  if all cached messages are in chronological order since last\nsnapshot -- the feed transitions back into the `Normal` state:\n\n![CME Flowchart](doc/images/CME_model_flowchart_1.svg)\n\nThe flowchart diagram above gives an overview of the \n\n- **`simulate`** : a toplevel function that recursively\n  applies the `one_step` function until `feed_state` is not changed by it.\n- **`one_step`** : performs one step of the simulation\n  of the `feed_state`. The function extracts the next message from the stream\n  of incoming packets and forwards it either to `process_msg_normal` and\n  `process_msg_recovery`, depending on the current `feed_status`. If we are\n  currently `InRecovery`, then recovery is attempted (via `attempt_recovery`)\n  after the message is processed.\n- **`process_msg_normal`** :  processes the next\n  message, when our feed is in the `Normal` state. All snapshot messages are\n  ignored. If a refresh message `is_msg_relevant` and has the sequence number\n  that is exactly (`last_processed + 1`), then it is applied to the books in the\n  exchange. If there is a gap in the sequence number, then the `feed_status` is\n  changed to `InRecovery`.\n- **`is_msg_relevant`** : used by the previous\n  function, it checks whether the next incremental refresh message is\n  \"relevant\": its security ID equals to this feed's security, if its sequirity\n  sequence number is greater that the last processed one and if the order level\n  number is smaller than the book size of the feed. \n- **`process_md_update_action`** : applies the\n  current incremental refresh action (could be creation, deletion or update of\n  some order) to the books in the exchange.\n- **`process_msg_recovery`** : processes the\n  next message, when our feed is `InRecovery`. The incremental refresh messages\n  are placed in the cache (cache is kept sorted). The snapshots for \"our\"\n  security are stored, and the snapshot for the \"reference\" security are used\n  to detect whether our security is `Liquid` or `Illiquid`.\n- **`attemp_recovery`** : there are three possibilities\n  for the status to change back from `InRecovery`\n\n\n## Simple Binary Encoding  \nThe Simple Binary Encoding (SBE) is a binary encoding format for\nserialization/deserialization of a sequence of hierachically structured\nmessages in a stream of raw binary data.  \n\nIn the SBE framework, one first defines an *XML schema* -- an XML description\nthe binary layout of the messages.  Typically, a special codegenerating tool\nthen reads this XML file, and produces a code (usually `Java` or `C++`) that\ncontains the message type declarations and read/write routines for them. The\ngenerated code can then be linked with the business logic software.  \n\nIn this chapter we'll describe such a code-generation tool for the OCaml\nprogramming language.\n \n## SBE types codegeneration \n\nAt the root of it, SBE relies on a small number of common types:\nASCII-characters, signed or unsigned integers of various sizes and IEE754\nfloating-point numbers. \n    \nIn the XML schema any such 'primitive' type might be augmented by one of the\nfollowing modifiers:\n\n- It can have a `nullValue`, which denotes the null or none value for the\n      field of that type. \n- It can have an associated `length` value, meaning that the given field is\n      a sequence of values of that type. \n- It can have a `constant`  \"presence\" -- in that case the corresponding\n      field is never read or written to the binary stream.  Instead, the field\n      is always equal to a constant value, provided in the XML file.\n\nThe following conversions between the SBE simple types and their modifiers is\nused.\n\n| SBE Type    | OCaml type  |\n| ----------- | ----------- |\n| int8        | int         |\n| int16       | int         |\n| int32       | Int32.t     |\n| int64       | Int63.t     |\n| char        | char        |\n\n\n| SBE type modifier | OCaml parametrized type |\n| ----------------- | ----------------------- |\n| length            | 'a list                 |\n| nullValue         | 'a option               |\n\nThree kinds of \"complex\" types can then be constructed based on the primitive\ntypes described above:\n\n- The `composite` type is a sequence (a record) of fields of various types.  \n- The `enum` represents a number of mutually-exclusive cases. Each case\n      encoded with a constant `case-id`, provided in the XML schema.\n- The `set` is a collection of boolean fields, packed into a single bit\n      field.  \n\nFor `enum` and `set` types, an `encodingType` name must be provided.\n\nThe composite types are represented as OCaml record-types with\neach record entry having the corresponding primitive type. \n\n```xml\n\u003ccomposite name=\"FLOAT\"\u003e\n    \u003ctype name=\"mantissa\" primitiveType=\"int64\"/\u003e\n    \u003ctype name=\"exponent\" primitiveType=\"int\"/\u003e\n\u003c/composite\u003e\n```\n```ocaml\ntype t_FLOAT = {\n     f_FLOAT_mantissa : int64;\n     f_FLOAT_exponent : int\n}\n```\n\nThe enum types are represented as OCaml variant-types (2a) with each variant\ncase being a constant.\n\n```xml\n\u003cenum name=\"LegSide\" encodingType=\"uInt8\"\u003e\n    \u003cvalidValue name=\"BuySide\" \u003e1\u003c/validValue\u003e\n    \u003cvalidValue name=\"SellSide\"\u003e2\u003c/validValue\u003e\n\u003c/enum\u003e\n```\n```ocaml\ntype t_LegSide =\n     | V_LegSide_BuySide\n     | V_LegSide_SellSide\n ```\n\nIf the `enum` type has a nullable `encodingType`, then one extra case is added to the variant.  \n\n```xml\n\u003cenum name=\"AggressorSide\" encodingType=\"uInt8NULL\"\u003e\n    \u003cvalidValue name=\"NoAggressor\"\u003e0\u003c/validValue\u003e\n    \u003cvalidValue name=\"Buy\"\u003e1\u003c/validValue\u003e\n    \u003cvalidValue name=\"Sell\"\u003e2\u003c/validValue\u003e\n\u003c/enum\u003e\n```\n\n```ocaml\ntype t_AggressorSide =\n    | V_AggressorSide_NoAggressor\n    | V_AggressorSide_Buy\n    | V_AggressorSide_Sell\n    | V_AggressorSide_Null\n```\n\nFinally, the set types are treated as records, but with all entries\nbeing of the `boolean` type.\n\n```xml\n\u003cset name=\"SettlPriceType\" encodingType=\"uInt8\"\u003e\n    \u003cchoice name=\"Final\"\u003e0\u003c/choice\u003e\n    \u003cchoice name=\"Actual\"\u003e1\u003c/choice\u003e\n\u003c/set\u003e\n```\n\n```ocaml\ntype t_SettlPriceType = {\n    r_SettlPriceType_Final : bool;\n    r_SettlPriceType_Actual : bool;\n}\n```\nIn the XML schema, the declaration of all the necessary simple and complex\ntypes is followed by the declaration of various messages. Each message contains\na block of fields that are always present in the message, followed by a number\nof variable-sized groups. Each group is stored as a sequence of repeated\nblocks, with each block containing the same fixed number of fields.\n\n## The `cme_codegen` tool\n\nThe OCaml codegenerator takes as an input the XML schema file (set with `-i`\nflag) and writes three files into a specified directory (set with `-d` flag)\n\n```bash\n$ _build/default/src-codegenerator/cme_codegen.bc -i templates.xml -d outputdir\n$ ls outputdir\nmessage_types.ml  readers.ml  writers.ml\n```\n\nThe `message_types.ml` file contains all the OCaml type declarations. At the\nvery bottom of the file, the `message` type is declared -- it encompasses all\nthe messages in a single variant type.\n\nThe `readers.ml` and `writers.ml` files contain the reading and writing\nroutines for various types, for individual messages and for the top-level\n`Message_types.message`. \n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimandra-ai%2Fcme-mdp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fimandra-ai%2Fcme-mdp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimandra-ai%2Fcme-mdp/lists"}