{"id":29376497,"url":"https://github.com/twitter/caladrius","last_synced_at":"2025-07-09T22:43:15.081Z","repository":{"id":33837340,"uuid":"134899982","full_name":"twitter/caladrius","owner":"twitter","description":"Performance modelling system for Distributed Stream Processing Systems (DSPS) such as Apache Heron and Apache Storm","archived":false,"fork":false,"pushed_at":"2023-04-10T11:46:19.000Z","size":1457,"stargazers_count":22,"open_issues_count":6,"forks_count":18,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-05-09T19:35:32.155Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/twitter.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-05-25T20:05:55.000Z","updated_at":"2023-10-12T19:57:45.000Z","dependencies_parsed_at":"2022-08-29T22:51:42.457Z","dependency_job_id":null,"html_url":"https://github.com/twitter/caladrius","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/twitter/caladrius","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twitter%2Fcaladrius","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twitter%2Fcaladrius/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twitter%2Fcaladrius/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twitter%2Fcaladrius/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/twitter","download_url":"https://codeload.github.com/twitter/caladrius/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twitter%2Fcaladrius/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264504616,"owners_count":23618831,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-09T22:43:14.384Z","updated_at":"2025-07-09T22:43:15.063Z","avatar_url":"https://github.com/twitter.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Caladrius\n\nPerformance modelling for Distributed Stream Processing Systems (DSPS)\nsuch as [Apache Heron](https://apache.github.io/incubator-heron/) and [Apache\nStorm](http://storm.apache.org/).\n\nFull details can be found on the [documentation\nsite](http://caladrius.readthedocs.io/).\n\n**NOTE**: Caladrius is a prototype project, which is the result of a 3 month\ninternship with Twitter's Real Time Compute Team. It should be considered alpha\nlevel software. All contributions are welcome, please see the contributing page\non the documentation website for more details.\n\n## Setup\n\n### Python\n\nCaladrius requires Python 3.6, additional Python dependencies are listed in\nthe Pipfile. Dependencies can be installed using\n[pipenv](https://docs.pipenv.org/) by running the following command in the\ncaladrius root directory:\n\n    $ pipenv install \n\nAdd the `--dev` flag to the above command to install development dependencies.\n\nCaladrius should also be added to your `PYTHONPATH`. The best way to do this is\nby adding the folder above the Caladrius repo to the `PYTHONPATH` environment\nvariable using a command like the one below:\n\n    $ export PYTHONPATH=$PYTHONPATH:\u003cpath/to/folder/above/caladrius\u003e\n\nThis line should be added to your `.profile` (or similar) start up script to\npreserve this across reboots.\n\n####Troubleshooting\nErrors such as the following entail that the PYTHONPATH is not set correctly. \n\n    File \"app.py\", line 15, in \u003cmodule\u003e\n        from caladrius import logs\n        \nAnother way to ensure that `pipenv` is able to read the environment variables\nis to create a `.env` file in the project directory and add the `PYTHONPATH` there.        \n\n### Graph Database\n\nCaladrius requires a [Gremlin\nServer](http://tinkerpop.apache.org/docs/current/reference/#gremlin-server)\ninstance running [TinkerPop](http://tinkerpop.apache.org/) 3.3.2 or higher. \n\nThe reference gremlin sever can be downloaded from \n[here](https://www.apache.org/dyn/closer.lua/tinkerpop/3.3.3/apache-tinkerpop-gremlin-server-3.3.3-bin.zip).\n\nThe Gremlin server should have the [Gremlin\nPython](http://tinkerpop.apache.org/docs/current/reference/#gremlin-python)\nplugin installed:\n\n    $ gremlin-server.sh install org.apache.tinkerpop gremlin-python 3.3.3\n\nStart the server with the gremlin python config (included in the standard\nserver distribution):\n\n    $ bin/gremlin-server.sh start conf/gremlin-server-modern-py.yaml\n\n*Please note:* The default settings for the Gremlin Server result in an\nin-memory TinkerPop Server instance. If graphs need to be persisted to disk\nthen these settings can be altered in the appropriate configuration file in the\n`conf` directory of the Gremlin Server distribution.\n    \n## Running Caladrius\n\n### Configuration\n\nAll configuration is done via the `yaml` file provided to the `app.py` script\n(see section below). This file defines the models run by the various API\nendpoints and any connection details, modelling variables or other\nconfigurations they may require.\n\nAn example configuration file with sensible defaults is provided in\n`config/main.yaml.example`. You should copy this and edit it with your specific\nconfigurations.\n\n### Starting the API Server\n\nThe Caladrius API server can be started by running the `app.py` script in the\nroot directory. This can be run in the appropriate virtual environment using\npipenv (make sure your `python` command points to Python 3):\n\n    $ pipenv run python app.py --config /path/to/config/file\n\nAdditional command line arguments are available via:\n\n    $ pipenv run python app.py --help\n\n## Documentation\n\nDocumentation for stable releases is hosted on\n[ReadTheDocs](http://caladrius.readthedocs.io/).\n\nIf you want to build the latest documentation then this can be done using\n[Sphinx](http://www.sphinx-doc.org/en/master/index.html). Assuming you have\ninstalled the development dependencies above, the docs can be built using the\nfollowing commands in the repository root:\n\n    $ pipenv run sphinx-apidoc -f -o docs/source . tests/*\n    $ cd docs\n    $ pipenv run make html\n\nThis will place the constructed html documentation in the `docs/build`\ndirectory.\n\n## Security\n\nIf you spot any security or other sensitive issues with the software please\nreport them via the [Twitter HackerOne](https://hackerone.com/twitter) bug\nbounty program.\n\n## Using the API\nThe software provides multiple endpoints for a user to find out how different\npacking plans will perform for a single topology. Here, we provide examples of how to \ncall the APIs from the command-line.\n\n### Heron Current API\nIn this example, the WindowedWordCountTopoology has three components (spouts -\u003e bolt -\u003e bolt).\nEach operator in the job has one running task/instance only.\n\n    curl -H 'Content-Type: application/json' -d '{ \"1\" : {\"default\": 101.4}, \"2\": {\"default\": 104.3}, \"3\" : {\"default\" : 101.5}  }'  \n    -X POST \"\u003cCaladrius URL\u003e:5000/model/topology/heron/current/WindowedWordCountTopology?cluster=\u003ccluster-name\u003e\u0026environ=test\u0026model=queueing_theory\u0026source_hours=2\"\n    \nCaveat: Caladrius currently does not support calculations for topologies that have only two levels. This is because\na topology with two levels consists of spouts (with only outgoing streams) and sink bolts (with possibly only incoming\nstreams). Some of Caladrius' calculations such as measuring the input to output tuple ratios cannot be applied \nto such operators.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwitter%2Fcaladrius","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftwitter%2Fcaladrius","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwitter%2Fcaladrius/lists"}