{"id":13585106,"url":"https://github.com/m-lab/murakami","last_synced_at":"2025-07-24T16:34:31.672Z","repository":{"id":30968132,"uuid":"207701726","full_name":"m-lab/murakami","owner":"m-lab","description":"Run automated internet measurement tests in a Docker container.","archived":false,"fork":false,"pushed_at":"2025-05-31T11:03:00.000Z","size":566,"stargazers_count":45,"open_issues_count":18,"forks_count":12,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-06-30T12:35:27.800Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":"docs/SUPPORTED-TEST-RUNNERS.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-09-11T02:04:59.000Z","updated_at":"2025-01-26T07:49:19.000Z","dependencies_parsed_at":"2024-04-15T05:46:08.113Z","dependency_job_id":"55d71df9-757a-4590-8176-680c565cd0ef","html_url":"https://github.com/m-lab/murakami","commit_stats":{"total_commits":284,"total_committers":10,"mean_commits":28.4,"dds":0.4859154929577465,"last_synced_commit":"23364d89eef50b6551c174d470f59735e8f354dd"},"previous_names":["throneless-tech/murakami"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/m-lab/murakami","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fmurakami","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fmurakami/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fmurakami/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fmurakami/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m-lab","download_url":"https://codeload.github.com/m-lab/murakami/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fmurakami/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264438491,"owners_count":23608337,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T15:04:44.774Z","updated_at":"2025-07-15T12:38:39.811Z","avatar_url":"https://github.com/m-lab.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Murakami\n\nMurakami is a tool for creating an automated Internet measurement service,\nrunning in a Docker container. A Murakami measurement container will\nautomatically run supported tests using a randomized schedule a configurable\nnumber of times a day, and can be configured to export each test result to a\nlocal storage device, to one or more remote servers via SCP, or to a Google\nCloud Storage bucket. Results are saved as individual files in JSON new line\nformat (`.jsonl`).\n\n## Contributing to Murakami\n\nM-Lab welcomes your contribution to Murakami development. We recommend forking\nthe repository and submitting pull requests to address\n[issues](https://github.com/m-lab/murakami/issues) we've identified. If there is\na feature you would like to contribute that isn't in our list of issues, please\ncreate an issue so that it can be discussed and integrated into our roadmap.\n\n## Supported Measurement Tests\n\n* [Network Diagnostic Tool](https://www.measurementlab.net/tests/ndt/) (NDT) -\n  NDT is diagnostic and single stream performance test, originally developed by\n  Internet2, and is now maintained by M-Lab. The supported NDT protocol versions\n  listed below measure a connection using [different TCP congestion control\n  algorithms, and have different port/firewall requirements](https://www.measurementlab.net/blog/evolution-of-ndt/#timeline).\n  * [ndt5-client-go](https://www.github.com/m-lab/ndt5-client-go) - The [ndt5\n    protocol](https://www.measurementlab.net/tests/ndt/ndt5/) test uses CUBIC\n    TCP congestion control, and is backward compatible with previous iterations\n    of the NDT server. ndt5 requires some non-standard ports to be open.\n  * [ndt7-client-go](https://www.github.com/m-lab/ndt7-client-go) - The [ndt7\n    protocol](https://www.measurementlab.net/tests/ndt/ndt7/) test uses BBR TCP\n    congestion control for the download measurement and for upload uses the\n    congestion control enabled in the client network. ndt7 uses standard ports 80/443.\n* [Neubot DASH](https://github.com/neubot/dash) - Dynamic Adaptive Streaming\n  over HTTP, hosted by M-Lab. [DASH](https://www.measurementlab.net/tests/neubot/) is designed to measure the quality of tested networks by emulating a video streaming player.\n* [Ookla's speedtest-cli](https://www.speedtest.net/apps/cli) - Ookla's official\n  command line test for testing internet bandwidth using the speedtest.net platform.\n  * Both multi-stream and single-stream tests are supported\n* [OONI Probe](https://ooni.org/install/cli) - The Open\n  [Observatory of Network Interference](https://ooni.org) (OONI) is a non-profit free software\n  project that aims to empower decentralized efforts in documenting Internet\n  censorship around the world. Collect data that can potentially serve as evidence of Internet censorship since it shows how, when, where, and by whom it is implemented.\n\nFor more information about each supported test see: [Supported Tests\nRunners](docs/SUPPORTED-TEST-RUNNERS.md).\n\nTo add new internet measurement tools to Murakami, see: [Adding New Measurement Tools to Murakami](docs/ADDING-NEW-TEST-RUNNERS.md)\n\n## Supported Operating Systems\n\nMurakami supports Linux operating systems like Ubuntu, Debian, etc. Windows is\nnot supported. Mac OS may work, but is yet untested.\n\n## Configuring Murakami\n\nA Murakami container can be configured flexibly depending on the deployment\nscenario. If you simply run a Murakami container using M-Lab's `latest` [image\non Dockerhub](https://hub.docker.com/r/measurementlab/murakami/), by default all\ntests are configured to run four times daily at randomized intervals, but with\nno data exporters enabled. To use Murakami and save test results, either using\nan individual device or a fleet of managed devices, you'll use one of the\noptions below to configure the container at runtime, or build your own local\ncontainer image.\n\nThe Murakami container can be configured/customized using:\n* in-line environment variables on the command line when running the Docker container\n* customizing the `configs/murakami.config.example` configuration file and\n  passing the variables it contains on the command line when running the\n  Docker container\n* customizing the `configs/murakami.toml.example` configuration file and including it in a\n  deployed Docker container\n* environment variables in a [Balena Cloud](https://www.balena.io/cloud) fleet\n  and/or per device\n\nSee [CONFIGURING-MURAKAMI](docs/CONFIGURING-MURAKAMI.md) for more information on\nall configuration variables, and in our documentation specific to each deployment scenario.\n\n## Deployment Scenarios\n\nMurakami supports two types of Docker container deployments on supported systems:\n\n* **[Standalone](docs/INSTALL-MURAKAMI-STANDALONE.md)**: a single Murakami device, configured on-device\n* **[Fleet Managed Murakami using Balena Cloud](docs/INSTALL-MURAKAMI-BALENA-CLOUD.md)**: one or more Murakami devices, configured and managed by the [Balena Cloud IoT platform](https://www.balena.io)\n\nIt is also possible to install Murakami directly on supported systems without\nDocker, however currently documenting direct system installation of Murakami is\nbeyond our project scope. Future testing and documentation is needed to test and\nconfirm supported systems and requirements.\n\n## Dockerhub Images and Tags\n\nMeasurement Lab publishes a pre-built Murakami Docker container image\non Dockerhub tagged `latest`. Please visit [our repo on\nDockerhub](hhttps://hub.docker.com/r/measurementlab/murakami) for more details.\n\n## Building Murakami Images\n\nIf you are interested in building your own Murakami Docker images, please see\nour [BUILD instructions](docs/BUILD.md).\n\n## Updating Your Murakami Images/Installations\n\nMurakami is periodically updated with new measurements and features, as well as\nupdates to existing measurement services. M-Lab recommends keeping your Murakami installations or\ncontainer images up to date with changes in the main branch of this repository,\nor with the `latest` image on Dockerhub. See the appropriate section on\n_Updating Murakami_ in the supported _Deployment Scenarios_ documents\nlinked above for more information.\n\n## Included Utility Scripts for Google Cloud and Balena Cloud Tasks\n\nUtility scripts that provide some automation to setup Google Cloud resources and\nprovisioning new devices for a Balena Cloud fleet are available in `utilities/`.\nThese are designed to support M-Lab demo fleets, so please inspect and customize\nthem to your liking.\n\n* `setup_bq.sh` - Intended to be run once to setup GCP resources for a new\n  measurement program or demo. The Google Cloud SDK must be installed and\n  authorized with an account that has IAM permissions to create GCS buckets\n  and BigQuery datasets, tables and views.\n* `setup_balena_device.sh` - This script can be used to provision new Murakami devices in\n  your Balena Cloud fleet. It requires the balena-cli to be installed and\n  authorized.\n* `bq_load_murakami.sh` - Intended to be scheduled daily using scheduled jobs or\n  cron, this script will load murakami test results files within a GCS bucket\n  from the previous day into BigQuery tables.\n\nFor more details, see [Murakami on Google Cloud Demo](docs/GCLOUD-DEMO.md).\n\n## Optional Prototype Data Visualization Service\n\nAn optional data visualization service,\n[Murakami-Viz](https://github.com/m-lab/murakami-viz/), can be used to receive\ntest results from Murakami measurement devices if desired. Please see\n[USING-MURAKAMI-VIZ](docs/USING_MURAKAMI_VIZ.md) for more information.\n\n### Included Convenience Scripts\n\nTwo convenience utilities are provided with Murakami:\n\n* [murakami_convert](docs/CONVERT-DATA.py) `scripts/convert.py`\n  * A script to convert `jsonl` test results to CSV.\n* [murakami_upload](docs/UPLOAD-UTILITY.py) `scripts/upload.py`\n  * A script to upload previously collected test results to a newly deployed\n    instance of Murakami-Viz.\n\n## What Does the Name Mean?\n\n**What do we talk about when we talk about running tests?** The M-Lab team talks\nabout this a lot, and the name \"Murakami\" pays homage to the renown author\nHaruki Murakami, who's book [_What I Talk About When I Talk About\nRunning_](http://www.harukimurakami.com/book/what-i-talk-about-when-i-talk-about-running-a-memoir)\nprovided us inspiration.\n\n## Acknowledgements\n\nM-Lab would like to thank the Institute for Museum and Library Services (IMLS),\nand Simmons University, whose partnership on IMLS Award\n[#LG-71-18-0110-18](https://www.imls.gov/grants/awarded/lg-71-18-0110-18)\nsupported the development of many current Murakami features, as well as\nMurakami-Viz.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fmurakami","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm-lab%2Fmurakami","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fmurakami/lists"}