{"id":19900108,"url":"https://github.com/m-lab/signal-searcher","last_synced_at":"2025-09-10T21:41:22.976Z","repository":{"id":57527392,"uuid":"63167882","full_name":"m-lab/signal-searcher","owner":"m-lab","description":"Find interesting and important patterns in M-Lab data","archived":false,"fork":false,"pushed_at":"2023-08-22T23:11:32.000Z","size":279,"stargazers_count":5,"open_issues_count":28,"forks_count":3,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-10-29T14:37:50.374Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://measurementlab.net","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null}},"created_at":"2016-07-12T15:06:38.000Z","updated_at":"2020-04-15T16:57:48.000Z","dependencies_parsed_at":"2024-02-23T05:42:50.776Z","dependency_job_id":null,"html_url":"https://github.com/m-lab/signal-searcher","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fsignal-searcher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fsignal-searcher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fsignal-searcher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fsignal-searcher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m-lab","download_url":"https://codeload.github.com/m-lab/signal-searcher/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223997594,"owners_count":17238570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T20:11:12.580Z","updated_at":"2024-11-12T20:11:13.134Z","avatar_url":"https://github.com/m-lab.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/m-lab/signal-searcher.svg?branch=master)](https://travis-ci.org/m-lab/signal-searcher)\n[![Coverage Status](https://coveralls.io/repos/github/m-lab/signal-searcher/badge.svg?branch=master)](https://coveralls.io/github/m-lab/signal-searcher?branch=master)\n\n\n# Requirements\n\n**Signal Searcher attempts to find interesting and important signals in MLab's Internet\nmeasurement data using computers instead of people.**\n\n[Measurement Lab](mlab) is collecting lots and lots of measurement data from all over the Internet. The project is drowning in data, and too frequently has to work too hard to acquire any of data's derivatives: information, knowledge, and wisdom.  In the past, issues were found because there were operator rumors of bad behavior. Depending on intensive investigations by humans spurred by rumor-mill dross is not a sustainable solution. As a process it is too random to get started reliably, and for its execution it depends on a very scarce resource (researcher time).\n\nSignal Searcher (this repo) is an attempt to create an automated solution that will permit the display and discovery of issues relating to end-to-end Internet performance and aid in Internet performance transparency.  Once performance and measurement has been made transparent, network management can proceed from a firm footing based on open and auditable data. Signal searcher tries to help bootstrap this process by automatically mining the data.\n\nLet's break down Signal Searcher's mission for our data:\n\n\u003e Signal Searcher attempts to find interesting and important signals in MLab's\n\u003e Internet measurement data using computers instead of people.\n\n## Interesting\n\nA signal is interesting if:\n\n1. It has patterns that indicate overload\n2. It gets worse over time\n3. It changes differently than its cohort\n4. It is out of control\n5. It is surprising\n\n## Important\n\nA signal is important if:\n\n1. It corresponds to an attribute that directly affects a user's Internet\n   experience, e.g. latency, download throughput, and upload throughput.\n\nand either\n\n2. It has a large effect on part of the Internet OR\n3. It affects a large part of the Internet\n\n## Signal\n\nEach signal is a time-series of measured values. Signals may be (and almost\ncertainly are) aggregates of time-series data. Let us imagine each value from a\ntest as a single row in a table of measured values. Each value has lots of\nattributes for the test environment and for the measured values that were\nrecorded. In data mining there is the concept of a [data cube](datacube): a\nhypercube where each dimension corresponds to a piece of the key specifying a\ndata value. To create a signal, you must specify an operation for every\nkey dimension in the data cube.  That is, for every dimension you must\n\"transform\" (apply an operation to each data value that e.g. aligns timestamps,\ntransforms IPs to ASNs, or buckets data), \"roll up\" (combine all data along the\ngiven dimension using some operation like \"average\" or \"median\"), or \"drill\ndown\" (specify a particular value or range of values for that particular\ndimension). For metadata dimensions (dimensions that are neither part of the\nkey or the value under test), there is another operator: \"drop\", which means to\nignore that dimension entirely. Once every dimension has an operation\nspecified, the result will be a table of values. If the operations are\nspecified properly for our purposes, then the result will be a signal: a\none-dimensional sequence of values.\n\n## MLab's Internet Measurement Data\n\nOur data is network measurement data, collected at the M-Lab measurement points\nacross the Internet, and stored in [BigQuery](bigquery). The two main\ncategories of data are measurements of performance and measurements of\ntopology. Each test is associated with a litany of metadata (although we would\nlike more metadata in BigQuery) regarding the environment in which the test was\nconducted. We will need to be able to join topology and performance metrics if\nwe would like to perform any sort of tomography about where in the network any\nobserved problems might lie. Because each performance data test is followed by\na traceroute, the topology data can be considered metadata of the performance\ndata. However, the topology data is also interesting in its own right and may\nalso be studied independently of the performance data. \n\nFor our first version, we will restrict ourselves to studying just sources and\ndestinations of traffic, leaving more complex topology-sensitive analyses for\nthe future.  In the first version we will also restrict ourselves to just the\nthree main measurements of interest for end-to-end user experience on the\nInternet: Download speed, upload speed, and RTT.\n\nA sample query to get this out of MLab's BigQuery tables might be:\n```sql\nSELECT\n  web100_log_entry.connection_spec.remote_ip AS server_ip_v4,\n  web100_log_entry.connection_spec.remote_ip AS client_ip_v4,\n  8 * (web100_log_entry.snap.HCThruOctetsAcked /\n        (web100_log_entry.snap.SndLimTimeRwin +\n         web100_log_entry.snap.SndLimTimeCwnd +\n         web100_log_entry.snap.SndLimTimeSnd)) AS download_mbps,\n  (web100_log_entry.snap.SumRTT /\n     web100_log_entry.snap.CountRTT) AS avg_rtt,\n  web100_log_entry.snap.MinRTT AS min_rtt,\n  (web100_log_entry.snap.SegsRetrans /\n     web100_log_entry.snap.DataSegsOut) AS packet_retransmit_rate\nFROM\n  [plx.google:m_lab.ndt.all]\nWHERE\n  web100_log_entry.snap.SndLimTimeSnd IS NOT NULL\n  AND web100_log_entry.snap.SndLimTimeCwnd IS NOT NULL\n  AND web100_log_entry.snap.SndLimTimeRwin IS NOT NULL\n  AND project = 0\n  AND web100_log_entry.is_last_entry = True\n  AND connection_spec.data_direction = 1\n  AND web100_log_entry.snap.CongSignals \u003e 0\n  AND web100_log_entry.snap.HCThruOctetsAcked \u003e= 8192\n  AND (web100_log_entry.snap.State == 1\n    OR (web100_log_entry.snap.State \u003e= 5\n        AND web100_log_entry.snap.State \u003c= 11))\n  AND (web100_log_entry.snap.SndLimTimeRwin +\n       web100_log_entry.snap.SndLimTimeCwnd +\n       web100_log_entry.snap.SndLimTimeSnd) \u003e= 9000000\n  AND (web100_log_entry.snap.SndLimTimeRwin +\n       web100_log_entry.snap.SndLimTimeCwnd +\n       web100_log_entry.snap.SndLimTimeSnd) \u003c 3600000000\n  AND web100_log_entry.snap.CountRTT \u003e 10\n```\n\n## Using computers\n\nWe specify a signal by a series of transforms, roll-ups, and drill downs. So\nthe space we are searching for interesting signals is the set of all possible\ntransforms, roll-ups, and drill-downs applied to each dimension in our data\nset. The size of this search space grows exponentially with the number of\ndimensions a measurement has (\"the curse of dimensionality\") and\nmultiplicatively with the number of values in each dimension.  Even with\ncomputational help, this is too large a space to use brute force. Instead, we\nwill specify roll-ups and drill-downs of interest and then use the derived time\nseries data.\n\nTo cover the project's remit of general search, we will train an ML\nmodel on our data and then run any new data through the model to see if the new\nresults are mispredicted (and therefore \"surprising\", which can be thought of\nas \"potentially interesting\").\n\n# Design\n\nWe use this insight to build a system which goes through our database\nlooking for data streams of interest that violate at least one of\nthese principles.  Because of our focus on end-to-end user experience,\nwe restrict ourselves to searching for signals in metrics which affect\nthe end-user experience: Download speed, upload speed, and RTT.\n\nExpanding on the initial super-generic statement, we come up with the\nfollowing assertions about the properties a healthy and open Internet\nshould have:\n\n1. Performance for a given ISP or ISP pair should not vary diurnally.\n   Diurnal performance variation was discovered to be a leading indicator\n   of network congestion problems.\n\n2. Performance should not get worse over time. If we ever find\n   performance to be degrading, then that is an interesting signal that\n   we should investigate.\n\n3. For a given access ISP, performance should be the same to each\n   content ISP in a given metro. If, for access ISP X and content ISPs Y\n   and Z, the expected (X,Y) performance is statistically significantly\n   different from the (X,Z) performance, then we have discovered an\n   Internet bottleneck outside of the last mile in X!\n\n4. Performance measurements should be under statistical control.  If\n   the variation in expected performance suddenly spikes, then that is\n   an important signal worth investigating.\n\n5. If we have a trained model of past performance, then it should be\n   able to predict current performance with high accuracy. If the\n   predictions of our model are very wrong, then it is likely that we\n   are seeing something new happening on the Internet.\n\nIn all of the above, we use the word \"performance\" and we don't say\nwhether it should go up or down, because RTT should go down, and bandwidth\nshould go up, but either way we can still say that performance should\nnot get worse.\n\n# Signal searcher\n\nSignal searcher will operate in batch mode. The time period between batch\nruns will be determined by the runtime of the system - if the batch job\ntakes days to complete, then it should be run less frequently than if\nit only takes minutes.  As command line parameters, signal searcher will\ntake in the IP ranges of interest and the time period of interest.\n\nThe output of signal searcher will be a performance report describing\nany observed problems in that specified subset of the data. The report\nwill clearly have to go through multiple UX iterations, but it will\ninitially be designed to output graphs similar to those in [the M-Lab\ninterconnection report][interconnection].\n\nIn the explanations below, whenever we specify a time-series, what we mean is\nthe raw data, bucketed appropriately, with the median operator applied to each\nbucket.\n\n## 24-hour cycles\n\nOne of the strongest findings in [the report][interconnection] was that\ndiurnal performance was a strong indicator of network overload. Therefore,\nour first metric of interest will be to load the time-varying signal\nof a single ISP's performance and to analyze to see if it has a strong\n24-hour cycle.\n\nWe will do this by performing a Fourier transform of each time series of\nperformance and then squaring the absolute value of the result to get a power\ncurve.  We then look in this curve to see if it has a bump at the peak\ncorresponding to a 24 hour frequency.\n\n## Non-degrading performance\n\nPerformance should not get worse over time. If we discover an instance where\nperformance does seem to get worse over time, then we should investigate.\n\n## Equivalent performance within a metro area\n\nIf we discover that the same access ISP has significantly different performance\nto different content ISPs within a metro area, then that is strong evidence\nthat in at least some instances the user's experienced performance is being\naffected by a bottleneck that is not part of their last mile.\n\nWe can test equivalence using the Kolmogorov-Smirnov test which will estimate\nhow different two distributions might be. For the time period of interest, we\nquery to find the distribution of our data and then use the K-S test to\ndetermine how likely it is that the two distributions are the same.\n\n## Statistical control\n\nUse process control theory to evaluate whether the fluctuations of our\ntime series represent standard levels of randomness around the same\nstatistical mean or whether they represent the system transitioning to a new,\ndifferent, \"out of control\" state.\n\nThis depends on a currently-unpublished paper, and so is blocked until that\nwork is published.\n\n## Machine learning to find surprises\n\nThe state of the art machine learning system is Google's TensorFlow. So, we\nshould build a TensorFlow system, train and validate it on our data according\nto standard best practices, and then ensure that our data set of interest does\nnot hold any surprises for the trained system. In this context, surprises for\nour ML system would imply that some aspect of the data has changed in an\nimportant way.\n\n(N.B. TensorFlow is only available in C++ or Python flavors, so this means\nthat the project as a whole needs to be written in C++ or written in Python or\nto swim upstream. Python it is.)\n\n# Signal searcher's reports\n\nThe resulting reports from Signal Searcher should be emitted in priority order.\nAn item of interest can be of high priority because it affects a large portion\nof the Internet, or because it strongly effects a part of the Internet. An item\nwhich does both would then be of the highest priority.  The relationship\nbetween size and severity in determining priority can only be determined\nempirically, so we leave the exact ranking function unspecified until we have\nseveral unordered reports to provide us a basis from which to reason.\n\nThe format of the reports is unspecified, but should at a minimum contain\ngraphs of the signals we claim are interesting and an indication of why they\nare interesting.\n\n# Implementation\n\nWe will begin by using BigQuery resources wastefully. For every site, write a\nquery to get every NDT result to that site.  Then, run the resulting time\nseries through each of: The 24-hour tests, the non-degradation test, and the\nstatistical control test.\n\nOnce those three tests are implemented, we should look for equivalent\nperformance in a metro area by using the GeoIP of each test as well as the\norigin AS for each test.  Finally, we will train a TensorFlow model to predict\nperformance and then run each new measurement through that same model.\n\nThe suggested implementation order is:\n\n1. First phase: reproduce MLab Interconnection report analyses\n  - [ ] Querying to get a single time series. (half done)\n  - [x] 24-hour cycles\n  - [x] Initial output reports\n2. Second phase: reproduce the more-understandable (but less sensitive) analyses of performance degradation\n  - [x] Performance non-degradation\n  - [ ] Ensure [Statistical control][statisticalcontrol]\n  - [ ] Improved output reports\n3. Third phase: end-to-end verification of neutrality of interconnections within a metro\n  - [ ] Querying to get a performance distribution\n  - [ ] Equivalent performance within a metro\n4. Fourth phase: using machine learning to find surprises\n  - [ ] Querying to get everything\n  - [ ] TensorFlow prediction model\n  - [ ] Cleaned up output reports\n\nNote that once querying to get a time-series works, implementing the\ntime-series tests can proceed in parallel.  Also, output reports should be\nconstructed immediately once we have a single test working, and then should be\nrefactored once we have multiple tests working, and finally should be cleaned\nup and made into something we are proud of once all tests are working.\n\n# Future work\n\n- Apply the tools of granger causation in an attempt to discover what\n  patterns end up being leading indicators for these metrics.\n\n- Online signal searcher, which informs of Internet measurement changes in\n  real-time by evaluating each incoming measurement and deciding whether\n  or not to fire an alert.\n\n- See what other metrics in our data set may be informative. Web100\n  has a lot of variables that are largely unexamined; maybe one of them\n  holds the key to understanding Internet performance in the large.\n\n- Use paris-traceroute path information to build up more in-depth\n  analyses which attempt to localize performance problems in the network.\n  In particular, each measurement can be read as an attestation of the\n  minimum performance available to each link on the path. By combining\n  these measurements we should be able to discover what links in\n  poorly-performing paths do not have another measurement attesting to\n  their quality.\n\n\n[interconnection]: https://www.measurementlab.net/blog/2014_interconnection_report/ \"ISP Interconnection and its Impact on Consumer Internet Performance\"\n[mlab]: //www.measurementlab.net\n[datacube]: https://en.wikipedia.org/wiki/OLAP_cube\n[bigquery]: https://bigquery.cloud.google.com\n[statisticalcontrol]: https://www.measurementlab.net/publications/CUSUMAnomalyDetection.pdf\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fsignal-searcher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm-lab%2Fsignal-searcher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fsignal-searcher/lists"}