{"id":18244025,"url":"https://github.com/jsommers/someta","last_synced_at":"2025-10-15T08:10:08.991Z","repository":{"id":57592197,"uuid":"131348638","full_name":"jsommers/someta","owner":"jsommers","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-16T22:54:03.000Z","size":192,"stargazers_count":5,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-14T13:43:51.750Z","etag":null,"topics":["measurement","metadata","network"],"latest_commit_sha":null,"homepage":"https://www.cs.colgate.edu/~jsommers/someta.html","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jsommers.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-04-27T21:59:16.000Z","updated_at":"2024-12-24T15:39:10.000Z","dependencies_parsed_at":"2024-02-20T18:46:35.328Z","dependency_job_id":"bc430209-1e21-48c7-84da-a7540a106b28","html_url":"https://github.com/jsommers/someta","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/jsommers/someta","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsommers%2Fsometa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsommers%2Fsometa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsommers%2Fsometa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsommers%2Fsometa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jsommers","download_url":"https://codeload.github.com/jsommers/someta/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsommers%2Fsometa/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271274190,"owners_count":24730961,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-20T02:00:09.606Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["measurement","metadata","network"],"created_at":"2024-11-05T09:04:51.406Z","updated_at":"2025-10-15T08:10:03.953Z","avatar_url":"https://github.com/jsommers.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"SoMeta\n======\n\nAutomatic collection of network measurement metadata.\n\nThis is a complete rewrite of SoMeta in go.  The earlier (Python) version of `SoMeta` can be found at https://github.com/jsommers/metameasurement.\n\nCurrent version is v1.4.1.\n\nInstalling\n----------\n\nThe easiest way to build and install is to do: `go install github.com/jsommers/someta@latest` (or replace `latest` with a specific version tag).  This command will download and compile all dependencies, and install the `someta` binary in your `$GOPATH/bin` directory.\n\nYou can, of course, also clone the source repository and use the `go` toolchain to build, etc.\n\n\nConfiguring\n-----------\n\nSoMeta can be configured using a YAML file.  This method of configuration is new in v1.3, and subsumes (nearly) all command-line parameters indicated below.  Configuring via a file allows (1) inclusion of URLs and details of associated data sets used in the measurement task(s), (2) inclusion of README-like details which aren't possible with the command-line, and (3) creation of a configuration template that can easily be reused.\n\nAn example configuration is provided in the repo, named `someta_config_example.yaml`.  It should, in general, be self-explanatory (see command-line arguments discussion below for details regarding config parameters for individual monitors).\n\nYou can also create a new yaml config using the `-I` option.  Any command-line flags to change defaults or to configure monitors will be parsed and included in the generated config, which will be written to stdout.  The `-c` (command) option is required; if any monitors are configured (through `-M` command-line arguments), those configurations will be included in the generated yaml, but if _no_ monitors are configured, default configurations for _all_ monitors will be included in the generated yaml. (New in v1.4.0.)\n\nTo start SoMeta with a configuration file, use the `-y` option:\n\n```\nsometa -y someta_config_example.yaml\n```\n\nAnother useful command-line parameter is `-Y`, which does some simple sanity checks on the configuration and exits prior to starting any monitoring.  This option can be used with or without a yaml configuration.\n\nNote that non-monitor command-line parameters are _overridden_ by config file options.  Note also that any monitor configurations provided on the command-line are _added_ to monitors configured via the yaml file.  It is recommended for your own sanity to just use the configuration file when and where possible and not to mix the two configuration styles.\n\n\nRunning\n-------\n\nThere are several possible command-line options.  See below for a listing of all parameters (i.e., the output of `someta -a`.  Some additional detail is below, specifically regarding monitors and options.\n\n\nUsage of ./someta:\n\n      -C int\n        \tSet CPU affinity (default is not to set affinity) (default -1)\n      -F duration\n         \tTime period after which in-memory metadata will be flushed to file (default 10m0s)\n      -M value\n        \tSelect monitors to include. Default=None. Valid monitors=cpu,io,mem,netstat,rtt\n      -R duration\n        \tTime period after which metadata output will rollover to a new file (default 1h0m0s)\n      -Y\tCheck configuration but don't start metadata collection\n      -c string\n        \tCommand line for external measurement program\n      -d\tDebug output (metadata is written to stdout)\n      -f string\n        \tOutput file basename; current date/time is included as part of the filename (default \"metadata\")\n      -l\tSend logging messages to a file (by default, they go to stdout)\n      -m duration\n        \tTime interval on which to gather metadata from monitors (default 1s)\n      -q\tQuiet output\n      -u duration\n        \tTime interval on which to show periodic status while running (default 5s)\n      -v\tVerbose output\n      -w duration\n        \tWait time before starting external tool, and wait time after external tool stops, during which metadata are collected (default 1s)\n      -y string\n        \tName of YAML configuration file\n\nThe ``-c`` option indicates the \"external\" measurement tool to start.  By default, \nSoMeta starts ``sleep 5``, which causes SoMeta simply to collect 5 seconds-worth of\nmetadata, given what ever monitors have been configured.  You'll almost certainly\nneed to quote the command line for the external tool, and some escaping may be required\nif there are embedded quotes needed for the tool (see the example with scamper, below).\n\nThe ``-M`` option specifies a monitor to start.  Standard available sources include cpu, mem, io, netstat, rtt (see the ``monitors/`` directory).\n\nTo configure a monitor, parameters may be specified along with each monitor name, each separated by a colon (`:`) or a comma (`,`).  Each parameter may be a single string, or a ``key=value`` pair.  The order of parameters doesn't matter.\n\nNote that if you are using the rtt monitor with IPv6, you'll need to use comma separators because the colon key-value separator can't be distinguished from the colon separator within an IPv6 address.\n\nHere's an example with turning on all monitors (io, netstat, cpu, mem, rtt):\n\n    sudo ./someta -M=io,disk0 -M=netstat,en0 -M=cpu -M=me -M=rtt,type=hoplimited,dest=149.43.80.25,maxttl=3,interface=en0 -R 1m -F 20s -f fulltest -m 1s -w 2s -v -c \"sleep 150\"\n\nAgain, type `./someta -h` for a list of command line options and their defaults.\n\nValid parameters for each standard monitor are:\n\n   * ``-M=cpu:interval=X``: set the periodic sampling interval (default 1 sec)\n   * ``-M=io:interval=X``: set the periodic sampling interval (default 1 sec)\n   * ``-M=mem:interval=X``: set the periodic sampling interval (default 1 sec)\n   * ``-M=netstat:interval=X``: set the periodic sampling interval.\n\n     Note that the interval time value is parsed by go's `time.parseDuration`\n     (https://golang.org/pkg/time/#ParseDuration), so any value must also\n     include a unit, like `interval=1s` (1 second interval).\n\n     Additional string arguments to the netstat monitor\n     can specify interface names to monitor (all\n     interfaces are included if none are specified).\n     For example, to monitor en0's netstat counters\n     every 5 seconds:\n     \n     * ``-M=netstat:interval=5s:en0``\n\n   * ``-M=rtt:interface=IfaceName:rate=R:dest=D:type=ProbeType:maxttl=MaxTTL:proto=Protocol:allhops:constflow``\n     \n     Monitor RTT along a path to destination ``D`` out of interface ``IfaceName``\n     with probe rate ``R``.  Probe interval is gamma distributed.  The default\n     destination is 8.8.8.8 and default probe rate is 1/sec.\n\n     ``ProbeType`` can either be ``ping`` or ``hoplimited`` (default is hoplimited)\n\n     ``MaxTTL`` is maximum ttl for hop-limited probes (pointless for ping probes).  \n     Default is maxttl = 1.\n\n     ``Protocol`` is (icmp | tcp | udp) (for hop-limited probes).  Default is icmp.\n\n     ``allhops``: probe all hops up to maxttl (for hop-limited probes)\n\n     ``constflow``: manipulate packet contents to force first 4 bytes of transport header to be constant (to make probes follow a constant path).  This parameter only has an affect on icmp; data are appended to force the checksum to be a constant value.  Note: udp/tcp probes always have const first 4 bytes.\n\n   * ``-M=ss``\n\n     Monitor socket statistics using the `ss` tool (linux only).  Thanks to Ricky Mok (CAIDA) for contributing this module.  \n\n\nHere are some examples:\n\n    # Monitor only CPU performance while emitting 100 ICMP echo request (ping) probes to\n    # www.google.com.\n    $ sudo ./someta -M=cpu -c \"ping -c 100 www.google.com\" \n\n    # Monitor CPU performance and netstat counters (for all interfaces) for traceroute\n    $ sudo ./someta -M=cpu -M=netstat -c \"traceroute www.google.com\" \n\n    # Monitor CPU, IO and Netstat counters for ping\n    # Set the metadata output file to start with \"ping_google\"\n    $ sudo ./someta -M=io -M=netstat -c \"ping www.google.com\" -f ping_google\n\n    # Monitor everything, including RTT for the first 3 hops of the network path toward\n    # 8.8.8.8.  As the external tool, use scamper to emit ICMP echo requests, dumping\n    # its output to a warts file.\n    $ sudo ./someta -M=cpu -M=mem -M=io -M=netstat:eth0 -M=rtt:interface=eth0:type=hoplimited:maxttl=3:dest=8.8.8.8 -f ping_metadata -l -c \"scamper -c \\\"ping -P icmp-echo -c 60 -s 64\\\" -o ping.warts -O warts -i 8.8.8.8\"\n\n    # An example with using the RTT monitor w/IPv6 (with the dummy command `sleep`).\n    # Note that in my example below I used an IPv6 (6-in-4) tunnel interface.\n    $ sudo ./someta -c \"sleep 5\" -M=rtt,dest=\"2607:f8b0:4006:805::200e\",type=hoplimited,interface=he-ipv6,maxttl=6  -v\n\n\n\nReconfiguring\n-------------\n\nSending SIGHUP to SoMeta will cause it to re-read its YAML configuration file.  This feature is in progress.\n\n\nAnalyzing metadata\n------------------\n\nThe ``analyzemeta.py`` script performs some simple analysis on SoMeta metadata, printing results to the console.  \n\n\nReading into a Pandas DataFrame \n-------------------------------\n\nFor more complex data analyses (or, if you prefer, metadata analyses), there is a Python module `read_someta.py` that provides a function `read_someta` for reading data in a SoMeta `.json` file into a dictionary of Pandas DataFrame objects.  There will be a different DataFrame object associated with each monitor.\n\nFor example:\n\n```\n\u003e\u003e\u003e from read_someta import read_someta\n\u003e\u003e\u003e d = read_someta('fulltest_2018-05-03T18:07:11-04:00.json')\n\u003e\u003e\u003e d.keys()\ndict_keys(['someta', 'cpu', 'mem', 'rtt', 'io', 'netstat'])\n\u003e\u003e\u003e d['cpu']\n                                     cpu0_idle  cpu1_idle  cpu2_idle  cpu3_idle\ntimestamp\n2018-05-03 18:07:12.978601317-04:00  62.037037  87.735849  68.224299  89.719626\n2018-05-03 18:07:13.979181597-04:00  70.000000  93.069307  71.000000  96.000000\n2018-05-03 18:07:14.980990941-04:00  82.828283  97.979798  86.000000  98.000000\n2018-05-03 18:07:15.980368940-04:00  74.000000  96.039604  79.000000  96.000000\n2018-05-03 18:07:16.981288271-04:00  69.306931  89.000000  75.000000  91.089109\n...                                        ...        ...        ...        ...\n2018-05-03 18:08:08.981608769-04:00  80.808081  94.000000  83.838384  90.000000\n2018-05-03 18:08:09.983457489-04:00  83.000000  94.000000  86.274510  89.000000\n2018-05-03 18:08:10.981178466-04:00  87.000000  97.000000  93.000000  98.000000\n2018-05-03 18:08:11.983964314-04:00  70.297030  92.079208  72.000000  91.000000\n2018-05-03 18:08:12.981282530-04:00  90.909091  98.000000  95.959596  99.000000\n\n[61 rows x 4 columns]\n\u003e\u003e\u003e\n```\n\n\nPlotting metadata\n-----------------\n\nNB: plotting tools need some updating still from the earlier Python versions.  \n\nThe ``plotmeta.py`` tool is designed to help plot various metrics collected through SoMeta *monitors*.  To see what metrics may be plotted, you can run the following::\n\n    $ python3 plotmeta.py -l meta.json\n\nwhere ``meta.json`` is a SoMeta metadata file.  The output of ``plotmeta.py`` with the ``-l`` option shows various *items* that can be plotted.  Each item is organized into *groups*.  You can either plot any number of individual items (``-i`` option), or plot each metric for an entire group (``-g`` option).  If you want everything, use the ``-a`` option.  In addition, ``-t`` option can be used to change the type of output plot. Use *ecdf* for empirical CDF or *timeseries* for simple scatter plot with timeline (which is default output of the plot tool). See ``plotmeta.py -h`` for all options.\n\nHere are some examples::\n\n    $ python3 plotmeta.py -t ecdf -i cpu:idle -i io:disk0_write_time meta.json\n    $ python3 plotmeta.py -t timeseries -g cpu meta.json\n    $ python3 plotmeta.py -a meta.json\n\n\n\nChanges\n-------\n\nChanges from the earlier Python version of SoMeta:\n\n * Because of Go's command-argument handling, flags to someta cannot be written like `-Mcpu`, but must rather be written as `-M=cpu` or `-M cpu`.\n * CPU affinity is not yet implemented\n * Metadata structure is changed to permit a less tightly-coupled architecture between the someta main and monitors\n   * The plotting tool hasn't been updated yet to handle these changes, though\n     the basic analysis tool has been updated.\n * There's even more rich data collected about the system when someta starts up\n\n\nv1.3\n\n * Addition of yaml configuration method\n * Some minor other code cleanup\n * Documentation update\n * Addition of `commandlinetool` monitor, which subsumes the contributed `ss` monitor\n\nv1.4\n\n * Addition of -I flag to create a new config and dump to stdout\n\n\nCredits\n-------\n\nI gratefully acknowledge support from the National Science Foundation.  The materials here are based upon work supported by the NSF under grant 1814537 (\"NeTS: Small: RUI: Automating Active Measurement Metadata Collection and Analysis\").\n\nAny opinions, findings, and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the National Science Foundation.\n\n\nLicense\n-------\n\nCopyright 2018-21  SoMeta authors.  All rights reserved.\n\nThe SoMeta software is distributed under terms of the GNU General Public License, version 3.  See below for the standard GNU GPL v3 copying text.\n\n    This program is free software: you can redistribute it and/or modify\n    it under the terms of the GNU General Public License as published by\n    the Free Software Foundation, either version 3 of the License, or\n    (at your option) any later version.\n\n    This program is distributed in the hope that it will be useful,\n    but WITHOUT ANY WARRANTY; without even the implied warranty of\n    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n    GNU General Public License for more details.\n\n    You should have received a copy of the GNU General Public License\n    along with this program.  If not, see \u003chttp://www.gnu.org/licenses/\u003e.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjsommers%2Fsometa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjsommers%2Fsometa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjsommers%2Fsometa/lists"}