{"id":13815020,"url":"https://github.com/reyjrar/es-utils","last_synced_at":"2025-04-04T17:05:38.189Z","repository":{"id":3989964,"uuid":"5086268","full_name":"reyjrar/es-utils","owner":"reyjrar","description":"ElasticSearch Utilities","archived":false,"fork":false,"pushed_at":"2025-01-11T06:06:41.000Z","size":1909,"stargazers_count":141,"open_issues_count":1,"forks_count":27,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-03-28T16:04:59.830Z","etag":null,"topics":["elasticsearch","perl"],"latest_commit_sha":null,"homepage":null,"language":"Perl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/reyjrar.png","metadata":{"files":{"readme":"README.mkdn","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2012-07-17T18:42:29.000Z","updated_at":"2024-07-26T19:43:38.000Z","dependencies_parsed_at":"2023-07-12T10:31:49.810Z","dependency_job_id":"92227b5c-2353-4df7-b5c8-8ef00d85cc15","html_url":"https://github.com/reyjrar/es-utils","commit_stats":{"total_commits":439,"total_committers":14,"mean_commits":"31.357142857142858","dds":0.04783599088838264,"last_synced_commit":"2479435ef001fbba32c55b3d207ae48562bd226c"},"previous_names":[],"tags_count":97,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reyjrar%2Fes-utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reyjrar%2Fes-utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reyjrar%2Fes-utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reyjrar%2Fes-utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/reyjrar","download_url":"https://codeload.github.com/reyjrar/es-utils/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247217174,"owners_count":20903008,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","perl"],"created_at":"2024-08-04T04:02:50.855Z","updated_at":"2025-04-04T17:05:38.169Z","avatar_url":"https://github.com/reyjrar.png","language":"Perl","funding_links":[],"categories":["Perl"],"sub_categories":[],"readme":"# NAME\n\nApp::ElasticSearch::Utilities - Utilities for Monitoring ElasticSearch\n\n# VERSION\n\nversion 8.8\n\n# SYNOPSIS\n\nThis library contains utilities for unified interfaces in the scripts.\n\nThis a set of utilities to make monitoring ElasticSearch clusters much simpler.\n\nIncluded are:\n\n**SEARCHING**:\n\n    scripts/es-search.pl - Utility to interact with LogStash style indices from the CLI\n\n**MONITORING**:\n\n    scripts/es-graphite-dynamic.pl - Perform index maintenance on daily indexes\n    scripts/es-status.pl - Command line utility for ES Metrics\n    scripts/es-storage-overview.pl - View how shards/data is aligned on your cluster\n    scripts/es-nodes.pl - View node information\n\n**MAINTENANCE**:\n\n    scripts/es-daily-index-maintenance.pl - Perform index maintenance on daily indexes\n    scripts/es-alias-manager.pl - Manage index aliases automatically\n    scripts/es-open.pl - Open any closed indices matching a index parameters\n\n**MANAGEMENT**:\n\n    scripts/es-apply-settings.pl - Apply settings to all indexes matching a pattern\n    scripts/es-cluster-settings.pl - Manage cluster settings\n    scripts/es-copy-index.pl - Copy an index from one cluster to another\n    scripts/es-storage-overview.pl - View how shards/data is aligned on your cluster\n\n**DEPRECATED**:\n\n    scripts/es-graphite-static.pl - Send ES Metrics to Graphite or Cacti\n\nThe App::ElasticSearch::Utilities module simply serves as a wrapper around the scripts for packaging and\ndistribution.\n\n# FUNCTIONS\n\n## es\\_utils\\_initialize()\n\nTakes an optional reference to an `@ARGV` like array. Performs environment and\nargument parsing.\n\n## es\\_globals($key)\n\nGrab the value of the global value from the es-utils.yaml files.\n\n## es\\_basic\\_auth($host)\n\nGet the user/password combination for this host.  This is called from LWP::UserAgent if\nit recieves a 401, so the auth condition must be satisfied.\n\nReturns the username and password as a list.\n\n## es\\_pass\\_exec(host, username)\n\nCalled from es\\_basic\\_auth to exec a program, capture the password\nand return it to the caller.  This allows the use of password vaults\nand keychains.\n\n## es\\_pattern\n\nReturns a hashref of the pattern filter used to get the indexes\n    {\n        string =\u003e '\\*',\n        re     =\u003e '.\\*',\n    }\n\n## es\\_connect\n\nWithout options, this connects to the server defined in the args.  If passed\nan array ref, it will use that as the connection definition.\n\n## es\\_master(\\[$handle\\])\n\nReturns true (1) if the handle is to the the cluster master, or false (0) otherwise.\n\n## es\\_request(\\[$handle\\],$command,{ method =\u003e 'GET', uri\\_param =\u003e { a =\u003e 1 } }, {})\n\nRetrieve URL from ElasticSearch, returns a hash reference\n\nFirst hash ref contains options, including:\n\n    uri_param           Query String Parameters\n    index               Index name\n    type                Index type\n    method              Default is GET\n\nIf the request is not successful, this function will throw a fatal exception.\nIf you'd like to proceed you need to catch that error.\n\n## es\\_nodes\n\nReturns the hash of index meta data.\n\n## es\\_indices\\_meta\n\nReturns the hash of index meta data.\n\n## es\\_indices\n\nReturns a list of active indexes matching the filter criteria specified on the command\nline.  Can handle indices named:\n\n    logstash-YYYY.MM.DD\n    dcid-logstash-YYYY.MM.DD\n    logstash-dcid-YYYY.MM.DD\n    logstash-YYYY.MM.DD-dcid\n\nMakes use of --datesep to determine where the date is.\n\nOptions include:\n\n- **state**\n\n    Default is 'open', can be used to find 'closed' indexes as well.\n\n- **check\\_state**\n\n    Default is 1, set to 0 to disable state checks.  The combination of the default\n    with this option and the default for **state** means only open indices are returned.\n\n- **check\\_dates**\n\n    Default is 1, set to 0 to disable checking index age.\n\n## es\\_index\\_strip\\_date( 'index-name' )\n\nReturns the index name with the date removed.\n\n## es\\_index\\_bases( 'index-name' )\n\nReturns an array of the possible index base names for this index\n\n## es\\_index\\_days\\_old( 'index-name' )\n\nReturn the number of days old this index is.\n\n## es\\_index\\_shards( 'index-name' )\n\nReturns the number of replicas for a given index.\n\n## es\\_index\\_valid( 'index-name' )\n\nChecks if the specified index is valid\n\n## es\\_index\\_fields('index-name')\n\nReturns a hash reference with the following data:\n\n    key_name:\n      type: field_data_type\n      # If the field is nested\n      nested_path: nested_path\n      nested_key: nested_key\n\n## es\\_close\\_index('index-name')\n\nCloses an index\n\n## es\\_open\\_index('index-name')\n\nOpen an index\n\n## es\\_delete\\_index('index-name')\n\nDeletes an index\n\n## es\\_optimize\\_index('index-name')\n\nOptimize an index to a single segment per shard\n\n## es\\_apply\\_index\\_settings('index-name', { settings })\n\nApply a HASH of settings to an index.\n\n## es\\_index\\_segments( 'index-name' )\n\nExposes GET /$index/\\_segments\n\nReturns the segment data from the index in hashref:\n\n## es\\_segment\\_stats($index)\n\nReturn the number of shards and segments in an index as a hashref\n\n## es\\_index\\_stats( 'index-name' )\n\nExposes GET /$index/\\_stats\n\nReturns a hashref\n\n## es\\_settings()\n\nExposes GET /\\_settings\n\nReturns a hashref\n\n## es\\_node\\_stats()\n\nExposes GET /\\_nodes/stats\n\nReturns a hashref\n\n## es\\_flatten\\_hash\n\nPerforms flattening that's compatible with Elasticsearch's flattening.\n\n## es\\_human\\_count\n\nTakes a number and returns the number as a string in docs, thousands, millions, or billions.\n\n    1_000     -\u003e \"1.00 thousand\",\n    1_000_000 -\u003e \"1.00 million\",\n\n## es\\_human\\_size\n\nTakes a number and returns the number as a string in bytes, Kb, Mb, Gb, or Tb using base 1024.\n\n    1024        -\u003e '1.00 Kb',\n    1048576     -\u003e '1.00 Mb',\n    1073741824  -\u003e '1.00 Gb',\n\n## def('key')\n\nExposes Definitions grabbed by options parsing\n\n## es\\_local\\_index\\_meta(key =\u003e 'base' || 'index')\n\nFetch meta-data from the local config file, i.e. `~/.es-utils.yaml`.\n\nFormat is:\n\n    ---\n    meta:\n      index_name:\n        key: value\n      index_basename:\n        key: value\n\nThe most specific version is searched first, followed by the index stripped of\nit's date, and then on through all the bases discovered with\n`es_index_bases()`.\n\nThis is used by the `es-search.pl` utility to do lookups of the **timestamp**\nfield it needs to sort documents, i.e.:\n\n    ---\n    meta:\n      logstash:\n        timestamp: '@timestamp'\n        host: es-cluster-01.int.example.com\n      bro:\n        timestamp: 'timestamp'\n\n# ARGS\n\nFrom App::ElasticSearch::Utilities:\n\n    --local         Use localhost as the elasticsearch host\n    --host          ElasticSearch host to connect to\n    --port          HTTP port for your cluster\n    --proto         Defaults to 'http', can also be 'https'\n    --http-username HTTP Basic Auth username\n    --password-exec Script to run to get the users password\n    --insecure      Don't verify TLS certificates\n    --cacert        Specify the TLS CA file\n    --capath        Specify the directory with TLS CAs\n    --cert          Specify the path to the client certificate\n    --key           Specify the path to the client private key file\n    --noop          Any operations other than GET are disabled, can be negated with --no-noop\n    --timeout       Timeout to ElasticSearch, default 10\n    --keep-proxy    Do not remove any proxy settings from %ENV\n    --index         Index to run commands against\n    --base          For daily indexes, reference only those starting with \"logstash\"\n                     (same as --pattern logstash-* or logstash-DATE)\n    --pattern       Use a pattern to operate on the indexes\n    --days          If using a pattern or base, how many days back to go, default: 1\n\nSee also the \"CONNECTION ARGUMENTS\" and \"INDEX SELECTION ARGUMENTS\" sections from App::ElasticSearch::Utilities.\n\n# ARGUMENT GLOBALS\n\nSome options may be specified in the **/etc/es-utils.yaml**, **$HOME/.es-utils.yaml**\nor **$HOME/.config/es-utils/config.yaml** file:\n\n    ---\n    base: logstash\n    days: 7\n    host: esproxy.example.com\n    port: 80\n    timeout: 10\n    proto: https\n    http-username: bob\n    password-exec: /home/bob/bin/get-es-passwd.sh\n\n# CONNECTION ARGUMENTS\n\nArguments for establishing a connection with the cluster.  Unless specified otherwise, these options\ncan all be set in the globals file.\n\n- **local**\n\n    Assume ElasticSearch is running locally, connect to localhost.\n\n- **host**\n\n    Use a different hostname or IP address to connect.\n\n- **port**\n\n    Defaults to 9200.\n\n- **proto**\n\n    Defaults to 'http', can also be 'https'.\n\n- **http-username**\n\n    If HTTP Basic Authentication is required, use this username.\n\n    See also the [\"HTTP Basic Authentication\"](#http-basic-authentication) section for more details\n\n- **password-exec**\n\n    If HTTP Basic Authentication is required, run this command, passing the arguments:\n\n        \u003ccommand_to_run\u003e \u003ces_host\u003e \u003ces_username\u003e\n\n    The script expects the last line to contain the password in plaintext.\n\n- **noop**\n\n    Prevents any communication to the cluster from making changes to the settings or data contained therein.\n    In short, it prevents anything but HEAD and GET requests, **except** POST requests to the \\_search endpoint.\n\n- **timeout**\n\n    Timeout for connections and requests, defaults to 10.\n\n- **keep-proxy**\n\n    By default, HTTP proxy environment variables are stripped. Use this option to keep your proxy environment variables\n    in tact.\n\n- **insecure**\n\n    Don't verify TLS certificates\n\n- **cacert**\n\n    Specify a file with the TLS CA certificates.\n\n- **capath**\n\n    Specify a directory containing the TLS CA certificates.\n\n- **cert**\n\n    Specify the path to the TLS client certificate file..\n\n- **key**\n\n    Specify the path to the TLS client private key file.\n\n# INDEX SELECTION ARGUMENTS\n\n- **base**\n\n    In an environment using monthly, weekly, daily, or hourly indexes.  The base index name is everything without the date.\n    Parsing for bases, also provides splitting and matching on segments of the index name delineated by the '-' character.\n    If we have the following indexes:\n\n        web-dc1-YYYY.MM.DD\n        web-dc2-YYYY.MM.DD\n        logstash-dc1-YYYY.MM.DD\n        logstash-dc2-YYYY.MM.DD\n\n    Valid bases would be:\n\n        web\n        web-dc1\n        web-dc2\n        logstash\n        logstash-dc1\n        logstash-dc2\n        dc1\n        dc2\n\n    Combining that with the days option can provide a way to select many indexes at once.\n\n- **days**\n\n    How many days backwards you want your operation to be relevant.\n\n- **datesep**\n\n    Default is '.' Can be set to an empty string for no separator.\n\n- **pattern**\n\n    A pattern to match the indexes.  Can expand the following key words and characters:\n\n        '*'    expanded to '.*'\n        'ANY'  expanded to '.*'\n        'DATE' expanded to a pattern to match a date,\n\n    The indexes are compared against this pattern.\n\n# HTTP Basic Authentication\n\nHTTP Basic Authorization is only supported when the `proto` is set to **https**\nas not to leak credentials all over.\n\nThe username is selected by going through these mechanisms until one is found:\n\n    --http-username\n    'http-username' in /etc/es-utils.yml or ~/.es-utils.yml\n    Netrc element matching the hostname of the request\n    CLI::Helpers prompt()\n\nOnce the username has been resolved, the following mechanisms are tried in order:\n\n    Netrc element matching the hostname of the request\n    Password executable defined by --password-exec\n    'password-exec' in /etc/es-utils.yml, ~/.es-utils.yml\n    CLI::Helpers prompt()\n\n## Password Exec\n\nIt is **BAD** practice to specify passwords as a command line argument, or store it in a plaintext\nfile.  There are cases where this may be necessary, but it is not recommended.  The best method for securing  your\npassword is to use the **password-exec** option.\n\nThis option must point to an executable script.  That script will be passed two arguments, the hostname and the username\nfor the request.  It expects the password printed to STDOUT as the last line of output.  Here's an example password-exec setup\nusing Apple Keychain:\n\n    #!/bin/sh\n\n    HOSTNAME=$1;\n    USERNAME=$2;\n\n    /usr/bin/security find-generic-password -w -a \"$USERNAME\" -s \"$HOSTNAME\"\n\nIf we save this to \"$HOME/bin/get-passwd.sh\" we can execute a script\nlike this:\n\n    $ es-search.pl --http-username bob --password-exec $HOME/bin/get-passwd.sh \\\n                    --base secure-data --fields\n\nThough it's probably best to set this in your ~/.es-utils.yml file:\n\n    ---\n    host: secured-cluster.example.org\n    port: 443\n    proto: https\n    http-username: bob\n    password-exec: /home/bob/bin/get-passwd.sh\n\n### CLI::Helpers and Password Prompting\n\nIf all the fails to yield a password, the last resort is to use CLI::Helpers::prompt() to ask the user for their\npassword.  If the user is using version 1.1 or higher of CLI::Helpers, this call will turn off echo and readline magic\nfor the password prompt.\n\n# INSTALL\n\n**This library attempts to provide scripts compatible with version 0.19 through 1.1 of ElasticSearch**.\n\nRecommended install with [CPAN Minus](http://cpanmin.us):\n\n    cpanm App::ElasticSearch::Utilities\n\nYou can also use CPAN:\n\n    cpan App::ElasticSearch::Utilities\n\nOr if you'd prefer to manually install:\n\n    export RELEASE=\u003cCurrentRelease\u003e\n\n    wget \"https://github.com/reyjrar/es-utils/blob/master/releases/App-ElasticSearch-Utilities-$RELEASE.tar.gz?raw=true\" -O es-utils.tgz\n\n    tar -zxvf es-utils.tgz\n\n    cd App-ElasticSearch-Utilities-$RELEASE\n\n    perl Makefile.PL\n\n    make\n\n    make install\n\nThis will take care of ensuring all the dependencies are satisfied and will install the scripts into the same\ndirectory as your Perl executable.\n\n## USAGE\n\nThe tools are all wrapped in their own documentation, please see:\n\n    $UTILITY --help\n    $UTILITY --manual\n\nFor individual options and capabilities\n\n## PATTERNS\n\nPatterns are used to match an index to the aliases it should have.  A few symbols are expanded into\nregular expressions.  Those patterns are:\n\n    *       expands to match any number of any characters.\n    DATE    expands to match YYYY.MM.DD, YYYY-MM-DD, or YYYYMMDD\n    ANY     expands to match any number of any characters.\n\n# AUTHOR\n\nBrad Lhotsky \u003cbrad@divisionbyzero.net\u003e\n\n# CONTRIBUTORS\n\n- Alexey Shatlovsky \u003calexey.shatlovsky@booking.com\u003e\n- Samit Badle \u003cSamit.Badle@gmail.com\u003e\n- Takumi Sakamoto \u003ctakumi.saka@gmail.com\u003e\n- Vitaly Shupak \u003cvitaly.shupak@deshaw.com\u003e\n- Alexey Surikov \u003cksurent@gmail.com\u003e\n- Andrei Grechkin \u003candrei.grechkin@booking.com\u003e\n- Daniel Ostermeier \u003cdaniel.ostermeier@gmail.com\u003e\n- Jason Rojas \u003cjason.rojas@mgo.com\u003e\n- Kang-min Liu \u003cgugod@gugod.org\u003e\n- Lisa Hare \u003clhare@inview.co.uk\u003e\n- Markus Linnala \u003cMarkus.Linnala@cybercom.com\u003e\n- Matthew Feinberg \u003cmattf@intex.com\u003e\n- Mohammad S Anwar \u003cmohammad.anwar@yahoo.com\u003e\n\n# SUPPORT\n\n## Websites\n\nThe following websites have more information about this module, and may be of help to you. As always,\nin addition to those websites please use your favorite search engine to discover more resources.\n\n- MetaCPAN\n\n    A modern, open-source CPAN search engine, useful to view POD in HTML format.\n\n    [https://metacpan.org/release/App-ElasticSearch-Utilities](https://metacpan.org/release/App-ElasticSearch-Utilities)\n\n- CPAN Testers\n\n    The CPAN Testers is a network of smoke testers who run automated tests on uploaded CPAN distributions.\n\n    [http://www.cpantesters.org/distro/A/App-ElasticSearch-Utilities](http://www.cpantesters.org/distro/A/App-ElasticSearch-Utilities)\n\n- CPAN Testers Matrix\n\n    The CPAN Testers Matrix is a website that provides a visual overview of the test results for a distribution on various Perls/platforms.\n\n    [http://matrix.cpantesters.org/?dist=App-ElasticSearch-Utilities](http://matrix.cpantesters.org/?dist=App-ElasticSearch-Utilities)\n\n## Bugs / Feature Requests\n\nThis module uses the GitHub Issue Tracker: [https://github.com/reyjrar/es-utils/issues](https://github.com/reyjrar/es-utils/issues)\n\n## Source Code\n\nThis module's source code is available by visiting:\n[https://github.com/reyjrar/es-utils](https://github.com/reyjrar/es-utils)\n\n# COPYRIGHT AND LICENSE\n\nThis software is Copyright (c) 2024 by Brad Lhotsky.\n\nThis is free software, licensed under:\n\n    The (three-clause) BSD License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freyjrar%2Fes-utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freyjrar%2Fes-utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freyjrar%2Fes-utils/lists"}