{"id":30294456,"url":"https://github.com/linkedin/sysops-api","last_synced_at":"2025-08-17T01:35:04.138Z","repository":{"id":11584416,"uuid":"14074431","full_name":"linkedin/sysops-api","owner":"linkedin","description":"sysops-api is a framework designed to provide visability from tens of thousands of machines in seconds.","archived":false,"fork":false,"pushed_at":"2020-07-24T03:23:43.000Z","size":528,"stargazers_count":75,"open_issues_count":3,"forks_count":11,"subscribers_count":14,"default_branch":"master","last_synced_at":"2024-04-13T23:22:27.987Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/linkedin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-11-02T20:28:25.000Z","updated_at":"2024-03-27T14:56:57.000Z","dependencies_parsed_at":"2022-09-23T00:30:51.371Z","dependency_job_id":null,"html_url":"https://github.com/linkedin/sysops-api","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/linkedin/sysops-api","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linkedin%2Fsysops-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linkedin%2Fsysops-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linkedin%2Fsysops-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linkedin%2Fsysops-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/linkedin","download_url":"https://codeload.github.com/linkedin/sysops-api/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linkedin%2Fsysops-api/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270796217,"owners_count":24647319,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-16T02:00:11.002Z","response_time":91,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-17T01:35:03.672Z","updated_at":"2025-08-17T01:35:04.129Z","avatar_url":"https://github.com/linkedin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"sysops-api\n===============================\n\n© [2013] LinkedIn Corp. All rights reserved.\nLicensed under the Apache License, Version 2.0 (the \"License\");\u2028you may not use this file except in compliance with the License.\u2028You may obtain a copy of the License at  http://www.apache.org/licenses/LICENSE-2.0\n \nUnless required by applicable law or agreed to in writing, software\u2028distributed under the License is distributed on an \"AS IS\" BASIS,\u2028WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n\nOperational Visibility\n===============================\nsysops-api is a framework that leverages Redis to provide visibility from tens of thousands of machines in seconds.  Instead of trying to SSH to remote machines to collect data (execute commands, grep through files), LinkedIn uses this framework to answer any arbitrary question about all of our infrastructure.\n\n*[The slides from the  LISA 2013 presentation which describes the architecture can be found here](http://www.slideshare.net/MikeSvoboda/lisa-2013-sysopsapi-leveraging-inmemory-key-value-stores-for-large-scale-operations-with-redis-and-cfengine)*\n\n*[Video from LISA 2013 of this presentation can be watched here](http://youtu.be/H1dVsSvKBlM)*\n\n\nThis project is basically a means for us to answer any arbitrary question about any production machine and get results returned to us in seconds.   We primarily use this for crawling tens of thousands of machines for information very quickly.   Its how we can confidently make automation changes.  We use this to audit production before a change is pushed, so we know the state of machines and what our change will impact.  \n\n    [msvoboda@esv4-infra01 ~]$ time extract_sysops_cache.py --search sysctl-a --contents --scope global | grep net.core.netdev_budget  | uniq -c | sort -n\n    24247 net.core.netdev_budget = 300\n\n    real\t0m28.337s\n    user\t0m45.244s\n    sys  \t0m2.837s\n\nThe above command searched the output of \"sysctl –a\" across 24 thousand machines in 28 seconds and reported that all machines had the same value for net.core.netdev_budget = 300.   This means if we wanted to use automation to control this kernel tunable, we would not impact any systems.\n\nWhat kind of data is this fetching?\n===============================\nWe use this to insert any type of information we want.  This could be remotely executed commands (netstat, installed packages, loaded kernel modules, etc) or any file off of the filesystem.     We take snapshots of our systems, and use these snapshots to answer any question we want.   Very fast.  \n\nInsert process tables, mount tables, loaded kernel modules, /proc things, or whatever else you use on an every day basis as a sysadmin to debug systems.  Once this data exists in sysops-api, you will find that you no longer have to log in to machines remotely again to debug them.  Provide as much data as possible up front so you can answer any question thrown at you.\n\nCache Insertion\n===============================\nAt a high level, we create a directory and symlink arbitrary objects into it.  We use CFEngine to populate Redis, so we've chosen /var/cfengine/outgoing,\nbut the directory could be anywhere.  The provided python script, module_populate_mps_cache, searches for all files in this directory and inserts them\ninto Redis.  If we want to collect remotely executed commands, we do so externally from this tool and dump them into a JSON object called\n/etc/hardware_identification.json.  On insertion, we populate metadata about the object (md5sum, os.stat, wordcount) and insert at the same time. \nOptionally, we also record the amount of time required for each Redis server interaction.   Each machine at LinkedIn sends data to 4x Redis servers.  The\nRedis servers themselves do not replicate data.  Replication is handled by the clients at data insertion time.   The data insertion script assumes that\n4x Redis servers are passed on the CLI.  Modify this if you want.  If we only want to send data to 2x Redis servers, we duplicate physical machine names, but still supply 4x.  This allows us to use the same CFEngine policy to insert data into 1, 2, or 4x Redis servers without code modification.\n\nWe use CFEngine to drive execution of module_populate_mps_cache, but there is no reason it couldn't be driven via crontab.  All that happens in this\nscript is that it finds objects to send into the cache, and does that.\n\nCache Extraction\n===============================\nFor data extraction, The following steps are required:\n\n Provided by RedisFinder.py:\n  1. Determine scope (local, site, or global)\n  2. For each level of my scope, determine what Redis servers I will contact.\n  3. Pick a randomized Redis server from this list (load balancing)\n  4. If the randomly chosen Redis server doesn't respond, pick another one at random (failover)\n  5. Test that a Redis server.info() call works.  If it does, push it into the list of Redis servers that we should querty to build our working set.\n\nLinkedIn uses a technology opensourced by Yahoo! called Range to perform this lookup.  https://github.com/ytoolshed/range\n\nThe method of lookup could be:\n DNS txt records\n Files on the filesystem\n LDAP\n Any other directory service\n\nPlease add any other lookup services into RedisFinder.py as needed.\n\n\n Provided by CacheExtractor.py:\n  1. Search each Redis server for matching keys based off of the search string passed to the CacheExtractor object.\n  2. (Optionally) Extract data from the matched keys. \n\nThe cache exists as a simple python dictionary (hash). The key is in the format of\n\u003chostname\u003e#\u003cfilename\u003e\n\nThe value of the key is an array.  \n    Array[0] = Contents of the file / command \n    Array[1] = md5sum\n    Array[2] = Python's os.stat()\n    Array[3] =  wordcount\n\nCacheExtractor\n===============================\nThe basic interaction with sysops-api is the CacheExtractor python object.  The \"swiss army tool\" called extract_sysops_cache.py imports this object,\nmakes the output pretty, and displays to standard out.  Typically this is how users interact with the CacheExtractor object from the command line. \n\nCacheExtractor gives you a programmatic method of interacting with the results pulled out of the cache.  For example, its trival to search for 8000\nJSON objects, (lshw --output json) and access the hardware characteristics of an entire datacenter by programatically.  Several other utilities are included in the contrib section showing how to use this module. \n\nMore Documentation\n===============================\n*[Usage of extract_sysops_cache.py](https://github.com/linkedin/sysops-api/wiki/Extracting-the-Sysops-cache-for-fun-and-profit)*\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinkedin%2Fsysops-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinkedin%2Fsysops-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinkedin%2Fsysops-api/lists"}