{"id":21743988,"url":"https://github.com/telefonica/iot-utils","last_synced_at":"2026-04-11T03:04:26.721Z","repository":{"id":66221866,"uuid":"54195933","full_name":"Telefonica/iot-utils","owner":"Telefonica","description":"IOT operation Utilities \u0026 systems management tools - Ansible Scripting Python WinSSH OpenStack VirtualBox MySQL Postgress MongoDB DevOPS Admin","archived":false,"fork":false,"pushed_at":"2017-09-20T17:17:04.000Z","size":181,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-01-25T22:21:39.115Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://github.com/Telefonica/iot-utils/wiki","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Telefonica.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-03-18T11:19:05.000Z","updated_at":"2022-04-22T05:43:43.000Z","dependencies_parsed_at":null,"dependency_job_id":"b850c6c6-c8f4-488a-8579-188bd2d5010f","html_url":"https://github.com/Telefonica/iot-utils","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Telefonica%2Fiot-utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Telefonica%2Fiot-utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Telefonica%2Fiot-utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Telefonica%2Fiot-utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Telefonica","download_url":"https://codeload.github.com/Telefonica/iot-utils/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244722653,"owners_count":20499151,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-26T07:09:39.212Z","updated_at":"2025-12-30T21:26:24.468Z","avatar_url":"https://github.com/Telefonica.png","language":"Shell","readme":"# General utilities\n\n## 1.- Easy manage environments hosts access\nFor easy manage environment hosts access\u003cbr\u003e\nNeeds:\n- Memory remember many clusters, environments, hostnames, ips, etc?\n- Quickly find a specific hosts to access it\n- Dynamic refresh of all existing hosts at the time\n- Inspect trace log of large number of hosts finding specific information\n- Command utilities information at the time of large number of hosts\n- Easy filter groups of hosts (by example name, type, etc)\n- Easy VPN access\n\n\n### 1.1.- Requirements\n- No root user for this\n- Compatibility for ZSH friends:\u003cbr\u003e\n  In general, all in this repository is compatible with zsh shell. I known a difference, with zsh, we don't need launch \". command.sh\" when we need to execute and load environments, only to run \"command.sh\" is sufficient.\n\n\n### 1.2.- Install basic software\nLaunch:\n\n```\n# The git client and dialog window text\nsudo yum install dialog git -y\n# Use passwords inside scripts\nsudo yum install sshpass -y\n# Create VPNs\nsudo yum install openconnect -y\n# A program for making large letters out of ordinary text\nsudo yum install figlet -y\n# Install NetCat utility for Centos/RH 7 and higher\nsudo yum install nmap-ncat -y\n# Install NetCat utility for Centos/RH 6 and lower\nsudo yum install nc -y\n\n# For use OpenStack clients and Ansible, we have two options:\n\n# Installing RPM packages\n# Create python virtual environments\nsudo yum install python-virtualenv -y\nsudo yum install python-virtualenvwrapper -y\n\n# Or installing Python packages (in system Python)\nsudo pip install --upgrade pbr\nsudo pip install --upgrade PyYAML\nsudo pip install --upgrade virtualenv\nsudo pip install --upgrade virtualenvwrapper\n\n# How to update all python packages:\nsudo pip list --outdated | sed 's/(.*//g' | xargs -n1 pip install -U\n```\n\n\n### 1.3.- Install Ansible WinRM and OpenStack client tools (Linux)\nFirst choose versions\n- For Ansible 2.3.2.0 and WinRM 0.2.2 (actual)\n\n```\nANSIBLE_VERSION=2.3.2.0\nWINRM_VERSION=0.2.2\n```\n\nThen install software\n\n```\nrm -rf ~/venv-ansible-${ANSIBLE_VERSION}\nvirtualenv ~/venv-ansible-${ANSIBLE_VERSION}\nsource ~/venv-ansible-${ANSIBLE_VERSION}/bin/activate\n# Update basic Python packages\npip install --upgrade pip\npip install --upgrade setuptools\npip install --upgrade wheel\n# Install Ansible package\npip install ansible==${ANSIBLE_VERSION}\n# Install WIN RM for work with Windows machines\npip install pywinrm==${WINRM_VERSION}\n```\n\n- Install KILO OpenStack client tools\u003cbr\u003e\n  If we will work inside OST environments, we need to install the OpenStack client tools. We have actually KILO supported.\n\n```\npip install --upgrade python-cinderclient\npip install --upgrade python-glanceclient\npip install --upgrade python-heatclient\npip install --upgrade python-keystoneclient\npip install --upgrade python-neutronclient\npip install --upgrade python-novaclient\npip install --upgrade python-openstackclient\npip install --upgrade python-swiftclient\npip install --upgrade python-designateclient\npip install --upgrade python-ironicclient\npip install --upgrade python-magnumclient\npip install --upgrade python-mistralclient\npip install --upgrade python-troveclient\npip install --upgrade shade\n```\n\n- Generate python requirements file for backup software versions\u003cbr\u003e\n  We recommend to generate the requirements file of this python environment\n\n```\nANSIBLE_VERSION=2.3.2.0\nWINRM_VERSION=0.2.2\n\npip freeze \u003e $HOME/requirements-ansible-${ANSIBLE_VERSION}-OST-kilo.txt\n```\n\n- To recreate other python virtual environment\n\n```\nANSIBLE_VERSION=2.3.2.0\nWINRM_VERSION=0.2.2\n\nvirtualenv ~/venv-ansible-${ANSIBLE_VERSION}\npip install -r $HOME/requirements-ansible-${ANSIBLE_VERSION}-OST-kilo.txt\n```\n\n\n### 1.4.- Install Ansible WinRM and OpenStack client tools (Windows with CygWin)\nWe need to use Python version 2.7.12 and architecture x86. With CygWin we use own Python installation, but we can have a Windows Python installation (https://www.python.org/ftp/python/2.7.12/python-2.7.12.msi)\n- Install CygWin: https://cygwin.com/setup-x86_64.exe (mininal install)\n- Setup Cygwin64 Terminal ICON\u003cbr\u003e\n  Check execute this program as Administrator\n- Disable access to Windows Python installation\u003cbr\u003e\n  Enter in a Cygwin64 session\n\n```\necho $'PATH=$(echo $PATH | tr \\':\\' \\'\\\\n\\' | grep -v \"/cygdrive/.*/Python27\" | paste -sd:)' \u003e\u003e .bash_profile\nexit\n```\n\n- Install base packages\u003cbr\u003e\n  Enter in a Cygwin64 session\n\n```\ncurl https://cygwin.com/setup-x86_64.exe -o setup-x86_64.exe\n./setup-x86_64.exe -q --packages python python-devel python-setuptools openssl-devel libffi-devel gcc-g++\nexit\n\n# Description packages\n# python: Python language interpreter\n# python-devel: Python language interpreter\n# python-setuptools: Python package management tool\n# openssl-devel: A general purpose cryptography toolkit with TLS implementation (development)\n# libffi-devel: Portable foreign function interface library\n# gcc-g++: GNU Compiler Collection (C++)\n```\n\n- Install Python packages\u003cbr\u003e\n  Enter in a Cygwin64 session\n\n```\neasy_install-2.7 pip\npip install --upgrade pip\npip install --upgrade setuptools\npip install --upgrade wheel\npip install virtualenv\nexit\n```\n\n- Install final software\u003cbr\u003e\n  Enter in a Cygwin64 session\n\n```\n# Information of outdated Python packages\npip list --format=columns --outdated\n\n# Workaround to compile with gcc (u_int header type unknown)\nexport CFLAGS=\"-D_DEFAULT_SOURCE\"\nANSIBLE_VERSION=2.3.2.0\nWINRM_VERSION=0.2.2\n\nrm -rf ~/venv-ansible-${ANSIBLE_VERSION}\nvirtualenv ~/venv-ansible-${ANSIBLE_VERSION}\nsource ~/venv-ansible-${ANSIBLE_VERSION}/bin/activate\npip install --upgrade pip\npip install --upgrade setuptools\npip install --upgrade wheel\n# Install Ansible package\npip install ansible==${ANSIBLE_VERSION}\n# Install WIN RM for work with Windows machines\npip install pywinrm==${WINRM_VERSION}\n\npip install --upgrade python-cinderclient\npip install --upgrade python-glanceclient\npip install --upgrade python-heatclient\npip install --upgrade python-keystoneclient\npip install --upgrade python-neutronclient\npip install --upgrade python-novaclient\npip install --upgrade python-openstackclient\npip install --upgrade python-swiftclient\npip install --upgrade python-designateclient\npip install --upgrade python-ironicclient\npip install --upgrade python-magnumclient\npip install --upgrade python-mistralclient\npip install --upgrade python-troveclient\npip install --upgrade shade\n\npip freeze \u003e $HOME/requirements-ansible-${ANSIBLE_VERSION}-OST-kilo.txt\n\n# Show list of installed packages\ncygcheck -c\n```\n\n\n### 1.5.- Install PyCharm IDE Python develop environment\nWith no root user\n\n```\nmkdir -p $HOME/software\ncd $HOME/software\nwget https://download.jetbrains.com/python/pycharm-community-5.0.4.tar.gz\ncd $HOME\ntar xvfz software/xvfz pycharm-community-5.0.4.tar.gz\n```\n\n\n### 1.6.- Howto install myenvironments tools\nLaunch:\n\n```\ncd $HOME\ngit clone git@github.com:Telefonica/iot-utils.git\ncp -rp iot-utils/myenvironments $HOME\ncp -rp iot-utils/tools $HOME\nrm -rf iot-utils\n```\n\n### 1.7.- Configure\nRead and apply all related task in: `$HOME/myenvironments/conf`\n\n```\nbashrcconfignoroot.cnf.template\nconfigetchosts.info\ngithubprivateenv.sh.template\nhowtoconfigpublicprivatekeys.info\nlistostenvs.cnf.template\nsshconfigclientnoroot.template\npycharm.info\nvpnconnect.info\n```\n\nAs final step we need to ensure that all session terminals are close, and open new terminals\n\n### 1.8.- Start and howto use\n\n#### 1.8.1.- Generate host lists of VMWARE environments (VMWARE envs only useful specific for IOT, not use for others)\nLaunch:\n`noostservers.sh`\nThe host lists are stored at `$HOME/myenvironments/envs/iotenvNOOST_*.hosts`\n#### 1.8.2.- Generate host lists of OST environments\nLaunch:\n`ostservers.sh`\nThe host lists are stored at `$HOME/myenvironments/envs/iotenvOST_*.hosts`\n\n#### 1.8.3.- Use of SSH access to hosts\nLaunch:\n`sshaccess.sh`\n\n#### 1.8.4.- Configure openstack environments\nLaunch:\n\n```\n. openstackenv\u003cEPG|DSNAH|PREDSN|PRODSN\u003e.sh\n```\n\nOr\n\n```\nsource openstackenv\u003cEPG|DSNAH|PREDSN|PRODSN\u003e.sh\n```\n\nClear current environment:\n\n```\n. openstackenvCLEAR.sh\n```\n\nOr\n\n```\nsource openstackenvCLEAR.sh\n```\n\n### 1.8.5.- Manage VPNs\nTo manage VPNs we can use vpnconnect.sh. Howto use and help, launch: `vpnconnect.sh` without parameters\n\n### 1.8.6.- Use PyCharm IDE\nThe first time we need to execute PyCharm IDE from startup script (as no root user):\n\n```\ncd $HOME/pycharm-community-5.0.4/bin\n./pycharm.sh\n```\n\nAfter, we close PyCharm IDE, and automatically PyCharm IDe creates a desktop menu launch at `Application-\u003eProgramming-\u003ePycharm Community Edition`\n\n\n## 2.- Tools\nWe have tools for work better. These tools are not robust (by now), then you need to modify it to work properly (Ex. paths)\u003cbr\u003e\nStored at $HOME/tools\n\n```\n# Show diffs (Version X.Y.Z and commit number/release number) of RPMs between Git and RPM installed machines\ndiffrpms/findagentbase.sh\nHOWTO USE (now only for iot-agent-base):\ncd $HOME/tools/diffrpms\n./findagentbase.sh\nExample output:\n*********************************************************\nVERIFYING RPM...\n*********************************************************\nRPMNAME \u003ciot-agent-base\u003e\nBRANCH \u003crelease/1.3.1\u003e\nVERSION \u003c1.3.1\u003e\nGITURLREPO \u003cgit@github.com:telefonicaid/fiware-IoTAgent-Cplusplus.git\u003e\nSHORTSHA1 \u003c57b0ee1\u003e\nNUMCOMMITS \u003c117\u003e\nOK: ENV \u003ciotenvOST_EPG_201-IOT-demo.hosts PKG \u003ciot-agent-base\u003e EQUAL GITREPO VS INSTALLED RPM\nOK: ENV \u003ciotenvOST_EPG_201-IOT-demo.hosts PKG \u003ciot-agent-base\u003e EQUAL GITREPO VS INSTALLED RPM\nERROR: ENV \u003ciotenvOST_EPG_201-IoT-int-ext.hosts\u003e PKG \u003ciot-agent-base\u003e DIFFER: GITREPO 1.3.1.117 VS INSTALLED RPM 1.3.1.115\nERROR: ENV \u003ciotenvOST_EPG_201-IoT-int-ext.hosts\u003e PKG \u003ciot-agent-base\u003e DIFFER: GITREPO 1.3.1.117 VS INSTALLED RPM 1.3.1.115\nOK: ENV \u003ciotenvOST_EPG_264-iot-int.hosts PKG \u003ciot-agent-base\u003e EQUAL GITREPO VS INSTALLED RPM\nOK: ENV \u003ciotenvOST_EPG_264-iot-int.hosts PKG \u003ciot-agent-base\u003e EQUAL GITREPO VS INSTALLED RPM\nOK: ENV \u003ciotenvOST_PREDSN_IOT-D.hosts PKG \u003ciot-agent-base\u003e EQUAL GITREPO VS INSTALLED RPM\nOK: ENV \u003ciotenvOST_PREDSN_IOT-D.hosts PKG \u003ciot-agent-base\u003e EQUAL GITREPO VS INSTALLED RPM\nUNKNOWN: ENV \u003ciotenvOST_PRODSN_IOT-L.hosts PKG \u003ciot-agent-base\u003e CANNOT ACCESS\nUNKNOWN: ENV \u003ciotenvOST_PRODSN_IOT-L.hosts PKG \u003ciot-agent-base\u003e CANNOT ACCESS\n\n\n# Show diffs of a encrypted file between actual git branch and a commit\nansible-vault-git-diff.sh\n# Decrypt encrypted Ansible files\ndecryptansiblefiles.sh\n# Encrypt files to use with Ansible\nencryptansiblefiles.sh\n# Show differences inside encrypted files of a current git branch and write to diffcreds directory, over git repo directory parent\nfindcredentialchanges.sh\n# Memory processes sum threads\nps_mem.py\n# Example:\nsudo python ps_mem.py | grep httpd\n  4.2 MiB +   6.3 MiB =  10.5 MiB\thttpd (6)\n# 10.5 Mib per 6 httpd processes\n# Update a git repo and decrypt Ansible encrypted files\nupdateiotansible.sh\n```\n\n\n## 3.- Windows and PowerShell\nWe can configure any Windows host to easy access with ssh and work with Windows console and PowerShell\n\n### 3.1.- Microsoft has finally integrated into Windows openssh\nTo install it, follow the steps:\n\n- Download a copy of Win-OpenSSH from https://github.com/PowerShell/Win32-OpenSSH\n- The last is here: https://github.com/PowerShell/Win32-OpenSSH/releases/download/4_5_2016/OpenSSH-Win64.zip\n\n- Unzip inside folder C:\\openssh\n\n- Open a PowerShell console with administrator privileges\n\n- Generate server keys with\n\n```\ncd C:\\openssh\n.\\ssh-keygen.exe -A\n```\n\n- If Windows Firewall is activated:\nOpen a port for the SSH server in Windows Firewall:\u003cbr\u003e\nEither run the following PowerShell command (Windows 8 and 2012 or newer only), as the Administrator:\n\n```\nNew-NetFirewallRule -Protocol TCP -LocalPort 22 -Direction Inbound -Action Allow -DisplayName SSHD\n```\n\nor go to Control Panel \u003e System and Security \u003e Windows Firewall \u003e Advanced Settings \u003e Inbound Rules and add a new rule for port 22.\u003cbr\u003e\nNOTE: New-NetFirewallRule is for servers only. If you're on a workstation try:\n\n```\nnetsh advfirewall firewall add rule name='SSH Port' dir=in action=allow protocol=TCP localport=22\n```\n\n- If you need key-based authentication, run the following to setup the key-auth package\n\n```\npowershell.exe .\\install-sshlsa.ps1\nRestart-Computer\n```\n\n- Edit C:\\openssh\\sshd_config to configure sftp server:\n\n```\nSubsystem sftp C:\\openssh\\sftp-server.exe\n```\n\n- Install SSHD server as service\n\n```\n.\\sshd.exe install\nStart-Service sshd\n```\n\nMake the service start on boot (PowerShell):\n\n```\nSet-Service sshd -StartupType Automatic\n```\n\nOr manually:\n- Start the service and/or configure automatic start:\n- Go to Control Panel \u003e System and Security \u003e Administrative Tools and open Services. Locate SSHD service.\n- If you want the server to start automatically when your machine is started: Go to Action \u003e Properties.\n- In the Properties dialog, change Startup type to Automatic and confirm.\n- Start the SSHD service by clicking the Start the service.\n\nIn your Windows account profile folder (typically in C:\\Users\\username\\.ssh):\n- Create the .ssh folder (for the authorized_keys file) in your Windows account profile folder (typically in C:\\Users\\username\\.ssh).\n- Do not change permissions for the .ssh and the authorized_keys.\n- Configure authorized_keys file and store the PEM file in .ssh folder.\n\n* Test ssh connection\n\n```\nEnter in Windows console\nssh user@winmachine\nEnter in Powershell console\npowershell -File -\n```\n\nExample:\n\n```\nssh myuser@caprica.hi.inet\nmyuser@caprica.hi.inet's password: \nMicrosoft Windows [Versi n 6.1.7601]\nCopyright (c) 2009 Microsoft Corporation. Reservados todos los derechos.\n\nadmin@CAPRICA C:\\Users\\admin\u003epowershell -File -\nPS C:\\Users\\admin\u003e Get-EventLog -Log \"Application\"\n\n   Index Time          EntryType   Source                 InstanceID Message                                                                                                                           \n   ----- ----          ---------   ------                 ---------- -------                                                                                                                           \n  104935 mar 09 16:04  0           Software Protecti...   1073742727 Se detuvo el servicio de protecci n de software....                                                                               \n  104934 mar 09 15:59  0           Software Protecti...   1073742726 Se inici  el servicio de protecci n de software....                                                                               \n  104933 mar 09 15:59  Information Software Protecti...   1073742827 El servicio de protecci n de software complet  la comprobaci n del estado de licencias....                                        \n  104932 mar 09 15:59  Information Software Protecti...   1073742890 Estado de inicializaci n para objetos de servicio....                                                                             \n  104931 mar 09 15:59  Information Software Protecti...   1073742724 Se est  iniciando el servicio de protecci n de software....                                                                       \n  104930 mar 09 15:59  Information sshd                            0 No se encontr  la descripci n del Id. de evento '0' en el origen 'sshd'. Es posible que el equipo local no tenga la informaci n...\n  104929 mar 09 15:59  Information sshd                            0 No se encontr  la descripci n del Id. de evento '0' en el origen 'sshd'. Es posible que el equipo local no tenga la informaci n...\n  104928 mar 09 15:56  Information sshd                            0 No se encontr  la descripci n del Id. de evento '0' en el origen 'sshd'. Es posible que el equipo local no tenga la informaci n...\n  104927 mar 09 15:56  Information sshd                            0 No se encontr  la descripci n del Id. de evento '0' en el origen 'sshd'. Es posible que el equipo local no tenga la informaci n...\n...\nPS C:\\Users\\myuser\u003e exit\n\nmyuser@CAPRICA C:\\Users\\myuser\u003eexit\nConnection to caprica.hi.inet closed.\n```\n\n### 3.2.- Uninstall Win-OpenSSH\n- Start Powershell as Administrator\n\n- Stop the service\n\n```\nStop-Service sshd\n```\n\n- Uninstall\n\n```\n.\\sshd.exe uninstall\npowershell .\\uninstall-sshlsa.ps1\nRestart-Computer\n```\n\n\n## 4.- Operation process\nInside iot-utils/operations we store processes to operate systems. By now, exists the following:\n\n```\nbin/changeloglevelnginx.sh\n```\n\n- Change (locally, inside a machine with SSH) the level log of nginx server online.\n- Help of use:\n\n```\n./changeloglevelnginx.sh \n****************************************\nNGINX CHANGE LOG LEVEL\n****************************************\n\nUsage: changeloglevelnginx.sh \u003cloglevel\u003e\nWhere loglevel can be:\n    emerg: Emergency situations where the system is in an unusable state.\n    alert: Severe situation where action is needed promptly.\n    crit: Important problems that need to be addressed.\n    error: An Error has occurred. Something was unsuccessful.\n    warn: Something out of the ordinary happened, but not a cause for concern.\n    notice: Something normal, but worth noting has happened.\n    info: An informational message that might be nice to know.\n    debug: Debugging information that can be useful to pinpoint where a problem is occurring.\n\n\nFor debug level visit http://nginx.org/en/docs/debugging_log.html\n```\n\n- Example of use:\n./changeloglevelnginx.sh warn\n\n```\n****************************************\nNGINX CHANGE LOG LEVEL\n****************************************\nINFO: Change loglevel to \u003cwarn\u003e in file \u003c/etc/nginx/conf.d/myweb1.conf\u003e\nINFO: Change loglevel to \u003cwarn\u003e in file \u003c/etc/nginx/conf.d/myweb2.conf\u003e\n```\n \n\n## 5.- Manage and administer databases\nHere we document tools for manage and administer databases\n\n### 5.1.- Manage and administer Postgres databases\nPostgres is an object-relational database management system (ORDBMS) with an emphasis on extensibility and standards-compliance\n\n#### 5.1.1.- Troubleshootings\nSolutions for Postgres failures\n\n##### 5.1.1.1.- POSTGRES FAILURE missing chunk number 0 in pg_toasts\nWhen appear: hardware failures, cluster failures, and obsoleted Linux Kernels. Steps:\n- Stop postgres\n\n- File system repair\n\n```\ne2fsck -fp /dev/\u003cdevice\u003e\n```\n\n- Add following line to /var/lib/pgsql/9.3/data/postgresql.conf\n\n```\nallow_system_table_mods = on\n```\n\n- Enter postgres session in the affected database\n\n```\npsql -U postgres db_affected\n```\n\n- From the toast table number of the log failed (here is 2619)\n\n```\nSELECT relname from pg_class where oid = 2619;\n   relname    \n--------------\n pg_statistic\n(1 row)\n```\n\n- Test that this table is failing\n\n```\nSELECT count (*) from pg_statistic;\nERROR:  invalid page in block 25 of relation base/16386/12629\n```\n\n- Repair:\n\n```\nSET zero_damaged_pages = on;\nOutput:\nSET\n\nREINDEX TABLE pg_statistic;\nOutput:\nWARNING:  invalid page in block 25 of relation base/16386/12629; zeroing out page\nERROR:  could not create unique index \"pg_statistic_relid_att_inh_index\"\nDETAIL:  Key (starelid, staattnum, stainherit)=(1259, 1, f) is duplicated.\n\nDELETE from pg_statistic;\nOutput:\nDELETE 443\n\nREINDEX table pg_statistic;\nOutput:\nREINDEX\n\nVACUUM analyze pg_statistic;\nOutput:\nWARNING:  relation \"pg_statistic\" page 25 is uninitialized --- fixing\nWARNING:  relation \"pg_toast_2619\" page 9 is uninitialized --- fixing\nWARNING:  relation \"pg_toast_2619\" page 10 is uninitialized --- fixing\nWARNING:  relation \"pg_toast_2619\" page 11 is uninitialized --- fixing\nVACUUM\n```\n\n- Exit session and remove from /var/lib/pgsql/9.3/data/postgresql.conf the parameter added previously\n\n```\nallow_system_table_mods = on\n```\n\n- Stop and start postgres. Now the database is repaired\n\n- Do yum update to prevent the same error (Steps only for RH/Centos 6.x)\n\n```\nyum update -y bind* coreutils* chkconfig* device-mapper* binutils* dracut* elf* fence* kernel* lvm*\n```\n\n- Reboot the machine\n\n```\nreboot\n```\n\n### 5.2.- Manage and administer MongoDB databases\nThe MongoDB database is a simple no-sql database, oriented to documents using JSON and BSON formats\n\n#### 5.2.1.- Tools for any version of MongoDB\nInside [managemongodb](managemongodb) we have processes to manage many task for MongoDB\u003cbr\u003e\nThese tools are designed to be used by many people: Developers, Dbas, Release Engineers and Support, under the following concepts: extensive use of regexp expressions, high flexibility, high separated modularity functions, easy configuration and low level design (only use MongoDB tools, nodejs software inside MongoDB tools and Bash)\u003cbr\u003e\nBecause this design, at beginning the use of these tools can be complicated and difficult\u003cbr\u003e\nBecause there are bugs in older MongoDB versions 2.x and 3.1.x and 3.2.x (not manage correctly error code status and not manage a lot of special characters, by example the slash \"/\"), we construct specific process to do backups and restore MongoDB databases with these characteristics\u003cbr\u003e\nFor MongoDB 3.3.x versions and higher these bugs are resolved, and for these versions we will develop better tools\n\n##### 5.2.1.1.- Howto use\nThe use is simple, given that for each task we have a separated and isolated process.\n- We need to configure according to our needs the files inside conf directory. The environment variables are self-explained\n- All database backups are stored inside backup folder, as specified by the --backupprefix parameter of backup processes\n- All commands are inside bin folder, and the use is self-explained. For howto use and help we can execute any command without parameters\n\n\n##### 5.2.1.2.- Commands\nHere we list all commands that we can use\n\n- listdbs.sh\n\n```\n*************************************************\nList mongo db names\n*************************************************\n\nUsage: listdbs.sh [--help | --alldbs | --db \"searchstring\"]\n```\n\n- listcolls.sh\n\n```\n*************************************************\nList mongo collection names of a db\n*************************************************\n\nUsage: listcolls.sh [--help | --db \"dbname\" [ --col \"searchstring\" ] ]\n```\n\n- copydb.sh\n\n```\n*************************************************\nCopy a mongo db\n*************************************************\n\nUsage: copydb.sh [--help | --dbfrom \"dbsource\" --dbto \"dbtarget\"]\n```\n\n- dropdb.sh\n\n```\n*************************************************\nDrop a mongo db\n*************************************************\n\nUsage: dropdb.sh [--help | --db \"dbname\"]\n```\n\n- rencolls.sh\n\n```\n*************************************************\nRename collection names of a mongo db\n*************************************************\n\nUsage: rencolls.sh [--help | --db \"dbname\" --find \"searchstring\" --replace \"replacestring\" [ --dochanges ] ]\nWhere searchstring can be a regexp, and replacepattern is a string to replace the pattern\nExamples:\n\n- Find collections that start with sth_ and next char not /, and replace by sth_/\n  rencolls.sh --db mydbname --find '^sth_(([^/])|$)' --replace 'sth_\\/\\1' --dochanges\n- Find collections that start with sth_/ and next any char and replace by sth_\n  rencolls.sh --db mydbname --find '^sth_\\/' --replace 'sth_' --dochanges\n\n- Find collections that start with sth_ and next char not /, and TRY replace by sth_/ (don't apply any changes)\n  rencolls.sh --db mydbname --find '^sth_(([^/])|$)' --replace 'sth_\\/\\1'\n- Find collections that start with sth_/ and next any char and TRY replace by sth_ (don't apply any changes)\n  rencolls.sh --db mydbname --find '^sth_\\/' --replace 'sth_'\n```\n\n- backuponedb.sh\n\n```\n*************************************************\nBackup a mongo db with BSON format\n*************************************************\n\nUsage: backuponedb.sh [--help | --db \"dbtobackup\" [--backupprefix \"prefix\" default: default] [--rotate \u003cTrue|othervalue\u003e default: True]]\n```\n\n- backupdbs.sh\n\n```\n*************************************************\nBackup mongo dbs with BSON format\n*************************************************\n\nUsage: backupdbs.sh [--help | --alldbs | --dbs \"searchstring\" [--backupprefix \"prefix\" default: default] [--rotate \u003cTrue|othervalue\u003e default: True]]\n```\n\n- restoreonedb.sh\n\n```\n*************************************************\nRestore a mongo db with BSON format\n*************************************************\n\nUsage: restoreonedb.sh [--help | --dirbackup \"path_abs_dir_backup\" --dbsource \"dbsource\" [--dbtarget \"dbtarget\"]]\n```\n\n- restoredbs.sh\n\n```\n*************************************************\nRestore mongo dbs with BSON format\n*************************************************\n\nUsage: restoredbs.sh [--help | --dirbackup \"path_abs_dir_backup\" \u003c--dbs \"searchstring\" | --alldbs]\u003e\n```\n\n- STHbackupdbs.sh (a specific process to backup MongoDB databases with slashes inside collection names)\n\n```\n*************************************************\nSTH backup databases\n*************************************************\n\nUsage: STHbackupdbs.sh [--help | --done]\n```\n\n- STHrestoreonedb.sh (a specific process to restore a MongoDB database with slashes inside collection names)\n\n```\n*************************************************\nWARNING: Restore databases is very dangerous...\nThe original database will be dropped...\n*************************************************\nExample:\nSTHrestoreonedb.sh --dirbackup /home/ec2-user/managemongodbs/backups/backupsth --dbbackup Bsth_db --dborigin sth_db\n```\n\n##### 5.2.1.3.- One example howto backup all MongoDB databases with slashes\nWe assume that the MongoDB databases are named as prefix '^sth_'\u003cbr\u003e\nThen we execute two steps:\n\n- Backup of all databases no '^sth_'\n\n```\n./backupdbs.sh  --dbs '^(?!sth_)' --backupprefix backupnosth\n```\n\n- Backup of all databases '^sth_'\n\n```\n./STHbackupdbs.sh --done\n```\n\n#### 5.2.2.- Tools for MongoDB versions 3.3.x and higher\nIn the future we realize these tools more simple and efficient that current tools\n\n\n## 6.- Create a virtual machine image for OpenStack\nWe show step by step howto create a virtual machine image for OpenStack. We will use VirtualBox for this task. This VM is compatible with virtualbox too, except that we need to configure cloud-user with password or ssh public key, or use sysadmin user with password or ssh public key\n\n- Create new machine as:\n\n```\nName: Centos7.4-1708-20170920\nLinux RedHat 64 bits\n512Mb RAM\n1CPU\nHard disk VDI dinamic reserved 20Gb\nBidirectional Share clipboard\nBidirectional Drag and Drop\nUncheck Enable audio\nThree net adapters:\n- First adapter: Bridge adapter Realtek PCIe GBE Family controller. Promiscuous mode. Allow all\n- Second adapter: Bridge adapter Intel(R) Dual Band Wireless-AC 3160. Promiscuous mode. Allow all\n- Third adapter: Bridge adapter NAT\nAdd CDROM ISO CentOS-7-x86_64-Minimal-1708.iso\n```\n\n- Start machine. Select Install Centos\n\n- Data for installer in order:\n\n```\n- Language installation process -\u003e Continue\n\n- Network and hostname\nNOTE: Configure name distinct as localhost.localdomain. Simple name: centos\nConfigure adapters:\n  - NIC enp0s3: Automatically connect to this network and require IPV4\n    Annotate IP and add to putty for later access\n  - NIC enp0s8: Automatically connect to this network and require IPV4\n  - NIC enp0s9: Automatically connect to this network and require IPV4\n\n- Date and time: Europe/Madrid. Then click in Done\n\n- Language support add Español España\n\n- Keyboard add Spanish Castillian Winkeys and configure to first keyboard\n\n- Installation destination -\u003e Done\n\n- Disable kdump\n\n- Security policy -\u003e Done\n\n- Minimal installation\n\n- Begin install\n\n- Create user admin\nname sysadmin\npass ...\nMake this user administrator\ngroups wheel, adm, systemd-journal\n```\n\n- Reboot\n\n- Enter SSH in new machine\n\n```\n# DNS utils\nyum -y install net-tools bind-utils deltarpm\n# Chrony/Chronyd\nsystemctl enable chronyd.service\nsystemctl reload-or-restart chronyd.service\n\n# DISABLE/ENABLE to permissive selinux (by now disable)\nsed -i 's/^SELINUX=.*$/SELINUX=disabled/g' /etc/selinux/config\nsetenforce 0\n\n# FIREWALLD disable\nsystemctl stop firewalld\nsystemctl disable firewalld\nsystemctl mask firewalld\n\n# EPEL\nyum -y install epel-release\n\n# Yum plugins\nyum -y install yum-plugin-remove-with-leaves yum-plugin-ovl yum-utils pv\n\n# OpenStack cloud\nyum -y install curl cloud-init cloud-utils-growpart acpid\nsystemctl disable cloud-init\nTODO fail this...\nsystemctl enable acpid\nsystemctl start acpid\n\n# Clean...\nyum clean all \u0026\u0026 rm -rf /var/lib/yum/yumdb \u0026\u0026 rm -rf /var/lib/yum/history \u0026\u0026 rm -rf /var/cache/yum \u0026\u0026 rpm -vv --rebuilddb\nsync \u0026\u0026 echo 3 \u003e /proc/sys/vm/drop_caches \u0026\u0026 xfs_fsr -v\ndd if=/dev/zero | pv | dd of=/bigemptyfile bs=4096k || sync \u0026\u0026 sleep 1 \u0026\u0026 sync \u0026\u0026 rm -rf /bigemptyfile\n\n# Shutdown machine\nshutdown -h now\n\n# Reduce image\nVBoxManage modifymedium \"D:\\VMs\\Centos7.4-1708-20170920\\Centos7.4-1708-20170920.vdi\" --compact\n\n# Start machine\n\n# Update all packages\nyum -y update\n\n# Clean...\nyum clean all \u0026\u0026 rm -rf /var/lib/yum/yumdb \u0026\u0026 rm -rf /var/lib/yum/history \u0026\u0026 rm -rf /var/cache/yum \u0026\u0026 rpm -vv --rebuilddb\nsync \u0026\u0026 echo 3 \u003e /proc/sys/vm/drop_caches \u0026\u0026 xfs_fsr -v\ndd if=/dev/zero | pv | dd of=/bigemptyfile bs=4096k || sync \u0026\u0026 sleep 1 \u0026\u0026 sync \u0026\u0026 rm -rf /bigemptyfile\n\n# Shutdown machine\nshutdown -h now\n\n# Reduce image\nVBoxManage modifymedium \"D:\\VMs\\Centos7.4-1708-20170920\\Centos7.4-1708-20170920.vdi\" --compact\n\n# Start machine\n\n# Create standard net interfaces\necho 'DEVICE=\"eth0\"\nBOOTPROTO=\"dhcp\"\nBOOTPROTOv6=\"dhcp\"\nONBOOT=\"yes\"\nTYPE=\"Ethernet\"\nUSERCTL=\"yes\"\nPEERDNS=\"yes\"\nIPV6INIT=\"yes\"\nPERSISTENT_DHCLIENT=\"1\"' \u003e /etc/sysconfig/network-scripts/ifcfg-eth0\necho 'DEVICE=\"eth1\"\nBOOTPROTO=\"dhcp\"\nBOOTPROTOv6=\"dhcp\"\nONBOOT=\"yes\"\nTYPE=\"Ethernet\"\nUSERCTL=\"yes\"\nPEERDNS=\"yes\"\nIPV6INIT=\"yes\"\nPERSISTENT_DHCLIENT=\"1\"' \u003e /etc/sysconfig/network-scripts/ifcfg-eth1\necho 'DEVICE=\"eth2\"\nBOOTPROTO=\"dhcp\"\nBOOTPROTOv6=\"dhcp\"\nONBOOT=\"yes\"\nTYPE=\"Ethernet\"\nUSERCTL=\"yes\"\nPEERDNS=\"yes\"\nIPV6INIT=\"yes\"\nPERSISTENT_DHCLIENT=\"1\"' \u003e /etc/sysconfig/network-scripts/ifcfg-eth2\n\nrm -f /etc/sysconfig/network-scripts/[ifcfg-enp0s3,ifcfg-enp0s8,ifcfg-enp0s9]\n\n# Configure correctly the network for good access to OST metadata\necho 'NETWORKING=yes\nNOZEROCONF=yes' \u003e /etc/sysconfig/network\n\n# Configure cloud-init\nEdit /etc/cloud/cloud.cfg\nReplace\n---\n    name: centos\n---\nby\n---\n    name: cloud-user\n---\nAnd comment line:\n# ssh_pwauth:   0\n\nIn cloud_init_modules (After  - ssh) add this line:\n---\n - resolv-conf\n---\n\n# Remove persistent net rules\nln -s /dev/null /etc/udev/rules.d/80-net-name-slot.rules\nrm -f /etc/udev/rules.d/70-persistent-ipoib.rules\n\n# Change grub config\nEdit /etc/default/grub and replace by:\nGRUB_TIMEOUT=1\nGRUB_DISTRIBUTOR=\"$(sed 's, release .*$,,g' /etc/system-release)\"\nGRUB_DEFAULT=saved\nGRUB_DISABLE_SUBMENU=true\nGRUB_TERMINAL=\"console serial\"\nGRUB_SERIAL_COMMAND=\"serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1\"\nGRUB_CMDLINE_LINUX=\"crashkernel=auto console=tty0 console=ttyS0,115200n8 vconsole.keymap=es nofb nomodeset vga=791\"\n\n# Launch:\ngrub2-mkconfig -o /boot/grub2/grub.cfg\n\n# Set keymap and model\nlocalectl set-keymap es-winkeys,us pc105\nlocalectl set-x11-keymap es-winkeys,us pc105\nlocalectl set-keymap es-winkeys,us pc105\nlocalectl set-x11-keymap es-winkeys,us pc105\n\nlocalectl status\n---\n   System Locale: LANG=en_US.UTF-8\n       VC Keymap: es-winkeys,us\nVC Toggle Keymap: pc105\n      X11 Layout: es-winkeys,us\n       X11 Model: pc105\n---\n\n# Add sysadmin to /etc/sudoers file\nsysadmin ALL=(ALL) NOPASSWD: ALL\n\n# NOTE: If we need to configure sysadmin to use specific public key, add an .ssh/authorized_keys\nAs root:\nmkdir -p /home/sysadmin/.ssh\nCreate /home/sysadmin/.ssh/authorized_keys\nssh-rsa ...\nchmod 700 /home/sysadmin/.ssh\nchmod 600 /home/sysadmin/.ssh/*\nchown -R sysadmin.sysadmin /home/sysadmin/.ssh\n\n# Remove cloud-init default domainname localdomain\nsed -i -e 's/localdomain//g' /usr/lib/python2.7/site-packages/cloudinit/sources/__init__.py\nrm -f /usr/lib/python2.7/site-packages/cloudinit/sources/__init__.py[co]\n\n# Clean...\nyum clean all \u0026\u0026 rm -rf /var/lib/yum/yumdb \u0026\u0026 rm -rf /var/lib/yum/history \u0026\u0026 rm -rf /var/cache/yum \u0026\u0026 rpm -vv --rebuilddb\nsync \u0026\u0026 echo 3 \u003e /proc/sys/vm/drop_caches \u0026\u0026 xfs_fsr -v\ndd if=/dev/zero | pv | dd of=/bigemptyfile bs=4096k || sync \u0026\u0026 sleep 1 \u0026\u0026 sync \u0026\u0026 rm -rf /bigemptyfile\n\n# Shutdown machine\nshutdown -h now\n\n# Reduce image\nVBoxManage modifymedium \"D:\\VMs\\Centos7.4-1708-20170920\\Centos7.4-1708-20170920.vdi\" --compact\n\n# Start machine\n\n# To enter with sysadmin remotely, after first reboot (cloud-init disable all password autentication for all users)\nEdit /etc/ssh/sshd_config\nAdd:\nMatch User sysadmin\n  PasswordAuthentication yes\n\nreboot\n\n# Enter in the machine and test OK and remove other things\npackage-cleanup -y --oldkernels --count=1\n\n# When the image is working properly (disable sysadmin access remotely, only for console)\nEdit and comment\n/etc/ssh/sshd_config\n#Match User sysadmin\n#  PasswordAuthentication yes\n\nEdit /etc/cloud/cloud.cfg\nUncomment\nssh_pwauth:   0\n\n# Clean data created and cloud-init\nuserdel -r cloud-user\nrm -f /etc/sudoers.d/90-cloud-init-users /etc/group- /etc/gshadow- /etc/passwd- /etc/shadow-\nrm -rf /var/lib/cloud\n\n# Clean logs\nrm -rf /tmp/*\nrm -f /var/log/cloud-init*.log\nrm -f /var/log/messages*\n\n# Clean...\nyum clean all \u0026\u0026 rm -rf /var/lib/yum/yumdb \u0026\u0026 rm -rf /var/lib/yum/history \u0026\u0026 rm -rf /var/cache/yum \u0026\u0026 rpm -vv --rebuilddb\nsync \u0026\u0026 echo 3 \u003e /proc/sys/vm/drop_caches \u0026\u0026 xfs_fsr -v\ndd if=/dev/zero | pv | dd of=/bigemptyfile bs=4096k || sync \u0026\u0026 sleep 1 \u0026\u0026 sync \u0026\u0026 rm -rf /bigemptyfile\n\n# Clean history\nrm -f /root/.bash_history\nrm -f /home/sysadmin/.bash_history\ncat /dev/null \u003e /home/sysadmin/.bash_history \u0026\u0026 chown sysadmin.sysadmin /home/sysadmin/.bash_history \u0026\u0026 cat /dev/null \u003e /root/.bash_history \u0026\u0026 history -c\nsync\n\n# Clean...\nyum clean all \u0026\u0026 rm -rf /var/lib/yum/yumdb \u0026\u0026 rm -rf /var/lib/yum/history \u0026\u0026 rm -rf /var/cache/yum \u0026\u0026 rpm -vv --rebuilddb\nsync \u0026\u0026 echo 3 \u003e /proc/sys/vm/drop_caches \u0026\u0026 xfs_fsr -v\ndd if=/dev/zero | pv | dd of=/bigemptyfile bs=4096k || sync \u0026\u0026 sleep 1 \u0026\u0026 sync \u0026\u0026 rm -rf /bigemptyfile\n\n# Shutdown machine\nshutdown -h now\n\n# Reduce image\nVBoxManage modifymedium \"D:\\VMs\\Centos7.4-1708-20170920\\Centos7.4-1708-20170920.vdi\" --compact\n\n# Start machine\n\n# From outside of VirtualBox (we use Cygwin)\n\n# Convert your virtual box image to raw format\nVBoxManage clonehd \"D:\\VMs\\Centos7.4-1708-20170920\\Centos7.4-1708-20170920.vdi\" \"D:\\compartido\\Centos7.4-1708-20170920.raw\" --format raw\n\n# In other VM with Centos 7 with shared folder \"D:\\compartido\"\nyum install kvm qemu-img\nyum install libguestfs-tools\n\n# Convert the image to qcow2 format\nqemu-img convert -f raw /media/sf_compartido/Centos7.4-1708-20170920.raw -O qcow2 /media/sf_compartido/Centos7.4-1708-20170920-big.qcow2 \u0026\u0026 rm -f /media/sf_compartido/Centos7.4-1708-20170920.raw\n\n# Reduce image size using the virt-sparsify command\nvirt-sparsify --compress /media/sf_compartido/Centos7.4-1708-20170920-big.qcow2 /media/sf_compartido/Centos7.4-1708-20170920.qcow2 \u0026\u0026 rm -f /media/sf_compartido/Centos7.4-1708-20170920-big.qcow2\n\n# For edit images... Skip this if we don't need\n# export LIBGUESTFS_BACKEND=direct\nEdit /etc/libvirt/qemu.conf\nUncomment user and group root\nsystemctl start libvirtd\nsystemctl enable libvirtd\nvirt-ls -a /media/sf_compartido/CentOS-7-x86_64-GenericCloud-1608.qcow2 -R /lib/systemd/system\nguestmount -a /var/lib/libvirt/images/xenserver.qcow2 -m /dev/sda1 /mnt\n\n\n- Upload to openstack (we use http://telefonica.github.io/iot-utils/)\nGoto tenant (with OST client tools in cygwin)\n. venv-ansible-2.3.2.0/bin/activate\n. openstackEPG.sh\n(venv-ansible-2.3.2.0) [admin@caprica ~][...]-[...]$\n\nopenstack image create Centos7.4-1708-20170920 --disk-format qcow2 --file \"D:\\compartido\\Centos7.4-1708-20170920.qcow2\"\n```\n\n\n## 7.- Enjoy it...\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftelefonica%2Fiot-utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftelefonica%2Fiot-utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftelefonica%2Fiot-utils/lists"}