https://github.com/lujun9972/auto-dispatcher.py
https://github.com/lujun9972/auto-dispatcher.py
Last synced: 9 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/lujun9972/auto-dispatcher.py
- Owner: lujun9972
- Created: 2016-08-10T10:49:26.000Z (almost 10 years ago)
- Default Branch: master
- Last Pushed: 2021-04-30T09:24:41.000Z (about 5 years ago)
- Last Synced: 2025-02-09T15:12:30.990Z (over 1 year ago)
- Language: Python
- Size: 22.5 KB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.org
Awesome Lists containing this project
README
#+TITLE: README
#+AUTHOR: lujun9972
#+CATEGORY: auto-dispatcher.py
#+DATE: [2016-07-25 周一 10:07]
#+OPTIONS: ^:{}
* 功能
当提交一个安装包到服务器上时,服务器自动根据包名称判断应该使用哪个用户FTP到哪台生产机上,并调用生产机上的升级程序进行升级.
* 设计
脚本分成两部分,一部分专门检查是否有安装包提交到服务器上,并找出哪些文件是新提交的.
第二部分根据配置信息上传新提交的文件并执行相应的更像脚本.
*由于生产环境不连外网,因此不能使用第三方库!*
* 实现
** 判断有新的安装包提交到服务器上
有两种实现方式,一种是通过VC服务器上的Hook实现,该方法要求服务器上安装VC服务器. 另一种方式是通过扫描并对比前后文件树的差别来判断
+ 通过svnlook changed来查询某次提交更改的文件
#+BEGIN_SRC sh :tangle "post-commit"
REPOS_PATH=$1
REVISION=$2
files=$(svnlook changed ${REPOS_PATH} -r ${REVISION}|egrep -v "^D"|egrep -v "/$"|cut -f2)
auto-dispatcher.py $files
#+END_SRC
+ 扫描并对比前后文件树的差别
#+BEGIN_SRC python :tangle "auto_dispatcher.py"
import sys
import os
import os.path
import time
from dispatcher import dispatch_files
def directory_files(directory):
'''Return a set of full paths of files in DIRECTORY.'''
results=[]
for root,dirs,files in os.walk(directory):
full_paths=[os.path.join(root,f) for f in files]
full_paths = [f for f in full_paths if os.path.isfile(f)]
results.extend(full_paths)
return set(results)
def directory_files_and_size(directory):
'''Return a set of full paths and sizes of files in DIRECTORY.'''
files = directory_files(directory)
files_and_sizes = set([(f,os.path.getsize(f)) for f in files if os.path.isfile(f)])
return files_and_sizes
def directory_files_until_nochange(directory,interval=5):
old = directory_files_and_size(directory)
time.sleep(interval)
new = directory_files_and_size(directory)
while old != new:
time.sleep(interval)
old,new = new,directory_files_and_size(directory)
return set([file_and_size[0] for file_and_size in new])
if __name__ == "__main__":
try:
directory = sys.argv[1]
except IndexError:
directory = os.getcwd()
old = directory_files(directory)
while True:
new = directory_files_until_nochange(directory)
diff = new - old
dispatch_files(diff)
time.sleep(60)
old = new
#+END_SRC
** 根据配置信息上传新提交的文件并执行相应的更新脚本.
:PROPERTIES:
:header-args: :tangle "dispatcher.py"
:END:
*** 配置日志记录器
初始化日志,日志存放在"auto-dispatcher.log"中,最大容量为10M,一共可以有5个备份日志.
#+BEGIN_SRC conf :tangle "logger.cfg"
[loggers]
keys=root,main
[logger_root]
level=DEBUG
handlers=stderr
[logger_main]
level=DEBUG
handlers=stderr,file
qualname="main"
propagate=0
[handlers]
keys=stderr,file
[handler_stderr]
class=StreamHandler
formatter=default
args=(sys.stderr,)
[handler_file]
class=handlers.RotatingFileHandler
formatter=default
args=("auto-dispatcher.log","a",10*1024*1024,5)
[formatters]
keys=default
[formatter_default]
format=%(asctime)s [%(thread)d] %(filename)s[line:%(lineno)d] %(levelname)s %(message)s
#+END_SRC
从logger.cfg中读取logger配置
#+BEGIN_SRC python
# -*- coding: utf-8 -*-
import logging
import logging.config
logging.config.fileConfig("logger.cfg")
logger = logging.getLogger("main")
#+END_SRC
*** 根据包名找出生产机的用户,地址和登录密码
需要有一个配置文件(暂时命名为general-dispatch-info.cfg),该配置文件的每一个section的名称都是个正则表达式用于匹配安装包的名称.
配置文件需要以下配置信息
+ 对应生产机IP,必选
+ 登录生产机的用户,可选,默认从 =~/.netrc= 中获取
+ 登录生产机的密码,可选,默认从 =~/.netrc= 中获取
+ 上传到生产机的目录,可选,默认为 =~/newcx/年月日_时分秒/=
下面是配置文件的内容:
#+BEGIN_SRC conf :tangle "general-dispatch-info.cfg"
[ibps]
host = 10.8.6.10
login = ibpsusr
password = 123456
install_command = touch /tmp/installed
[cnaps2]
host = 10.8.6.10
login = cnaps2
password = 123456
#+END_SRC
下面定义函数根据包名找出FTP的相关信息
#+BEGIN_SRC python
import netrc
import configparser
import re
import time
import sys
def get_section_by_package(package,config):
'''从config中找出匹配PACKAGE的section. config是ConfigParser.read后的结果'''
for section in config.sections():
reg = re.compile(section)
if reg.match(package):
return section
def get_ftp_info_by_package(package,cfg_file,netrc_file=None):
'''根据PACKAGE,从CFG_FILE及NETRC_FILE中找出对应ftp的HOST,LOGIN,ACCOUNT以及PASSWORD
return host,login,account,password,dest_dir,install_command'''
config = configparser.ConfigParser()
config.read(cfg_file)
section = get_section_by_package(package,config)
if not section:
logger.warning("%s中未找到匹配%s的section",cfg_file,package)
exit(-1)
else:
try:
netrc_info = netrc.netrc(netrc_file)
login,account,password = netrc_info.authenticators(host)
except Exception:
login,account,password = None,None,None
host = config.get(section,"host")
login = config.get(section,"login",fallback=None) or login
account = config.get(section,"account",fallback=None) or account
password = config.get(section,"password",fallback=None) or password
dest_dir = config.get(section,"dest_dir",fallback=None) or "~/newcx/{0}".format(time.strftime("%Y%m%d_%H%M%S"))
install_command = config.get(section,"install_command",fallback=None)
return host,login,account,password,dest_dir,install_command
#+END_SRC
#+RESULTS:
*** 登录生产机并在指定目录下上传安装包
**** 若生产机开启ssh服务,则通过scp上传
但是这里遇到两个问题,第一个问题是,执行像ssh,scp这类secure command时,必须手工输入密码,而且它们是直接从控制终端而不是stdin中读取密码的,这也意味着无法通过脚本的方式传送密码給这些程序.
万幸的是,python中有个名为 =pty= 的modual,它有一个 =spawn= 函数,manual中对它的描述是:
#+BEGIN_QUOTE
pty.spawn(argv[, master_read[, stdin_read]])
Spawn a process, and connect its controlling terminal with the current process’s standard io. This is often used to baffle programs which insist on reading from the controlling terminal.
#+END_QUOTE
这就好办了,我们只要创建一个名为"pty-process.py"脚本,在这个脚本中用pty.spawn调用secure command,然后再通过写入该脚本stdin的方式就可以变相地給这些secure command发送密码了.
pty-process.py脚本的实现如下:
#+BEGIN_SRC python :tangle "pty-process.py"
#!/bin/env python3
import pty
import sys
pty.spawn(sys.argv[1:])
#+END_SRC
借助于这个pty-process.py我们可以实现一个函数用于执行secure command
#+BEGIN_SRC python
def execute_externel_secure_command(command,password=""):
secure_command = "echo {} |python3 pty-process.py {}".format(password,command)
logger.debug("execute:%s",secure_command)
result = subprocess.check_output(secure_command,shell=True)
logger.debug("result:%s",result)
return result
#+END_SRC
第二个问题是scp并不能自动在远程创建新目录,需要先在远程手工创建目录. 这个解决方案也很简单,直接通过ssh登录远程服务器执行mkdir命令就行:
#+BEGIN_SRC python
def execute_remote_command_by_ssh(host,login,password,command):
ssh_command = "ssh -o StrictHostKeyChecking=no {}@{} '{}'".format(login,host,command)
return execute_externel_secure_command(ssh_command,password)
#+END_SRC
最后通过scp上传文件的实现为:
#+BEGIN_SRC python
import subprocess
def upload_by_scp (file_path,host,dest_dir,login,password):
execute_remote_command_by_ssh(host,login,password,"mkdir -p {}".format(dest_dir))
scp_command = "scp -o StrictHostKeyChecking=no {0} {1}@{2}:{3}/".format(file_path,login,host,dest_dir,password)
return execute_externel_secure_command(scp_command,password)
#+END_SRC
**** 若生产机开启FTP服务则通过ftp上传
#+BEGIN_SRC python
import ftplib
import os.path
def upload_by_ftp(file_path,host,dest_dir,login="anonymous",password="",account=""):
'''upload FILE_PATH to DEST_DIR in HOST,though ftp protocol'''
with ftplib.FTP(host=host,user=login,passwd=password,acct=account) as ftp:
ftp.set_debuglevel(2) # A value of 2 or higher produces the maximum amount of debugging output, logging each line sent and received on the control connection.
logger.debug(ftp.getwelcome())
try:
ftp.mkd(dest_dir) # 创建目标文件夹
except ftplib.error_perm:
logger.debug("%s:%s already exist",host,dest_dir)
ftp.cwd(dest_dir) # 进入目标文件夹
with open(file_path,"rb") as file_handler:
ftp.storbinary("STOR {0}".format(os.path.basename(file_path)), file_handler)
logger.debug("ftp {0} to {1}:{2} done".format(file_path,host,dest_dir))
#+END_SRC
**** 上传时,优先使用scp上传,若失败则再换成通过ftp上传
#+BEGIN_SRC python
def upload(file_path,host,dest_dir,login,password):
'''upload FILE_PATH to DEST_DIR in HOST'''
try:
upload_by_scp(file_path,host,dest_dir,login,password)
except:
upload_by_ftp(file_path,host,dest_dir,login,password)
#+END_SRC
*** 调用生产机上的升级程序
只需要用上面定义的 =execute_remote_command_by_ssh= 就能实现调用生产机上的升级程序了.
*** 分发package
#+BEGIN_SRC python
def dispatch_file(file_path,cfg_file="general-dispatch-info.cfg",netrc_file=None):
package = os.path.basename(file_path)
host,login,account,password,dest_dir,install_command = get_ftp_info_by_package(package,cfg_file,netrc_file)
upload(file_path,host,dest_dir,login,password)
if install_command:
execute_remote_command_by_ssh(host,login,password,install_command)
import threading
def dispatch_files(file_paths,cfg_file="general-dispatch-info.cfg",netrc_file=None):
threads = (threading.Thread(target=dispatch_file,args=(file_path,cfg_file,netrc_file)) for file_path in file_paths)
for thread in threads:
thread.start()
return threads
#+END_SRC
*** main
#+BEGIN_SRC python
if __name__ == "__main__":
if len(sys.argv) == 2:
dispatch_file(sys.argv[1])
else:
dispatch_files(sys.argv[1:])
#+END_SRC
* Local Variables Setting:
# Local Variables:
# org-babel-default-header-args:python: ((:session . "auto_dispatcher") (:results . "output") (:exports . "code"))
# org-babel-python-command: "python3"
# End: