{"id":13625911,"url":"https://github.com/songcser/sanic-ms","last_synced_at":"2025-04-16T10:33:47.790Z","repository":{"id":48151151,"uuid":"101978825","full_name":"songcser/sanic-ms","owner":"songcser","description":"基于sanic的微服务基础架构","archived":false,"fork":false,"pushed_at":"2018-11-30T15:21:26.000Z","size":479,"stargazers_count":395,"open_issues_count":3,"forks_count":116,"subscribers_count":18,"default_branch":"master","last_synced_at":"2024-08-02T22:21:52.807Z","etag":null,"topics":["aiohttp","aiohttp-client","asyncio","microservice","peewee","sanic","swagger","unittest","zipkin"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/songcser.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-08-31T08:36:22.000Z","updated_at":"2024-06-28T14:41:26.000Z","dependencies_parsed_at":"2022-09-05T04:01:13.083Z","dependency_job_id":null,"html_url":"https://github.com/songcser/sanic-ms","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/songcser%2Fsanic-ms","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/songcser%2Fsanic-ms/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/songcser%2Fsanic-ms/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/songcser%2Fsanic-ms/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/songcser","download_url":"https://codeload.github.com/songcser/sanic-ms/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223708442,"owners_count":17189776,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aiohttp","aiohttp-client","asyncio","microservice","peewee","sanic","swagger","unittest","zipkin"],"created_at":"2024-08-01T21:02:05.298Z","updated_at":"2024-11-08T15:30:45.565Z","avatar_url":"https://github.com/songcser.png","language":"Python","funding_links":[],"categories":["zipkin","Python"],"sub_categories":[],"readme":"# Sanic Micro Service\n\n基于sanic的微服务基础架构\n\n## Introduce\n\n使用python做web开发面临的一个最大的问题就是性能，在解决C10K问题上显的有点吃力。有些异步框架Tornado、Twisted、Gevent 等就是为了解决性能问题。这些框架在性能上有些提升，但是也出现了各种古怪的问题难以解决。\n\n在python3.6中，官方的异步协程库asyncio正式成为标准。在保留便捷性的同时对性能有了很大的提升,已经出现许多的异步框架使用asyncio。\n\n使用较早的异步框架是aiohttp，它提供了server端和client端，对asyncio做了很好的封装。但是开发方式和最流行的微框架flask不同，flask开发简单，轻量，高效。将两者结合起来就有了sanic。\n\nSanic框架是和Flask相似异步协程框架，简单轻量，并且性能很高。本项目就是以sanic为基础搭建的微服务框架。微服务最近很火，它解决了复杂性问题，提高开发效率，便于部署等优点。\n\n正是结合这些优点, 以sanic为基础，集成多个流行的库搭建微服务框架。\n\n## Feature\n\n* **使用sanic异步框架，简单，轻量，高效。**\n* **使用uvloop为核心引擎，使sanic在很多情况下单机并发甚至不亚于Golang。**\n* **使用asyncpg为数据库驱动，进行数据库连接，执行sql语句执行。**\n* **使用aiohttp为Client，对其他微服务进行访问。**\n* **使用peewee为ORM，但是只是用来做模型设计和migration。**\n* **使用opentracing为分布式追踪系统。**\n* **使用unittest做单元测试，并且使用mock来避免访问其他微服务。**\n* **使用swagger做API标准，能自动生成API文档。**\n\n## Usage\n\n[Example](https://github.com/songcser/sanic-ms/tree/master/examples)\n\n#### Swagger API\n![image](https://github.com/songcser/sanic-ms/raw/master/examples/images/1514528294957.jpg)\n\n#### Zipkin Server\n![image](https://github.com/songcser/sanic-ms/raw/master/examples/images/1514528423339.jpg)\n![image](https://github.com/songcser/sanic-ms/raw/master/examples/images/1514528479787.jpg)\n\n## Config\n\n\u003e 设置配置文件和Django相似，通过设置环境变量值SANIC_CONFIG_MODULE\n\n```\nexport SANIC_CONFIG_MODULE='mysite.configs'\n```\n\n## Server\n\n\u003e 使用sanic异步框架，有较高的性能，但是使用不当会造成blocking, 对于有IO请求的都要选用异步库。**添加库要慎重**。\n\u003e sanic使用uvloop异步驱动，uvloop基于libuv使用Cython编写，性能比nodejs还要高。\n\n功能说明：\n\n#### Before Server Start\n\n* 创建DB连接池\n* 创建Client连接\n* 创建queue, 用于日志追踪\n* 创建opentracing.tracer进行日志追踪\n\n#### Middleware\n\n* 处理跨域请求\n* 创建span, 用于日志追踪\n* 对response进行封装，统一格式\n\n#### Error Handler\n\n对抛出的异常进行处理，返回统一格式\n\n#### Task\n\n创建task消费queue中对span，用于日志追踪\n\n#### Asynchronous Handler\n由于使用的是异步框架，可以将一些IO请求并行处理\n\nExample:\n\n```\nasync def async_request(datas):\n    # async handler request\n    results = await asyncio.gather(*[data[2] for data in datas])\n    for index, obj in enumerate(results):\n        data = datas[index]\n        data[0][data[1]] = results[index]\n\n@user_bp.get('/\u003cid:int\u003e')\n@doc.summary(\"get user info\")\n@doc.description(\"get user info by id\")\n@doc.produces(Users)\nasync def get_users_list(request, id):\n    async with request.app.db.acquire(request) as cur:\n        record = await cur.fetch(\n            \"\"\" SELECT * FROM users WHERE id = $1 \"\"\", id)\n        datas = [\n            [record, 'city_id', get_city_by_id(request, record['city_id'])]\n            [record, 'role_id', get_role_by_id(request, record['role_id'])]\n        ]\n        await async_request(datas)\n        return record\n```\nget_city_by_id, get_role_by_id是并行处理。\n\n\n#### 相关连接\n[sanic](https://github.com/channelcat/sanic)\n\n\n## Model \u0026 Migration\n\n\u003e Peewee is a simple and small ORM. It has few (but expressive) concepts, making it easy to learn and intuitive to use。\n\u003e\n\u003e ORM使用peewee, 只是用来做模型设计和migration, 数据操作使用asyncpg。\n\nExample:\n\n```\n# models.py\n\nclass Users(Model):\n    id = PrimaryKeyField()\n    create_time = DateTimeField(verbose_name='create time',\n                                default=datetime.datetime.utcnow)\n    name = CharField(max_length=128, verbose_name=\"user's name\")\n    age = IntegerField(null=False, verbose_name=\"user's age\")\n    sex = CharField(max_length=32, verbose_name=\"user's sex\")\n    city_id = IntegerField(verbose_name='city for user', help_text=CityApi)\n    role_id = IntegerField(verbose_name='role for user', help_text=RoleApi)\n\n    class Meta:\n        db_table = 'users'\n\n\n# migrations.py\n\nfrom sanicms.migrations import MigrationModel, info, db\n\nclass UserMigration(MigrationModel):\n    _model = Users\n\n    # @info(version=\"v1\")\n    # def migrate_v1(self):\n    #     migrate(self.add_column('sex'))\n\ndef migrations():\n    try:\n        um = UserMigration()\n        with db.transaction():\n            um.auto_migrate()\n            print(\"Success Migration\")\n    except Exception as e:\n        raise e\n\nif __name__ == '__main__':\n    migrations()\n```\n\n* 运行命令 python migrations.py\n* migrate_v1函数添加字段sex, 在BaseModel中要先添加name字段\n* info装饰器会创建表migrate_record来记录migrate，version每个model中必须唯一，使用version来记录是否执行过，还可以记录author，datetime\n* migrate函数必须以**migrate_**开头\n\n#### 相关连接\n\n[peewee](http://docs.peewee-orm.com/en/latest/)\n\n\n## DB \n\n\u003e **asyncpg is the fastest driver among common Python, NodeJS and Go implementations**\n\u003e\n\u003e 使用asyncpg为数据库驱动, 对数据库连接进行封装, 执行数据库操作。\n\u003e\n\u003e 不使用ORM做数据库操作，一个原因是性能，ORM会有性能的损耗，并且无法使用asyncpg高性能库。另一个是单个微服务是很简单的，表结构不会很复杂，简单的SQL语句就可以处理来，没必要引入ORM。\n\nExample:\n\n```\nsql = \"SELECT * FROM users WHERE name=$1\"\nname = \"test\"\nasync with request.app.db.acquire(request) as cur:\n    data = await cur.fetchrow(sql, name)\n\nasync with request.app.db.transaction(request) as cur:\n    data = await cur.fetchrow(sql, name)\n```\n\n* acquire() 函数为非事务, 对于只涉及到查询的使用非事务，可以提高查询效率\n* tansaction() 函数为事务操作，对于增删改必须使用事务操作\n* 传入request参数是为了获取到span，用于日志追踪\n* **TODO**  数据库读写分离\n\n#### 相关连接\n[asyncpg](https://github.com/MagicStack/asyncpg)\n[benchmarks](https://magic.io/blog/asyncpg-1m-rows-from-postgres-to-python/)\n\n## Client\n\n\u003e 使用aiohttp中的client，对客户端进行了简单的封装，访问其他微服务。\n\n\u003e Don’t create a session per request. Most likely you need a session per application which performs all requests altogether.\n\u003e A session contains a connection pool inside, connection reusage and keep-alives (both are on by default) may speed up total performance.\n\nExample: \n\n```\n@app.listener('before_server_start')\nasync def before_srver_start(app, loop):\n    app.client =  Client(loop, url='http://host:port')\n\nasync def get_role_by_id(request, id):\n    cli = request.app.client.cli(request)\n    async with cli.get('/cities/{}'.format(id)) as res:\n        return await res.json()\n\n@app.listener('before_server_stop')\nasync def before_server_stop(app, loop):\n    app.client.close()\n\n```\n\n对于访问不同的微服务可以创建多个不同的client，这样每个client都会keep-alives\n\n#### 相关连接\n\n[aiohttp](http://aiohttp.readthedocs.io/en/stable/client.html)\n\n\n## LOG \u0026 Distributed Tracing System\n\n\u003e 使用官方logging, 配置文件为logging.yml, sanic版本要0.6.0及以上。JsonFormatter将日志转成json格式，用于输入到ES\n\u003e\n\u003e Enter OpenTracing: by offering consistent, expressive, vendor-neutral APIs for popular platforms, OpenTracing makes it easy for developers to add (or switch) tracing implementations with an O(1) configuration change. OpenTracing also offers a lingua franca for OSS instrumentation and platform-specific tracing helper libraries. Please refer to the Semantic Specification.\n\n### 装饰器logger\n\n```\n@logger(type='method', category='test', detail='detail', description=\"des\", tracing=True, level=logging.INFO)\nasync def get_city_by_id(request, id):\n    cli = request.app.client.cli(request)\n```\n\n* type: 日志类型，如 method, route\n* category: 日志类别，默认为app的name\n* detail: 日志详细信息\n* description: 日志描述，默认为函数的注释\n* tracing: 日志追踪，默认为True\n* level: 日志级别，默认为INFO\n\n### 分布式追踪系统\n\n* OpenTracing是以Dapper，Zipkin等分布式追踪系统为依据, 为分布式追踪建立了统一的标准。\n* Opentracing跟踪每一个请求，记录请求所经过的每一个微服务，以链条的方式串联起来，对分析微服务的性能瓶颈至关重要。\n* 使用opentracing框架，但是在输出时转换成zipkin格式。 因为大多数分布式追踪系统考虑到性能问题，都是使用的thrift进行通信的，本着简单，Restful风格的精神，没有使用RPC通信。以日志的方式输出, 可以使用fluentd, logstash等日志收集再输入到Zipkin。Zipkin是支持HTTP输入的。\n* 生成的span先无阻塞的放入queue中，在task中消费队列的span。后期可以添加上采样频率。\n* 对于DB，Client都加上了tracing\n\n#### 相关连接\n\n[opentracing](https://github.com/opentracing/opentracing-python)\n[zipkin](https://github.com/openzipkin/zipkin)\n[jaeger](https://uber.github.io/jaeger/)\n\n\n## API\n\n\u003e api文档使用swagger标准。\n\nExample:\n\n```\nfrom sanicms import doc\n\n@user_bp.post('/')\n@doc.summary('create user')\n@doc.description('create user info')\n@doc.consumes(Users)\n@doc.produces({'id': int})\nasync def create_user(request):\n    data = request['data']\n    async with request.app.db.transaction(request) as cur:\n        record = await cur.fetchrow(\n            \"\"\" INSERT INTO users(name, age, city_id, role_id)\n                VALUES($1, $2, $3, $4, $5)\n                RETURNING id\n            \"\"\", data['name'], data['age'], data['city_id'], data['role_id']\n        )\n        return {'id': record['id']}\n```\n\n* summary: api概要\n* description: 详细描述\n* consumes: request的body数据\n* produces: response的返回数据\n* tag: API标签\n* 在consumes和produces中传入的参数可以是peewee的model,会解析model生成API数据, 在field字段的help_text参数来表示引用对象\n* http://host:ip/openapi/spec.json 获取生成的json数据\n\n#### 相关连接\n\n[swagger](https://swagger.io/)\n\n## Response\n\n在返回时，不要返回response，直接返回原始数据，会在Middleware中对返回的数据进行处理，返回统一的格式，具体的格式可以[查看](https://gist.github.com/songcser/ae8af65f33f34f09f265879e107cb584)\n\n## Unittest\n\n\u003e 单元测试使用unittest\n\nExample:\n\n```\nfrom sanicms.tests import APITestCase\nfrom service.server import app\n\nclass TestCase(APITestCase):\n    _app = app\n    _blueprint = 'visit'\n\n    def setUp(self):\n        super(TestCase, self).setUp()\n        self._mock.get('/cities/1',\n                       payload={'id': 1, 'name': 'shanghai'})\n        self._mock.get('/roles/1',\n                       payload={'id': 1, 'name': 'shanghai'})\n\n    def test_create_user(self):\n        data = {\n            'name': 'test',\n            'age': 2,\n            'city_id': 1,\n            'role_id': 1,\n        }\n        res = self.client.create_user(data=data)\n        body = ujson.loads(res.text)\n        self.assertEqual(res.status, 200)\n```\n\n* 其中_blueprint为blueprint名称\n* 在setUp函数中，使用_mock来注册mock信息, 这样就不会访问真实的服务器, payload为返回的body信息\n* 使用client变量调用各个函数, data为body信息，params为路径的参数信息，其他参数是route的参数\n\n### coverage\n\n```\ncoverage erase\ncoverage run --source . -m sanicms tests\ncoverage xml -o reports/coverage.xml\ncoverage2clover -i reports/coverage.xml -o reports/clover.xml\ncoverage html -d reports\n```\n\n* coverage2colver 是将coverage.xml 转换成 clover.xml，bamboo需要的格式是clover的。\n\n#### 相关连接\n\n[unittest](https://docs.python.org/3/library/unittest.html)\n[coverage](https://coverage.readthedocs.io/en/coverage-4.4.1/)\n## Exception\n\n\u003e 使用 app.error_handler = CustomHander() 对抛出的异常进行处理\n\nExample:\n\n```\nfrom sanicms.exception import ServerError\n\n@visit_bp.delete('/users/\u003cid:int\u003e')\nasync def del_user(request, id):\n    raise ServerError(error='内部错误',code='10500', message=\"msg\")\n```\n\n* code: 错误码，无异常时为0，其余值都为异常\n* message: 状态码信息\n* error: 自定义错误信息\n* status_code: http状态码，使用标准的http状态码\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsongcser%2Fsanic-ms","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsongcser%2Fsanic-ms","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsongcser%2Fsanic-ms/lists"}