{"id":13429381,"url":"https://github.com/JackTheMico/ruia-peewee-async","last_synced_at":"2025-03-16T03:31:25.783Z","repository":{"id":57910833,"uuid":"526460701","full_name":"JackTheMico/ruia-peewee-async","owner":"JackTheMico","description":"A Ruia plugin that uses the peewee-async to store data to MySQL or PostgreSQL or both of them","archived":false,"fork":false,"pushed_at":"2024-01-15T20:40:55.000Z","size":245,"stargazers_count":1,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-14T14:54:43.834Z","etag":null,"topics":["mysql","plugin","postgresql","ruia"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JackTheMico.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-08-19T04:20:02.000Z","updated_at":"2024-02-03T14:17:24.000Z","dependencies_parsed_at":"2023-02-19T15:31:41.477Z","dependency_job_id":"5a69fd73-f1af-4b5f-b7a2-7bd8cabc3fef","html_url":"https://github.com/JackTheMico/ruia-peewee-async","commit_stats":{"total_commits":80,"total_committers":4,"mean_commits":20.0,"dds":"0.15000000000000002","last_synced_commit":"34ac3c5f7a60f54ad05fb1ce33843140848e649c"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JackTheMico%2Fruia-peewee-async","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JackTheMico%2Fruia-peewee-async/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JackTheMico%2Fruia-peewee-async/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JackTheMico%2Fruia-peewee-async/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JackTheMico","download_url":"https://codeload.github.com/JackTheMico/ruia-peewee-async/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243597817,"owners_count":20316843,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mysql","plugin","postgresql","ruia"],"created_at":"2024-07-31T02:00:36.197Z","updated_at":"2025-03-16T03:31:25.403Z","avatar_url":"https://github.com/JackTheMico.png","language":"Python","funding_links":[],"categories":["Uncategorized"],"sub_categories":["Uncategorized"],"readme":"# ruia-peewee-async\n[![996.icu](https://img.shields.io/badge/link-996.icu-red.svg)](https://996.icu)\n[![LICENSE](https://img.shields.io/badge/license-Anti%20996-blue.svg)](https://github.com/996icu/996.ICU/blob/master/LICENSE)\n\n[![Testing on main branch](https://github.com/JackTheMico/ruia-peewee-async/actions/workflows/codecov.yml/badge.svg?branch=main)](https://github.com/JackTheMico/ruia-peewee-async/actions/workflows/codecov.yml)\n[![Testing on develop branch](https://github.com/JackTheMico/ruia-peewee-async/actions/workflows/codecov.yml/badge.svg?branch=develop)](https://github.com/JackTheMico/ruia-peewee-async/actions/workflows/codecov.yml)\n\n[![Semantic Release](https://github.com/JackTheMico/ruia-peewee-async/actions/workflows/release.yml/badge.svg)](https://github.com/JackTheMico/ruia-peewee-async/actions/workflows/release.yml)\n\nA [Ruia](https://github.com/howie6879/ruia) plugin that uses [peewee-async](https://github.com/05bit/peewee-async) to store data to MySQL or PostgreSQL or both of them.\n\n\n## Installation\n\nUsing [pip](https://pip.pypa.io/en/stable/) or [ pipenv ](https://pipenv.pypa.io/en/latest/) or [ poetry ](https://python-poetry.org/) to install.\n\n```shell\npip install ruia-peewee-async[aiomysql]\npipenv install ruia-peewee-async[aiomysql]\npoetry add ruia-peewee-async[aiomysql]\n\nor\n\npip install ruia-peewee-async[aiopg]\npipenv install ruia-peewee-async[aiopg]\npoetry add ruia-peewee-async[aiopg]\n\nor\n\npip install ruia-peewee-async[all]\npipenv install ruia-peewee-async[all]\npoetry install ruia-peewee-async[all]\n```\n`ruia-peewee-async[all]` means to install both aiomysql and aiopg.\n\n## Usage\n\nA complete example is in [the example directory](./examples/douban.py).\n\nThere's a `create_model` method to create the Peewee model based on database configuration.\nYou can use the `create_model` method to manipulate tables before starting the spider.\n```python\nfrom ruia_peewee_async import create_model\n\nmysql_model, mysql_manager, postgres_model, postgres_manager = create_model(mysql=mysql) # or postgres=postgres or both\n# create the table at the same time\nmysql_model, mysql_manager, postgres_model, postgres_manager = create_model(mysql=mysql, create_table=True) # or postgres=postgres or both\nrows = mysql_model.select().count()\nprint(rows)\n```\n\nAnd class `Spider` from `ruia_peewee_async` has attributes below related to database you can use.\n```python\nfrom peewee import Model\nfrom typing import Callable, Dict\nfrom typing import Optional as TOptional\nfrom peewee_async import (\n    AsyncQueryWrapper,\n    Manager,\n    MySQLDatabase,\n    PooledMySQLDatabase,\n    PooledPostgresqlDatabase,\n    PostgresqlDatabase,\n)\nfrom ruia import Spider as RuiaSpider\n\nclass Spider(RuiaSpider):\n    mysql_model: Union[Model, Dict] # It will be a Model instance after spider started.\n    mysql_manager: Manager\n    postgres_model: Union[Model, Dict] # same above\n    postgres_manager: Manager\n    mysql_db: MySQLDatabase\n    postgres_db: PostgresqlDatabase\n    mysql_filters: TOptional[AsyncQueryWrapper]\n    postgres_filters: TOptional[AsyncQueryWrapper]\n```\nFor more information, check out [peewee's documentation](http://docs.peewee-orm.com/en/latest/) and [peewee-async's documentation](https://peewee-async.readthedocs.io/en/latest/).\n\n## Development\nUsing `pyenv` to install the version of python that you need.\nFor example\n```shell\npyenv install 3.7.9\n```\nThen go to the root of the project and run:\n```shell\npoetry install \u0026\u0026 poetry install -E aiomysql -E aiopg\n```\nto install all dependencies.\n\nMacOS users have to run `brew install postgresql` to install postgresql and export the `pg_config` to the PATH,\nso that the `psycorg2` dependency can be installed successfully with pip.\n\n- Using `poetry shell` to enter the virtual environment.\n  Or open your favorite editor and select the virtual environment to start coding.\n- Using `pytest` to run unit tests under `tests` folder.\n- Using `pytest --cov .` to run all tests and generate coverage report in terminal.\n\n## Thanks\n- [ruia](https://github.com/howie6879/ruia)\n- [peewew](https://github.com/coleifer/peewee)\n- [peewee-async](https://github.com/05bit/peewee-async)\n- [aiomysql](https://github.com/aio-libs/aiomysql)\n- [aiopg](https://github.com/aio-libs/aiopg)\n- [schema](https://github.com/keleshev/schema)\n- [pytest and its awesome plugins](https://github.com/pytest-dev/pytest)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJackTheMico%2Fruia-peewee-async","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FJackTheMico%2Fruia-peewee-async","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJackTheMico%2Fruia-peewee-async/lists"}